Skip to content

Commit

Permalink
Add camunda Exporter from PR 4715
Browse files Browse the repository at this point in the history
  • Loading branch information
mesellings committed Dec 5, 2024
1 parent 87f46f2 commit 1d55821
Show file tree
Hide file tree
Showing 3 changed files with 28 additions and 2 deletions.
30 changes: 28 additions & 2 deletions docs/reference/announcements/870.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,35 @@ For organizations that do not use cloud-native platforms such as Kubernetes or c

The architecture will include details on optimal instance sizing, network configurations, and security best practices, to ensure robust performance and reliability.

### Exporters
### Camunda Exporter

A new Exporter architecture removes the need to deploy Operate, Tasklist and Identity as separate services.
A new Camunda Exporter brings the importer and archiving logic of web components (Tasklist and Operate) closer to the distributed platform (Zeebe). The index schema is also being harmonized.

#### Harmonized index schema

Camunda is harmonizing our index structure and usage.

- This removes unnecessary duplications over multiple indices due to the previous architecture.
- With this change, several Operate indices can and will be used by Tasklist.
- New indices have been created to integrate Identity into the system.

![Harmonized indices schema](../img/harmonized-indices-schema.png)

#### Camunda Exporter

The exporter can consume Zeebe records (mostly events created by the engine), aggregate data, and store the related data into shared and harmonized indices.

- Data is archived in the background, coupled to the exporter but without blocking the exporter's progress.
- Indices can be located in either ElasticSearch (ES) or Opensearch (OS). Our web components (Tasklist and Operate) then use the new harmonized indices to show data to the user.

The following diagram shows a simplified version of this work.

![Camunda Exporter diagram](../img/target-camunda-exporter.png)

- For example, Tasklist and Operate Importers are still required for old data to be imported, but the Camunda exporter writes all new data into ES/OS. After old indices are drained, importers can be turned off.
- The archiver, which takes care of the archiving of completed process instances, will be moved into the Zeebe system as well, to reduce the installation complexity and provide a better scaling and replication factor (based on partitions).
- This helps achieve a streamlined architecture, and improves platform performance and stability (especially regarding ES/OS).
- A new separate component covers the migration, which will be part of the single application but can also deployed separately. It will adjust the previous Operate indices to make them more harmonized and usable by Tasklist.

<!-- :::info
Learn more about these updates in Streamlined Deployment with 8.7.
Expand Down
Binary file added docs/reference/img/harmonized-indices-schema.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/reference/img/target-camunda-exporter.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 1d55821

Please sign in to comment.