Skip to content

Commit

Permalink
chore: Remove Optimize OpenSearch warnings
Browse files Browse the repository at this point in the history
  • Loading branch information
@grlimacan
grlimacan committed Dec 11, 2024
1 parent 6088f25 commit b13be7a
Show file tree
Hide file tree
Showing 7 changed files with 37 additions and 43 deletions.
42 changes: 21 additions & 21 deletions docs/self-managed/operational-guides/backup-restore/optimize-backup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,20 @@ This release introduces breaking changes, including the utilized URL.
For example, `curl 'http://localhost:8092/actuator/backups'` rather than the previously used `backup`.
:::

Optimize stores its data over multiple indices in Elasticsearch. To ensure data integrity across indices, a backup of Optimize data consists of two Elasticsearch snapshots, each containing a different set of Optimize indices. Each backup is identified by a positive integer backup ID. For example, a backup with ID `123456` consists of the following Elasticsearch snapshots:
Optimize stores its data over multiple indices in the database. To ensure data integrity across indices, a backup of Optimize data consists of two ElasticSearch/OpenSearch snapshots, each containing a different set of Optimize indices. Each backup is identified by a positive integer backup ID. For example, a backup with ID `123456` consists of the following snapshots:

```
camunda_optimize_123456_3.9.0_part_1_of_2
camunda_optimize_123456_3.9.0_part_2_of_2
```

Optimize provides an API to trigger a backup and retrieve information about a given backup's state. During backup creation Optimize can continue running. The backed up data can later be restored using the standard Elasticsearch snapshot restore API.
Optimize provides an API to trigger a backup and retrieve information about a given backup's state. During backup creation Optimize can continue running. The backed up data can later be restored using the standard ElasticSearch/OpenSearch snapshot restore API.

## Prerequisites

The following prerequisites must be set up before using the backup API:

1. A snapshot repository of your choice must be registered with Elasticsearch.
1. A snapshot repository of your choice must be registered with ElasticSearch/OpenSearch.
2. The repository name must be specified using the `CAMUNDA_OPTIMIZE_BACKUP_REPOSITORY_NAME` environment variable, or by adding it to your Optimize [`environment-config.yaml`]($optimize$/self-managed/optimize-deployment/configuration/system-configuration/):

```yaml
Expand All @@ -48,13 +48,13 @@ POST actuator/backups

### Response

| Code | Description |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| 202 Accepted | Backup process was successfully initiated. To determine whether backup process was completed refer to the GET API. |
| 400 Bad Request | Indicates issues with the request, for example when the `backupId` contains invalid characters. |
| 409 Conflict | Indicates that a backup with the same `backupId` already exists. |
| 500 Server Error | All other errors, e.g. issues communicating with Elasticsearch for snapshot creation. Refer to the returned error message for more details. |
| 502 Bad Gateway | Optimize has encountered issues while trying to connect to Elasticsearch. |
| Code | Description |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| 202 Accepted | Backup process was successfully initiated. To determine whether backup process was completed refer to the GET API. |
| 400 Bad Request | Indicates issues with the request, for example when the `backupId` contains invalid characters. |
| 409 Conflict | Indicates that a backup with the same `backupId` already exists. |
| 500 Server Error | All other errors, e.g. issues communicating with the database for snapshot creation. Refer to the returned error message for more details. |
| 502 Bad Gateway | Optimize has encountered issues while trying to connect to the database. |

### Example request

Expand Down Expand Up @@ -96,8 +96,8 @@ GET actuator/backup
| 200 OK | Backup state could be determined and is returned in the response body (see example below). |
| 400 Bad Request | There is an issue with the request, for example the repository name specified in the Optimize configuration does not exist. Refer to returned error message for details. |
| 404 Not Found | If a backup ID was specified, no backup with that ID exists. |
| 500 Server Error | All other errors, e.g. issues communicating with Elasticsearch for snapshot state retrieval. Refer to the returned error message for more details. |
| 502 Bad Gateway | Optimize has encountered issues while trying to connect to Elasticsearch. |
| 500 Server Error | All other errors, e.g. issues communicating with the database for snapshot state retrieval. Refer to the returned error message for more details. |
| 502 Bad Gateway | Optimize has encountered issues while trying to connect to the database. |

### Example request

Expand Down Expand Up @@ -135,8 +135,8 @@ Possible states of the backup:

- `COMPLETE`: The backup can be used for restoring data.
- `IN_PROGRESS`: The backup process for this backup ID is still in progress.
- `FAILED`: Something went wrong when creating this backup. To find out the exact problem, use the [Elasticsearch get snapshot status API](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/get-snapshot-status-api.html) for each of the snapshots included in the given backup.
- `INCOMPATIBLE`: The backup is incompatible with the current Elasticsearch version.
- `FAILED`: Something went wrong when creating this backup. To find out the exact problem, use the [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-snapshot-status-api.html) / [OpenSearch](https://opensearch.org/docs/latest/api-reference/snapshots/get-snapshot-status/) get snapshot status API for each of the snapshots included in the given backup.
- `INCOMPATIBLE`: The backup is incompatible with the current ElasticSearch/OpenSearch version.
- `INCOMPLETE`: The backup is incomplete (this could occur when the backup process was interrupted or individual snapshots were deleted).

## Delete backup API
Expand All @@ -154,10 +154,10 @@ DELETE actuator/backups/{backupId}
| Code | Description |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 204 No Content | The delete request for the associated snapshots was submitted to Elasticsearch successfully. |
| 204 No Content | The delete request for the associated snapshots was submitted to the database successfully. |
| 400 Bad Request | There is an issue with the request, for example the repository name specified in the Optimize configuration does not exist. Refer to returned error message for details. |
| 500 Server Error | An error occurred, for example the snapshot repository does not exist. Refer to the returned error message for details. |
| 502 Bad Gateway | Optimize has encountered issues while trying to connect to Elasticsearch. |
| 502 Bad Gateway | Optimize has encountered issues while trying to connect to ElasticSearch/OpenSearch. |
### Example request
Expand All @@ -167,22 +167,22 @@ curl --request DELETE 'http://localhost:8092/actuator/backups/123456'

## Restore backup

There is no Optimize API to perform the backup restore. Instead, the standard [Elasticsearch restore snapshot API](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/restore-snapshot-api.html) can be used. Note that the Optimize versions of your backup snapshots must match the currently running version of Optimize. You can identify the version at which the backup was taken by the version tag included in respective snapshot names; for example, a snapshot with the name`camunda_optimize_123456_3.9.0_part_1_of_2` was taken of Optimize version `3.9.0`.
There is no Optimize API to perform the backup restore. Instead, the standard [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-snapshot-api.html) / [OpenSearch](https://opensearch.org/docs/latest/api-reference/snapshots/restore-snapshot) restore snapshot API can be used. Note that the Optimize versions of your backup snapshots must match the currently running version of Optimize. You can identify the version at which the backup was taken by the version tag included in respective snapshot names; for example, a snapshot with the name`camunda_optimize_123456_3.9.0_part_1_of_2` was taken of Optimize version `3.9.0`.

:::note
Optimize must NOT be running while a backup is being restored.
:::

To restore an existing backup, all the snapshots this backup contains (as listed in the response of the [create backup API request](#example-response)) must be restored using the Elasticsearch API.
To restore an existing backup, all the snapshots this backup contains (as listed in the response of the [create backup API request](#example-response)) must be restored using the restore API.

To restore a given backup, the following steps must be performed:

1. Stop Optimize.
2. Ensure no Optimize indices are present in Elasticsearch (or the restore process will fail).
3. Iterate over all Elasticsearch snapshots included in the desired backup and restore them using the Elasticsearch restore snapshot API.
2. Ensure no Optimize indices are present in the database (or the restore process will fail).
3. Iterate over all ElasticSearch/OpenSearch snapshots included in the desired backup and restore them using the restore snapshot API mentioned above.
4. Start Optimize.

Example Elasticsearch request:
Example request:

```shell
curl --request POST `http://localhost:9200/_snapshot/repository_name/camunda_optimize_123456_3.9.0_part_1_of_2/_restore?wait_for_completion=true`
Expand Down
8 changes: 6 additions & 2 deletions optimize/self-managed/optimize-deployment/advanced-features/import-guide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -191,15 +191,19 @@ Therefore, the Optimize `ProcessInstance` is an aggregation of the engine's hist
:::note
Optimize uses [nested documents](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html), the above mentioned data is an example of documents that are nested within Optimize's `ProcessInstance` index.

Elasticsearch applies restrictions regarding how many objects can be nested within one document. If your data includes too many nested documents, you may experience import failures. To avoid this, you can temporarily increase the nested object limit in Optimize's [index configuration](./../configuration/system-configuration.md#index-settings). Note that this might cause memory errors.
Elasticsearch and OpenSearch apply restrictions regarding how many objects can be nested within one document. If your data includes too many nested documents, you may experience import failures. To avoid this, you can temporarily increase the nested object limit in Optimize's [index configuration](./../configuration/system-configuration.md#index-settings). Note that this might cause memory errors.
:::

Import executions per engine entity are actually independent from another. Each follows a [producer-consumer-pattern](https://dzone.com/articles/producer-consumer-pattern), where the type specific `ImportService` is the single producer and a dedicated single `ImportJobExecutor` is the consumer of its import jobs, decoupled by a queue. So, both are executed in different threads. To adjust the processing speed of the executor, the queue size and the number of threads that process the import jobs can be configured:

:::note
Although the parameters below include `ElasticSearch` in their name, they apply to both ElasticSearch and OpenSearch installations. For backward compatibility reasons, the parameters have not been renamed.
:::

```yaml
import:
# Number of threads being used to process the import jobs per data type that are writing
# data to elasticsearch.
# data to the database.
elasticsearchJobExecutorThreadCount: 1
# Adjust the queue size of the import jobs per data type that store data to elasticsearch.
# A too large value might cause memory problems.
Expand Down
4 changes: 0 additions & 4 deletions .../self-managed/optimize-deployment/configuration/shared-elasticsearch-cluster.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,6 @@ The following illustration demonstrates this use case with two Optimize instance
Changing the value of `*.settings.index.prefix` after an instance was already running results in new indexes being created with the new prefix value. There is no support in migrating data between indexes based on different prefixes.
:::

:::note
Not all Optimize features are supported when using OpenSearch as a database. For a full list of the features that are currently supported, please refer to the [Camunda 7](https://github.com/camunda/issues/issues/705) and [Camunda 8](https://github.com/camunda/issues/issues/635) OpenSearch features.
:::

\* Elasticsearch index prefix settings path: `es.settings.index.prefix`<br/> \* OpenSearch index prefix settings path: `opensearch.settings.index.prefix`

![Shared Elasticsearch Cluster Setup](img/shared-elasticsearch-cluster.png)
Loading

0 comments on commit b13be7a

Please sign in to comment.