From fcd6669e74c4a5cc07b85e2fdc5707cfa1f9d543 Mon Sep 17 00:00:00 2001 From: Nick Sanford Date: Wed, 13 Nov 2024 12:11:19 -0500 Subject: [PATCH 1/6] [DOCS-3095] - local mongo collection --- docs/services/data/_index.md | 71 ++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/docs/services/data/_index.md b/docs/services/data/_index.md index 19607133a0..c74b9e2efe 100644 --- a/docs/services/data/_index.md +++ b/docs/services/data/_index.md @@ -120,6 +120,74 @@ You can also control how local data is deleted if your machine's local storage b {{< /expand >}} +{{< expand "Capture Directly to MongoDB" >}} + +Data Capture supports capturing tabular data directly to MongoDB in addition to capturing to disk. + +This feature is intended to support use cases like offline dashboards which don't require strong data +delivery or consistency guarantees. + +Here is a sample configuration that will capture fake sensor readings both to the configured MongoDB URI +as well as to the Viam capture directory on disk: + +```json +{ + "components": [ + { + "name": "sensor-1", + "namespace": "rdk", + "type": "sensor", + "model": "fake", + "attributes": {}, + "service_configs": [ + { + "type": "data_manager", + "attributes": { + "capture_methods": [ + { + "method": "Readings", + "capture_frequency_hz": 0.5, + "additional_params": {} + } + ] + } + } + ] + } + ], + "services": [ + { + "name": "data_manager-1", + "namespace": "rdk", + "type": "data_manager", + "attributes": { + "mongo_capture_config": { + "uri": "mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000" + } + } + } + ] +} +``` + +When `mongo_capture_config.uri` is configured, Data Capture will attempt to connect to the configured MongoDB +server and write captured tabular data to the configured `mongo_capture_config.database` and `mongo_capture_config.collection` +(or their defaults if unconfigured) after enqueuing that data to be written to disk. + +If writes to MongoDB fail for any resson, Data Capture will log an error for each failed write and continue capturing. + +Failing to write to MongoDB doesn't affect capturing & syncing data to cloud storage other than adding capture latency. + +{{< alert title="Caution" color="caution" >}} + +- Capturing directly to MongoDB may write data to MongoDB that later fails to be written to disk (and therefore never gets synced to cloud stroage). +- Capturing directly to MongoDB does not retry failed writes to MongoDB. As a consequence, it is NOT guaranteed all data captured will be written to MongoDB. This can happen in cases such MongoDB being inaccessible to viam-server or writes timing out. +- Capturing directly to MongoDB may reduce the maximum frequency that Data Capture can capture data due to the added latency of writing to MongoDB. If your use case needs to support very high capture rates, this feature may not be appropriate. + +{{< /alert >}} + +{{< /expand >}} + ## Configuration To capture data from one or more machines, you must first [configure the data management service](#data-management-service-configuration). @@ -204,6 +272,9 @@ The following attributes are available for the data management service: | `sync_interval_mins` | float | Optional | Time interval in minutes between syncing to the cloud. Viam does not impose a minimum or maximum on the frequency of data syncing. However, in practice, your hardware or network speed may impose limits on the frequency of data syncing.
Default: `0.1`, meaning once every 6 seconds. |

| | `delete_every_nth_when_disk_full` | int | Optional | How many files to delete when local storage meets the [fullness criteria](/services/data/#storage). The data management service will delete every Nth file that has been captured upon reaching this threshold. Use JSON mode to configure this attribute.
Default: `5`, meaning that every fifth captured file will be deleted. |

| | `maximum_num_sync_threads` | int | Optional | Max number of CPU threads to use for syncing data to the Viam Cloud.
Default: [runtime.NumCPU](https://pkg.go.dev/runtime#NumCPU)/2 so half the number of logical CPUs available to viam-server |

| +| `mongo_capture_config.uri` | string | Optional | The [MongoDB URI](https://www.mongodb.com/docs/v6.2/reference/connection-string/) data capture will attempt to write tabular data to after it is enqueued to be written to disk. When non empty, data capture will capture tabular data to the configured MongoDB database & collection at that URI.
See `mongo_capture_config.database` and `mongo_capture_config.collection` below for database & collection defaults.
See [Data Capture Directly To MongoDB](/services/data/#capture-directly-to-mongodb) for an example config.|

| +| `mongo_capture_config.database` | string | Optional | When `mongo_capture_config.uri` is non empty, changes the database data capture will write tabular data to.
Default: `"sensorData"` |

| +| `mongo_capture_config.collection` | string | Optional | When `mongo_capture_config.uri` is non empty, changes the collection data capture will write tabular data to.
Default: `"readings"` |

| | `cache_size_kb` | float | Optional | `viam-micro-server` only. The maximum amount of storage bytes (in kilobytes) allocated to a data collector.
Default: `1` KB. |

| ### Resource data capture configuration From cb5a7d339f5894a71bc8738f2e73c39b24fea799 Mon Sep 17 00:00:00 2001 From: Nick Sanford Date: Thu, 14 Nov 2024 11:02:42 -0500 Subject: [PATCH 2/6] wip --- docs/services/data/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/services/data/_index.md b/docs/services/data/_index.md index c74b9e2efe..93abd1c821 100644 --- a/docs/services/data/_index.md +++ b/docs/services/data/_index.md @@ -180,7 +180,7 @@ Failing to write to MongoDB doesn't affect capturing & syncing data to cloud sto {{< alert title="Caution" color="caution" >}} -- Capturing directly to MongoDB may write data to MongoDB that later fails to be written to disk (and therefore never gets synced to cloud stroage). +- Capturing directly to MongoDB may write data to MongoDB that later fails to be written to disk (and therefore never gets synced to cloud storage). - Capturing directly to MongoDB does not retry failed writes to MongoDB. As a consequence, it is NOT guaranteed all data captured will be written to MongoDB. This can happen in cases such MongoDB being inaccessible to viam-server or writes timing out. - Capturing directly to MongoDB may reduce the maximum frequency that Data Capture can capture data due to the added latency of writing to MongoDB. If your use case needs to support very high capture rates, this feature may not be appropriate. From 86dafe979fb374b475afb685a0cd7fe5c897b083 Mon Sep 17 00:00:00 2001 From: Nick Sanford Date: Thu, 14 Nov 2024 11:04:13 -0500 Subject: [PATCH 3/6] spell --- docs/services/data/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/services/data/_index.md b/docs/services/data/_index.md index 93abd1c821..632f51b0ca 100644 --- a/docs/services/data/_index.md +++ b/docs/services/data/_index.md @@ -174,7 +174,7 @@ When `mongo_capture_config.uri` is configured, Data Capture will attempt to conn server and write captured tabular data to the configured `mongo_capture_config.database` and `mongo_capture_config.collection` (or their defaults if unconfigured) after enqueuing that data to be written to disk. -If writes to MongoDB fail for any resson, Data Capture will log an error for each failed write and continue capturing. +If writes to MongoDB fail for any reason, Data Capture will log an error for each failed write and continue capturing. Failing to write to MongoDB doesn't affect capturing & syncing data to cloud storage other than adding capture latency. From d3e38a16e8dd24f29dfd6c90bc737b894d679894 Mon Sep 17 00:00:00 2001 From: Nick Sanford Date: Thu, 14 Nov 2024 11:07:23 -0500 Subject: [PATCH 4/6] wip --- docs/services/data/_index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/services/data/_index.md b/docs/services/data/_index.md index 632f51b0ca..bd1a4419e4 100644 --- a/docs/services/data/_index.md +++ b/docs/services/data/_index.md @@ -122,7 +122,7 @@ You can also control how local data is deleted if your machine's local storage b {{< expand "Capture Directly to MongoDB" >}} -Data Capture supports capturing tabular data directly to MongoDB in addition to capturing to disk. +Data capture supports capturing tabular data directly to MongoDB in addition to capturing to disk. This feature is intended to support use cases like offline dashboards which don't require strong data delivery or consistency guarantees. @@ -170,11 +170,11 @@ as well as to the Viam capture directory on disk: } ``` -When `mongo_capture_config.uri` is configured, Data Capture will attempt to connect to the configured MongoDB +When `mongo_capture_config.uri` is configured, Data capture will attempt to connect to the configured MongoDB server and write captured tabular data to the configured `mongo_capture_config.database` and `mongo_capture_config.collection` (or their defaults if unconfigured) after enqueuing that data to be written to disk. -If writes to MongoDB fail for any reason, Data Capture will log an error for each failed write and continue capturing. +If writes to MongoDB fail for any reason, Data capture will log an error for each failed write and continue capturing. Failing to write to MongoDB doesn't affect capturing & syncing data to cloud storage other than adding capture latency. @@ -182,7 +182,7 @@ Failing to write to MongoDB doesn't affect capturing & syncing data to cloud sto - Capturing directly to MongoDB may write data to MongoDB that later fails to be written to disk (and therefore never gets synced to cloud storage). - Capturing directly to MongoDB does not retry failed writes to MongoDB. As a consequence, it is NOT guaranteed all data captured will be written to MongoDB. This can happen in cases such MongoDB being inaccessible to viam-server or writes timing out. -- Capturing directly to MongoDB may reduce the maximum frequency that Data Capture can capture data due to the added latency of writing to MongoDB. If your use case needs to support very high capture rates, this feature may not be appropriate. +- Capturing directly to MongoDB may reduce the maximum frequency that Data capture can capture data due to the added latency of writing to MongoDB. If your use case needs to support very high capture rates, this feature may not be appropriate. {{< /alert >}} @@ -272,7 +272,7 @@ The following attributes are available for the data management service: | `sync_interval_mins` | float | Optional | Time interval in minutes between syncing to the cloud. Viam does not impose a minimum or maximum on the frequency of data syncing. However, in practice, your hardware or network speed may impose limits on the frequency of data syncing.
Default: `0.1`, meaning once every 6 seconds. |

| | `delete_every_nth_when_disk_full` | int | Optional | How many files to delete when local storage meets the [fullness criteria](/services/data/#storage). The data management service will delete every Nth file that has been captured upon reaching this threshold. Use JSON mode to configure this attribute.
Default: `5`, meaning that every fifth captured file will be deleted. |

| | `maximum_num_sync_threads` | int | Optional | Max number of CPU threads to use for syncing data to the Viam Cloud.
Default: [runtime.NumCPU](https://pkg.go.dev/runtime#NumCPU)/2 so half the number of logical CPUs available to viam-server |

| -| `mongo_capture_config.uri` | string | Optional | The [MongoDB URI](https://www.mongodb.com/docs/v6.2/reference/connection-string/) data capture will attempt to write tabular data to after it is enqueued to be written to disk. When non empty, data capture will capture tabular data to the configured MongoDB database & collection at that URI.
See `mongo_capture_config.database` and `mongo_capture_config.collection` below for database & collection defaults.
See [Data Capture Directly To MongoDB](/services/data/#capture-directly-to-mongodb) for an example config.|

| +| `mongo_capture_config.uri` | string | Optional | The [MongoDB URI](https://www.mongodb.com/docs/v6.2/reference/connection-string/) data capture will attempt to write tabular data to after it is enqueued to be written to disk. When non empty, data capture will capture tabular data to the configured MongoDB database & collection at that URI.
See `mongo_capture_config.database` and `mongo_capture_config.collection` below for database & collection defaults.
See [Data capture Directly To MongoDB](/services/data/#capture-directly-to-mongodb) for an example config.|

| | `mongo_capture_config.database` | string | Optional | When `mongo_capture_config.uri` is non empty, changes the database data capture will write tabular data to.
Default: `"sensorData"` |

| | `mongo_capture_config.collection` | string | Optional | When `mongo_capture_config.uri` is non empty, changes the collection data capture will write tabular data to.
Default: `"readings"` |

| | `cache_size_kb` | float | Optional | `viam-micro-server` only. The maximum amount of storage bytes (in kilobytes) allocated to a data collector.
Default: `1` KB. |

| From 768e59b3a22d43126ed41acf5d8b8b0f88d6bedf Mon Sep 17 00:00:00 2001 From: Sierra Guequierre Date: Thu, 14 Nov 2024 11:46:12 -0500 Subject: [PATCH 5/6] Update _index.md mostly line spacing and caps --- docs/services/data/_index.md | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/docs/services/data/_index.md b/docs/services/data/_index.md index bd1a4419e4..5d8b5f681e 100644 --- a/docs/services/data/_index.md +++ b/docs/services/data/_index.md @@ -120,15 +120,13 @@ You can also control how local data is deleted if your machine's local storage b {{< /expand >}} -{{< expand "Capture Directly to MongoDB" >}} +{{< expand "Capture directly to MongoDB" >}} Data capture supports capturing tabular data directly to MongoDB in addition to capturing to disk. -This feature is intended to support use cases like offline dashboards which don't require strong data -delivery or consistency guarantees. +This feature is intended to support use cases like offline dashboards which don't require strong data delivery or consistency guarantees. -Here is a sample configuration that will capture fake sensor readings both to the configured MongoDB URI -as well as to the Viam capture directory on disk: +Here is a sample configuration that will capture fake sensor readings both to the configured MongoDB URI as well as to the Viam capture directory on disk: ```json { @@ -170,19 +168,19 @@ as well as to the Viam capture directory on disk: } ``` -When `mongo_capture_config.uri` is configured, Data capture will attempt to connect to the configured MongoDB -server and write captured tabular data to the configured `mongo_capture_config.database` and `mongo_capture_config.collection` -(or their defaults if unconfigured) after enqueuing that data to be written to disk. +When `mongo_capture_config.uri` is configured, data capture will attempt to connect to the configured MongoDB server and write captured tabular data to the configured `mongo_capture_config.database` and `mongo_capture_config.collection` (or their defaults if unconfigured) after enqueuing that data to be written to disk. -If writes to MongoDB fail for any reason, Data capture will log an error for each failed write and continue capturing. +If writes to MongoDB fail for any reason, data capture will log an error for each failed write and continue capturing. -Failing to write to MongoDB doesn't affect capturing & syncing data to cloud storage other than adding capture latency. +Failing to write to MongoDB doesn't affect capturing and syncing data to cloud storage other than adding capture latency. {{< alert title="Caution" color="caution" >}} - Capturing directly to MongoDB may write data to MongoDB that later fails to be written to disk (and therefore never gets synced to cloud storage). -- Capturing directly to MongoDB does not retry failed writes to MongoDB. As a consequence, it is NOT guaranteed all data captured will be written to MongoDB. This can happen in cases such MongoDB being inaccessible to viam-server or writes timing out. -- Capturing directly to MongoDB may reduce the maximum frequency that Data capture can capture data due to the added latency of writing to MongoDB. If your use case needs to support very high capture rates, this feature may not be appropriate. +- Capturing directly to MongoDB does not retry failed writes to MongoDB. As a consequence, it is NOT guaranteed all data captured will be written to MongoDB. + This can happen in cases such as MongoDB being inaccessible to `viam-server` or writes timing out. +- Capturing directly to MongoDB may reduce the maximum frequency that data capture can capture data due to the added latency of writing to MongoDB. + If your use case needs to support very high capture rates, this feature may not be appropriate. {{< /alert >}} From 4cd92cbaa58e046da73e4692f9b7e89a77d6b060 Mon Sep 17 00:00:00 2001 From: nicksanford Date: Thu, 14 Nov 2024 11:48:42 -0500 Subject: [PATCH 6/6] Update docs/services/data/_index.md Co-authored-by: Sierra Guequierre --- docs/services/data/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/services/data/_index.md b/docs/services/data/_index.md index 5d8b5f681e..4804298b5b 100644 --- a/docs/services/data/_index.md +++ b/docs/services/data/_index.md @@ -126,7 +126,7 @@ Data capture supports capturing tabular data directly to MongoDB in addition to This feature is intended to support use cases like offline dashboards which don't require strong data delivery or consistency guarantees. -Here is a sample configuration that will capture fake sensor readings both to the configured MongoDB URI as well as to the Viam capture directory on disk: +Here is a sample configuration that will capture fake sensor readings both to the configured MongoDB URI as well as to the `~/.viam/capture` directory on disk: ```json {