Skip to content

Commit

Permalink
[DOCS-3095] - local mongo collection
Browse files Browse the repository at this point in the history
  • Loading branch information
nicksanford committed Nov 13, 2024
1 parent bcbdd0f commit 2e21eab
Showing 1 changed file with 71 additions and 0 deletions.
71 changes: 71 additions & 0 deletions docs/services/data/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,74 @@ You can also control how local data is deleted if your machine's local storage b

{{< /expand >}}

{{< expand "Capture Directly to MongoDB" >}}

Data Capture supports capturing tabular data directly to MongoDB in addition to capturing to disk.

This feature is intended to support usecases like offline dashboards which don't require strong data
delivery or consistency guarantees.

Here is a sample cloud config that will capture fake sensor readings both to the configured MongoDB URI
as well as to the viam capture directory on disk:

```json
{
"components": [
{
"name": "sensor-1",
"namespace": "rdk",
"type": "sensor",
"model": "fake",
"attributes": {},
"service_configs": [
{
"type": "data_manager",
"attributes": {
"capture_methods": [
{
"method": "Readings",
"capture_frequency_hz": 0.5,
"additional_params": {}
}
]
}
}
]
}
],
"services": [
{
"name": "data_manager-1",
"namespace": "rdk",
"type": "data_manager",
"attributes": {
"mongo_capture_config": {
"uri": "mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000"
}
}
}
]
}
```

When `mongo_capture_config.uri` is configured, Data Capture will attempt to connect to the configured MongoDB
server and write captured tabular data to the configured `mongo_capture_config.database` and `mongo_capture_config.collection`
(or their defaults if unconfigured) after enqueuing that data to be written to disk.

If writes to MongoDB fail for any resson, Data Capture will log an error for each failed write and continue capturing.

Failing to write to MongoDB doesn't affect capturing & syncing data to app.viam.com other than adding capture latency.

{{< alert title="Caution" color="caution" >}}

- Capturing Directly to MongoDB may write data to MongoDB that later fails to be written to disk (and therefore never gets synced to app.viam.com).
- Capturing Directly to MongoDB does not retry failed writes to MongoDB. As a consequence, it is NOT guaranteed all data captured will be written to MongoDB. This can happen in cases such MongoDB being inaccessible to viam-server or writes timing out.
- Capturing Directly to MongoDB may reduce the maximum frequency that Data Capture can capture data due to the added latency of writing to MongoDB. If your usecase needs to support very high capture rates, this feature may not be appropriate.

{{< /alert >}}

{{< /expand >}}

## Configuration

To capture data from one or more machines, you must first [configure the data management service](#data-management-service-configuration).
Expand Down Expand Up @@ -204,6 +272,9 @@ The following attributes are available for the data management service:
| `sync_interval_mins` | float | Optional | Time interval in minutes between syncing to the cloud. Viam does not impose a minimum or maximum on the frequency of data syncing. However, in practice, your hardware or network speed may impose limits on the frequency of data syncing. <br> Default: `0.1`, meaning once every 6 seconds. | <p class="center-text"><i class="fas fa-check" title="yes"></i></p> |
| `delete_every_nth_when_disk_full` | int | Optional | How many files to delete when local storage meets the [fullness criteria](/services/data/#storage). The data management service will delete every Nth file that has been captured upon reaching this threshold. Use JSON mode to configure this attribute. <br> Default: `5`, meaning that every fifth captured file will be deleted. | <p class="center-text"><i class="fas fa-times" title="no"></i></p> |
| `maximum_num_sync_threads` | int | Optional | Max number of CPU threads to use for syncing data to the Viam Cloud. <br> Default: [runtime.NumCPU](https://pkg.go.dev/runtime#NumCPU)/2 so half the number of logical CPUs available to viam-server | <p class="center-text"><i class="fas fa-times" title="no"></i></p> |
| `mongo_capture_config.uri` | string | Optional | The [MongoDB URI](https://www.mongodb.com/docs/v6.2/reference/connection-string/) data capture will attempt to write tabular data to after it is enqueued to be written to disk. When non empty, data capture will capture tabular data to the configured MongoDB database & collection at that URI.<br>See `mongo_capture_config.database` and `mongo_capture_config.collection` below for database & collection defaults.<br>See [Data Capture Directly To MongoDB](/services/data/#capture-directly-to-mongodb) for an example config.| <p class="center-text"><i class="fas fa-times" title="no"></i></p> |
| `mongo_capture_config.database` | string | Optional | When `mongo_capture_config.uri` is non empty, changes the database data capture will write tabular data to. <br> Default: `"sensorData"` | <p class="center-text"><i class="fas fa-times" title="no"></i></p> |
| `mongo_capture_config.collection` | string | Optional | When `mongo_capture_config.uri` is non empty, changes the collection data capture will write tabular data to.<br> Default: `"readings"` | <p class="center-text"><i class="fas fa-times" title="no"></i></p> |
| `cache_size_kb` | float | Optional | `viam-micro-server` only. The maximum amount of storage bytes (in kilobytes) allocated to a data collector. <br> Default: `1` KB. | <p class="center-text"><i class="fas fa-check" title="yes"></i></p> |

### Resource data capture configuration
Expand Down

0 comments on commit 2e21eab

Please sign in to comment.