-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DOCS-3095] - local mongo collection #3673
Merged
nicksanford
merged 6 commits into
viamrobotics:main
from
nicksanford:DOCS-3095-local-mongo-collection
Nov 14, 2024
Merged
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
fcd6669
[DOCS-3095] - local mongo collection
nicksanford cb5a7d3
wip
nicksanford 86dafe9
spell
nicksanford d3e38a1
wip
nicksanford 768e59b
Update _index.md mostly line spacing and caps
sguequierre 4cd92cb
Update docs/services/data/_index.md
nicksanford File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||
---|---|---|---|---|
|
@@ -120,6 +120,74 @@ You can also control how local data is deleted if your machine's local storage b | |||
|
||||
{{< /expand >}} | ||||
|
||||
{{< expand "Capture Directly to MongoDB" >}} | ||||
|
||||
Data capture supports capturing tabular data directly to MongoDB in addition to capturing to disk. | ||||
|
||||
This feature is intended to support use cases like offline dashboards which don't require strong data | ||||
delivery or consistency guarantees. | ||||
|
||||
Here is a sample configuration that will capture fake sensor readings both to the configured MongoDB URI | ||||
as well as to the Viam capture directory on disk: | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||
|
||||
```json | ||||
{ | ||||
"components": [ | ||||
{ | ||||
"name": "sensor-1", | ||||
"namespace": "rdk", | ||||
"type": "sensor", | ||||
"model": "fake", | ||||
"attributes": {}, | ||||
"service_configs": [ | ||||
{ | ||||
"type": "data_manager", | ||||
"attributes": { | ||||
"capture_methods": [ | ||||
{ | ||||
"method": "Readings", | ||||
"capture_frequency_hz": 0.5, | ||||
"additional_params": {} | ||||
} | ||||
] | ||||
} | ||||
} | ||||
] | ||||
} | ||||
], | ||||
"services": [ | ||||
{ | ||||
"name": "data_manager-1", | ||||
"namespace": "rdk", | ||||
"type": "data_manager", | ||||
"attributes": { | ||||
"mongo_capture_config": { | ||||
"uri": "mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000" | ||||
} | ||||
} | ||||
} | ||||
] | ||||
} | ||||
``` | ||||
|
||||
When `mongo_capture_config.uri` is configured, Data capture will attempt to connect to the configured MongoDB | ||||
server and write captured tabular data to the configured `mongo_capture_config.database` and `mongo_capture_config.collection` | ||||
(or their defaults if unconfigured) after enqueuing that data to be written to disk. | ||||
|
||||
If writes to MongoDB fail for any reason, Data capture will log an error for each failed write and continue capturing. | ||||
|
||||
Failing to write to MongoDB doesn't affect capturing & syncing data to cloud storage other than adding capture latency. | ||||
|
||||
{{< alert title="Caution" color="caution" >}} | ||||
|
||||
- Capturing directly to MongoDB may write data to MongoDB that later fails to be written to disk (and therefore never gets synced to cloud storage). | ||||
- Capturing directly to MongoDB does not retry failed writes to MongoDB. As a consequence, it is NOT guaranteed all data captured will be written to MongoDB. This can happen in cases such MongoDB being inaccessible to viam-server or writes timing out. | ||||
- Capturing directly to MongoDB may reduce the maximum frequency that Data capture can capture data due to the added latency of writing to MongoDB. If your use case needs to support very high capture rates, this feature may not be appropriate. | ||||
|
||||
{{< /alert >}} | ||||
|
||||
{{< /expand >}} | ||||
|
||||
## Configuration | ||||
|
||||
To capture data from one or more machines, you must first [configure the data management service](#data-management-service-configuration). | ||||
|
@@ -204,6 +272,9 @@ The following attributes are available for the data management service: | |||
| `sync_interval_mins` | float | Optional | Time interval in minutes between syncing to the cloud. Viam does not impose a minimum or maximum on the frequency of data syncing. However, in practice, your hardware or network speed may impose limits on the frequency of data syncing. <br> Default: `0.1`, meaning once every 6 seconds. | <p class="center-text"><i class="fas fa-check" title="yes"></i></p> | | ||||
| `delete_every_nth_when_disk_full` | int | Optional | How many files to delete when local storage meets the [fullness criteria](/services/data/#storage). The data management service will delete every Nth file that has been captured upon reaching this threshold. Use JSON mode to configure this attribute. <br> Default: `5`, meaning that every fifth captured file will be deleted. | <p class="center-text"><i class="fas fa-times" title="no"></i></p> | | ||||
| `maximum_num_sync_threads` | int | Optional | Max number of CPU threads to use for syncing data to the Viam Cloud. <br> Default: [runtime.NumCPU](https://pkg.go.dev/runtime#NumCPU)/2 so half the number of logical CPUs available to viam-server | <p class="center-text"><i class="fas fa-times" title="no"></i></p> | | ||||
| `mongo_capture_config.uri` | string | Optional | The [MongoDB URI](https://www.mongodb.com/docs/v6.2/reference/connection-string/) data capture will attempt to write tabular data to after it is enqueued to be written to disk. When non empty, data capture will capture tabular data to the configured MongoDB database & collection at that URI.<br>See `mongo_capture_config.database` and `mongo_capture_config.collection` below for database & collection defaults.<br>See [Data capture Directly To MongoDB](/services/data/#capture-directly-to-mongodb) for an example config.| <p class="center-text"><i class="fas fa-times" title="no"></i></p> | | ||||
| `mongo_capture_config.database` | string | Optional | When `mongo_capture_config.uri` is non empty, changes the database data capture will write tabular data to. <br> Default: `"sensorData"` | <p class="center-text"><i class="fas fa-times" title="no"></i></p> | | ||||
| `mongo_capture_config.collection` | string | Optional | When `mongo_capture_config.uri` is non empty, changes the collection data capture will write tabular data to.<br> Default: `"readings"` | <p class="center-text"><i class="fas fa-times" title="no"></i></p> | | ||||
nicksanford marked this conversation as resolved.
Show resolved
Hide resolved
|
||||
| `cache_size_kb` | float | Optional | `viam-micro-server` only. The maximum amount of storage bytes (in kilobytes) allocated to a data collector. <br> Default: `1` KB. | <p class="center-text"><i class="fas fa-check" title="yes"></i></p> | | ||||
|
||||
### Resource data capture configuration | ||||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.