Moving the training data from one environment to another

[razakpm]

I have a staging environment and a production environment.
I have trained all information on vanna.ai in the staging environment.
I need to move the training data to the production vanna.ai.
Is there any script or easy way to move my training data to production environment?

[chrishage]

Hi team, I am also interested in this information. Regards, Chris

…

On Tue, Jul 30, 2024 at 9:30 AM Abdul Razak ***@***.***> wrote: I have a staging environment and a production environment. I have trained all information on vanna.ai in the staging environment. I need to move the training data to the production vanna.ai. Is there any script or easy way to move my training data to production environment? — Reply to this email directly, view it on GitHub <#575>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZBBTW36QR3KXSOCD6PCKY3ZO6BOBAVCNFSM6AAAAABLWIU7X2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGQZTONZSGM2DOOI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[andreped]

Hello, @razakpm and @chrishage! :]

I am not part of the vanna team but contributing when I can and might be able to assist.

Which vector database are you using and which configuration?

A while back we used Chroma with persistent storage as our vector DB. By doing persistent storage, a chroma.sqlite3 file was created, which we then could save somewhere else (e.g., Azure Storage) and then in a production environment we could download the persisted chroma.sqlite3 file and point to it during Chroma construction during app launch. That way, we didn't need to recreate the embeddings either, and the vector DB was already preconfigured in the new environment.

I believe something like that could work for you as well?

Even for Chroma, there are other options like using an HttpClient approach instead of persistent storage, but if you want two separate vector store instances, it would make sense to persist the vector DB data somewhere else. A similar idea would likely apply for other vector DBs, depending on what you are trying to do.

I hope it was helpful! Let me know if you need further assistance :]

[sinjup]

Hi, As another option, You can use a .json file to store all the training data and then use vn.train() to train it in the production environment. Here’s a general approach to achieve this:

Export Training Data from Staging Environment:
First, create a script to export your training data from the staging environment into a .json file. You can obtain this by using the API -'/api/v0/get_training_data. '

Transfer the .json File to Production:
Transfer the exported .json file from your staging environment to the production environment.

Import and Train in Production Environment:
Create a script to read the .json file in the production environment and use the vn.train() method to train the model with the imported data.