[Feat]: RAG support

[sotwi]

Description
Add a simple vectorDB rag to enhance memory.

Use Case
It would be possible to retrieve information from outside of the context window

[EXTREMOPHILARUM]

Do you have any ideas for implementing this? I thought it can maybe something like sqlite with vector embedding addon.

[the-rich-piana]

I'm not sure a full-fledged database is necessary. IMO just a JSON object that has values as float vectors.
Not sure how performant it would be, but probably lighter weight. But this probably means no HNSW indexing.

Also it would require the user to install the nomic-embed-text model or some other embed model. But ya happy to collaborate on this.

[EXTREMOPHILARUM]

Hello would love to collaborate. I was looking at vector db approach and found this. The article is for javascript and transformers.js but I believe it can be done in a similar manner with react native and llama.rn . llama.rn does have an embeddings function. We can connect over a call to discuss further.

[the-rich-piana]

Sure thing. Those both seem like good options.

[bi4key]

You can try this method:
Vali-98/ChatterUI#41 (comment)

[Vali-98]

Do you have any ideas for implementing this? I thought it can maybe something like sqlite with vector embedding addon.

Hey there, this is actually something which I've already done in ChatterUI under the hood alongside expo-sqlite with the sqlite-vec plugin, so I can say for certain its 100% possible to be done in react-native (you might want to instead use a non-expo library like op-sqlite).

For android, all you need to do is compile the plugin for the 4 common ABIs and have them in your source files, then patch or fork the sqlite library you are using to accept said plugin. Same goes for Swift, but it needs to be compiled to the correct objc or whatever source format iOS uses.

The issue however (and why I have not added RAG to mainline ChatterUI) is that small embedding models kinda suck. They barely retrieve the right information and are unreliable for anything beyond the most basic query. Embedding performance also isn't that amazing.