How to reduce the memory usage?

[xiaobingxia-at]

Hello Milvus experts, I'm trying to reduce the memory usage while maintaining a relatively high recall rate, I have tried following options:

• Use in-memory index: IVF-SQ8 (IVF-PQ recall rate is a little bit low), the memory usage is about 40% of raw data size.
• Use HNSW + MMAP: The memory usage is 28% of raw data size. (When hosting 160m 1536-d vectors, raw data is 919GB, query node memory usage is 260GB, and Disk Usage is 610GB)
• Use Disk ANN: with default config, the memory usage is 42.6% of raw data size. (When hosting 134.948m 1536-d vectors, raw data is 772.18GB, disk ann index on disk is: 749.5GB, Query Node memory usage is: 319GB). This is concerning, the reason I come to diskann is because of the limited memory resource. If anyway it will take memory which is >40% of raw data, then there is no difference from using in-memory index. Is there anything I did wrong here?

My goal is to reduce the memory usage to be of 10% of raw data size given my data volume. Is there any good advice?
Thanks!

[yhmo]

Typically, the index size compares to the original data size:

• IVF_FLAT/HNSW, a bit larger than the original data size.
• IVF_SQ8/IVF_PQ, 25% ~ 30% of the original data size.
• DISKANN, 1/4 ~ 1/6 of the original data size.

You got 40% memory usage with IVF_SQ8, which is higher than expected. Is your milvus standalone? If so, maybe some compaction/index tasks are not finished.
Normally, if MMAP is enabled, the memory usage is much less than no-mmap. You got 28% for HNSW+MMAP, I think this is expected.
42.6% for DISKANN is also unexpected. Ideally, it should be 20% ~ 25%. Did you change the configuration of PQCodeBugetGBRatio?

[xiaofan-luan]

Hello Milvus experts, I'm trying to reduce the memory usage while maintaining a relatively high recall rate, I have tried following options:

• Use in-memory index: IVF-SQ8 (IVF-PQ recall rate is a little bit low), the memory usage is about 40% of raw data size.
• Use HNSW + MMAP: The memory usage is 28% of raw data size. (When hosting 160m 1536-d vectors, raw data is 919GB, query node memory usage is 260GB, and Disk Usage is 610GB)
• Use Disk ANN: with default config, the memory usage is 42.6% of raw data size. (When hosting 134.948m 1536-d vectors, raw data is 772.18GB, disk ann index on disk is: 749.5GB, Query Node memory usage is: 319GB). This is concerning, the reason I come to diskann is because of the limited memory resource. If anyway it will take memory which is >40% of raw data, then there is no difference from using in-memory index. Is there anything I did wrong here?

My goal is to reduce the memory usage to be of 10% of raw data size given my data volume. Is there any good advice? Thanks!

I don't recommend to use IVFSQ/PQ in your use case.

You can tune diskANN params to reduce the memory foot print(disable the cache and increase the quantazitio ratio)
For HNSW I believe you can insert more data until it reaches OOM.

[xiaobingxia-at]

How about the IVF-SQ8 + MMAP? I noticed that the search latency is 50ms compared to 10ms when using HNSW + MMAP. But 50ms is good enough for us.

[xiaobingxia-at]

And why does increasing quantatizition ratio can help reduce the memory footprint? I thought it will increase the recall by using more memory?

[xiaofan-luan]

quantazation change 32bit fp to 8bit, 4bit, 2bit or even 1bit.
the less the bit number, the less memory you will need and potentially better performance , but on the other side lower the recall.

[xiaofan-luan]

IVF SQ8 is usually much slower compared to Graph index due to he IO granularity