Nebula Graph write performance tuning

[Shishqa]

Hi! I am tuning Nebula Graph to handle the scenario with huge upload rate. Can you please help me better understand the bottlenecks of my experimental setup?

The setup can't process more than 1 million entries insert (vertices and edges) per second in total (about 300k entries per graphd process). However, CPU/disk_io/net_io seem underutilized: CPU utilization is around 60%, disk writes are under 5MB/s and reads are around 150MB/s,

RAM is utilized fully (mostly file cache). However I have configured frequent flushes based on Ceph's recommendations, which also uses rocksdb: https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive/. It seems like Nebula utilizes and tracks all available RAM via its memory tracker.

Our setup consists of 3 containerized nodes, each hosting all three services: graphd, metad, and storaged. I've allocated most of the memory to graphd (0.7), leaving 0.2 (24GB) for storaged.

Resource allocations are:

• 48vCPU for each node
• 128GB RAM
• 1GB/s net bw
• 512 MB/s SSD bw
• 1 TB SSD nvme

Configurations:

metad:

--daemonize=false
--containerized=true
--meta_server_addrs=ip1:9559,ip2:9559,ip3:9559
--local_ip=ip
--ws_ip=127.0.0.1
--port=9559
--ws_http_port=19559
--data_path=/data/meta
--log_dir=/logs/metad
--v=0
--minloglevel=0

graphd:

--daemonize=false
--containerized=true
--memory_tracker_detail_log=true
--memory_tracker_limit_ratio=0.7
--memory_tracker_untracked_reserved_memory_mb=1024
--system_memory_high_watermark_ratio=2.0
--meta_server_addrs=ip1:9559,ip2:9559,ip3:9559
--local_ip=ip
--port=9669
--ws_ip=127.0.0.1
--ws_http_port=19669
--log_dir=/logs/graphd
--v=0
--minloglevel=0
--max_job_size=24
--max_allowed_query_size=40194304
--storage_client_timeout_ms=600000

storaged:

--daemonize=false
--containerized=true
--memory_tracker_detail_log=true
--memory_tracker_limit_ratio=0.2
--memory_tracker_untracked_reserved_memory_mb=1024
--meta_server_addrs=ip1:9559,ip2:9559,ip3:9559
--local_ip=ip
--ws_ip=127.0.0.1
--port=9779
--ws_http_port=19779
--data_path=/data/storage
--log_dir=/logs/storaged
--v=0
--minloglevel=0
--rocksdb_batch_size=4096
--rocksdb_block_cache=8192
--max_batch_size=4096
--num_io_threads=24
--rocksdb_compression=no
--enable_partitioned_index_filter=true
--rocksdb_db_options='{"max_subcompactions": "16", "max_background_jobs": "16", "max_total_wal_size": "1073741824", "writable_file_max_buffer_size": "0", "compaction_readahead_size": "2MB", "skip_checking_sst_file_sizes_on_db_open":"true"}'
--rocksdb_column_family_options='{"disable_auto_compactions":"false", "write_buffer_size": "8388608", "max_write_buffer_number": "128", "min_write_buffer_number_to_merge": "16", "compaction_style": "kCompactionStyleLevel", "level0_file_num_compaction_trigger": "8", "max_bytes_for_level_base": "1073741824", "max_bytes_for_level_multiplier": "8"}'
--enable_rocksdb_statistics
--rocksdb_stats_level=kAll

Here's the current load setup:

The graph space has 3x replication and 60 partitions.

~40 loader processes are created, each containing 3 threads. Each thread writes to a separate graphd. The loader reads data from the disk and forms batch INSERT requests. Threads send requests in parallel but are limited by a semaphore to prevent high inflight.

Right now, each batch contains 1000 entries (nodes and edges combined), and I limit inflight per process to 24 requests. Thus, the total inflight is limited to 1k. With more requests, either the raft buffer overflows, or graphd starts limiting requests with GraphMemoryLimitExceeded.

Can Nebula handle more load with 4+ nodes? Or would it make sense to separate services into different containers? Or am I missing an obvious bottleneck that could be resolved by simply adding resources to the node?

Thank you in advance for any advice or suggestions!

[critical27]

1. the raft buffer overflows can't be addressed totally (there is a parameter, but it only reserve more space for buffers). But usually the bottleneck is raft, not the buffer itself.
2. GraphMemoryLimitExceeded could be relieved by adding more container nodes.
3. According to what you described, I guess the network used by raft is the bottleneck. Have you checked it yet? (Usually 100Mb/s is a threshold)

Can Nebula handle more load with 4+ nodes?

Yes, adding more nodes with more partitions will help.

[Shishqa]

Thank you for the response!

The guess is quite precise, as everything slows down when I change replication from x1 to x3. But how can I address the raft network issue? I've tried to increase snapshot_part_rate_limit from 10MB to 1GB, but this didn't help

The network usage is just around 100-150mb/s