Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MongoDB(with RocksDB engine) RangeDeleter is too slow. #142

Open
SunguckLee opened this issue Apr 12, 2018 · 10 comments
Open

MongoDB(with RocksDB engine) RangeDeleter is too slow. #142

SunguckLee opened this issue Apr 12, 2018 · 10 comments

Comments

@SunguckLee
Copy link

SunguckLee commented Apr 12, 2018 •
edited
Loading

After chunk migration,
MongoDB call RangeDeleter(Not RocksDB, it's mongodb background thread name) to remove obsolete documents moved to another shards.

With MongoRocks, RangeDeleter is too slow. Sometimes it takes over 2 hours to remove only 1 chunk's documents. About 20000 ~ 80000 documents in a chunk. Some shards, RangeDeleter of 1 chunk takes 7~8 hour.

2018-04-12T07:03:36.110+0900 I SHARDING [RangeDeleter] Deleter starting delete for: mydb.mycollection from { user: -7881283402969115800 } -> { user: -7873819790711520591 }, with opId: 2371519422
2018-04-12T09:08:58.056+0900 I SHARDING [RangeDeleter] rangeDeleter deleted 83593 documents for mydb.mycollection from { user: -7881283402969115800 } -> { user: -7873819790711520591 }

So now, chunk balancing is blocked because stacked task in RangeDeleter queue is full.

And I also cleanup these obsolete documents(after moved to another shard) with "cleanupOrphaned" command. It also takes a lot of time.

// CASE-1
shard01:PRIMARY (admin) 17:31:47> db.getSisterDB("mydb").mycollection.count()
189984

shard01:PRIMARY (admin) 17:16:49> var nextKey = { };
shard01:PRIMARY (admin) 17:17:04> var result;
shard01:PRIMARY (admin) 17:17:04> while ( nextKey != null ) {
...   result = db.adminCommand( { cleanupOrphaned: "mydb.mycollection", startingFromKey: nextKey } );
...
...   if (result.ok != 1)
...      print("Unable to complete at this time: failure or timeout.")
...
...   printjson(result);
...
...   nextKey = result.stoppedAtKey;
... }
{
  "stoppedAtKey" : {
    "_id" : NumberLong("-3689348814741910320")
  },
  "ok" : 1
}
{ 
  "stoppedAtKey" : { 
    "_id" : { "$maxKey" : 1 }
   }, 
   "ok" : 1 
}
{ "ok" : 1 }

shard01:PRIMARY (admin) 17:31:47> db.getSisterDB("mydb").mycollection.count()
151656


// CASE-2
shard01:PRIMARY (admin) 17:45:15> db.getSisterDB("mydb").mycollection1.count().count()
151097

shard01:PRIMARY (admin) 17:46:04> var nextKey = { };
shard01:PRIMARY (admin) 17:46:04> var result;
shard01:PRIMARY (admin) 17:46:04>
shard01:PRIMARY (admin) 17:46:04> while ( nextKey != null ) {
...   result = db.adminCommand( { cleanupOrphaned: "mydb.mycollection1", startingFromKey: nextKey } );
...
...   if (result.ok != 1)
...      print("Unable to complete at this time: failure or timeout.")
...
...   printjson(result);
...
...   nextKey = result.stoppedAtKey;
... }
{
  "stoppedAtKey" : {
    "_id" : NumberLong("-936846091942123690")
  },
  "ok" : 1
}
{ 
  "stoppedAtKey" : { 
    "_id" : { "$maxKey" : 1 } 
  }, 
  "ok" : 1 }
{ "ok" : 1 }

shard01:PRIMARY (admin) 18:26:26> db.getSisterDB("mydb").mycollection1.count().count()
38379

There's only 189984 documents in this collection. And removed only 38328 documents with "cleanupOrphaned" command, But it took 14 minutes.
What is wrong with my MongoRocks setup ?

MongoDB Configuration

    engine: rocksdb
    rocksdb:
        cacheSizeGB: 30
        compression: lz4
        maxWriteMBPerSec : 1024
        configString: "block_based_table_factory={cache_index_and_filter_blocks=1};max_background_jobs=12;max_background_compactions=10;target_file_size_base=134217728;write_buffer_size=134217728;level0_file_num_compaction_trigger=3;level0_slowdown_writes_trigger=10"

RocksDB OPTIONS

[Version]
  rocksdb_version=5.11.0
  options_file_version=1.1

[DBOptions]
  manual_wal_flush=false
  allow_ingest_behind=false
  avoid_flush_during_shutdown=false
  avoid_flush_during_recovery=false
  info_log_level=INFO_LEVEL
  access_hint_on_compaction_start=NORMAL
  write_thread_max_yield_usec=100
  write_thread_slow_yield_usec=3
  wal_recovery_mode=kPointInTimeRecovery
  allow_concurrent_memtable_write=true
  enable_pipelined_write=false
  fail_if_options_file_error=false
  stats_dump_period_sec=600
  wal_bytes_per_sync=0
  max_total_wal_size=0
  delayed_write_rate=1073741824
  two_write_queues=false
  bytes_per_sync=1048576
  writable_file_max_buffer_size=1048576
  wal_dir=/data/mongodb/db/journal
  log_file_time_to_roll=0
  keep_log_file_num=1000
  WAL_ttl_seconds=0
  db_write_buffer_size=0
  table_cache_numshardbits=6
  max_open_files=-1
  max_file_opening_threads=16
  WAL_size_limit_MB=0
  max_background_flushes=2
  db_log_dir=
  max_background_compactions=10
  max_subcompactions=1
  max_background_jobs=12
  random_access_max_buffer_size=1048576
  delete_obsolete_files_period_micros=21600000000
  skip_stats_update_on_db_open=false
  skip_log_error_on_recovery=false
  dump_malloc_stats=false
  paranoid_checks=true
  is_fd_close_on_exec=true
  max_manifest_file_size=18446744073709551615
  error_if_exists=false
  use_adaptive_mutex=false
  enable_thread_tracking=true
  create_missing_column_families=false
  create_if_missing=true
  manifest_preallocation_size=4194304
  base_background_compactions=-1
  use_fsync=false
  allow_2pc=false
  recycle_log_file_num=0
  use_direct_io_for_flush_and_compaction=false
  compaction_readahead_size=0
  use_direct_reads=false
  allow_mmap_writes=false
  preserve_deletes=false
  enable_write_thread_adaptive_yield=true
  max_log_file_size=0
  allow_fallocate=true
  allow_mmap_reads=false
  new_table_reader_for_compaction_inputs=false
  advise_random_on_open=true


[CFOptions "default"]
  compaction_style=kCompactionStyleLevel
  compaction_filter_factory=PrefixDeletingCompactionFilterFactory
  memtable_factory=SkipListFactory
  memtable_insert_with_hint_prefix_extractor=nullptr
  merge_operator=nullptr
  prefix_extractor=nullptr
  comparator=leveldb.BytewiseComparator
  bottommost_compression=kDisableCompressionOption
  compression=kSnappyCompression
  max_bytes_for_level_multiplier_additional=1:1:1:1:1:1:1
  max_sequential_skip_in_iterations=8
  memtable_prefix_bloom_size_ratio=0.000000
  bloom_locality=0
  memtable_huge_page_size=0
  inplace_update_num_locks=10000
  arena_block_size=16777216
  target_file_size_multiplier=1
  num_levels=7
  min_write_buffer_number_to_merge=1
  max_write_buffer_number_to_maintain=0
  write_buffer_size=134217728
  level0_stop_writes_trigger=36
  level0_slowdown_writes_trigger=10
  level0_file_num_compaction_trigger=3
  max_bytes_for_level_multiplier=10.000000
  hard_pending_compaction_bytes_limit=274877906944
  max_compaction_bytes=3355443200
  force_consistency_checks=false
  optimize_filters_for_hits=true
  compaction_options_fifo={allow_compaction=false;max_table_files_size=1073741824;ttl=0;}
  max_bytes_for_level_base=536870912
  level_compaction_dynamic_level_bytes=true
  target_file_size_base=134217728
  max_write_buffer_number=4
  soft_pending_compaction_bytes_limit=68719476736
  inplace_update_support=false
  compression_per_level=kNoCompression:kNoCompression:kLZ4Compression
  paranoid_file_checks=false
  table_factory=BlockBasedTable
  disable_auto_compactions=false
  compaction_options_universal={allow_trivial_move=false;size_ratio=1;min_merge_width=2;max_size_amplification_percent=200;max_merge_width=4294967295;compression_size_percent=-1;stop_style=kCompactionStopStyleTotalSize;}
  compaction_filter=nullptr
  compaction_pri=kByCompensatedSize
  max_successive_merges=0
  report_bg_io_stats=false

[TableOptions/BlockBasedTable "default"]
  read_amp_bytes_per_bit=8589934592
  format_version=2
  whole_key_filtering=true
  filter_policy=rocksdb.BuiltinBloomFilter
  verify_compression=false
  block_size_deviation=10
  block_size=16384
  partition_filters=false
  checksum=kCRC32c
  hash_index_allow_collision=true
  index_block_restart_interval=1
  block_restart_interval=16
  no_block_cache=false
  pin_l0_filter_and_index_blocks_in_cache=false
  cache_index_and_filter_blocks_with_high_priority=false
  metadata_block_size=4096
  cache_index_and_filter_blocks=true
  index_type=kBinarySearch
  flush_block_policy_factory=FlushBlockBySizePolicyFactory
@SunguckLee
Copy link
Author

SunguckLee commented Apr 18, 2018 •
edited
Loading

I have found the reason why RangeDeleter of MongoDB(with RocksDB) is so slow.

MongoDB use "indexScan" instance for RangeDeleter(of MongoDB not RocksDB) processing.
MongoDB RangeDeleter is not re-use indexScan for all documents of chunk, but re-create indexScan object for each document of chunk. This is also documented in MongoDB JIRA(https://jira.mongodb.org/browse/SERVER-33237).

But in this code(https://github.com/mongodb/mongo/blob/r3.4.14/src/mongo/db/dbhelpers.cpp#L366), worse thing is the first exec->getNext() call is really slow with RocksDB storage engine. Not for Wiredtiger storage engine.
I have profiled the elapsed time of the very first exec->getNext() call on busy server.

>> The first exec->getNext() call
  - Wiredtiger : 50~100 us (micro second)
  - RocksDB : 12000 ~ 600000 us

>> Following exec->getNext() call is almost same speed for both storage engine.

Because of current implementation of RangeDeleter, getNext() func is called only once for each indexScan object. Meaning RangeDeleter use very frist(very slow) exec->getNext().
So RangeDeleter with RocksDB is so slow but not slow for Wiredtiger storage engine.

Why first call of indexScan->getNext() is so slow with RocksDB ?

Regards,

@SunguckLee
Copy link
Author

This is happen the last release version of MongoDB and MongoRocks (and RocksDB)

MongoDB 3.4.14, RocksDB 5.12.2

@igorcanadi
Copy link
Contributor

Good find. The problem here is that getNext() needs to walk through all the tombstones at the beginning of the index (of which there are many, since RangeDeleter is just creating them). This yields O(N^2) behavior and is one of the most common anti-pattern when using LSM trees. Looks like https://jira.mongodb.org/browse/SERVER-33237 would optimize the performance of RangeDeleter in RocksDB case a lot.

@SunguckLee
Copy link
Author

Hi Igor.

I think tombstone is spreaded everywhere in the index. So I have expected every getNext() takes same time. I am curios that why first getNext() call only slow, following getNext() call is not.

We can avoid this RangeDeleter issue with simple modifying MongoDB RangeDeleter code.
And I know this is most common anti-pattern for LSM, But still RocksDB shows better performance than Wiredtiger for same workload.

Regards,

@igorcanadi
Copy link
Contributor

I think tombstone is spreaded everywhere in the index

That might be true at the beginning of the operation, but as soon as RangeDeleter kicks in it starts adding new tombstones at the beginning of the range, right?

But still RocksDB shows better performance than Wiredtiger for same workload.

That's good to hear.

@q96456
Copy link

q96456 commented Mar 3, 2021 •
edited
Loading

I I ran into the same problem using version 3.4.18.
I have confirmed the code,https://jira.mongodb.org/browse/SERVER-33237 is already included.
Is there any other solution?

2021-03-03T07:53:04.558+0800 I SHARDING [RangeDeleter] Deleter starting delete for: rs40.rs2 from { _id: 6862256481019475218 } -> { _id: 6862276335282767516 }, with opId: 2168351339
2021-03-03T07:53:04.558+0800 D SHARDING [RangeDeleter] begin removal of { : 6862256481019475218 } to { : 6862276335282767516 } in rs40.rs2 with write concern: { w: 1, j: false, wtimeout: 0 }
2021-03-03T15:13:37.414+0800 D SHARDING [RangeDeleter] end removal of { : 6862256481019475218 } to { : 6862276335282767516 } in rs40.rs2 (took 26432856ms)
2021-03-03T15:13:37.414+0800 I SHARDING [RangeDeleter] rangeDeleter deleted 333767 documents for rs40.rs2 from { _id: 6862256481019475218 } -> { _id: 6862276335282767516 }

@eharry
Copy link
Contributor

eharry commented Mar 3, 2021

@q96456
periodly run db.runCommand({setParameter: 1, rocksdbCompact:1}) should help.
this command do a global-scope rocksdb manual compaction.
All the tombstones should be cleared.

@q96456
Copy link

q96456 commented Mar 4, 2021 •
edited
Loading

@eharry
Will this command affect normal online read and write access?
And is there a command to see the number of tombstone entries？I use "db.serverStatus().rocksdb" ,output:

{
"stats" : [
"",
"** Compaction Stats [default] ",
"Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop",
"----------------------------------------------------------------------------------------------------------------------------------------------------------",
" L0 1/0 149.97 MB 0.3 0.0 0.0 0.0 19890.4 19890.4 0.0 1.0 0.0 55.9 364137 137705 2.644 0 0",
" L2 2/0 401.72 MB 0.8 26645.0 17628.2 9016.8 26421.6 17404.7 0.0 1.5 54.8 54.4 497536 33331 14.927 332G 216M",
" L3 3/0 290.41 MB 0.5 29140.3 19666.4 9473.9 20899.6 11425.7 0.0 1.1 61.2 43.9 487805 62242 7.837 453G 79M",
" L4 25/0 5.13 GB 1.0 105238.5 10214.8 95023.7 104148.2 9124.5 1211.2 10.2 53.2 52.7 2024409 37606 53.832 1146G 127M",
" L5 252/0 46.30 GB 1.0 107102.1 9702.9 97399.2 105615.7 8216.6 631.6 10.9 43.1 42.5 2542494 33409 76.102 2251G 9503M",
" L6 2274/0 556.84 GB 0.0 72281.1 8844.5 63436.6 63512.3 75.7 0.0 7.2 44.3 39.0 1669418 20767 80.388 2262G 239G",
" Sum 2557/0 609.10 GB 0.0 340407.0 66056.8 274350.2 340487.9 66137.7 1842.8 17.1 46.0 46.0 7585800 325060 23.337 6446G 249G",
" Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0",
"Uptime(secs): 47054460.2 total, 0.1 interval",
"Flush(GB): cumulative 19890.444, interval 0.000",
"AddFile(GB): cumulative 0.000, interval 0.000",
"AddFile(Total Files): cumulative 0, interval 0",
"AddFile(L0 Files): cumulative 0, interval 0",
"AddFile(Keys): cumulative 0, interval 0",
"Cumulative compaction: 340487.87 GB write, 7.41 MB/s write, 340406.98 GB read, 7.41 MB/s read, 7585799.7 seconds",
"Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds",
"Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count",
"",
" File Read Latency Histogram By Level [default] ",
" Level 0 read latency histogram (micros):",
"Count: 1295796917 Average: 60.5612 StdDev: 2.77",
"Min: 0 Median: 5.4195 Max: 29670336",
"Percentiles: P50: 5.42 P75: 10.79 P99: 719.82 P99.9: 6722.46 P99.99: 8832.06",
"------------------------------------------------------",
"[ 0, 1 ) 291549 0.022% 0.022% ",
"[ 1, 2 ) 32055228 2.474% 2.496% ",
"[ 2, 3 ) 56659337 4.373% 6.869% #",
"[ 3, 4 ) 205068374 15.826% 22.694% ###",
"[ 4, 5 ) 279901723 21.601% 44.295% ####",
"[ 5, 6 ) 176201722 13.598% 57.893% ###",
"[ 6, 7 ) 100322842 7.742% 65.635% ##",
"[ 7, 8 ) 42838182 3.306% 68.941% #",
"[ 8, 9 ) 19914755 1.537% 70.478% ",
"[ 9, 10 ) 25061476 1.934% 72.412% ",
"[ 10, 12 ) 84776982 6.542% 78.955% #",
"[ 12, 14 ) 45710282 3.528% 82.482% #",
"[ 14, 16 ) 21117662 1.630% 84.112% ",
"[ 16, 18 ) 7306757 0.564% 84.676% ",
"[ 18, 20 ) 4103952 0.317% 84.993% ",
"[ 20, 25 ) 7184271 0.554% 85.547% ",
"[ 25, 30 ) 7273907 0.561% 86.108% ",
"[ 30, 35 ) 7647191 0.590% 86.698% ",
"[ 35, 40 ) 8701190 0.671% 87.370% ",
"[ 40, 45 ) 10566180 0.815% 88.185% ",
"[ 45, 50 ) 10662804 0.823% 89.008% ",
"[ 50, 60 ) 15232207 1.176% 90.184% ",
"[ 60, 70 ) 9364251 0.723% 90.906% ",
"[ 70, 80 ) 6798875 0.525% 91.431% ",
"[ 80, 90 ) 5185664 0.400% 91.831% ",
"[ 90, 100 ) 3130610 0.242% 92.073% ",
"[ 100, 120 ) 4258387 0.329% 92.402% ",
"[ 120, 140 ) 3151842 0.243% 92.645% ",
"[ 140, 160 ) 3330865 0.257% 92.902% ",
"[ 160, 180 ) 9194135 0.710% 93.611% ",
"[ 180, 200 ) 12721330 0.982% 94.593% ",
"[ 200, 250 ) 22389073 1.728% 96.321% ",
"[ 250, 300 ) 9479653 0.732% 97.052% ",
"[ 300, 350 ) 7988809 0.617% 97.669% ",
"[ 350, 400 ) 4471637 0.345% 98.014% ",
"[ 400, 450 ) 2786185 0.215% 98.229% ",
"[ 450, 500 ) 2441024 0.188% 98.417% ",
"[ 500, 600 ) 4297044 0.332% 98.749% ",
"[ 600, 700 ) 2829991 0.218% 98.968% ",
"[ 700, 800 ) 2124337 0.164% 99.131% ",
"[ 800, 900 ) 1168697 0.090% 99.222% ",
"[ 900, 1000 ) 677493 0.052% 99.274% ",
"[ 1000, 1200 ) 983620 0.076% 99.350% ",
"[ 1200, 1400 ) 769725 0.059% 99.409% ",
"[ 1400, 1600 ) 611324 0.047% 99.456% ",
"[ 1600, 1800 ) 457980 0.035% 99.492% ",
"[ 1800, 2000 ) 333785 0.026% 99.518% ",
"[ 2000, 2500 ) 389848 0.030% 99.548% ",
"[ 2500, 3000 ) 170498 0.013% 99.561% ",
"[ 3000, 3500 ) 114797 0.009% 99.570% ",
"[ 3500, 4000 ) 95607 0.007% 99.577% ",
"[ 4000, 4500 ) 101676 0.008% 99.585% ",
"[ 4500, 5000 ) 380864 0.029% 99.614% ",
"[ 5000, 6000 ) 2672735 0.206% 99.820% ",
"[ 6000, 7000 ) 1425940 0.110% 99.931% ",
"[ 7000, 8000 ) 613515 0.047% 99.978% ",
"[ 8000, 9000 ) 188627 0.015% 99.992% ",
"[ 9000, 10000 ) 56676 0.004% 99.997% ",
"[ 10000, 12000 ) 25515 0.002% 99.999% ",
"[ 12000, 14000 ) 7177 0.001% 99.999% ",
"[ 14000, 16000 ) 3908 0.000% 100.000% ",
"[ 16000, 18000 ) 1564 0.000% 100.000% ",
"[ 18000, 20000 ) 669 0.000% 100.000% ",
"[ 20000, 25000 ) 1270 0.000% 100.000% ",
"[ 25000, 30000 ) 758 0.000% 100.000% ",
"[ 30000, 35000 ) 146 0.000% 100.000% ",
"[ 35000, 40000 ) 88 0.000% 100.000% ",
"[ 40000, 45000 ) 45 0.000% 100.000% ",
"[ 45000, 50000 ) 23 0.000% 100.000% ",
"[ 50000, 60000 ) 17 0.000% 100.000% ",
"[ 60000, 70000 ) 15 0.000% 100.000% ",
"[ 70000, 80000 ) 4 0.000% 100.000% ",
"[ 80000, 90000 ) 5 0.000% 100.000% ",
"[ 90000, 100000 ) 8 0.000% 100.000% ",
"[ 100000, 120000 ) 3 0.000% 100.000% ",
"[ 140000, 160000 ) 3 0.000% 100.000% ",
"[ 160000, 180000 ) 2 0.000% 100.000% ",
"[ 200000, 250000 ) 1 0.000% 100.000% ",
"[ 450000, 500000 ) 2 0.000% 100.000% ",
"[ 25000000, 30000000 ) 2 0.000% 100.000% ",
"",
"** Level 2 read latency histogram (micros):",
"Count: 1725204370 Average: 49.5815 StdDev: 1.56",
"Min: 0 Median: 4.9987 Max: 29552335",
"Percentiles: P50: 5.00 P75: 6.77 P99: 647.94 P99.9: 6359.34 P99.99: 8698.95",
"------------------------------------------------------",
"[ 0, 1 ) 302495 0.018% 0.018% ",
"[ 1, 2 ) 36027130 2.088% 2.106% ",
"[ 2, 3 ) 69828968 4.048% 6.153% #",
"[ 3, 4 ) 236334430 13.699% 19.852% ###",
"[ 4, 5 ) 520774407 30.186% 50.039% ######",
"[ 5, 6 ) 281288446 16.305% 66.343% ###",
"[ 6, 7 ) 192889830 11.181% 77.524% ##",
"[ 7, 8 ) 68704548 3.982% 81.506% #",
"[ 8, 9 ) 27180825 1.576% 83.082% ",
"[ 9, 10 ) 28951062 1.678% 84.760% ",
"[ 10, 12 ) 56886403 3.297% 88.057% #",
"[ 12, 14 ) 20912143 1.212% 89.269% ",
"[ 14, 16 ) 10014916 0.581% 89.850% ",
"[ 16, 18 ) 3879168 0.225% 90.075% ",
"[ 18, 20 ) 2109265 0.122% 90.197% ",
"[ 20, 25 ) 3929219 0.228% 90.425% ",
"[ 25, 30 ) 5004321 0.290% 90.715% ",
"[ 30, 35 ) 6856007 0.397% 91.112% ",
"[ 35, 40 ) 6814254 0.395% 91.507% ",
"[ 40, 45 ) 5331889 0.309% 91.816% ",
"[ 45, 50 ) 3893831 0.226% 92.042% ",
"[ 50, 60 ) 4326892 0.251% 92.293% ",
"[ 60, 70 ) 2175901 0.126% 92.419% ",
"[ 70, 80 ) 1756821 0.102% 92.521% ",
"[ 80, 90 ) 1936533 0.112% 92.633% ",
"[ 90, 100 ) 1428938 0.083% 92.716% ",
"[ 100, 120 ) 1775476 0.103% 92.819% ",
"[ 120, 140 ) 1422396 0.082% 92.901% ",
"[ 140, 160 ) 2044037 0.118% 93.020% ",
"[ 160, 180 ) 7470301 0.433% 93.453% ",
"[ 180, 200 ) 13080519 0.758% 94.211% ",
"[ 200, 250 ) 28462833 1.650% 95.861% ",
"[ 250, 300 ) 15226922 0.883% 96.743% ",
"[ 300, 350 ) 14876623 0.862% 97.606% ",
"[ 350, 400 ) 9853848 0.571% 98.177% ",
"[ 400, 450 ) 3884376 0.225% 98.402% ",
"[ 450, 500 ) 3332846 0.193% 98.595% ",
"[ 500, 600 ) 5179247 0.300% 98.895% ",
"[ 600, 700 ) 3763626 0.218% 99.114% ",
"[ 700, 800 ) 2567260 0.149% 99.262% ",
"[ 800, 900 ) 1470191 0.085% 99.348% ",
"[ 900, 1000 ) 928178 0.054% 99.401% ",
"[ 1000, 1200 ) 1326038 0.077% 99.478% ",
"[ 1200, 1400 ) 1003101 0.058% 99.536% ",
"[ 1400, 1600 ) 805291 0.047% 99.583% ",
"[ 1600, 1800 ) 605501 0.035% 99.618% ",
"[ 1800, 2000 ) 475149 0.028% 99.646% ",
"[ 2000, 2500 ) 656955 0.038% 99.684% ",
"[ 2500, 3000 ) 371857 0.022% 99.705% ",
"[ 3000, 3500 ) 286322 0.017% 99.722% ",
"[ 3500, 4000 ) 244543 0.014% 99.736% ",
"[ 4000, 4500 ) 247912 0.014% 99.751% ",
"[ 4500, 5000 ) 360783 0.021% 99.771% ",
"[ 5000, 6000 ) 1765079 0.102% 99.874% ",
"[ 6000, 7000 ) 1261507 0.073% 99.947% ",
"[ 7000, 8000 ) 610415 0.035% 99.982% ",
"[ 8000, 9000 ) 191825 0.011% 99.993% ",
"[ 9000, 10000 ) 59554 0.003% 99.997% ",
"[ 10000, 12000 ) 30981 0.002% 99.999% ",
"[ 12000, 14000 ) 8713 0.001% 99.999% ",
"[ 14000, 16000 ) 5026 0.000% 99.999% ",
"[ 16000, 18000 ) 2380 0.000% 100.000% ",
"[ 18000, 20000 ) 1456 0.000% 100.000% ",
"[ 20000, 25000 ) 2437 0.000% 100.000% ",
"[ 25000, 30000 ) 1533 0.000% 100.000% ",
"[ 30000, 35000 ) 618 0.000% 100.000% ",
"[ 35000, 40000 ) 514 0.000% 100.000% ",
"[ 40000, 45000 ) 429 0.000% 100.000% ",
"[ 45000, 50000 ) 329 0.000% 100.000% ",
"[ 50000, 60000 ) 389 0.000% 100.000% ",
"[ 60000, 70000 ) 175 0.000% 100.000% ",
"[ 70000, 80000 ) 78 0.000% 100.000% ",
"[ 80000, 90000 ) 41 0.000% 100.000% ",
"[ 90000, 100000 ) 25 0.000% 100.000% ",
"[ 100000, 120000 ) 28 0.000% 100.000% ",
"[ 120000, 140000 ) 18 0.000% 100.000% ",
"[ 140000, 160000 ) 4 0.000% 100.000% ",
"[ 160000, 180000 ) 2 0.000% 100.000% ",
"[ 180000, 200000 ) 6 0.000% 100.000% ",
"[ 200000, 250000 ) 11 0.000% 100.000% ",
"[ 250000, 300000 ) 5 0.000% 100.000% ",
"[ 300000, 350000 ) 7 0.000% 100.000% ",
"[ 350000, 400000 ) 7 0.000% 100.000% ",
"[ 400000, 450000 ) 2 0.000% 100.000% ",
"[ 450000, 500000 ) 2 0.000% 100.000% ",
"[ 25000000, 30000000 ) 1 0.000% 100.000% ",
"",
"** Level 3 read latency histogram (micros):",
"Count: 2375297687 Average: 36.3652 StdDev: 1.51",
"Min: 0 Median: 2.7602 Max: 29792279",
"Percentiles: P50: 2.76 P75: 4.56 P99: 377.82 P99.9: 5189.85 P99.99: 7887.83",
"------------------------------------------------------",
"[ 0, 1 ) 72682395 3.060% 3.060% #",
"[ 1, 2 ) 553357089 23.296% 26.356% #####",
"[ 2, 3 ) 738793014 31.103% 57.459% ######",
"[ 3, 4 ) 293955165 12.376% 69.835% ##",
"[ 4, 5 ) 217429375 9.154% 78.989% ##",
"[ 5, 6 ) 87119976 3.668% 82.656% #",
"[ 6, 7 ) 64567958 2.718% 85.375% #",
"[ 7, 8 ) 45739052 1.926% 87.300% ",
"[ 8, 9 ) 26636299 1.121% 88.422% ",
"[ 9, 10 ) 15468933 0.651% 89.073% ",
"[ 10, 12 ) 19373430 0.816% 89.889% ",
"[ 12, 14 ) 7027020 0.296% 90.184% ",
"[ 14, 16 ) 3344376 0.141% 90.325% ",
"[ 16, 18 ) 1721303 0.072% 90.398% ",
"[ 18, 20 ) 1298989 0.055% 90.452% ",
"[ 20, 25 ) 4623013 0.195% 90.647% ",
"[ 25, 30 ) 10433630 0.439% 91.086% ",
"[ 30, 35 ) 15392840 0.648% 91.734% ",
"[ 35, 40 ) 13848593 0.583% 92.317% ",
"[ 40, 45 ) 9833797 0.414% 92.731% ",
"[ 45, 50 ) 7006417 0.295% 93.026% ",
"[ 50, 60 ) 7891656 0.332% 93.359% ",
"[ 60, 70 ) 4756953 0.200% 93.559% ",
"[ 70, 80 ) 2347383 0.099% 93.658% ",
"[ 80, 90 ) 932700 0.039% 93.697% ",
"[ 90, 100 ) 480637 0.020% 93.717% ",
"[ 100, 120 ) 569153 0.024% 93.741% ",
"[ 120, 140 ) 592954 0.025% 93.766% ",
"[ 140, 160 ) 3938582 0.166% 93.932% ",
"[ 160, 180 ) 20086897 0.846% 94.778% ",
"[ 180, 200 ) 28773915 1.211% 95.989% ",
"[ 200, 250 ) 43267677 1.822% 97.811% ",
"[ 250, 300 ) 14165553 0.596% 98.407% ",
"[ 300, 350 ) 11667464 0.491% 98.898% ",
"[ 350, 400 ) 4350236 0.183% 99.081% ",
"[ 400, 450 ) 1745781 0.073% 99.155% ",
"[ 450, 500 ) 1354421 0.057% 99.212% ",
"[ 500, 600 ) 2336954 0.098% 99.310% ",
"[ 600, 700 ) 1776698 0.075% 99.385% ",
"[ 700, 800 ) 1297415 0.055% 99.440% ",
"[ 800, 900 ) 978831 0.041% 99.481% ",
"[ 900, 1000 ) 738226 0.031% 99.512% ",
"[ 1000, 1200 ) 1157255 0.049% 99.561% ",
"[ 1200, 1400 ) 920192 0.039% 99.599% ",
"[ 1400, 1600 ) 752268 0.032% 99.631% ",
"[ 1600, 1800 ) 583487 0.025% 99.656% ",
"[ 1800, 2000 ) 479229 0.020% 99.676% ",
"[ 2000, 2500 ) 802354 0.034% 99.710% ",
"[ 2500, 3000 ) 686642 0.029% 99.738% ",
"[ 3000, 3500 ) 773641 0.033% 99.771% ",
"[ 3500, 4000 ) 913603 0.038% 99.809% ",
"[ 4000, 4500 ) 972563 0.041% 99.850% ",
"[ 4500, 5000 ) 895633 0.038% 99.888% ",
"[ 5000, 6000 ) 1489446 0.063% 99.951% ",
"[ 6000, 7000 ) 697183 0.029% 99.980% ",
"[ 7000, 8000 ) 263463 0.011% 99.991% ",
"[ 8000, 9000 ) 94049 0.004% 99.995% ",
"[ 9000, 10000 ) 40403 0.002% 99.997% ",
"[ 10000, 12000 ) 33160 0.001% 99.998% ",
"[ 12000, 14000 ) 14787 0.001% 99.999% ",
"[ 14000, 16000 ) 7225 0.000% 99.999% ",
"[ 16000, 18000 ) 4174 0.000% 99.999% ",
"[ 18000, 20000 ) 2687 0.000% 100.000% ",
"[ 20000, 25000 ) 4525 0.000% 100.000% ",
"[ 25000, 30000 ) 2163 0.000% 100.000% ",
"[ 30000, 35000 ) 1167 0.000% 100.000% ",
"[ 35000, 40000 ) 924 0.000% 100.000% ",
"[ 40000, 45000 ) 740 0.000% 100.000% ",
"[ 45000, 50000 ) 545 0.000% 100.000% ",
"[ 50000, 60000 ) 586 0.000% 100.000% ",
"[ 60000, 70000 ) 329 0.000% 100.000% ",
"[ 70000, 80000 ) 137 0.000% 100.000% ",
"[ 80000, 90000 ) 69 0.000% 100.000% ",
"[ 90000, 100000 ) 44 0.000% 100.000% ",
"[ 100000, 120000 ) 60 0.000% 100.000% ",
"[ 120000, 140000 ) 27 0.000% 100.000% ",
"[ 140000, 160000 ) 12 0.000% 100.000% ",
"[ 160000, 180000 ) 12 0.000% 100.000% ",
"[ 180000, 200000 ) 11 0.000% 100.000% ",
"[ 200000, 250000 ) 24 0.000% 100.000% ",
"[ 250000, 300000 ) 22 0.000% 100.000% ",
"[ 300000, 350000 ) 20 0.000% 100.000% ",
"[ 350000, 400000 ) 24 0.000% 100.000% ",
"[ 400000, 450000 ) 23 0.000% 100.000% ",
"[ 450000, 500000 ) 22 0.000% 100.000% ",
"[ 500000, 600000 ) 4 0.000% 100.000% ",
"[ 25000000, 30000000 ) 3 0.000% 100.000% ",
"",
"** Level 4 read latency histogram (micros):",
"Count: 12550877697 Average: 36.7118 StdDev: 1.15",
"Min: 0 Median: 2.1652 Max: 29780367",
"Percentiles: P50: 2.17 P75: 2.97 P99: 354.89 P99.9: 5678.14 P99.99: 8204.73",
"------------------------------------------------------",
"[ 0, 1 ) 846370266 6.744% 6.744% #",
"[ 1, 2 ) 4784610403 38.122% 44.865% ########",
"[ 2, 3 ) 3900279938 31.076% 75.941% ######",
"[ 3, 4 ) 1054761492 8.404% 84.345% ##",
"[ 4, 5 ) 463485711 3.693% 88.038% #",
"[ 5, 6 ) 109298747 0.871% 88.909% ",
"[ 6, 7 ) 52006098 0.414% 89.323% ",
"[ 7, 8 ) 45580811 0.363% 89.686% ",
"[ 8, 9 ) 32486917 0.259% 89.945% ",
"[ 9, 10 ) 21244089 0.169% 90.114% ",
"[ 10, 12 ) 26511263 0.211% 90.325% ",
"[ 12, 14 ) 12753749 0.102% 90.427% ",
"[ 14, 16 ) 7080767 0.056% 90.483% ",
"[ 16, 18 ) 4677062 0.037% 90.521% ",
"[ 18, 20 ) 3324800 0.026% 90.547% ",
"[ 20, 25 ) 44720927 0.356% 90.904% ",
"[ 25, 30 ) 146233634 1.165% 92.069% ",
"[ 30, 35 ) 195807649 1.560% 93.629% ",
"[ 35, 40 ) 136129497 1.085% 94.713% ",
"[ 40, 45 ) 69672093 0.555% 95.269% ",
"[ 45, 50 ) 43189677 0.344% 95.613% ",
"[ 50, 60 ) 39715746 0.316% 95.929% ",
"[ 60, 70 ) 15145786 0.121% 96.050% ",
"[ 70, 80 ) 6838728 0.054% 96.104% ",
"[ 80, 90 ) 2516981 0.020% 96.124% ",
"[ 90, 100 ) 1294132 0.010% 96.135% ",
"[ 100, 120 ) 1693857 0.013% 96.148% ",
"[ 120, 140 ) 1724332 0.014% 96.162% ",
"[ 140, 160 ) 10156665 0.081% 96.243% ",
"[ 160, 180 ) 46176552 0.368% 96.611% ",
"[ 180, 200 ) 77447001 0.617% 97.228% ",
"[ 200, 250 ) 126055471 1.004% 98.232% ",
"[ 250, 300 ) 52035513 0.415% 98.647% ",
"[ 300, 350 ) 43059290 0.343% 98.990% ",
"[ 350, 400 ) 13133493 0.105% 99.094% ",
"[ 400, 450 ) 5542405 0.044% 99.139% ",
"[ 450, 500 ) 4264933 0.034% 99.173% ",
"[ 500, 600 ) 6978471 0.056% 99.228% ",
"[ 600, 700 ) 5735080 0.046% 99.274% ",
"[ 700, 800 ) 4662948 0.037% 99.311% ",
"[ 800, 900 ) 3898019 0.031% 99.342% ",
"[ 900, 1000 ) 3287653 0.026% 99.368% ",
"[ 1000, 1200 ) 5840038 0.047% 99.415% ",
"[ 1200, 1400 ) 5561575 0.044% 99.459% ",
"[ 1400, 1600 ) 4913717 0.039% 99.498% ",
"[ 1600, 1800 ) 3588642 0.029% 99.527% ",
"[ 1800, 2000 ) 3058529 0.024% 99.551% ",
"[ 2000, 2500 ) 4355190 0.035% 99.586% ",
"[ 2500, 3000 ) 3560035 0.028% 99.614% ",
"[ 3000, 3500 ) 4451962 0.035% 99.650% ",
"[ 3500, 4000 ) 6575131 0.052% 99.702% ",
"[ 4000, 4500 ) 8485591 0.068% 99.770% ",
"[ 4500, 5000 ) 8377833 0.067% 99.836% ",
"[ 5000, 6000 ) 11752642 0.094% 99.930% ",
"[ 6000, 7000 ) 5512837 0.044% 99.974% ",
"[ 7000, 8000 ) 1865403 0.015% 99.989% ",
"[ 8000, 9000 ) 658607 0.005% 99.994% ",
"[ 9000, 10000 ) 275317 0.002% 99.996% ",
"[ 10000, 12000 ) 202719 0.002% 99.998% ",
"[ 12000, 14000 ) 100582 0.001% 99.999% ",
"[ 14000, 16000 ) 50591 0.000% 99.999% ",
"[ 16000, 18000 ) 25465 0.000% 99.999% ",
"[ 18000, 20000 ) 17398 0.000% 100.000% ",
"[ 20000, 25000 ) 27560 0.000% 100.000% ",
"[ 25000, 30000 ) 12556 0.000% 100.000% ",
"[ 30000, 35000 ) 5518 0.000% 100.000% ",
"[ 35000, 40000 ) 3859 0.000% 100.000% ",
"[ 40000, 45000 ) 2911 0.000% 100.000% ",
"[ 45000, 50000 ) 2064 0.000% 100.000% ",
"[ 50000, 60000 ) 2211 0.000% 100.000% ",
"[ 60000, 70000 ) 975 0.000% 100.000% ",
"[ 70000, 80000 ) 469 0.000% 100.000% ",
"[ 80000, 90000 ) 264 0.000% 100.000% ",
"[ 90000, 100000 ) 166 0.000% 100.000% ",
"[ 100000, 120000 ) 198 0.000% 100.000% ",
"[ 120000, 140000 ) 113 0.000% 100.000% ",
"[ 140000, 160000 ) 51 0.000% 100.000% ",
"[ 160000, 180000 ) 52 0.000% 100.000% ",
"[ 180000, 200000 ) 30 0.000% 100.000% ",
"[ 200000, 250000 ) 53 0.000% 100.000% ",
"[ 250000, 300000 ) 46 0.000% 100.000% ",
"[ 300000, 350000 ) 26 0.000% 100.000% ",
"[ 350000, 400000 ) 29 0.000% 100.000% ",
"[ 400000, 450000 ) 21 0.000% 100.000% ",
"[ 450000, 500000 ) 35 0.000% 100.000% ",
"[ 500000, 600000 ) 32 0.000% 100.000% ",
"[ 25000000, 30000000 ) 8 0.000% 100.000% ",
"",
"** Level 5 read latency histogram (micros):",
"Count: 14426302466 Average: 91.3285 StdDev: 0.96",
"Min: 0 Median: 2.5449 Max: 29813234",
"Percentiles: P50: 2.54 P75: 25.54 P99: 1239.71 P99.9: 6160.71 P99.99: 10677.96",
"------------------------------------------------------",
"[ 0, 1 ) 962438829 6.671% 6.671% #",
"[ 1, 2 ) 4566424402 31.653% 38.325% ######",
"[ 2, 3 ) 3090729006 21.424% 59.749% ####",
"[ 3, 4 ) 1180952794 8.186% 67.935% ##",
"[ 4, 5 ) 489538889 3.393% 71.329% #",
"[ 5, 6 ) 159304352 1.104% 72.433% ",
"[ 6, 7 ) 79363683 0.550% 72.983% ",
"[ 7, 8 ) 68799596 0.477% 73.460% ",
"[ 8, 9 ) 52702655 0.365% 73.825% ",
"[ 9, 10 ) 33862892 0.235% 74.060% ",
"[ 10, 12 ) 35685054 0.247% 74.307% ",
"[ 12, 14 ) 16035911 0.111% 74.419% ",
"[ 14, 16 ) 8095048 0.056% 74.475% ",
"[ 16, 18 ) 5139496 0.036% 74.510% ",
"[ 18, 20 ) 4115760 0.029% 74.539% ",
"[ 20, 25 ) 50312268 0.349% 74.888% ",
"[ 25, 30 ) 150215198 1.041% 75.929% ",
"[ 30, 35 ) 195653880 1.356% 77.285% ",
"[ 35, 40 ) 136370804 0.945% 78.230% ",
"[ 40, 45 ) 68570199 0.475% 78.706% ",
"[ 45, 50 ) 38799796 0.269% 78.975% ",
"[ 50, 60 ) 33376376 0.231% 79.206% ",
"[ 60, 70 ) 11314446 0.078% 79.284% ",
"[ 70, 80 ) 5008013 0.035% 79.319% ",
"[ 80, 90 ) 2195595 0.015% 79.334% ",
"[ 90, 100 ) 1371381 0.010% 79.344% ",
"[ 100, 120 ) 2024702 0.014% 79.358% ",
"[ 120, 140 ) 2116434 0.015% 79.373% ",
"[ 140, 160 ) 28821438 0.200% 79.572% ",
"[ 160, 180 ) 317347062 2.200% 81.772% ",
"[ 180, 200 ) 541002045 3.750% 85.522% #",
"[ 200, 250 ) 939107167 6.510% 92.032% #",
"[ 250, 300 ) 402223338 2.788% 94.820% #",
"[ 300, 350 ) 337402363 2.339% 97.159% ",
"[ 350, 400 ) 97004742 0.672% 97.831% ",
"[ 400, 450 ) 29439088 0.204% 98.035% ",
"[ 450, 500 ) 21166694 0.147% 98.182% ",
"[ 500, 600 ) 28122368 0.195% 98.377% ",
"[ 600, 700 ) 23258911 0.161% 98.538% ",
"[ 700, 800 ) 18432354 0.128% 98.666% ",
"[ 800, 900 ) 14561954 0.101% 98.767% ",
"[ 900, 1000 ) 11714344 0.081% 98.848% ",
"[ 1000, 1200 ) 18747943 0.130% 98.978% ",
"[ 1200, 1400 ) 15966912 0.111% 99.089% ",
"[ 1400, 1600 ) 13390793 0.093% 99.182% ",
"[ 1600, 1800 ) 10554685 0.073% 99.255% ",
"[ 1800, 2000 ) 8411021 0.058% 99.313% ",
"[ 2000, 2500 ) 13658000 0.095% 99.408% ",
"[ 2500, 3000 ) 10177912 0.071% 99.478% ",
"[ 3000, 3500 ) 9542612 0.066% 99.544% ",
"[ 3500, 4000 ) 10106501 0.070% 99.614% ",
"[ 4000, 4500 ) 11149082 0.077% 99.692% ",
"[ 4500, 5000 ) 11000830 0.076% 99.768% ",
"[ 5000, 6000 ) 17652518 0.122% 99.890% ",
"[ 6000, 7000 ) 8686708 0.060% 99.951% ",
"[ 7000, 8000 ) 3400532 0.024% 99.974% ",
"[ 8000, 9000 ) 1380987 0.010% 99.984% ",
"[ 9000, 10000 ) 692164 0.005% 99.988% ",
"[ 10000, 12000 ) 646964 0.004% 99.993% ",
"[ 12000, 14000 ) 339542 0.002% 99.995% ",
"[ 14000, 16000 ) 184935 0.001% 99.997% ",
"[ 16000, 18000 ) 105702 0.001% 99.997% ",
"[ 18000, 20000 ) 72688 0.001% 99.998% ",
"[ 20000, 25000 ) 115543 0.001% 99.999% ",
"[ 25000, 30000 ) 63796 0.000% 99.999% ",
"[ 30000, 35000 ) 38339 0.000% 99.999% ",
"[ 35000, 40000 ) 26187 0.000% 100.000% ",
"[ 40000, 45000 ) 20167 0.000% 100.000% ",
"[ 45000, 50000 ) 14671 0.000% 100.000% ",
"[ 50000, 60000 ) 15610 0.000% 100.000% ",
"[ 60000, 70000 ) 7285 0.000% 100.000% ",
"[ 70000, 80000 ) 3467 0.000% 100.000% ",
"[ 80000, 90000 ) 1960 0.000% 100.000% ",
"[ 90000, 100000 ) 1129 0.000% 100.000% ",
"[ 100000, 120000 ) 1357 0.000% 100.000% ",
"[ 120000, 140000 ) 599 0.000% 100.000% ",
"[ 140000, 160000 ) 399 0.000% 100.000% ",
"[ 160000, 180000 ) 218 0.000% 100.000% ",
"[ 180000, 200000 ) 148 0.000% 100.000% ",
"[ 200000, 250000 ) 240 0.000% 100.000% ",
"[ 250000, 300000 ) 169 0.000% 100.000% ",
"[ 300000, 350000 ) 191 0.000% 100.000% ",
"[ 350000, 400000 ) 213 0.000% 100.000% ",
"[ 400000, 450000 ) 194 0.000% 100.000% ",
"[ 450000, 500000 ) 170 0.000% 100.000% ",
"[ 500000, 600000 ) 49 0.000% 100.000% ",
"[ 25000000, 30000000 ) 7 0.000% 100.000% ",
"",
"** Level 6 read latency histogram (micros):",
"Count: 22753304220 Average: 258.3514 StdDev: 1.69",
"Min: 0 Median: 207.7721 Max: 29818431",
"Percentiles: P50: 207.77 P75: 285.50 P99: 2886.97 P99.9: 7243.77 P99.99: 16010.44",
"------------------------------------------------------",
"[ 0, 1 ) 653620769 2.873% 2.873% #",
"[ 1, 2 ) 2825637444 12.419% 15.291% ##",
"[ 2, 3 ) 2147094547 9.436% 24.728% ##",
"[ 3, 4 ) 694868111 3.054% 27.782% #",
"[ 4, 5 ) 247204153 1.086% 28.868% ",
"[ 5, 6 ) 121129642 0.532% 29.400% ",
"[ 6, 7 ) 105034536 0.462% 29.862% ",
"[ 7, 8 ) 77298201 0.340% 30.202% ",
"[ 8, 9 ) 49683588 0.218% 30.420% ",
"[ 9, 10 ) 24535948 0.108% 30.528% ",
"[ 10, 12 ) 21616407 0.095% 30.623% ",
"[ 12, 14 ) 9434002 0.041% 30.664% ",
"[ 14, 16 ) 5287817 0.023% 30.688% ",
"[ 16, 18 ) 4238641 0.019% 30.706% ",
"[ 18, 20 ) 4872712 0.021% 30.728% ",
"[ 20, 25 ) 41101209 0.181% 30.908% ",
"[ 25, 30 ) 101980198 0.448% 31.356% ",
"[ 30, 35 ) 123168040 0.541% 31.898% ",
"[ 35, 40 ) 82808705 0.364% 32.262% ",
"[ 40, 45 ) 41747586 0.183% 32.445% ",
"[ 45, 50 ) 24347514 0.107% 32.552% ",
"[ 50, 60 ) 26010143 0.114% 32.667% ",
"[ 60, 70 ) 20789253 0.091% 32.758% ",
"[ 70, 80 ) 21080789 0.093% 32.851% ",
"[ 80, 90 ) 16470559 0.072% 32.923% ",
"[ 90, 100 ) 10420731 0.046% 32.969% ",
"[ 100, 120 ) 15322106 0.067% 33.036% ",
"[ 120, 140 ) 15964854 0.070% 33.106% ",
"[ 140, 160 ) 85229513 0.375% 33.481% ",
"[ 160, 180 ) 1020644981 4.486% 37.967% #",
"[ 180, 200 ) 2063154675 9.067% 47.034% ##",
"[ 200, 250 ) 4341524829 19.081% 66.115% ####",
"[ 250, 300 ) 2847597044 12.515% 78.630% ###",
"[ 300, 350 ) 2519426411 11.073% 89.703% ##",
"[ 350, 400 ) 787568320 3.461% 93.164% #",
"[ 400, 450 ) 252201831 1.108% 94.273% ",
"[ 450, 500 ) 187756141 0.825% 95.098% ",
"[ 500, 600 ) 152426888 0.670% 95.768% ",
"[ 600, 700 ) 116197336 0.511% 96.278% ",
"[ 700, 800 ) 92761937 0.408% 96.686% ",
"[ 800, 900 ) 73255312 0.322% 97.008% ",
"[ 900, 1000 ) 57825263 0.254% 97.262% ",
"[ 1000, 1200 ) 89167445 0.392% 97.654% ",
"[ 1200, 1400 ) 73336180 0.322% 97.976% ",
"[ 1400, 1600 ) 60826024 0.267% 98.244% ",
"[ 1600, 1800 ) 47844000 0.210% 98.454% ",
"[ 1800, 2000 ) 36682211 0.161% 98.615% ",
"[ 2000, 2500 ) 58035753 0.255% 98.870% ",
"[ 2500, 3000 ) 38169142 0.168% 99.038% ",
"[ 3000, 3500 ) 31564615 0.139% 99.177% ",
"[ 3500, 4000 ) 28440708 0.125% 99.302% ",
"[ 4000, 4500 ) 28331035 0.125% 99.426% ",
"[ 4500, 5000 ) 28097582 0.123% 99.550% ",
"[ 5000, 6000 ) 50424664 0.222% 99.771% ",
"[ 6000, 7000 ) 26513216 0.117% 99.888% ",
"[ 7000, 8000 ) 11402850 0.050% 99.938% ",
"[ 8000, 9000 ) 4837203 0.021% 99.959% ",
"[ 9000, 10000 ) 2545708 0.011% 99.970% ",
"[ 10000, 12000 ) 2496844 0.011% 99.981% ",
"[ 12000, 14000 ) 1238523 0.005% 99.987% ",
"[ 14000, 16000 ) 734102 0.003% 99.990% ",
"[ 16000, 18000 ) 459296 0.002% 99.992% ",
"[ 18000, 20000 ) 322115 0.001% 99.993% ",
"[ 20000, 25000 ) 519311 0.002% 99.996% ",
"[ 25000, 30000 ) 301160 0.001% 99.997% ",
"[ 30000, 35000 ) 185486 0.001% 99.998% ",
"[ 35000, 40000 ) 133060 0.001% 99.998% ",
"[ 40000, 45000 ) 103689 0.000% 99.999% ",
"[ 45000, 50000 ) 75366 0.000% 99.999% ",
"[ 50000, 60000 ) 82849 0.000% 100.000% ",
"[ 60000, 70000 ) 39008 0.000% 100.000% ",
"[ 70000, 80000 ) 18353 0.000% 100.000% ",
"[ 80000, 90000 ) 9985 0.000% 100.000% ",
"[ 90000, 100000 ) 5932 0.000% 100.000% ",
"[ 100000, 120000 ) 6997 0.000% 100.000% ",
"[ 120000, 140000 ) 3255 0.000% 100.000% ",
"[ 140000, 160000 ) 1820 0.000% 100.000% ",
"[ 160000, 180000 ) 1277 0.000% 100.000% ",
"[ 180000, 200000 ) 777 0.000% 100.000% ",
"[ 200000, 250000 ) 1412 0.000% 100.000% ",
"[ 250000, 300000 ) 1174 0.000% 100.000% ",
"[ 300000, 350000 ) 1009 0.000% 100.000% ",
"[ 350000, 400000 ) 1231 0.000% 100.000% ",
"[ 400000, 450000 ) 1265 0.000% 100.000% ",
"[ 450000, 500000 ) 1324 0.000% 100.000% ",
"[ 500000, 600000 ) 428 0.000% 100.000% ",
"[ 600000, 700000 ) 38 0.000% 100.000% ",
"[ 700000, 800000 ) 5 0.000% 100.000% ",
"[ 25000000, 30000000 ) 107 0.000% 100.000% ",
"",
"",
"** DB Stats **",
"Uptime(secs): 47054460.2 total, 0.1 interval",
"Cumulative writes: 37G writes, 252G keys, 37G commit groups, 1.0 writes per commit group, ingest: 21147.27 GB, 0.46 MB/s",
"Cumulative WAL: 37G writes, 0 syncs, 37190266782.00 writes per sync, written: 21147.27 GB, 0.46 MB/s",
"Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent",
"Interval writes: 41 writes, 280 keys, 41 commit groups, 1.0 writes per commit group, ingest: 0.03 MB, 0.28 MB/s",
"Interval WAL: 41 writes, 0 syncs, 41.00 writes per sync, written: 0.00 MB, 0.28 MB/s",
"Interval stall: 00:00:0.000 H:M:S, 0.0 percent"
],
"compaction-stats" : {
"level-stats" : {
"L0" : {
"num-files" : 1,
"being-compacted" : 0,
"total-file-size-MB" : 149.97,
"score" : 0.3,
"bytes-read-GB" : 0,
"bytes-read-non-output-levels-GB" : 0,
"bytes-read-output-levels-GB" : 0,
"bytes-written-GB" : 19890.4,
"bytes-new-GB" : 19890.4,
"bytes-moved-GB" : 0,
"w-amp" : 1,
"bytes-read-MB-s" : 0,
"bytes-written-MB-s" : 55.9,
"compactions-sec" : 364137,
"compactions-cnt" : 137705,
"compaction-avg-len-sec" : 2.644,
"num-input-records" : "0",
"num-dropped-records" : "0"
},
"L2" : {
"num-files" : 2,
"being-compacted" : 0,
"total-file-size-MB" : 401.72,
"score" : 0.8,
"bytes-read-GB" : 26645,
"bytes-read-non-output-levels-GB" : 17628.2,
"bytes-read-output-levels-GB" : 9016.8,
"bytes-written-GB" : 26421.6,
"bytes-new-GB" : 17404.7,
"bytes-moved-GB" : 0,
"w-amp" : 1.5,
"bytes-read-MB-s" : 54.8,
"bytes-written-MB-s" : 54.4,
"compactions-sec" : 497536,
"compactions-cnt" : 33331,
"compaction-avg-len-sec" : 14.927,
"num-input-records" : "332G",
"num-dropped-records" : "216M"
},
"L3" : {
"num-files" : 3,
"being-compacted" : 0,
"total-file-size-MB" : 290.41,
"score" : 0.5,
"bytes-read-GB" : 29140.3,
"bytes-read-non-output-levels-GB" : 19666.4,
"bytes-read-output-levels-GB" : 9473.9,
"bytes-written-GB" : 20899.6,
"bytes-new-GB" : 11425.7,
"bytes-moved-GB" : 0,
"w-amp" : 1.1,
"bytes-read-MB-s" : 61.2,
"bytes-written-MB-s" : 43.9,
"compactions-sec" : 487805,
"compactions-cnt" : 62242,
"compaction-avg-len-sec" : 7.837,
"num-input-records" : "453G",
"num-dropped-records" : "79M"
},
"L4" : {
"num-files" : 25,
"being-compacted" : 0,
"total-file-size-MB" : 5253.12,
"score" : 1,
"bytes-read-GB" : 105238.5,
"bytes-read-non-output-levels-GB" : 10214.8,
"bytes-read-output-levels-GB" : 95023.7,
"bytes-written-GB" : 104148.2,
"bytes-new-GB" : 9124.5,
"bytes-moved-GB" : 1211.2,
"w-amp" : 10.2,
"bytes-read-MB-s" : 53.2,
"bytes-written-MB-s" : 52.7,
"compactions-sec" : 2024409,
"compactions-cnt" : 37606,
"compaction-avg-len-sec" : 53.832,
"num-input-records" : "1146G",
"num-dropped-records" : "127M"
},
"L5" : {
"num-files" : 252,
"being-compacted" : 0,
"total-file-size-MB" : 47411.2,
"score" : 1,
"bytes-read-GB" : 107102.1,
"bytes-read-non-output-levels-GB" : 9702.9,
"bytes-read-output-levels-GB" : 97399.2,
"bytes-written-GB" : 105615.7,
"bytes-new-GB" : 8216.6,
"bytes-moved-GB" : 631.6,
"w-amp" : 10.9,
"bytes-read-MB-s" : 43.1,
"bytes-written-MB-s" : 42.5,
"compactions-sec" : 2542494,
"compactions-cnt" : 33409,
"compaction-avg-len-sec" : 76.102,
"num-input-records" : "2251G",
"num-dropped-records" : "9503M"
},
"L6" : {
"num-files" : 2274,
"being-compacted" : 0,
"total-file-size-MB" : 570204.16,
"score" : 0,
"bytes-read-GB" : 72281.1,
"bytes-read-non-output-levels-GB" : 8844.5,
"bytes-read-output-levels-GB" : 63436.6,
"bytes-written-GB" : 63512.3,
"bytes-new-GB" : 75.7,
"bytes-moved-GB" : 0,
"w-amp" : 7.2,
"bytes-read-MB-s" : 44.3,
"bytes-written-MB-s" : 39,
"compactions-sec" : 1669418,
"compactions-cnt" : 20767,
"compaction-avg-len-sec" : 80.388,
"num-input-records" : "2262G",
"num-dropped-records" : "239G"
},
"Sum" : {
"num-files" : 2557,
"being-compacted" : 0,
"total-file-size-MB" : 623718.4,
"score" : 0,
"bytes-read-GB" : 340407,
"bytes-read-non-output-levels-GB" : 66056.8,
"bytes-read-output-levels-GB" : 274350.2,
"bytes-written-GB" : 340487.9,
"bytes-new-GB" : 66137.7,
"bytes-moved-GB" : 1842.8,
"w-amp" : 17.1,
"bytes-read-MB-s" : 46,
"bytes-written-MB-s" : 46,
"compactions-sec" : 7585800,
"compactions-cnt" : 325060,
"compaction-avg-len-sec" : 23.337,
"num-input-records" : "6446G",
"num-dropped-records" : "249G"
},
"Int" : {
"num-files" : 0,
"being-compacted" : 0,
"total-file-size-MB" : 0,
"score" : 0,
"bytes-read-GB" : 0,
"bytes-read-non-output-levels-GB" : 0,
"bytes-read-output-levels-GB" : 0,
"bytes-written-GB" : 0,
"bytes-new-GB" : 0,
"bytes-moved-GB" : 0,
"w-amp" : 0,
"bytes-read-MB-s" : 0,
"bytes-written-MB-s" : 0,
"compactions-sec" : 0,
"compactions-cnt" : 0,
"compaction-avg-len-sec" : 0,
"num-input-records" : "0",
"num-dropped-records" : "0"
}
},
"uptime-total-sec" : 47054460.2,
"uptime-interval-sec" : 0.1,
"flush-cumulative-GB" : 19890.444,
"flush-interval-GB" : 0,
"addfile-cumulative-GB" : 0,
"addfile-interval-GB" : 0,
"addfile-cumulative-cnt" : 0,
"addfile-interval-cnt" : 0,
"addfile-cumulative-l0-cnt" : 0,
"addfile-interval-l0-cnt" : 0,
"addfile-cumulative-key-cnt" : 0,
"addfile-interval-key-cnt" : 0,
"cumulative-written-GB" : 340487.87,
"cumulative-written-MB-s" : 7.41,
"cumulative-read-GB" : 340406.98,
"cumulative-read-MB-s" : 7.41,
"cumulative-seconds" : 7585799.7,
"interval-written-GB" : 0,
"interval-written-MB-s" : 0,
"interval-read-GB" : 0,
"interval-read-MB-s" : 0,
"interval-seconds" : 0,
"stalls-level0-slowdown" : 0,
"stalls-level0-slowdown-with-compaction" : 0,
"stalls-level0-numfiles" : 0,
"stalls-level0-numfiles-with-compaction" : 0,
"stalls-stop-for-pending-compaction-bytes" : 0,
"stalls-slowdown-for-pending-compaction-bytes" : 0,
"stalls-memtable-compaction" : 0,
"stalls-memtable-slowdown" : 0,
"stalls-interval-total-count" : 0
},
"db-stats" : {
"uptime-total-sec" : 47054460.2,
"uptime-interval-sec" : 0.1,
"cumulative-writes-cnt" : "37G",
"cumulative-writes-keys" : "252G",
"cumulative-writes-commit-groups" : "37G",
"cumulative-writes-per-commit-group" : 1,
"cumulative-writes-ingest-GB" : 21147.27,
"cumulative-writes-ingest-MB-s" : 0.46,
"cumulative-WAL-writes" : "37G",
"cumulative-WAL-syncs" : "0",
"cumulative-WAL-writes-per-sync" : 37190266782,
"cumulative-WAL-written-GB" : 21147.27,
"cumulative-WAL-written-MB-s" : 0.46,
"cumulative-stall-sec" : 0,
"cumulative-stall-percent" : 0,
"interval-writes-cnt" : "41",
"interval-writes-keys" : "280",
"interval-writes-commit-groups" : "41",
"interval-writes-per-commit-group" : 1,
"interval-writes-ingest-MB" : 0.03,
"interval-writes-ingest-MB-s" : 0.28,
"interval-WAL-writes" : "41",
"interval-WAL-syncs" : "0",
"interval-WAL-writes-per-sync" : 41,
"interval-WAL-written-MB" : 0,
"interval-WAL-written-MB-s" : 0.28,
"interval-stall-sec" : 0,
"interval-stall-percent" : 0
},
"num-immutable-mem-table" : "0",
"mem-table-flush-pending" : "0",
"compaction-pending" : "0",
"background-errors" : "0",
"cur-size-active-mem-table" : "126MB",
"cur-size-all-mem-tables" : "126MB",
"num-entries-active-mem-table" : "1193166",
"num-entries-imm-mem-tables" : "0",
"estimate-table-readers-mem" : "3060MB",
"num-snapshots" : "0",
"oldest-snapshot-time" : "0",
"num-live-versions" : "1",
"total-live-recovery-units" : 5,
"block-cache-usage" : "19GB",
"transaction-engine-keys" : NumberLong(0),
"transaction-engine-snapshots" : NumberLong(0),
"thread-status" : [ ],
"counters" : {
"block-cache-misses" : NumberLong("55117747493"),
"block-cache-hits" : NumberLong("1012070612832"),
"block-cache-add" : NumberLong("19729844769"),
"block-cache-add-failures" : NumberLong(0),
"block-cache-index-miss" : NumberLong(0),
"block-cache-index-hit" : NumberLong(0),
"block-cache-index-add" : NumberLong(0),
"block-cache-index-bytes-insert" : NumberLong(0),
"block-cache-index-bytes-evict" : NumberLong(0),
"block-cache-filter-miss" : NumberLong(0),
"block-cache-filter-hit" : NumberLong(0),
"block-cache-filter-add" : NumberLong(0),
"block-cache-filter-bytes-insert" : NumberLong(0),
"block-cache-filter-bytes-evict" : NumberLong(0),
"block-cache-data-miss" : NumberLong("55117747493"),
"block-cache-data-hit" : NumberLong("1012070612832"),
"block-cache-data-add" : NumberLong("19729844769"),
"block-cache-data-bytes-insert" : NumberLong("323258630004656"),
"block-cache-bytes-read" : NumberLong("16583933343299440"),
"block-cache-bytes-write" : NumberLong("323258630004656"),
"bloom-filter-useful" : NumberLong("539046924060"),
"persistent-cache-hit" : NumberLong(0),
"persistent-cache-miss" : NumberLong(0),
"sim-block-cache-hit" : NumberLong(0),
"sim-block-cache-miss" : NumberLong(0),
"memtable-hit" : NumberLong("62517535519"),
"memtable-miss" : NumberLong("167295690838"),
"l0-hit" : NumberLong(1400097260),
"l1-hit" : NumberLong(0),
"l2andup-hit" : NumberLong("113317193502"),
"compaction-key-drop-new" : NumberLong("125284514344"),
"compaction-key-drop-obsolete" : NumberLong("124605396897"),
"compaction-key-drop-range_del" : NumberLong(0),
"compaction-key-drop-user" : NumberLong(0),
"compaction-range_del-drop-obsolete" : NumberLong(0),
"num-keys-written" : NumberLong("252503884995"),
"num-keys-read" : NumberLong("229813226357"),
"number-keys-updated" : NumberLong(0),
"bytes-written" : NumberLong("22706710496614"),
"bytes-read-point-lookup" : NumberLong("44219118687772"),
"num-seeks" : NumberLong("143911179221"),
"num-forward-iterations" : NumberLong("331184231795"),
"num-backward-iterations" : NumberLong(6829221),
"number-db-seek-found" : NumberLong("143911120397"),
"number-db-next-found" : NumberLong("280679777271"),
"number-db-prev-found" : NumberLong(6829221),
"bytes-read-iteration" : NumberLong("89852122408121"),
"no-file-closes" : NumberLong(0),
"no-file-opens" : NumberLong(1860291),
"no-file-errors" : NumberLong(0),
"l0-slowdown-micros" : NumberLong(0),
"memtable-compaction-micros" : NumberLong(0),
"l0-num-files-stall-micros" : NumberLong(0),
"stall-micros" : NumberLong(0),
"db-mutex-wait-micros" : NumberLong(0),
"rate-limit-delay-millis" : NumberLong(0),
"num-iterators" : NumberLong(0),
"number-multiget-get" : NumberLong(0),
"number-multiget-keys-read" : NumberLong(0),
"number-multiget-bytes-read" : NumberLong(0),
"number-deletes-filtered" : NumberLong(0),
"number-merge-failures" : NumberLong(0),
"bloom-filter-prefix-checked" : NumberLong(0),
"bloom-filter-prefix-useful" : NumberLong(0),
"number-reseeks-iteration" : NumberLong(1378816),
"getupdatessince-calls" : NumberLong(0),
"block-cachecompressed-miss" : NumberLong(0),
"block-cachecompressed-hit" : NumberLong(0),
"block-cachecompressed-add" : NumberLong(0),
"block-cachecompressed-add-failures" : NumberLong(0),
"wal-synced" : NumberLong("2823032053"),
"wal-bytes" : NumberLong("22706710496614"),
"write-self" : NumberLong("37003104641"),
"write-other" : NumberLong(187162141),
"write-timeout" : NumberLong(0),
"write-wal" : NumberLong("74380533564"),
"compaction-bytes-read" : NumberLong("338957749989762"),
"compaction-bytes-written" : NumberLong("344238887561273"),
"flush-bytes-written" : NumberLong("21357202056503"),
"number-direct-load-table-properties" : NumberLong(0),
"number-superversion_acquires" : NumberLong(538227926),
"number-superversion_releases" : NumberLong(74967),
"number-superversion_cleanups" : NumberLong(4681),
"number-block-compressed" : NumberLong(0),
"number-block-decompressed" : NumberLong(0),
"number-block-not_compressed" : NumberLong(0),
"merge-operation-time-nanos" : NumberLong(0),
"filter-operation-time-nanos" : NumberLong(0),
"row-cache-hit" : NumberLong(0),
"row-cache-miss" : NumberLong(0),
"read-amp-estimate-useful-bytes" : NumberLong(0),
"read-amp-total-read-bytes" : NumberLong(0),
"number-rate_limiter-drains" : NumberLong(34697703)
},
"concurrentTransactions" : {
"write" : {
"out" : 0,
"available" : 128,
"totalTickets" : 128
},
"read" : {
"out" : 1,
"available" : 127,
"totalTickets" : 128
}
}
}

@sunlike-Lipo
Copy link
Contributor

@q96456
it do impact r/w performance of the node running this command. it can be avoid by doing compaction on secondary nodes and then step down the primary.
Currently showing the number of tombstone entries is not supported by db.serverStatus() command yet. You can check it by get num-deletes of every sst file in the MANIFEST with rocksdb ldb tool.

@q96456
Copy link

q96456 commented Mar 5, 2021

@sunlike-Lipo
I tried，on secondary，but disk space is not enough. This is a dangerous operation.The compression function needs to implement the data range interface.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@igorcanadi @eharry @SunguckLee @q96456 @sunlike-Lipo