Nodes stop syncing blocks when processing the debug_* method

[Mummyhap]

The nodes we are running stop syncing blocks when processing the debug_* method, causing them to only returning timeout errors, like 'RPC exec write tcp 127.0.0.1:21661->127.0.0.1:37036: i/o timeout', which then leads to all other RPC requests timing out as well.
Our charts show that the program spends most of its time in the "trie.(Trie).tryGet" function.
Currently, the performance debug_ is unable to meet Nansen's requirements. The data processing speed cannot keep up with the block height growth. We hope your tech team can optimize the client as soon as possible.
This is the node running command:

/node/archive/tomo/bin/tomo --networkid 88 --syncmode full --gcmode archive --announce-txs --datadir /node/archive/tomo/geth/ --tomox.datadir /node/archive/tomo/geth/tomox --txpool.globalslots 1000000 --txpool.globalqueue 3000000 --rpc --rpccorsdomain '*' --rpcvhosts '*' --rpcapi 'eth,net,web3,debug,trace,txpool,tomox' --wsapi 'eth,net,web3,debug,trace,txpool,tomox' --rpcaddr 0.0.0.0 --ws --wsaddr 0.0.0.0 --wsorigins '*' --port 21660 --rpcport 21661 --wsport 21662 --gasprice 250000000 --bootnodes enode://fd3da177f9492a39d*****3ee25ebffc3a@*.*.*.*:30303,enode://97f0ca95a653e3c44*****543f89b2425700a68bc8876@*.*.*.*:30301,enode://b72927f349f3a27b*****8dd77ad85a869032dea@*.*.*.*:30301,enode://c8f2f0643****8b8df351@*.*.*.*:30301 --unlock 0x407638f566802d698117973817d867e3e9604e06 --identity blockpi --mine --store-reward --keystore /node/archive/tomo/geth/keystore --password /node/archive/tomo/geth/passwd --ethstats blockpi:getty-site-pablo-auger-room-sos-blair-shin-whiz-delhi@stats.tomochain.com:443

[NghiaNDD98]

To investigate, you may

• provide tomo's log, chart and which debug_* method
• reproduce this error

[tungng98]

Hello, after some weeks running nodes to stress tests Viction with Debug API. I observed that Viction doesn't have many mechanism to balance the number of requests it can process. Debug API is very resource intensive, mostly on CPU and Disk I/O.

I used haproxy to load balancing and limit maximum connection that a node should handle, and a good value maxconn in my case is from 7 to 10 depend on the hardware of the VM. When I tried to set it higher, 12 for example, block sync can't catchup with the blockchain tip. The delay keeps increasing until the node cannot sync more blocks.