Mempool disorder txs cause deliver tx failed in ApplyBlock

[withzoo]

Summary of Bug

1. Describe

I use 5 accounts to sending txs to private cluster, each account tps is 200, total tps is 1000, but i got this error
invalid sequence: invalid nonce; got 23505, expected in the range of [23158, 23158]
nonce of account is managed by client, send one tx , nonce++, so nonce shouldn't be wrong

2. Impact

This causes some sented transactions to be lost

The reason of error is that txs in mempool is disorder, and disorder txs will pack in block, finally cause deliverTx failed in ApplyBlock, then mempool will discard some txs in recheck

the order of txs in mempool
-> addr A(nonce 23158) -> addr A(nonce 23159) -> addr B(nonce xxxx) => addr A(nonce 23157) ->

Log:

D[2022-03-15|17:55:48.959][45955:311] Invalid tx. module=state code=3 log="invalid sequence: invalid nonce; got 23158, expected 23157" index=943
D[2022-03-15|17:55:48.975][45955:311] Invalid tx. module=state code=3 log="invalid sequence: invalid nonce; got 23159, expected 23157" index=944
D[2022-03-15|17:55:49.010][45955:311] Invalid tx. module=state code=3 log="invalid sequence: invalid nonce; got 23160, expected 23158" index=948
D[2022-03-15|17:55:49.026][45955:311] Invalid tx. module=state code=3 log="invalid sequence: invalid nonce; got 23161, expected 23158" index=949
D[2022-03-15|17:55:49.026][45955:311] Invalid tx. module=state code=3 log="invalid sequence: invalid nonce; got 23162, expected 23158" index=950

I[2022-03-15|17:55:49.592][45868:74] Tx is no longer valid. module=mempool tx=CF32D93947451479E0617DABAAC382D0A060CE1B1B5C038178703EF9526CAA1B res="&{CheckTx:code:3 log:"invalid sequence: invalid nonce; got 23319, expected 23158" gas_used:6524144 codespace:"sdk" }" err=null
I[2022-03-15|17:55:49.600][45868:74] Tx is no longer valid. module=mempool tx=6A48B4CD129077AAED973B449F1F19DF22B68821D0AEF28F90AE48E4186CE0D9 res="&{CheckTx:code:3 log:"invalid sequence: invalid nonce; got 23320, expected 23158" gas_used:6640297 codespace:"sdk" }" err=null
I[2022-03-15|17:55:49.601][45868:74] Tx is no longer valid. module=mempool tx=7DD1A4E084EB22410CDEBA1DD21E10491D4478E835AB545606378372E9E80DC3 res="&{CheckTx:code:3 log:"invalid sequence: invalid nonce; got 23321, expected 23158" gas_used:6809154 codespace:"sdk" }" err=null
I[2022-03-15|17:55:49.601][45868:74] Tx is no longer valid. module=mempool tx=4B076CAD7CF2C8A2C26A448402229DD600D8CA69F6B4165D2A163A2DEE47F867 res="&{CheckTx:code:3 log:"invalid sequence: invalid nonce; got 23322, expected 23158" gas_used:6952559 codespace:"sdk" }" err=null
I[2022-03-15|17:55:49.601][45868:74] Tx is no longer valid. module=mempool tx=5188012D9BBD1466965053364C4E45DBF6A4459F896B2D71BD196813F659AF3B res="&{CheckTx:code:3 log:"invalid sequence: invalid nonce; got 23323, expected 23158" gas_used:7095964 codespace:"sdk" }" err=null
I[2022-03-15|17:55:49.602][45868:74] Tx is no longer valid. module=mempool tx=8BD11843E4DB4DCF4DB164A0531B5F8FEA13FC95D992DF7BA82C6B1EEF4F3181 res="&{CheckTx:code:3 log:"invalid sequence: invalid nonce; got 23324, expected 23158" gas_used:7239279 codespace:"sdk" }" err=null
I[2022-03-15|17:55:49.602][45868:74] Tx is no longer valid. module=mempool tx=47783D7FCCBC28C47CAA4B2871B0A7514025C62FEE62AE124F306EB450C9B209 res="&{CheckTx:code:3 log:"invalid sequence: invalid nonce; got 23325, expected 23158" gas_used:7275476 codespace:"sdk" }" err=null

3. Recover

lost txs can't recover

Expected Result

txs in mempool are ordered, and all txs deliver success in ApplyBlock

Version

v1.1.10

Steps to Reproduce

I start 4 node private cluster by using testnet.sh script, and launch parameters as below

echorun nohup exchaind start
--home cache/node${index}/exchaind
--p2p.seed_mode=$seed_mode
--p2p.allow_duplicate_ip
--enable-dynamic-gp=false
--enable-wtx=${WRAPPEDTX}
--mempool.node_key_whitelist ${WHITE_LIST}
--mempool.recheck true
--mempool.size 20000
--mempool.max_tx_num_per_block 2000
--p2p.pex=false
--p2p.addr_book_strict=false
$p2p_seed_opt $p2p_seed_arg
--p2p.laddr tcp://${IP}:${p2pport}
--rpc.laddr tcp://${IP}:${rpcport}
--consensus.timeout_commit 600ms
--log_level *:debug
--chain-id ${CHAIN_ID}
--upload-delta=false
--enable-gid
--append-pid=true
${LOG_SERVER}
--elapsed DeliverTxs=0,Round=1,CommitRound=1,Produce=1
--rest.laddr tcp://localhost:$restport
--enable-preruntx=$PRERUN
--consensus-role=v$index
${Test_CASE}
--keyring-backend test >cache/val${index}.log 2>&1 &

then, use 5 accounts to sending txs to private cluster, each account tps is 200, total tps is 1000, sometime will get error like this
invalid sequence: invalid nonce; got 23505, expected in the range of [23158, 23158]

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[cwbhhjl]

You can try turning off recheck