Time to load keys

[yorickdowne]

web3signer w/ 10k keys that were created by staking-deposit-cli, Holesky, loaded via keymanager API

I am wondering whether the time to load keys can be improved on startup. Every startup web3signer takes ~20 mins

eth-holesky-web3signer-1  | 2023-09-15 15:42:24.472+00:00 | ForkJoinPool.commonPool-worker-2 | INFO  | SignerLoader | 9980 signing metadata processed
eth-holesky-web3signer-1  | 2023-09-15 15:42:26.832+00:00 | pool-2-thread-1 | INFO  | SignerLoader | 9990 signing metadata processed
eth-holesky-web3signer-1  | 2023-09-15 15:42:29.404+00:00 | ForkJoinPool.commonPool-worker-10 | INFO  | SignerLoader | 10000 signing metadata processed
eth-holesky-web3signer-1  | 2023-09-15 15:42:29.979+00:00 | pool-2-thread-1 | INFO  | SignerLoader | Total Artifact Signer loaded via configuration files: 10000
eth-holesky-web3signer-1  | Error count 0
eth-holesky-web3signer-1  | Time Taken: 00:19:14.756.

[jframe]

Does it take the same time to load keys using the keyManager API as does loading by restarting web3signer?

The keys that are generated using the staking-deposit-cli can take a long time to load as encrypted using scrypt. There might not be much we do and can still take a look.

[yorickdowne]

I believe it does take just as long, yes.

I know scrypt decode takes a minute. Clients like Teku store this in an optimized format somehow while still keeping it encrypted - maybe you can crib from how it's handled there?

[usmansaleem]

@yorickdowne the main reason for slow loading at start up is due to scrypt encryption on the v4 wallet files. Teku also doesn't convert any of the v4 imported parameters. We also identify one potential enhancement during the import phase though and will be incorporating it via #921. You should be able to convert scrypt to pbkdf2 to decrease the decryption time, but with the caveat of lower security. Let us know if you need more information around how to convert or lower the v4 file parameters, it would need a small Java program to achieve it.

[yorickdowne]

Got it. Yes that Java program for conversion would be helpful.

[usmansaleem]

@yorickdowne I created a small Java program that should help you to convert existing SCRYPT v4 keystore files to either SCRYPT or PBKDF2 with lower n or c respectively. https://github.com/usmansaleem/v4keystore_converter . Let me know if you need more information.

[yorickdowne]

Thank you, amazing! Will give this a try next week.

[usmansaleem]

Closing the ticket. Feel free reopen the ticket or reach out on Consensys Discord channel if you have further queries regarding the java program. Discord handle: usmans.info.

[yorickdowne]

Great success! Converting to pbkdf2 with c=10 loads keys almost instantly. Thank you!