HDDS-11773. Prevent frequent DataNode Ratis snapshotting.

[jojochuang]

What changes were proposed in this pull request?

HDDS-11773. Bump hdds.ratis.snapshot.threshold and hdds.container.ratis.statemachine.max.pending.apply-transactions to 100k

Please describe your PR in detail:

• DataNode is configured to snapshot every 10k transactions, which is too frequent for HBase workloads where DataNode can do thousands and eventually 10s of thousands of transactions per second.
• Bump hdds.ratis.snapshot.threshold to 100k for now. Update hdds.container.ratis.statemachine.max.pending.apply-transactions to make it consistent.
• We have to revisit and make it 1M eventually.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11773

How was this patch tested?

Applied the change to a HBase cluster. Previously it was snapshotting every 4-5 seconds, and now it is doing it about every minute.

[smengcl]

Thanks @jojochuang .

But this also changes OM's default. It could be a concern when Ratis snapshot interval is set to a value too high for followers to catch up, thus failing OM bootstrapping? Is there an existing mechanism to tune this for Datanodes only?

What do you think? @szetszwo

[jojochuang]

isn't it DataNode only?
SCM uses ozone.scm.ha.ratis.snapshot.threshold.
I guess OM uses the default value which is 400000.

[smengcl]

isn't it DataNode only? SCM uses ozone.scm.ha.ratis.snapshot.threshold. I guess OM uses the default value which is 400000.

Right. Pls amend the config tag

[jojochuang]

The snapshotting itself takes around the same amount of time as before, 1-3ms.

2024-11-22 22:31:12,432 INFO [e693615a-d484-4165-8446-dff08cac5978@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.
ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:179, i:7007978) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/s
napshot.179_7007978
2024-11-22 22:31:12,434 INFO [e693615a-d484-4165-8446-dff08cac5978@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.
ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:179, i:7007978) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0
a206/sm/snapshot.179_7007978 took: 1 ms
2024-11-22 22:32:46,285 INFO [e693615a-d484-4165-8446-dff08cac5978@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.
ContainerStateMachine: group-D595E2D0A206: Taking a snapshot at:(t:179, i:7107980) file /var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0a206/sm/s
napshot.179_7107980
2024-11-22 22:32:46,287 INFO [e693615a-d484-4165-8446-dff08cac5978@group-D595E2D0A206-StateMachineUpdater]-org.apache.hadoop.ozone.container.common.transport.server.ratis.
ContainerStateMachine: group-D595E2D0A206: Finished taking a snapshot at:(t:179, i:7107980) file:/var/lib/hadoop-ozone/datanode/ratis/data/c420af11-2786-4f5a-9b5a-d595e2d0
a206/sm/snapshot.179_7107980 took: 2 ms

[smengcl]

Thanks @jojochuang

[jojochuang]

Thanks for the review, @smengcl