Clear object store between rounds

[yankevn]

Summary

These changes clear the object store at the end of each round during multi-round compaction. This replaces the existing behavior, which calls delete_many() on the list of object refs created during that round.

Rationale

In E2E testing, delete_many showed to take far too long when deleting a large number of object refs. This made compaction latency infeasible, and so clear() is being used instead. For this to work, only one partition may be running compaction at a time, otherwise clearing the shared object store will lead to issues.

Changes

• Switch delete_many to clear
• Move the pull request template

Impact

Executing clear() rather than delete_many() should lead to better performance. However, any jobs that run multi-round compaction with multiple partitions compacting in parallel will fail.

Testing

Unit tests were written.

Regression Risk

There is a risk the clear(), like delete_many(), will also take an extremely long time to run. To mitigate this, we will have to perform additional E2E testing on a large table.

The multi-round tests also had to be made more lax, as the FileObjectStore class cannot support clear(). Thus, we cannot check if the files were actually deleted, although we still check if clear() was called.

Checklist

• Unit tests covering the changes have been added

  • If this is a bugfix, regression tests have been added

• E2E testing has been performed

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite	Current: f4af160	Previous: 9e01066	Ratio
deltacat/tests/compute/test_compact_partition_incremental.py::test_compact_partition_incremental[1-incremental-pkstr-sknone-norcf_V1]	0.47873611444451214 iter/sec (stddev: 0)	1.1053474023542524 iter/sec (stddev: 0)	2.31

This comment was automatically generated by workflow using github-action-benchmark.

[raghumdani]

LGTM!