indexer-alt: separate updates for consistent sequential pipelines #20482
+907
−181
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Description Although postgres supports bulk-updating rows using `VALUES`, diesel does not natively support it. This change adds support for this. It is in its own crate so that we can limit the fallout of depending on the diesel breaking changes feature (which we need to depend on to access the types that diesel converts collections of model types into, ready to be inserted into the DB). ## Test plan Unit tests for the query generation, and E2E tests for running updates on a DB with this new DSL: ``` sui$ cargo nextest run -p diesel-update-from ```
## Description Use object changes from transaction effects to figure out which changes to consistent tables correspond to new rows and which changes correspond to updates. This means we can avoid using `INSERT ... ON CONFLICT DO UPDATE` which requires postgres to try an insert, detect constraints, and then go for the update, which should hopefully improve performance. On the other hand, it means that these pipelines will not work at all if they are started at an arbitrary point in time (because the `UPDATE`-s will fail). ## Test plan Run the indexer before and after the change, dump the resulting tables and make sure the results are the same: ``` sui$ cargo run -p sui-indexer-alt -- generate-config > /tmp/indexer.toml sui$ cargo run -p sui-indexer-alt -- indexer \ --remote-store-url https://checkpoints.mainnet.sui.io \ --last-checkpoint 50000 --config /tmp/indexer.toml \ --pipeline sum_obj_types --pipeline sum_coin_balances sui$ psql postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt sui_indexer_alt=# COPY (SELECT object_id, object_version, owner_kind, owner_id FROM sum_obj_types ORDER BY object_id) TO '/tmp/objs.csv' WITH DELIMITER ',' CSV HEADER; sui_indexer_alt=# COPY (SELECT object_id, object_version, owner_id, coin_balance FROM sum_coin_balances ORDER BY object_id) TO '/tmp/coins.csv' WITH DELIMITER ',' CSV HEADER; ```
## Description Add `#[diesel(treat_none_as_default_value = false)]` to model types that include optional fields. This affects how those fields are written out to SQL when they contain `None`. Previously (and by default), those fields would be represented by the keyword `DEFAULT VALUE`, and after this change, they will be represented by a parameter binding, which will be bound to `NULL`. This is semantically identical in our case, because we don't set default values, but it also results in less variety in prepared statements (because regardless of the content of fields, they will now all be represented by a binding), which will improve grouping of statistics per-statement, and could also improve performance, if those prepared statements can be cached and re-used. ## Test plan Re-run indexer on first 100000 checkpoints.
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Skipped Deployments
|
amnn
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
Avoid the dependency on `pq-sys`.
amnn
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
8 tasks
amnn
commented
Dec 2, 2024
There was a problem hiding this comment.
I actually think I don't need this anymore.
amnn
temporarily deployed
to
sui-typescript-aws-kms-test-env
GitHub Actions
Inactive
gegaowp
reviewed
Dec 3, 2024
| @@ -59,38 +60,49 @@ impl Processor for SumCoinBalances { | |||
| } | |||
| } | |||
|
|
|||
| // Deleted and wrapped coins | |||
| // Do a fist pass to add updates without their associated contents into the `values` | |||
| // mapping, based on the transaction's object changes. | |||
| for change in tx.effects.object_changes() { | |||
There was a problem hiding this comment.
TransactionEffectsAPI has methods of
fn created(&self) -> Vec<(ObjectRef, Owner)>;
fn mutated(&self) -> Vec<(ObjectRef, Owner)>;
fn unwrapped(&self) -> Vec<(ObjectRef, Owner)>;
fn deleted(&self) -> Vec<ObjectRef>;
fn unwrapped_then_deleted(&self) -> Vec<ObjectRef>;
fn wrapped(&self) -> Vec<ObjectRef>;
it seems cleaner to use that instead of classifying object changes again?
| @@ -14,6 +14,7 @@ use crate::schema::{ | |||
|
|
|||
| #[derive(Insertable, Debug, Clone, FieldCount)] | |||
| #[diesel(table_name = kv_objects, primary_key(object_id, object_version))] | |||
| #[diesel(treat_none_as_default_value = false)] | |||
| }, | ||
| ]); | ||
|
|
||
| assert_display_snapshot!(debug_query::<Pg, _>(&query), @r###"UPDATE "objects" SET "version" = excluded."version", "kind" = excluded."kind", "owner" = excluded."owner", "type_" = excluded."type_" FROM (VALUES ($1, $2, $3, $4, $5), ($6, $7, $8, $9, $10)) AS excluded ("object_id", "version", "kind", "owner", "type_") WHERE ("objects"."object_id" = excluded."object_id") -- binds: [[1, 2, 3], 1, 2, None, Some("type"), [4, 5, 6], 2, 3, Some([7, 8, 9]), None]"###); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Use object changes from transaction effects to figure out which changes to consistent tables correspond to new rows and which changes correspond to updates. This means we can avoid using
INSERT ... ON CONFLICT DO UPDATEwhich requires postgres to try an insert, detect constraints, and then go for the update, which should hopefully improve performance. On the other hand, it means that these pipelines will not work at all if they are started at an arbitrary point in time (because theUPDATE-s will fail).The largest part of this change was adding support for bulk-updates to diesel (see diesel-rs/diesel#2879). This requires opting in to breaking changes by exposing diesel's internals. To limit the fall out of that, this support has been added in its own crate.
Finally, as part of this change, I ran into a flag that can be set on model types:
#[diesel(treat_none_as_default_value = ...)]which defaults totrue. Setting this tofalseon models that contain optional values should improve statistics collection and may improve performance through prepared statement caching.Test plan
Unit tests for
update_fromquery generation, and E2E tests for running updates on a DB with the new DSL:Run the indexer before and after the change, dump the resulting tables and make sure the results are the same:
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.