Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add active utxo indexer #299

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
Draft

Add active utxo indexer #299

wants to merge 15 commits into from

Conversation

ana-pantilie
Copy link
Contributor

@ana-pantilie ana-pantilie commented Jan 26, 2024

I have managed to finish the implementation of the UtxoWithSpent indexer before heading off to the Plutus Core team. This means that the remaining changes to be done in this PR are minimal, and if one decides to revive the Marconi project this PR can be taken over quite easily.

What is done:

  1. generalized the SQLite indexer to accept complex insertion and deletion queries, as is required for this feature
  2. generalized the WithCatchup mechanism: this way an indexer can change its behavior once it has caught up with the node
  3. implemented the UtxoWithSpent indexer which work in the following way:
    • if the indexer hasn't caught up with the node, we index utxo and spent information into two separate tables, 'utxo' and 'spent'
    • once the indexer catches up with the node, it switches to a one-table representation of the data, utxo_with_spent, where each utxo contains the information on whether it was spent or not; inserting requires both adding rows and making "spent" updates to the table, which in consequence means that rollbacking must do the inverse of those operations

The reasoning behind this design is efficiency: during the catching up phase we want fast insertions into the tables, while after catch-up querying should be as quick as possible.

What needs to be done:

  • add SQL indexes to the utxo_with_spent table
  • write a Queryable instance for extracting all active utxos
  • add a event to remove all spent utxos from the table after some interval of time
  • write an executable which runs this indexer, and test against testing networks and mainnet

Pre-submit checklist:

  • Branch
    • Tests are provided (if possible)
    • Commit sequence broadly makes sense and have useful messages
    • Important changes are reflected in changelog.d of the affected packages
    • Relevant tickets are mentioned in commit messages
  • PR
    • (For external contributions) Corresponding issue exists and is linked in the description
    • Targeting main unless this is a cherry-pick backport
    • Self-reviewed the diff
    • Useful pull request description
    • If relevant, reference the ADR in the PR and reference the PR in the ADR
    • Reviewer requested

Signed-off-by: Ana Pantilie <[email protected]>
Signed-off-by: Ana Pantilie <[email protected]>
Signed-off-by: Ana Pantilie <[email protected]>
Signed-off-by: Ana Pantilie <[email protected]>
Signed-off-by: Ana Pantilie <[email protected]>
Signed-off-by: Ana Pantilie <[email protected]>
Signed-off-by: Ana Pantilie <[email protected]>
Signed-off-by: Ana Pantilie <[email protected]>
Signed-off-by: Ana Pantilie <[email protected]>
Signed-off-by: Ana Pantilie <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant