You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would like to create page indexes (per this doc) for a column when writing a parquet file, and then use that index to seek to a particular row. I have a case where the file is sorted by a particular ID, and the queries often want to start with a particular ID and then read all rows thereafter. Without an index, I can find the right row group using statistics, but then have to scan through all values in the column in the row group to find the ID and determine the right row.
I've gone through all of the code and API in the parquet package and sub-packages, and all I can find are columns in Thrift-generated code for this and accessors in the column chunk metadata that return the file offset for where the index is stored. But there seems to be no API to actually read the index and use it. And there is no configuration, on the write side, for whether to create an index or not.
When will this be supported? How active is development on the Go runtime?
The text was updated successfully, but these errors were encountered:
Hey @jhump, you're correct that the page indexes are not yet supported in the Go implementation. It's been on my to do list for a while but I haven't found the bandwidth. I don't have a timeline for it, but I would also happily review a PR if you wanted to contribute it yourself!
Development on the Go implementation has been mostly maintenance lately, but I fully intend to get back to it and add page indexes, bloom filters, and so on. I just don't have a timeline yet
@zeroshade, thank you for the quick response! I don't have quite enough experience in this codebase to offer a PR yet, but maybe I will later, after using it more and doing more development with it.
I would like to create page indexes (per this doc) for a column when writing a parquet file, and then use that index to seek to a particular row. I have a case where the file is sorted by a particular ID, and the queries often want to start with a particular ID and then read all rows thereafter. Without an index, I can find the right row group using statistics, but then have to scan through all values in the column in the row group to find the ID and determine the right row.
I've gone through all of the code and API in the parquet package and sub-packages, and all I can find are columns in Thrift-generated code for this and accessors in the column chunk metadata that return the file offset for where the index is stored. But there seems to be no API to actually read the index and use it. And there is no configuration, on the write side, for whether to create an index or not.
When will this be supported? How active is development on the Go runtime?
The text was updated successfully, but these errors were encountered: