-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support protobuf struct type #206
base: main
Are you sure you want to change the base?
support protobuf struct type #206
Conversation
@zeroshade can I get your thoughts on this? spark follows this pattern: https://spark.apache.org/docs/latest/sql-data-sources-protobuf.html#handling-circular-references-protobuf-fields
but parquet can't do null types my idea, is if we get to depth = x, then we drop the field? e.g. for 2
|
@zeroshade otherwise, as this is used for json type fields, we could convert to a json string. and not need to think about recursive depths |
if the intent here is to use it for specifically JSON fields, then i would say we just use a string and utilize the Canonical JSON extension type so that it gets stored in parquet as a JSON typed field. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good, just a few nit picks
"google.golang.org/protobuf/encoding/protojson" | ||
"google.golang.org/protobuf/types/known/structpb" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you shift these includes to be grouped with the others of the same prefix below?
"fmt" | ||
"google.golang.org/protobuf/types/known/structpb" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as above, please place this with the sorted imports below
@tscottcoombes1 Running |
Rationale for this change
extra support for protobuf common types, converting to parquet
What changes are included in this PR?
support for protobuf struct and tests
Are these changes tested?
yes
Are there any user-facing changes?
no, just support for new type