Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write null counts in Parquet statistics when they are known to be zero #6502

Closed
alamb opened this issue Oct 2, 2024 · 1 comment
Closed
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate

Comments

@alamb
Copy link
Contributor

alamb commented Oct 2, 2024

Describe the bug
The Rust parquet writer writes the thrift equivalent of None when the null count is zero rather than the thrift equivalent of Some(0)

This is inconsistent with the spec and other major implementations (Java and C++)

To Reproduce

Expected behavior
Write Some(0) when null count is known to be zero

Additional context
Part of #6256, but we split out the read/write changes into smaller PRs so adding a special issue here to track it / show up in the release notes

@alamb alamb added parquet Changes to the parquet crate bug labels Oct 2, 2024
@alamb
Copy link
Contributor Author

alamb commented Oct 2, 2024

Fixed by @etseidl in #6490

@alamb alamb closed this as completed Oct 2, 2024
@alamb alamb added enhancement Any new improvement worthy of a entry in the changelog and removed bug labels Oct 2, 2024
@alamb alamb changed the title Write null counts in Parquet statistics when they are known Write null counts in Parquet statistics when they are known to be zero Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate
Projects
None yet
Development

No branches or pull requests

1 participant