-
Notifications
You must be signed in to change notification settings - Fork 99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Directly read ADBC to Spark Dataframe #1801
Comments
I think this should be a Spark feature request? What I would like to do here is provide a JNI driver that can leverage the better-optimized postgresql/snowflake drivers from Java, though. |
Right, it should be a spark feature. I'm posting here to check if it is a meaningful feature, and someone from the arrow team is already working on it. :) |
I don't believe anyone is working on this. Best to take it to the Spark community. The ADBC driver for postgres, snowflake in Java just wraps JDBC. It doesn't provide any benefits. If we had JNI bindings to the C++/Go drivers we might see some performance benefits. |
make sense. let me post it in the Spark repo. |
Hey @tokoko ! Great to see you here as well. Not a direct use case on work, but thinking about using ADBC in a project to read data on spark. Do you want to directly contribute to spark or keep it a plugin? |
@HaoXuAI My goal at the time was to get it to mostly working condition as a plugin and then contribute, but we can do it either way. @lidavidm Even with JNI drivers, the adbc java interface itself will still look the same, right? spark data source implementation will be independent of how drivers are implemented. |
Yes, the idea of JNI would be to implement the same Java-side interface |
What feature or improvement would you like to see?
Similar to JDBC, something like:
That way help to leverage ADBC in Spark compute environment.
The text was updated successfully, but these errors were encountered: