Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Directly read ADBC to Spark Dataframe #1801

Open
HaoXuAI opened this issue Apr 30, 2024 · 8 comments
Open

Directly read ADBC to Spark Dataframe #1801

HaoXuAI opened this issue Apr 30, 2024 · 8 comments
Labels
Type: enhancement New feature or request

Comments

@HaoXuAI
Copy link

HaoXuAI commented Apr 30, 2024

What feature or improvement would you like to see?

Similar to JDBC, something like:

jdbcDF = spark.read \
    .format("adbc") \
    .option("url", "adbc:postgresql") \
    .option("dbtable", "schema.tablename") \
    .option("user", "username") \
    .option("password", "password") \
    .load()

That way help to leverage ADBC in Spark compute environment.

@HaoXuAI HaoXuAI added the Type: enhancement New feature or request label Apr 30, 2024
@HaoXuAI HaoXuAI changed the title Directly load ADBC to Spark Dataframe Directly read ADBC to Spark Dataframe Apr 30, 2024
@lidavidm
Copy link
Member

lidavidm commented May 1, 2024

I think this should be a Spark feature request?

What I would like to do here is provide a JNI driver that can leverage the better-optimized postgresql/snowflake drivers from Java, though.

@HaoXuAI
Copy link
Author

HaoXuAI commented May 1, 2024

I think this should be a Spark feature request?

What I would like to do here is provide a JNI driver that can leverage the better-optimized postgresql/snowflake drivers from Java, though.

Right, it should be a spark feature. I'm posting here to check if it is a meaningful feature, and someone from the arrow team is already working on it. :)
What do you mean by JNI driver?

@lidavidm
Copy link
Member

lidavidm commented May 1, 2024

I don't believe anyone is working on this. Best to take it to the Spark community.

The ADBC driver for postgres, snowflake in Java just wraps JDBC. It doesn't provide any benefits. If we had JNI bindings to the C++/Go drivers we might see some performance benefits.

@HaoXuAI
Copy link
Author

HaoXuAI commented May 1, 2024

make sense. let me post it in the Spark repo.

@tokoko
Copy link
Contributor

tokoko commented May 7, 2024

@HaoXuAI hey, fancy seeing you here 😄 I've started this a while ago and then abandoned it (changed jobs and was no longer using Dremio). Can help you bring it back from the dead if you have a use case.

@HaoXuAI
Copy link
Author

HaoXuAI commented May 7, 2024

Hey @tokoko ! Great to see you here as well. Not a direct use case on work, but thinking about using ADBC in a project to read data on spark. Do you want to directly contribute to spark or keep it a plugin?

@tokoko
Copy link
Contributor

tokoko commented May 7, 2024

@HaoXuAI My goal at the time was to get it to mostly working condition as a plugin and then contribute, but we can do it either way.

@lidavidm Even with JNI drivers, the adbc java interface itself will still look the same, right? spark data source implementation will be independent of how drivers are implemented.

@lidavidm
Copy link
Member

lidavidm commented May 7, 2024

Yes, the idea of JNI would be to implement the same Java-side interface

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants