chore: upgrade to datafusion 43

[ion-elgreco]

Description

Bump kernel

• closes Schema metadata descriptions cause write failures in deltalake >= 0.19.0 #2850

[rtyler]

We need Datafusion to upgrade as well before we can do this. I have some patches to make the latest datafusion work but we're blocked on them to adopt the newest arrow and kernel

[Xuanwo]

Hi, datafusion has been released. Let's rock!

[alamb]

Thank you @ion-elgreco -- let me know if there is anything I can to do help with this PR (e.g. I could work on updating the code to use non deprecated APIs in the core for parquet statistics...)

[ion-elgreco]

Hey @alamb, I am not working on this atm, but @rtyler has picked it up from here

[ion-elgreco]

@alamb there seems to be a regression in DF, see https://github.com/delta-io/delta-rs/actions/runs/10959345721/job/30431504526?pr=2886#step:7:948.

A literal with a list is creating a dtype list with inner type using the name "item", we however have our data internally read according to the parquet spec, so that when we write the parquet files are meeting the field naming of the spec.

I had a quick glance at the Changelog but can't really pinpoint where this change may have occurred, do you have any ideas?

DFSchema { inner: 
Schema { fields: [
Field { name: "items_in_bucket", data_type: List(Field { name: "element", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, 
new schema: 
DFSchema { inner: Schema { fields: [
Field { name: "items_in_bucket", data_type: List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },

[alamb]

@alamb there seems to be a regression in DF, see https://github.com/delta-io/delta-rs/actions/runs/10959345721/job/30431504526?pr=2886#step:7:948.

A literal with a list is creating a dtype list with inner type using the name "item", we however have our data internally read according to the parquet spec, so that when we write the parquet files are meeting the field naming of the spec.

It sounds somewhat similar to a discussion happening here about nullability (but I think field name is similar in terms of how it is treated): apache/datafusion#11989 (comment)

Update; filed apache/datafusion#12560 to track the regression more

[rtyler]

@alamb thanks for following up! I feel a little sheepish 🐑 insofar that my internal CI showed these errors weeks ago in my integration testing between datafusion main with delta-rs main, I just didn't have the time to triage them like @ion-elgreco has 😄

I've subscribed to the issue you created and can help testing if necessary! Depending on my availability this weekend I can try fixing it in datafusion too 🤔

[matthewmturner]

Hi there - I believe a fix was implemented in datafusion and this PR is not blocked any longer.

[ion-elgreco]

Hi there - I believe a fix was implemented in datafusion and this PR is not blocked any longer.

It's not yet released though

[rtyler]

I can confirm that the latest datafusion main passes tests in CI 🥳

[alamb]

fix

I filed apache/datafusion#12813 to consider releasing a patch version of DataFusion

[rtyler]

@timsaucer I believe the error is introduced further up the optimization rules stack FWIW.

Going into optimization

Projection: target.id, target.temp, CASE __delta_rs_update_predicate WHEN Boolean(true) THEN CAST(make_parquet_array(Int64(100)) AS List(Field { name: "element", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, met
adata: {} })) ELSE target.items END AS items, __delta_rs_update_predicate

After invoking simplify_expressions

 DEBUG datafusion_optimizer::utils                                 > simplify_expressions:
Projection: target.id, target.temp, CASE __delta_rs_update_predicate WHEN Boolean(true) THEN List([100]) ELSE target.items END AS items, __delta_rs_update_predicate
  MetricObserver id=update_source_count
    Projection: target.id, target.temp, target.items, target.id = Int32(1) OR target.id != Int32(1) AND Boolean(NULL) AS __delta_rs_update_predicate
      TableScan: target

SimplifyExpressions appears to erase the defined field names which only causes problems later in OptimizeProjections.

I might not be qualified to fix this sticky of a bug 😆

[timsaucer]

FYI this does correct the error in the unit test test_update_with_array. I need to add unit tests in DataFusion still.
apache/datafusion#13468

[alamb]

We plan to make another DataFusion patch release to unblock this upgrade:

• Release Minor DataFusion 43.1.0 release apache/datafusion#13499

[alamb]