Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dayofyear, weekofyear, month, dayofmonth, minute, second, next_da… #268

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

Avasil
Copy link
Contributor

@Avasil Avasil commented Mar 12, 2018

…y Column functions

Related to #164

*
* apache/spark
*/
def next_day[T](date: AbstractTypedColumn[T, String], dayOfWeek: String): date.ThisType[T, Option[java.sql.Date]] =
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now it doesn't compile.
In Spark it returns java.sql.Date, I'm not sure whether I should add TypedEncoder for that or use something else.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any issue with an encoder for java.sql.Date, if this is what's return in vanilla we can simply follow.

@OlivierBlanvillain
Copy link
Contributor

@Avasil Sorry for taking so long to have a look at your PR... The diff looks pretty good at a quick glance, do you need help on anything to get the CI green before the review?

@Avasil
Copy link
Contributor Author

Avasil commented Apr 9, 2018

Thanks and no problem @OlivierBlanvillain I was a bit busy lately too - I think I'm good, If I have any issues with java.sql.Date I'll let you guys know on gitter. :)

@Avasil
Copy link
Contributor Author

Avasil commented Apr 12, 2018

@OlivierBlanvillain Hmm, now it fails the test related to #205:

  test("#205: comparing literals encoded using Injection") {
    import org.apache.spark.sql.catalyst.util.DateTimeUtils
    implicit val dateAsInt: Injection[java.sql.Date, Int] =
      Injection(DateTimeUtils.fromJavaDate, DateTimeUtils.toJavaDate)

    val today = new java.sql.Date(System.currentTimeMillis)
    val data = Vector(P(42, today))
    val tds = TypedDataset.create(data)

    tds.filter(tds('d) === today).collect().run()
  }
}

final case class P(i: Int, d: java.sql.Date)

I'ts failing with:

[info]   Cause: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 86, Column 6: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 86, Column 6: Assignment conversion not possible from type "java.lang.Integer" to type "java.sql.Date"
(...)
[info] /* 009 */   private java.sql.Date argValue;
(...)
[info] /* 086 */     final java.sql.Date value2 = resultIsNull ? null : org.apache.spark.sql.catalyst.util.DateTimeUtils.fromJavaDate(argValue);

Any tips how to debug stuff like that? I need to somehow figure out why Spark tries to generate code this way. :D

Avasil added 2 commits April 25, 2018 09:14
# Conflicts:
#	dataset/src/main/scala/frameless/functions/NonAggregateFunctions.scala
#	dataset/src/test/scala/frameless/functions/NonAggregateFunctionsTests.scala
@Avasil
Copy link
Contributor Author

Avasil commented Sep 19, 2018

@OlivierBlanvillain @imarios Any ideas how to proceed? :) IIRC many other column functions could use java.sql.Date encoder in similar way to next_day

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants