switching version check to spark session

[fdosani]

Fix for #320

Seems like Databricks does something to the pyspark.version.__version__. Switching to the actual spark session for the check in this case.

@achrusciel FYI, in case you can try out this specific branch to make sure it works for you.

[achrusciel]

I am testing the change right now. The code is now executing and I can see in the Spark UI of the cluster that jobs are being executed, and it works.

This is the first time I am switching from the LegacySparkCompare to the new SparkSQLCompare. The same comparison for about 5 million rows completes in ca. 2 minutes when using LegacySparkCompare and with the new SparkSQLCompare it has completed after 8 minutes successfully.

[fdosani]

million rows completes in ca. 2 minutes when using LegacySparkCompare and with the new SparkSQLCompare it has completed after 8 minutes successfull

This is expected. The legacy version dropped duplicates and doesn't have some functionality which the new one has (aligns to the Pandas version). The main thing is the dropping duplicates.