CHANGELOG.md

Changelog

v2.0.2 (2024-10-10)

Bug Fixes

• Fix ingress capability discovery (#2201 by @jacobsalway)
• fix: imagePullPolicy was ignored (#2222 by @missedone)
• fix: spark-submission failed due to lack of permission by user spark (#2223 by @missedone)
• Remove cap_net_bind_service from image (#2216 by @jacobsalway)
• fix: webhook panics due to logging (#2232 by @ChenYi015)

Misc

• Bump github.com/aws/aws-sdk-go-v2 from 1.30.5 to 1.31.0 (#2207 by @dependabot[bot])
• Bump golang.org/x/net from 0.28.0 to 0.29.0 (#2205 by @dependabot[bot])
• Bump github.com/docker/docker from 27.0.3+incompatible to 27.1.1+incompatible (#2125 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.58.3 to 1.63.3 (#2206 by @dependabot[bot])
• Update integration test workflow and add golangci lint check (#2197 by @ChenYi015)
• Bump github.com/aws/aws-sdk-go-v2 from 1.31.0 to 1.32.0 (#2229 by @dependabot[bot])
• Bump cloud.google.com/go/storage from 1.43.0 to 1.44.0 (#2228 by @dependabot[bot])
• Bump manusa/actions-setup-minikube from 2.11.0 to 2.12.0 (#2226 by @dependabot[bot])
• Bump golang.org/x/time from 0.6.0 to 0.7.0 (#2227 by @dependabot[bot])
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.33 to 1.27.42 (#2231 by @dependabot[bot])
• Bump github.com/prometheus/client_golang from 1.19.1 to 1.20.4 (#2204 by @dependabot[bot])
• Add check for generating manifests and code (#2234 by @ChenYi015)

Full Changelog

v2.0.1 (2024-09-26)

New Features

• FEATURE: build operator image as non-root (#2171 by @ImpSy)

Bug Fixes

• Update controller RBAC for ConfigMap and PersistentVolumeClaim (#2187 by @ChenYi015)

Misc

• Bump github.com/onsi/ginkgo/v2 from 2.19.0 to 2.20.2 (#2188 by @dependabot[bot])
• Bump github.com/onsi/gomega from 1.33.1 to 1.34.2 (#2189 by @dependabot[bot])

Full Changelog

v2.0.0 (2024-09-23)

Breaking Changes

• Use controller-runtime to reconsturct spark operator (#2072 by @ChenYi015)
• feat: support driver and executor pod use different priority (#2146 by @Kevinz857)

New Features

• Support gang scheduling with Yunikorn (#2107) by @jacobsalway
• Reintroduce option webhook.enable (#2142 by @ChenYi015)
• Add default batch scheduler argument (#2143 by @jacobsalway)
• Support extended kube-scheduler as batch scheduler (#2136 by @ChenYi015)
• Set schedulerName to Yunikorn (#2153 by @jacobsalway)
• Feature: Add pprof endpoint (#2164 by @ImpSy)

Bug Fixes

• fix: Add default values for namespaces to match usage descriptions (#2128 by @snappyyouth)
• Fix: Spark role binding did not render properly when setting spark service account name (#2135 by @ChenYi015)
• fix: unable to set controller/webhook replicas to zero (#2147 by @ChenYi015)
• Adding support for setting spark job namespaces to all namespaces (#2123 by @ChenYi015)
• Fix: e2e test failes due to webhook not ready (#2149 by @ChenYi015)
• fix: webhook not working when settings spark job namespaces to empty (#2163 by @ChenYi015)
• fix: The logger had an odd number of arguments, making it panic (#2166 by @tcassaert)
• fix the make kind-delete-custer to avoid accidental kubeconfig deletion (#2172 by @ImpSy)
• Add specific error in log line when failed to create web UI service (#2170 by @tcassaert)
• Account for spark.executor.pyspark.memory in Yunikorn gang scheduling (#2178 by @jacobsalway)
• Fix: spark application does not respect time to live seconds (#2165 by @ChenYi015)

Misc

• Update workflow and docs for releasing Spark operator (#2089 by @ChenYi015)
• Fix broken integration test CI (#2109 by @ChenYi015)
• Fix CI: environment variable BRANCH is missed (#2111 by @ChenYi015)
• Update Makefile for building sparkctl (#2119 by @ChenYi015)
• Update release workflow and docs (#2121 by @ChenYi015)
• Run e2e tests on Kind (#2148 by @jacobsalway)
• Upgrade to Go 1.23.1 (#2155 by @jacobsalway)
• Upgrade to Spark 3.5.2 (#2154 by @jacobsalway)
• Bump sigs.k8s.io/scheduler-plugins from 0.29.7 to 0.29.8 (#2159 by @dependabot[bot])
• Bump gocloud.dev from 0.37.0 to 0.39.0 (#2160 by @dependabot[bot])
• Update e2e tests (#2161 by @ChenYi015)
• Upgrade to Spark 3.5.2(#2012) (#2157 by @ha2hi)
• Bump github.com/aws/aws-sdk-go-v2/config from 1.27.27 to 1.27.33 (#2174 by @dependabot[bot])
• Bump helm.sh/helm/v3 from 3.15.3 to 3.16.1 (#2173 by @dependabot[bot])
• implement workflow to scan latest released docker image (#2177 by @ImpSy)

What's Changed

• Cherry pick #2081 #2046 #2091 #2072 by @ChenYi015 in #2108
• Cherry pick #2089 #2109 #2111 by @ChenYi015 in #2110
• Release v2.0.0-rc.0 by @ChenYi015 in #2115
• Cherry pick commits for releasing v2.0.0 by @ChenYi015 in #2156
• Release v2.0.0 by @ChenYi015 in #2182

Full Changelog

v2.0.0-rc.0 (2024-08-09)

Breaking Changes

• Use controller-runtime to reconsturct spark operator (#2072 by @ChenYi015)

Misc

• Fix CI: environment variable BRANCH is missed (#2111 by @ChenYi015)

• Fix broken integration test CI (#2109 by @ChenYi015)

• Update workflow and docs for releasing Spark operator (#2089 by @ChenYi015)

What's Changed

• Release v2.0.0-rc.0 (#2115 by @ChenYi015)
• Cherry pick #2089 #2109 #2111 (#2110 by @ChenYi015)
• Cherry pick #2081 #2046 #2091 #2072 (#2108 by @ChenYi015)

Full Changelog

spark-operator-chart-1.4.6 (2024-07-26)

• Add topologySpreadConstraints (#2091 by @jbhalodia-slack)
• Add Alibaba Cloud to adopters (#2097 by @ChenYi015)
• Update Stale bot settings (#2095 by @andreyvelich)
• Add @ChenYi015 to approvers (#2096 by @ChenYi015)
• Add CHANGELOG.md file and use python script to generate it automatically (#2087 by @ChenYi015)

Full Changelog

spark-operator-chart-1.4.5 (2024-07-22)

• Update the process to build api-docs, generate CRD manifests and code (#2046 by @ChenYi015)
• Add workflow for closing stale issues and PRs (#2073 by @ChenYi015)

Full Changelog

spark-operator-chart-1.4.4 (2024-07-22)

• Update helm docs (#2081 by @csp33)

Full Changelog

spark-operator-chart-1.4.3 (2024-07-03)

• Add PodDisruptionBudget to chart (#2078 by @csp33)
• Update README and documentation (#2047 by @ChenYi015)
• Add code of conduct and update contributor guide (#2074 by @ChenYi015)
• Remove .gitlab-ci.yml (#2069 by @jacobsalway)
• Modified README.MD as per changes discussed on #2062 (#2066 by @vikas-saxena02)

Full Changelog

spark-operator-chart-1.4.2 (2024-06-17)

• Support objectSelector on mutating webhook (#2058 by @Cian911)

Full Changelog

spark-operator-chart-1.4.1 (2024-06-15)

• Adding an option to set the priority class for spark-operator pod (#2043 by @pkgajulapalli)
• Update minikube version in CI (#2059 by @Cian911)

Full Changelog

spark-operator-chart-1.4.0 (2024-06-05)

• Certifictes are generated by operator rather than gencerts.sh (#2016 by @ChenYi015)
• Add ChenYi015 as spark-operator reviewer (#2045 by @ChenYi015)

Full Changelog

spark-operator-chart-1.3.2 (2024-06-05)

• Bump appVersion to v1beta2-1.5.0-3.5.0 (#2044 by @ChenYi015)
• Add restartPolicy field to SparkApplication Driver/Executor initContainers CRDs (#2022 by @mschroering)
• 📝 Add Inter&Co to who-is-using.md (#2040 by @ignitz)

Full Changelog

spark-operator-chart-1.3.1 (2024-05-31)

• Chart: add POD_NAME env for leader election (#2039 by @Aakcht)

Full Changelog

spark-operator-chart-1.3.0 (2024-05-20)

• Support exposing extra TCP ports in Spark Driver via K8s Ingress (#1998 by @hiboyang)
• Fixes a bug with dynamic allocation forcing the executor count to be 1 even when minExecutors is set to 0 (#1979 by @peter-mcclonski)
• Remove outdated PySpark experimental warning in example (#2014 by @andrejpk)
• Update Spark Job Namespace docs (#2000 by @matthewrossi)

Full Changelog

spark-operator-chart-1.2.15 (2024-05-07)

• Fix examples (#2010 by @peter-mcclonski)

Full Changelog

spark-operator-chart-1.2.14 (2024-04-26)

• feat: add support for service labels on driver-svc (#1985 by @Cian911)

Full Changelog

spark-operator-chart-1.2.13 (2024-04-24)

• fix(chart): remove operator namespace default for job namespaces value (#1989 by @t3mi)
• Fix Docker Hub Credentials in CI (#2003 by @andreyvelich)

Full Changelog

spark-operator-chart-1.2.12 (2024-04-19)

• Add emptyDir sizeLimit support for local dirs (#1993 by @jacobsalway)
• fix: Removed publish-image dependency on publishing the helm chart (#1995 by @vara-bonthu)

Full Changelog

spark-operator-chart-1.2.11 (2024-04-19)

• fix: Update Github workflow to publish Helm charts on chart changes, irrespective of image change (#1992 by @vara-bonthu)
• chore: Add Timo to user list (#1615 by @vanducng)
• Update spark operator permissions for CRD (#1973 by @ChenYi015)
• fix spark-rbac (#1986 by @Aransh)
• Use Kubeflow Docker Hub for Spark Operator Image (#1974 by @andreyvelich)
• fix: fixed serviceaccount annotations (#1972 by @AndrewChubatiuk)

Full Changelog

spark-operator-chart-1.2.7 (2024-04-16)

• fix: upgraded k8s deps (#1983 by @AndrewChubatiuk)
• chore: remove k8s.io/kubernetes replaces and adapt to v1.29.3 apis (#1968 by @ajayk)
• Add some helm chart unit tests and fix spark service account render failure when extra annotations are specified (#1967 by @ChenYi015)
• feat: Doc updates, Issue and PR templates are added (#1970 by @vara-bonthu)

Full Changelog

spark-operator-chart-1.2.5 (2024-04-14)

• fixed docker image tag and updated chart docs (#1969 by @AndrewChubatiuk)

Full Changelog

spark-operator-chart-1.2.4 (2024-04-13)

• publish chart independently, incremented both chart and image versions to trigger build of both (#1964 by @AndrewChubatiuk)
• Update helm chart README (#1958 by @ChenYi015)
• fix: add containerPort declaration for webhook in helm chart (#1961 by @zevisert)
• added id for a build job to fix digests artifact creation (#1963 by @AndrewChubatiuk)
• support multiple namespaces (#1955 by @AndrewChubatiuk)
• chore: replace GoogleCloudPlatform/spark-on-k8s-operator with kubeflow/spark-operator (#1937 by @zevisert)
• Chart: add patch permissions for spark operator SA to support spark 3.5.0 (#1884 by @Aakcht)
• Cleanup after golang upgrade (#1956 by @AndrewChubatiuk)
• feat: add support for custom service labels (#1952 by @Cian911)
• upgraded golang and dependencies (#1954 by @AndrewChubatiuk)
• README for installing operator using kustomize with custom namespace and service name (#1778 by @shahsiddharth08)
• BUGFIX: Added cancel method to fix context leak (#1917 by @fazledyn-or)
• remove unmatched quotes from user-guide.md (#1584 by @taeyeopkim1)
• Add PVC permission to Operator role (#1889 by @wyangsun)
• Allow to set webhook job resource limits (#1429,#1300) (#1946 by @karbyshevds)
• Create OWNERS (#1927 by @zijianjoy)
• fix: fix issue #1723 about spark-operator not working with volcano on OCP (#1724 by @disaster37)
• Add Rokt to who-is-using.md (#1867 by @jacobsalway)
• Handle invalid API resources in discovery (#1758 by @wiltonsr)
• Fix docs for Volcano integration (#1719 by @VVKot)
• Added qualytics to who is using (#1736 by @josecsotomorales)
• Allowing optional annotation on rbac (#1770 by @cxfcxf)
• Support seccompProfile in Spark application CRD and fix pre-commit jobs (#1768 by @ordukhanian)
• Updating webhook docs to also mention eks (#1763 by @JunaidChaudry)
• Link to helm docs fixed (#1783 by @haron)
• Improve getMasterURL() to add [] to IPv6 if needed (#1825 by @LittleWat)
• Add envFrom to operator deployment (#1785 by @matschaffer-roblox)
• Expand ingress docs a bit (#1806 by @matschaffer-roblox)
• Optional sidecars for operator pod (#1754 by @qq157755587)
• Add Roblox to who-is (#1784 by @matschaffer-roblox)
• Molex started using spark K8 operator. (#1714 by @AshishPushpSingh)
• Extra helm chart labels (#1669 by @kvanzuijlen)

Full Changelog

spark-operator-chart-1.1.27 (2023-03-17)

• Added permissions for leader election #1635 (#1647 by @ordukhanian)
• Fix #1393 : fix tolerations block in wrong segment for webhook jobs (#1633 by @zhiminglim)
• add dependabot (#1629 by @monotek)
• Add support for ephemeral.volumeClaimTemplate in helm chart CRDs (#1661 by @ArshiAAkhavan)
• Add Kognita to "Who is using" (#1637 by @claudino-kognita)
• add lifecycle to executor (#1674 by @tiechengsu)
• Fix signal handling for non-leader processes (#1680 by @antonipp)

Full Changelog

spark-operator-chart-1.1.26 (2022-10-25)

• update go to 1.19 + k8s.io libs to v0.25.3 (#1630 by @ImpSy)
• Update README - secrets and sidecars need mutating webhooks (#1550 by @djdillon)

Full Changelog

spark-operator-chart-1.1.25 (2022-06-08)

• Webhook init and cleanup should respect nodeSelector (#1545 by @erikcw)
• rename unit tests to integration tests in Makefile#integration-test (#1539 by @dcoliversun)

Full Changelog

spark-operator-chart-1.1.24 (2022-06-01)

• Fix: use V1 api for CRDs for volcano integration (#1540 by @Aakcht)

Full Changelog

spark-operator-chart-1.1.23 (2022-05-18)

• fix: add pre-upgrade hook to rbac resources (#1511 by @cwyl02)

Full Changelog

spark-operator-chart-1.1.22 (2022-05-16)

• Fixes issue #1467 (issue when deleting SparkApplication without metrics server) (#1530 by @aneagoe)
• Implement --logs and --delete flags on 'sparkctl create' and a timeout on 'sparkctl log' to wait a pod startup (#1506 by @alaurentinoofficial)
• Fix Spark UI URL in app status (#1518 by @gtopper)
• remove quotes from yaml file (#1524 by @zencircle)
• Added missing manifest yaml, point the manifest to the right direction (#1504 by @RonZhang724)

Full Changelog

spark-operator-chart-1.1.21 (2022-05-12)

• Ensure that driver is deleted prior to sparkapplication resubmission (#1521 by @khorshuheng)

Full Changelog

spark-operator-chart-1.1.20 (2022-04-11)

• Add ingress-class-name controller flag (#1482 by @voyvodov)

Full Changelog

spark-operator-chart-1.1.19 (2022-02-14)

• Add Operator volumes and volumeMounts in chart (#1475 by @ocworld)

Full Changelog

spark-operator-chart-1.1.18 (2022-02-13)

• Updated default registry to ghcr.io (#1454 by @aneagoe)
• Github actions workflow fix for Helm chart deployment (#1456 by @vara-bonthu)
• Kubernetes v1.22 extensions/v1beta1 API removal (#1427 by @aneagoe)
• Fixes an issue with github action in job build-spark-operator (#1452 by @aneagoe)
• use github container registry instead of gcr.io for releases (#1422 by @TomHellier)
• Fixes an error that was preventing the pods from being mutated (#1421 by @ssullivan)
• Make github actions more feature complete (#1418 by @TomHellier)
• Resolves an error when deploying the webhook where the k8s api indica… (#1413 by @ssullivan)

Full Changelog

spark-operator-chart-1.1.15 (2021-12-02)

• Add docker build to github action (#1415 by @TomHellier)

Full Changelog

spark-operator-chart-1.1.14 (2021-11-30)

• Updating API version of admissionregistration.k8s.io (#1401 by @sairamankumar2)
• Add C2FO to who is using (#1391 by @vanhoale)

Full Changelog

spark-operator-chart-1.1.13 (2021-11-18)

• delete-service-accounts-and-roles-before-creation (#1384 by @TiansuYu)

Full Changelog

spark-operator-chart-1.1.12 (2021-11-14)

• webhook timeout variable (#1387 by @sairamankumar2)

Full Changelog

spark-operator-chart-1.1.11 (2021-11-12)

• [FIX] add service account access to persistentvolumeclaims (#1390 by @mschroering)
• Add DeepCure to who is using (#1389 by @mschroering)

Full Changelog

spark-operator-chart-1.1.10 (2021-11-09)

• Add custom toleration support for webhook jobs (#1383 by @korjek)
• fix container name in addsecuritycontext patch (#1377 by @lybavsky)

Full Changelog

spark-operator-chart-1.1.9 (2021-11-01)

• Role and RoleBinding not installed for webhook-init in Helm pre-hook (#1379 by @zzvara)

Full Changelog

spark-operator-chart-1.1.8 (2021-10-26)

• Regenerate deleted cert after upgrade (#1373 by @simplylizz)
• Make manifests usable by Kustomize (#1367 by @karpoftea)
• #1329 update the operator to allow subpaths to be used with the spark ui ingress. (#1330 by @TomHellier)

Full Changelog

spark-operator-chart-1.1.7 (2021-10-21)

• serviceAccount annotations (#1350 by @moskitone)
• Update Dockerfile (#1369 by @Sadagopan88)
• [FIX] tolerations are not directly present in Driver(/Executor)Spec (#1365 by @s-pedamallu)
• fix running metrics for application deletion (#1358 by @Aakcht)
• Update who-is-using.md (#1338 by @Juandavi1)
• Update who-is-using.md (#1082 by @Juandavi1)
• Add support for executor service account (#1322 by @bbenzikry)
• fix NPE introduce on #1280 (#1325 by @ImpSy)

Full Changelog

spark-operator-chart-1.1.6 (2021-08-04)

• Add hook deletion policy for spark-operator service account (#1313 by @pdrastil)

Full Changelog

spark-operator-chart-1.1.5 (2021-07-28)

• Add user defined pod labels (#1288 by @pdrastil)

Full Changelog

spark-operator-chart-1.1.4 (2021-07-25)

• Migrate CRDs from v1beta1 to v1. Add additionalPrinterColumns (#1298 by @drazul)
• Explain "signal: kill" errors during submission (#1292 by @zzvara)
• fix the invalid repo address (#1291 by @william-wang)
• add failure context to recordExecutorEvent (#1280 by @ImpSy)
• Update pythonVersion to fix example (#1284 by @stratus)
• add crds drift check between chart/ and manifest/ (#1272 by @ImpSy)

Full Changelog

spark-operator-chart-1.1.3 (2021-05-25)

• Allow user to specify service annotation on Spark UI service (#1264 by @khorshuheng)

Full Changelog

spark-operator-chart-1.1.2 (2021-05-25)

• implement shareProcessNamespace in SparkPodSpec (#1262 by @ImpSy)

Full Changelog

spark-operator-chart-1.1.1 (2021-05-19)

• Enable UI service flag for disabling UI service (#1261 by @sairamankumar2)
• Add DiDi to who-is-using.md (#1255 by @Run-Lin)
• doc: update who is using page (#1251 by @luizm)
• Add Tongdun under who-is-using (#1249 by @lomoJG)
• [#1239] Custom service port name for spark application UI (#1240 by @marcozov)
• fix: do not remove preemptionPolicy in patcher when not present (#1246 by @HHK1)

Full Changelog

spark-operator-chart-1.1.0 (2021-04-28)

• Updating Spark version from 3.0 to 3.1.1 (#1153 by @chethanuk)

Full Changelog

spark-operator-chart-1.0.10 (2021-04-28)

• Add support for blue/green deployments (#1230 by @flupke)
• Update who-is-using.md: Fossil is using Spark Operator for Production (#1244 by @duyet)

Full Changelog

spark-operator-chart-1.0.9 (2021-04-23)

• Link to Kubernetes Slack (#1234 by @jsoref)
• fix: remove preemptionPolicy when priority class name is used (#1236 by @HHK1)
• Spelling (#1231 by @jsoref)
• Add support to expose custom ports (#1205 by @luizm)
• Fix the error of hostAliases when there are more than 2 hostnames (#1209 by @cdmikechen)
• remove multiple prefixes for 'p' (#1210 by @chaudhryfaisal)
• added --s3-force-path-style to force path style URLs for S3 objects (#1206 by @chaudhryfaisal)
• Allow custom bucket path (#1207 by @bribroder)
• fix: Remove priority from the spec when using priority class (#1203 by @HHK1)
• Fix go get issue with "unknown revision v0.0.0" (#1198 by @hongshaoyang)

Full Changelog

spark-operator-chart-1.0.8 (2021-03-07)

• Helm: Put service account into pre-install hook. (#1155 by @tandrup)
• correct hook annotation for webhook job (#1193 by @chaudhryfaisal)
• Update who-is-using.md (#1174 by @tarek-izemrane)
• add Carrefour as adopter and contributor (#1156 by @AliGouta)

Full Changelog

spark-operator-chart-1.0.7 (2021-02-05)

• fix issue #1131 (#1142 by @kz33)

Full Changelog

spark-operator-chart-1.0.6 (2021-02-04)

• Add Fossil to who-is-using.md (#1152 by @duyet)
• #1143 Helm issues while deploying using argocd (#1145 by @TomHellier)
• Include Gojek in who-is-using.md (#1146 by @pradithya)
• add hostAliases for SparkPodSpec (#1133 by @ImpSy)
• Adding MavenCode (#1128 by @charlesa101)
• Add MongoDB to who-is-using.md (#1123 by @chickenPopcorn)
• update go version to 1.15 and k8s deps to v0.19.6 (#1119 by @stpabhi)

Full Changelog

spark-operator-chart-1.0.5 (2020-12-15)

• Add prometheus containr port name (#1099 by @nicholas-fwang)

Full Changelog

spark-operator-chart-1.0.4 (2020-12-12)

• Upgrade the Chart version to 1.0.4 (#1113 by @ordukhanian)
• Support Prometheus PodMonitor Deployment (#1106) (#1112 by @ordukhanian)
• update executor status if pod is lost while app is still running (#1111 by @ImpSy)
• Add scheduler func for clearing batch scheduling on completed (#1079 by @nicholas-fwang)
• Add configuration for SparkUI service type (#1100 by @jutley)

Full Changelog

spark-operator-chart-1.0.3 (2020-12-07)

• Update docs with new helm instructions (#1105 by @hagaibarel)

Full Changelog