Scala is nice for doing things with Spark compared to say, Java, but I really think that Kotlin can do just as well. The only advantage Scala has with Spark is the syntactic sugar. Once there is an API for Kotlin (beyond the preview version), I will be dropping Scala like a bad habit. Kotlin is also concise, has better IntelliJ support, less "implicit magic", and better Java interop.
Kotlin has lots of syntax sugar but the semantics are crazy. It's essentially an imperative language where the only way to reason about effects (suspend functions) is by reasoning about control flow. That already makes things like error handling incomprehensible, and it'll be a really bad fit for Spark's way of doing things (where values have to be serializable and shipped around the cluster) - imagine trying to use a suspend function from a Spark anything, it's completely impossible to implement in a sane way.
I'm curious what your use case is for using a suspend function in spark. I have written several spark jobs with Kotlin and have never had a reason to create one. The only difference from the Scala/Java jobs is that I had to be much more explicit with certain things (encoders in higher order funcs, etc.).
All of the reasons you'd want to use a suspend function at all - e.g. mutually recursive functions, async I/O (e.g. database queries), error handling - suspend functions seem to be kotlin's answer to anything even slightly non-vanilla. Writing in Scala I can, with care, use async and iteratees in spark and be able to reason about what's going to get executed where and when, because these things are represented as values.
That's not actually true, IntelliJ support for Java has always been much better, and still is today! Just try auto-completion in Java VS Kotlin in your IDE... it's much faster in Java files, something I've been complaining about as I've been using more Kotlin than Java, recently. Check also how many refactorings are available in your Java files compared to the Kotlin ones.
Jetbrains knows its main source of income is still, by far, Java, so they don't drop that ball.
If only Kotlin would introduce a non-ridiculous way to declare map and list literals it would be a great data science language. As it is, it falls over at the first hurdle where you want to specify some data inline. One reason I've really stuck with Groovy for JVM-based data sciency stuff.
https://github.com/JetBrains/kotlin-spark-api