The scala type system is incredibly strong (and undecidable) but the implementation itself is not conducive to compilation speed. Two of the primary motivations for Dotty are 1) reducing the complexity of the language and 2) starting clean slate on a more reasonable compiler.
Im of the opinion that most groups don't want the complexity that scala brings, and we're going to see that in the community growing around spark. The databricks style guide clearly avoids a lot the complexity of the language, and the people who are coming to scala for the tech and not the language are going to follow their lead.
Dotty is unlikely to attain relevance - for me or as a future version of scala - because there's no reason to expect it to be any less of an implementation nightmare than is scalac. It is being written by the same person, applying the same engineering standards and the same checks and balances, so starting with a less unsound type system isn't going to matter.
Yeah, the problem is that the theoretical underpinnings of DOT do not represent Scala as we know it; that would be and is a different compiler (Dotty).
If Dotty works out then we will, for the first time, have a fully specified Scala on sound theoretical footing. Along for the ride: much faster compiler, better tooling, streamlined type system, and nice-to-haves like union types and implicit function types, among other useful features.
The first is that the linked post answers a lot of points, but not the ones I made. It makes Dotty sounds really exciting, which it is. It makes it sound as if everybody working on Dotty believes it to be the future of Scala, and I believe that to be true, too. What it does not do (that I could see) is give reassurance that Dotty and Scala will not drift appart and become two similar but incompatible languages. I felt it'd be rude to point this out, since there really is no way to provide such guarantees (just like there really is no way for me to back up my worries with anything concrete).
The second part is, I'm not casting doubts, but expressing worries. I'm not saying it will happen, I fear it might. I'm sorry if this offends - I really am, my goal is not to insult anyone or demean anyone's work. I have no hidden agenda, and I'm not trying to be one of these anti-Scala trolls that sometime crop up. I love Scala. Look at my github profile, most of my public work is in Scala, and some of my libraries represent a lot of work (which obviously does not mean they're actually any good). I really, really hope everything works out the way people mean it to - as I said, Dotty sounds really exciting. I've seen too many "let's rewrite this from scratch and then make it backward compatible, it'll be amazing" projects to be able to believe it blindly, though.
As a huge Scala fan, I fear that Dotty will suck the oxygen away. In particular, freezing the syntax for as long as the proposed timeframe for Dotty to be delivered would kill any language.
I would urge typesa^H^Hlightbend to bring forward the introduction of vital syntactic features from dotty (e.g. existentials as HKTs, if they really think that's the best way to express type lambdas). Even if that involves making large changes to scalac that will have to be thrown away, the Scala community cannot survive with the syntax the way it is for much longer.
IMO Scala has a lot of very well thought out features that complement each other, and this is only going to get better with the move towards Dotty.
For me it feels not as bloated as other languages which added features without thinking about how they fit to the existing ones. But I work a lot on Scala, so I'm probably biased since I know the language better than i.e. Typescript :)
There's some weird history there. For most of the project's existence, Scala 3 was known as Dotty. It was basically a research project, and it was recognized from the beginning that it was a different language than Scala, largely due to the type system semantics...the goal being a formally defined, fully sound type system (Scala had some edge cases that made it not sound, even if it was more sound than most languages).
However, the language resembled Scala, was created by the same person, had some cool capabilities that a lot of Scala users longed for, and it was a chance to "start over" with some of the more questionable design decisions that had been made previously, so there was a lot of questions and pressure to make it Scala 3.
So it was decided to no longer treat Dotty as a second language, but rather to find a path to migrate the existing Scala ecosystem to this new language. This transitioned the project from a relatively slow-paced research language into a full fledged and funded project with an urge for engineering not just tool and ecosystem compatibility, but also bytecode compatibility with Scala.
So for most of those 8 years, there was no objective but to create a better language than Scala. Then very rapidly (maybe the last two years), it went from "this is a completely new language" to "these language ideas will be incorporated into Scala 3", to "this will actually become Scala 3 and we're working on a transition path".
And from the perspective of someone who was aware of Dotty and a regular user of Scala, that felt like an extremely fast transition. Especially for a language that took years to make much smaller changes in the past (2.10->2.11->2.12 felt like an eternity).
The language is in a great state in terms of what's possible, and in terms of the underlying abstractions (partly why Dotty doesn't seem like an urgent requirement to me). It's in a much less state in terms of what those things look like in code. Shapeless is great in terms of what you can do with it, but much less great in terms of what the compiler errors look like when you make a mistake. Contrast with Ceylon where similar functionality was built in from the start.
Python 3 was very much at the front of my mind as I wrote that - I think their "plateau" has killed the language (I hope I'm wrong). People were complaining about language churn, but I wasn't one of them. With Dotty an unknown number of years away and no committed schedule, I simply don't think Scala can afford to freeze for that long.
My english is obviously not as fluent as I thought it was. None of this was meant to be implied by my comments. I have no opinion on what the best way to advance Scala might be, nor am I even qualified to hold such an opinion. I have no idea whether Odersky's continued involvement with the language is needed or even desirable. I've no concept how much work is being done on Scala - none whatsoever? more than I could possibly imagine? probably somewhere in the middle.
What I said, and mean as my opinion and not an absolute truth, is that a non-insignificant chunk of the brains that used to work on Scala now seems to be busy on Dotty, and that the resulting work, amazing though it may be, might not make it back to Scala.
I have nothing to back this opinion up. Everybody involved with Dotty seems absolutely committed to it being the future Scala, and I've no reason to believe they're not in earnest. But I can't take the switch for granted until it's happened.
* C++ language specification: 1374 pages (and still includes plenty of "unspecified behavior" / "undefined behavior")
* Java language specification: 788 pages
* Scala language specification: 191 pages
> Odersky himself gave up on scalac and he's writing a compiler and a new language from scratch.
Odersky created a new language to be able to do research without sacrificing Scala stability, which is already mainstream and should not undergo any more major revolutions. So Dotty is for research / incubating language features, while Scala stays on the stable side of things, letting companies like Twitter or LinkedIn develop without fear that 2.13 will not compile their 2.12 code.
> also poorly designed overall
This is an opinion, not a fact.
> With so many features that they interact with each other in ways that are pretty much unpredictable.
Name three. I'm coding in it for several years, and didn't notice a single one issue. I heard this complaint about Scala being too hard a few times very long time ago (when Scala was at version 2.6, and its tooling was in fact terrible), but it was always from programmers, who had also major problems coding in Java, so I never treated this seriously.
Jokes aside, even Scala proper sees itself as an experimental testing ground more than a complete idea realized. That's why dotty became Scala 3, with a huge amount of backtracking.
It pushed the state of the art across the JVM landscape, but I think it's lost its purpose with Java, Kotlin, and Clojure around.
- Tooling needs improvements. He mentioned how the Scala plugin for IntelliJ IDEA will flag working code as incorrect. It will also fail to flag code that won't compile as written. He didn't mention sbt as a pain point, perhaps because it goes without saying.
- Scala 3 (Dotty) has the potential to split the community and libraries for years, much like the Python 2/3 transition. That said, I'm still kind of excited about Dotty because I only have 10s of thousands of lines of code that potentially need porting work, not 10s of millions, and that seems worth it in exchange for stable macros, faster compilation, etc.
- The Scala community is losing people who just want "better Java" to Kotlin, and losing people who want pure FP to Haskell. I came to Scala because of Spark and I keep using it because I have grown to really like it, despite the pain points. I'm not eager to jump ship toward Kotlin, plain old Java, or Haskell. But if the community keeps shrinking, and the ecosystem shrinks with it, I will eventually be forced to jump toward something else.
In the Spark world, you can use a tiny subset of the Scala features and enjoy huge productivity gains over the other language APIs (Java & Python). Those productivity gains are wiped out as more crazy language features get used.
I don't think the super complex language features should be removed. Li's libs do some crazy stuff under the hood, but provide a clean, Python-like public interface. Most devs aren't that good and complex underlying implementations leak and yield complex public interfaces.
Lots of folks would love Scala codebases that only use 10% of the available language features and none of the complex frameworks. But, like you mentioned, it's a hard language to use responsibly.
This isn't as bad as it looks. Think of it like an additional experimental scalac that is more ready to integrate into mainline scala than Dotty is.
I'm fully on board with you...I think Scalaz as a library is a virus, and it's community is toxic, and it's desperate desire to turn Scala into Haskell ends up turning an awesome ML-inspired language into a vastly inferior religious (...ahem, pure) language.
But Cats and Shapeless aren't Scalaz. They have concepts that happen to exist in Haskell, but they aren't Haskell's concepts to own any more than C owns imperative programming. If anything, Mathematics owns the concepts and Haskell happened to borrow them to make a terrible purist concept slightly more practical. It turns out that when you need the concepts (and I have, on a couple of occasions), it makes Scala slightly more practical as well.
I'm not super jazzed about where Scala sits on the syntactic complexity vs expressive power plot. Seems uncomfortably close to C++ from what I've seen of it, although I know that Dotty/Scala 3 is supposed to clean it up quite a bit conceptually.
It also means that the main designer of the Scala language is not working on Scala but on a language that will "eventually" become the future of Scala, which means that, right now, dotty is not the future of Scala.
A lot of very smart people seem to be very excited about that, I can't help but find it ominous.
It's certainly true that today (3 March 2017) there is still a gap between DOT and the current Scala compiler (version 2.12.1) -- just as there is a gap between Haskell and the lazy lambda calculus. However, as a conceptual guide what Scala is, DOT is quite helpful. With time DOT will become even more helpful
In general, it's OK for there to be a gap between a real programming language, and its theoretical toy model. That's the gap between theory and practise. Why would you care about theory if it was as complex as practise? The ideal case (embodied in the lambda-calculus) is that the theory is much easier, but looses only a modest amount of precision.
* The Mill build tool looks a lot better than SBT, but seems like everyone is still using SBT
* Scala minor version are binary incompatible, so maintaining Scala projects is a big pain. Upgrading Spark from Scala 2.11 to Scala 2.12 was a massive undertaking for example.
* Scala has tons of language features and lets people do crazy things in the code. Hard to win technical arguments with Scala geniuses that like using complicated language features.
* Scalatest is stil used by most projects and is annoying to use, as described here: https://github.com/lihaoyi/utest#why-utest. The overuse of DSLs in Scala is really annoying. Too many DSLs is another example of something I consider to be an antipattern, but there is no Scala community consensus on the responsible use of DSLs.
I'm optimistic about Scala. There are some folks that love the language and are continuously improving the ecosystem. Scala 3 will have to sell a better story about ditching legacy tooling and giving users a better default stack if it wants to compete with modern Go/Rust/Python.
Yes, part of the reason why I am generally excited about Scala 3 is because of the work on the theoretical underpinnings of the language on the DOT calculus.
Unfortunately I don't know much more about this other than "this is a Good Thing" and has helped/will help with dealing with edge cases in the language and the compiler, better type inference etc.
But Scala 3 still is overwhelmingly compatible with Scala 2.x (which is a required because of the tons of legacy code out there). Given that Scala 3 continues to be essentially the same language as Scala 2, the overall complexity of the language has not gone down very much even though the core of the language is now more consistent.
Put another way, the emergent complexity of the (tad more uniform) building blocks of Scala 3 still needs to be tackled by programmers.
I also want to point out that Scala 3 compilation speed is supposed to be faster but generally speaking the compiler is still slow-ish.
All in all, Scala 3 is more compelling than before. I may still adopt it in the future for a project. But I'm not as starry eyed about it than some others may be...
Im of the opinion that most groups don't want the complexity that scala brings, and we're going to see that in the community growing around spark. The databricks style guide clearly avoids a lot the complexity of the language, and the people who are coming to scala for the tech and not the language are going to follow their lead.
reply