mostly because legacy systems contain a lot of implicit features and undocumented decisions that are hard to re-discover and customers get really annoyed. Maybe less of a problem in this case since it is young and rather well defined domain.
It's normally death for a company initiative. The problem is that it takes time for the rewrite to happen. While it's tempting to believe that since you wrote it once that it will be quick to write it again, usually it takes about the same amount of time to rewrite. This leads to vastly underestimating the time scale required. At the same time, the business tends to look about 6 weeks out. For those 6 weeks they are fine, but usually a "disaster" happens and you are required to make a "quick fix" to the original. "Disaster" after "disaster" happens and normally the rewrite is abandoned. Sometimes it continues, and when you finish you discover that nobody remembered about a really crucial set of requirements that are all but impossible to implement in the new system.
However, in an open source project rewrites are often successful. Sometimes it frustrates the users in the process (see GTK/Gnome/KDE rewrites) but often there is not enough pressure to kill the rewrite so it succeeds. Whether or not you'd be better off with the original is often an open question, though.
I don't know enough about Matrix/Riot to comment about whether their rewrite was a good idea. I hope they are happy with it.
I was involved in a rewrite of a ~10 year old vb app to C#. A lot of it involved rewriting the code almost line for line in C# then adding unit tests and refactoring. It took a team of 4 about 6 months for most of the work. At the end we had a solid and now much easier to maintain app that was consistent with the rest of our application suite.
Highly successful project. Sometimes a rewrite is the right choice.
The difference for a project like Matrix however is that our product is not really any single software codebase but the protocol spec itself. In fact, it's a bad smell about the spec if it's too hard to implement it - whether that's clients or servers. So part of the point of the Dendrite (golang server) project is to dogfood the spec and fix its various shortcomings (as per the OP's complaints) and make sure it's fit for purpose. For context, the core Matrix team has already written 4 entirely disjoint Matrix client codebases (Matrix Console on Web, Riot/Web, Riot/iOS and Riot/Android) as part of developing the spec and ensuring it's fit for purpose, so it's not that unreasonable for us to also write 2 server implementations (3, if you include the failed Dendron project).
It's worth noting however that Dendrite progress has not been entirely smooth, though. Particular problems have been:
* When we lost funding last year (https://matrix.org/blog/2017/07/07/a-call-to-arms-supporting...) it inevitably had an impact on the team - and ironically the only two people who left the project ended up being the two folks who were working on (and created) Dendrite. This genuinely wasn't a reflection on Dendrite itself (which is ironically one of the most fun projects to work on in Matrix, given it's all blue skies and green fields in terms of implementing a server that learns from Synapse's many mistakes), but more a sad coincidence. Either way, it ended up with the project being under-resourced (although we're finally fixing that during May having re-hired :)
* Matrix traffic accelerated massively during 2017, and Dendrite wasn't yet ready to use in production - but meanwhile Synapse wasn't performant enough to keep the matrix.org server running smoothly, so we had to pull folks off Dendrite in order to keep Synapse scaling (ironically, ending up taking inspiration to some extent from Dendrite in improving the Synapse codebase)
* In general, it's hard to make progress on an R&D project which isn't yet in production when there's an existing project in production which is on fire.
* We deliberately scoped Dendrite to implement a server against the current Matrix spec that Synapse implements - rather than trying to improve the Matrix spec at the same time. This is good in terms of feature creep, but possibly a bit demoralising as you find yourself failing to innovate and improve the spec and instead re-implementing it, warts and all. I'm not sure what the right balance is here, and in practice we've ended up with the Matrix spec evolving significantly under Synapse, and meanwhile Dendrite then becomes a moving target as a result.
* We've almost certainly fallen foul of the Osborne Effect (https://en.wikipedia.org/wiki/Osborne_effect) too, where folks have held off on using Matrix or running Synapse because they're waiting for Dendrite to land, RSN.
But on the plus side, there's a small but active FOSS community contributing to Dendrite - much more than we've ever had on the relatively impenetrable Synapse codebase. And Dendrite itself works (about 80% of an MVP complete) and is demonstrably ~2 orders of magnitude faster than Synapse (even whilst Synapse's performance is also speeding up). And as we find algorithmic breakthroughs in performance as part of improving Synapse, these breakthroughs equally apply to Dendrite and will make it even crazier fast.
So, I think it's too early to say whether this is a textbook example of "second system syndrome", or whether instead it's a mature and sensible approach to ensuring a heterogenous ecosystem of implementations for an open standard and dogfooding one's own protocol. My hunch is that if we hadn't hit funding nightmares last year it would be a no-brainer win; but even now, it's likely to eventually work out for the best in the end. The best analogy is not the excruciating migration from Netscape to Mozilla that Joel wrote about in 2000, but the more spectacular and successful migration from Gecko to Servo happening in Mozilla right now. It's a massive massive architectural improvement and rewrite in a new language which requires a lot of work, but pays off in the end.
(This is my personal take on it, at least, as project lead for Matrix; those actually working on Synapse & Dendrite probably have different viewpoints! :)
Unlikely, sadly, having got jobs elsewhere before we secured long-term funding. However, it’s not like Dendrite is ownerless - we’ve made good progress on it since they left up until the point we had to temporarily double down on Synapse. Meanwhile, folks from the community who have been hacking on Dendrite are coming on board in the next few weeks. And the old devs are also still around in a FOSS contributor capacity; answering pings via Github and Matrix where needed.
reply