Client- and server-side are very different in terms of network connectivity, CPU and other resources. On the backend you can get by with N+1 queries if the DB is running on the adjacent VM. On the client side, hardly.

That was my first reaction.

Keep in mind that high load applications have limited database connections. Running queries in parallel will be an anti-pattern in such applications.

I don't know if you are aware, but their DB can't handle more than 1000 requests/sec, so if you are iterating through a list of nodes and requesting data for each one you can hit that limit (not a good practice, but sometimes you have to). Additionally, once you hit that limit, the DB slows down but keeps accepting requests meaning, if you keep hitting it, even at a slower rate you make the backlog worse. Seriously, be very very careful not to go over that limit, we found out the hard way.

This sounds odd to me. Assuming 100k independent web requests, a reasonable web app ought to be 100k transactions, and the database is the bottleneck. Suggesting that the web app should rework all this into one bulk transaction is ignoring the supposed value-add of the database abstraction and re-implementing it in the web app layer.

And most attempts are probably going to do it poorly. They're going to fail at maintaining the coherence and reliability of the naive solution that leaves this work to the database.

Of course, one ought to design appropriate bulk operations in a web app where there is really one client trying to accomplish batches of things. Then, you can also push the batch semantics up the stack, potentially all the way to the UX. But that's a lot different than going through heroics to merge independent client request streams into batches while telling yourself the database is not actually the bottleneck...

So we introduced so many optimizations just because we can't persist state. I can't help but think this is being penny wise pound foolish; the problems this article is solving wouldn't have been problems in the first place if you choose a boring architecture.

Yeah, but when that happens I usually don't think, oh hey they are lacking an optimal in memory distributed database solution.

I think, well... their engineers suck. Or they don't care. Pick one.

edit: His point is vague, so there is nothing technical to respond to. I am very much interested in a good technical example - but the things mentioned so far are by all appearances relatively straight-forward and linear, hence lack of effort or bad engineering are the only reasonable assumptions left.

concurrency issues, latency issues, poor database schemas, scaling issues, there's a lot of reasons internet scale things can grind to a halt.

I emphatically disagree.

I've seen this evolve into tightly coupled microservices that could be deployed independently in theory, but required exquisite coordination to work.

If you want them to be on a single server, that's fine, but having multiple databases or schemas will help enforce separation.

And, if you need one single place for analytics, push changes to that space asynchronously.

Having said that, I've seen silly optimizations being employed that make sense when you are Twitter, and to nobody else. Slice services up to the point they still do something meaningful in terms of the solution and avoid going any further.

You would query those first surely, before loading the DB? Regardless, I've been using Redis and Memcache for years from PHP, so mute point.

> I use multiple threads in a single request flow frequently.

So how do you track those? And for how long do they live after the parent request has been processed, or do they block the parent? Gets complicated quickly.

> It's ugly and inconsistent, takes time to memorize, and leads to errors.

Rasmus will admit the same, but he will also admit he does not care (I've seen him say this during a talk). Nobody bought your product because it had beautiful, consistent code.

> This limits you to writing basic CRUD. And so many other languages offer this.

99.9% of web apps are CRUD.

Huh?

Most likely this. When you have a horizontally sharded MySQL database that spans multiple datacenters - managing replication, failover and scaling out is not something you would expect most developers to be able to handle.

Let alone fine-tuning various kernel parameters, in depth monitoring and other performance tunings.

EDIT: Sure most experienced developers would have a high level and abstract idea about various things that may be outta whack - but it takes some one with expertise in the field to go beyond guesses.

The reason you avoid the db query isn't about trying to keep the db from being overloaded, it's that you can handle the request far away from the DB without needing to wait for the latency of hitting the DB.

Which is also still a complete premature and unnecessary optimization for most people. And then someone realizes their use case requires revocation so they add a DB check anyways.

Rather than applications you've worked on (many of us spend years on end working on a narrow range of applications), consider software you use (most of us flip between multiple applications every day spanning the gamut of uses). Ignore, for a moment, whatever you know about their implementation, and focus on the user-facing functionality.

Do even half of them logically communication between application nodes?

(Consider that there is still a rich class of software that is fully capable of running locally on the "client"'s computer!)

So I have a little consultancy gig for a few decades now where I spend a few days a month optimising bad software for performance (it is what I like; I don’t do anything else but ‘make shit faster’). I can tell you that the the past 10 years 99% of optimisations I did are fixing MySQL queries and indexes that table scan. I had projects that literally have table scanning queries over 50% of the queries ran. The result, as you know but apparently is not very common knowledge, is that these sites and apps run to a grinding halt (after incurring bizarre bills on aws rds; I moved many app from $100k/month bills to $10/month) when even a little traffic comes in.

Or; table scans should be rare but are not.

Edit; removed ‘time’ as that was not a good way of expressing this

Bane of my life. A few gigs ago, they had a global Redis cache, a Redis cache per server, an in-app cache, and then MySQL. Needless to say, there were many, MANY bugs that came down to cache coherency and race conditions between them.

[edit: it was a global Redis, not clustered.]

No, its not. I was responsible for implementing, deploying and managing the infrastructure at lever.co when we were a tiny fledgeling startup. The entire application is built on top of JSON OT. I took some measurements at one point when we had ~thousands of active browser sessions of our app. At the time we were seeing about 1-2 OT merges (transform + re-apply) per day. All the other non-concurrent operations can get sent straight to the database. I don't know what the numbers are now, but I'd be shocked if OT ever becomes the bottleneck.

For text based OT you'll see more concurrent edits. But for text based OT, I have a little C library that can comfortably do 20 million simple text OT transforms/second on a single core of my old 2012 macbook air. Good luck making that a bottleneck.

For the love of god, how could they not have thought of using a database?

You'd guess wrong. The example above is not the only query our server runs. It's an example of some of the queries that can be run. We have a VERY complex relationship graph, far more than what you'll typically find. This is finance, after all.

I used the user example for something relatable without getting into the weeds of the domain.

We are particularly read heavy and write light. The issue is quiet literally that we have too many applications doing too many reads. We are literally running into problems where our tempDb can't keep up with the requests because there are too many of them doing too complex of work.

You are assuming we can just partition a table here or there and everything will just work swimmingly, that's simply not the case. Our tables do not so easily partition. (perhaps our users table would, but again, that was for illustrative purposes and by no means the most complex example).

Do you think that such a simple solution hasn't been explored by a team of 50 DBAs? Or that this sort of obvious problem wouldn't have been immediately fixed?

Agreed, but it means it's not really a viable option for digging yourself out of that hole if you're already in it. Most of the time if you're desperately trying to split up your database it's because you're already hitting performance issues.

> Adding client query logging is cheaper and can be distributed.

Right, but that only works if you've got a good handle on what all your clients are. If you've got a random critical script that you don't know about, client logging isn't going to catch that one's queries.

> But then, that's just me having actually done that a few times. You're welcome to complain about how it's actually unsolvable and will never get better, of course.

I've done it a few times too, it's always been a shitshow. Query logging is a useful tool to have in some cases but it's often not an option, and even when it is not a quick or easy fix. You're far better off not getting into that situation in the first place, by enforcing proper datastore ownership and scalable data models from the start, or at least from well before you start hitting the performance limits of your datastores.