From my very superficial reading of the article it would seem like the win is that you're getting huge amounts of density.
I think their argument is that a typical web-style datacenter is throwing up hundreds and hundreds of commodity servers to do some sort of distributed processing tasks (think of something like a hadoop cluster or a ton of app servers behind a load balancer).
With this setup their argument may be you're no longer in need of a giant network/switching infrastructure to do these types of compute tasks. Plus all the power/cost savings of only needing a couple boxes just to get those thousands of CPUs. I wonder what kind of RAM these things are going to have, as obviously that's pretty crucial.
But yeah, losing one of these things would cause more than a little headache of an outage. Seems like an interesting approach that's worth more thought though.
In addition to privacy, performance, and portability, also:
* Servers in a datacenter are much more reliable than a network of PCs (power goes off, someone decides to play Crysis, etc)
* People will find ways to scam you (pretend like they’re doing the calculation while not actually doing it)
* Economies of scale means a datacenter will probably be cheaper than what you’d have to pay the PC owners (power consumption, network, wear, etc)
* The PC owners will have to trust this arbitrary code that they’re running (can you assure that there’s never a jailbreak resulting in a huge botnet?)
I’ve pitched this idea more than a decade ago and quickly realized it has many issues.
Everybody is focusing on what happens on top of the CPU, but very few people talk about what happens beneath it.
Datacenters are fucking expensive to build and run, buying network and compute hardware is a pain in the ass, maintaining sufficient capacity to sustain growth while operating on a 3-6 month deployment lead time is atrociously expensive, and that's without all of the financial calculation around depreciation/taxes/etc.
The fact that cloud providers are able to ball all of that shit up into a flat opex rate for a server is unbelievably attractive.
Datacenters are usually operating under either I/O bottlenecks or memory bandwidth bottlenecks, it is rarely bound by compute. So I think if anything they'd just want to have more on-chip memory for the server chips, and keep them at a low TDP, which would massively safe on power and cooling.
their schematics of datacenter reminds about schematics of a big server 15 years ago. Server racks instead of CPU-boards. "The datacenter is the computer."
It is still true that most cloud-provider datacenters house racks of commodity hardware? If so, I could definitely imagine a shift to hardware that was designed to support running virtualized environments while keeping power and cooling costs down.
I'm not sure what that would look like. Mainframe-esque, perhaps?
The upside is that these folks are extremely, extremely space-and-power constrained. Cloud data centers cycle through hardware much faster than any other industry (enterprise users, engineering workstations, render farms, etc) and when they're done with it they dump it on ebay for pennies. If you look at the large-volume sellers of rackmount servers on ebay, you'll notice that there are one or two near each major data center cluster.
The best part is that they really do not care at all about any kind of cost recovery, so they dump truckloads of this stuff all at once and tank the price. If you can afford to wait you can get some utterly amazing deals on five-year-old top-of-the-line gear. If you buy enough to justify freight the (amortized) shipping is really cheap too.
If I had to guess, I'd guess that, similar to their datacenter setups in some cases, they chose the cheaper, more easily replaceable option over a more efficient but {constrained, expensive} option.
I really, really, want to go into a bunch of detail on exactly why this calculation is so incredibly naive. More as a personal thought exercise than for internet fame (since this will be buried under a buried comment).
Maybe I'll find the time...
But, like everyone else is saying, putting things in a datacenter in a resilient way for a high profile, high bandwidth, multi-national app is not the same as buying some ssd, or even running a hetzner instance.
> A good chunk of network capital is also amortized into these costs, it isn't just the 20 cents per megabit for IP settlement, it would also be all the routers those ports are connected to, the racks those sit in, the electricity those consume, and the networks beneath that from the hypervisor upwards.
Some of that is capex, and some is opex. But it's worth noting that the hyperscalers are doing something far, for more complex than almost any single-tenant physical installation would do.
In a hyperscaler cloud, I can run an instance anywhere in the AZ, and I can connect it to my VPC, and I get what appears to be, functions like, and performs as if I'm plugged in to conventional network infrastructure. This is amazing, and the clouds don't even charge for this service as such. There are papers written about how this works, presentations are given, costs are bragged about, and entire industries of software-defined networking are devoted to enabling use cases like this.
But an on-prem or normal datacenter installation doesn't need any of this. Your favorite networking equipment vendor will happily sell you a single switch with tens of Tbps of switching bankwidth and individual links that are pushing 1Tbps. It takes maybe 2RU. If you are serving 1 billion DAU, you can serve up a respectable amount of content on that one switch. You could stick all the content in one rack (good luck at that scale), or you can scale across a few racks, and you don't need any of the amazing things that hyperscalers use.
And, realistically, very very few companies will ever need to scale past that in a single datacenter -- there aren't a whole lot of places were even 100% market penetration can find 1 billion people all close enough to a single datacenter that it makes sense.
So the apples-to-oranges comparison cuts both ways. You really can get a lot of mileage out of just plugging some off-the-shelf hardware into a transit port or two.
That exact argument could be made against disaster recovery data centers. Why bother building a new data center and buying two servers when you could just have one and save tons of money?
Co-lo for a single server is not practical if you are relying on the datacenter techs for repair/replacement. When it's their own hardware and the have hundreds (or thousands) of identical systems, much easier/cheaper.
They're apparently doing hardware-software co-design for rack-scale servers, integrating some kind of state-of-the-art systems-management features of the sort that "cloud" suppliers like AWS are assumed to be relying on for their offerings.
Kinda interesting ofc, but how many enterprises actually need something like this? If there was an actual perceived need for datacenter equipment to be designed (and hardware-software co-designed) on the "rack scale", we would probably be doing it right and running mainframe hardware for everything.
Simply saying their hardware is more expensive doesn't say much. Amazon and Google aren't running their data-centers on $50 Gig-E switches they picked up on sale from Fry's either - the switches in their data centers are also high-end switches that cost more than a car.
What about Merrill Lynch-grade expensive hardware actually avoid problems on this list? Do they have super-redundant systems where I can swap out a bad motherboard without any downtime - which exists, is super expensive, but still wouldn't have prevented half the issues on the list (in-and-of-itself anyway. Having a single monolithic system that 'never' goes down vs an N-machine HA setup would have avoided heartbeat-related issues since there is no heartbeat to do.)
> to rack it in tightly controlled datacenters, and to, should they desire,
Google and Amazon owns the datacenters in their entirety for some locations, so I'm curious what those tighter controls are/do.
> rehearse network transitions on isolated redundant hardware.
Isolated redundant hardware would help in certain test cases, but that doesn't test everything - often (and especially in the case of such esoteric corner-case issues), a live load is the only way to test the system.
> They have more predictable load and slower-changing application goals than most startups.
Oh absolutely, but that's not due to money or the engineers, but simply because their 'product' is strictly defined, whereas a startup may not even know what their product is, and which may change during their lifetime.
> They also have fixed operating hours for many services, which helps them meet strict SLAs.
Ah, that is certainly an advantage for keeping systems up - scheduled downtime. Taking the system offline to perform upgrades/other maintenance is certainly something that does not get enough attention.
> We just wanted to acknowledge it in the post.
Sure. I'm taking umbrage at phrase "on the other hand, some networks really are reliable" without solid evidence to back it up. Major financial firms have very little motivation to doing the same sort of public-disclosure and post-mortem that this article is really about, and furthermore the limited feature set and user base of their 'product' only serves to hamper any exposure it could have.
Somewhere inside that major financial firms who's networks 'rarely' exhibit partition behavior, is a sysadmin who h stories will never see the light of day, and we are seeing 'how the meat is made', so to speak.
When was the last time you did a search via Google that threw an exception vs. how much do we hear about their software breaking or their equipment failing (in that pseudo open fashion of theirs - who knows how many servers they actually have now).
I think their argument is that a typical web-style datacenter is throwing up hundreds and hundreds of commodity servers to do some sort of distributed processing tasks (think of something like a hadoop cluster or a ton of app servers behind a load balancer).
With this setup their argument may be you're no longer in need of a giant network/switching infrastructure to do these types of compute tasks. Plus all the power/cost savings of only needing a couple boxes just to get those thousands of CPUs. I wonder what kind of RAM these things are going to have, as obviously that's pretty crucial.
But yeah, losing one of these things would cause more than a little headache of an outage. Seems like an interesting approach that's worth more thought though.
reply