No doubt about it. Pumping megawatts into any box will bring up cooling issues. I read a quote from the director of operations or some such at Cray Research back in the mid-1990s: "Basically, I run a refrigerator company."
Surely also in the cooling systems. I remember on my visits to the Edinburgh supercomputing center, the quip was that Cray were a refrigeration company first. Like Tesla is a battery manufacturer.
Well if the energy is cheap enough then that certainly sounds attractive for high powered chillers. Maybe they can move their more energy hungry GPU gear there as well.
I cannot help but chuckle at how what is old is new again here. In the 70's and early 80's all of IBM's mainframes supported liquid cooling. Basically when a computer "uses" X kW of power it really means that it generates that many kW of heat while it is operating. Removing heat at scale has been a thing for a long time.
And what was alluded to in the video is the thermal mass of 'air' kind of sucks. So the old design of chilling air down to 67 degrees and filling a room with that air so that it can circulate around electronics putting out prodigious amounts of heat, and then collected and re-cooled, is not nearly as efficient as one would like.
Cooling water, piping it to the heat exchanger in the back door of the rack and then (unlike the video's idea) sucking air through it first, and then pushing the cooled air over the electronics to 're-heat' it, works better. Then you don't really care what temperature the air in the data center itself is as long as the heat exchanger can remove 'x' watts of heat from it before it gets blown over the computers. Suck air in from the floor (the coolest air) and blow it out the top (where it continues on to the ceiling).
Still, that only doubles the power capacity of the racks (maybe 2.5x if the heat exchanger is filled with actively chilled water)
Prior to heat exchanger doors people would have "cold" aisles and "hot" aisles. The cold air from the CRAC units would come up from the floor behind the servers, get sucked through them and exhausted forward into the "hot" aisle. There is a whole little mini industry of "cold air containment" which has stuff to build doors/covers for the cold aisle so that all of that air is sucked through servers.
> As a result the most efficient way to cool a server is to put a water filled heat exchanger in front of it and let the computer's fans draw air through that into the chassis.
What standard are you using for "efficient"? Thermo-electric coolers are neat, but the last thing they are whatsoever is efficient from a power-usage stand point.
If you're doing chilled water cooling, the only real option to consider is a mechanical heat pump (e.g. the ones in a refrigerator), but even that is silly. Hardware doesn't mind being somewhat warm, and using ambient air for cooling is far more efficient then either.
There's a reason thermoelectric refrigerators aren't really a thing. Peltiers are horrible. Mechanical heat pumps easily achieve twice the efficiency or more.
The heat can be tackled in part by pumping water through holes in the CPU. I believe it was IBM that came up with this. Can't tell if it's feasible or not.
Seymour Cray sold Freon cooling systems with an integrated supercomputer. Famous for it. Any bio of Cray will speak of his pioneering work in refrigeration. “Remove heat” was practically design rule #1. Hilarious stories are told of Cray machines where the processor crashed but not the refrigeration, and a block of supercomputing ice was found the next morning.
So it is with Tesla, who will be remembered for their batteries.
Yet I wonder what really has changed that makes it commercially viable now. Pumping fluids around isn't a technology that has significantly advanced, after all. Is it just that Moore's law has ceased to apply to heat and power usage? Is it that CPU manufacturers finally got around to testing and falsifying the long-assumed-true hypothesis that servers need to be kept in ultra cool rooms to begin with?
It's absolutely delightful to see concepts come many full circles like this. I remember the Cray-2 cooling towers as the most futuristic computer in the world even in the 90s, and it inspired me to sink my old system in mineral oil in the early 00s. I did the research and found the stuff in the Cray towers (Fluorinert) was $many/gallon, and the smallest volume they sold it was something like 55gal drums. Mineral oil was $cheap enough, also inert, and so I got a 5 gallon tub of the stuff. I built a plexiglass case, sealed it with silicone caulk, overclocked the CPU, hooked up an industrial-strength aquarium circulation pump and... rapidly went nowhere with a disgusting, oily motherboard. And a ridiculously heavy case filled with, essentially, hazmat.
It was a terrible idea back then and still seems bad today. I'll be interested to see what the next cycle brings, if it's back to spot cooling or if immersion really is the way of the future.
I would imagine that the first roll-out of this would be with the quantum computer division, as they already have experience with cryogenic cooling. This requires some major redesigns in chassis, encapsulation, boards, and manufacturing processes, so I would assume this would happen first in some esoteric HPC application before hitting POWER. Mainframes would most likely be the last ones - an unreliable fast mainframe is a worse value proposition than a reliable slow one.
If it works, it'd be a great cooler for electronics that could put back into the circuit some of the electricity that had been turned into heat. CPU and GPU coolers could become built into the structure of the ICs themselves.
In the old days Cray had a fluid-cooled supercomputer. It was immersed in a witches brew called Fluorinert, which if I understand correctly was banned under various CFC-related treaties (edit: no mention of such banning on Wikipedia, I may be thinking of other cooling liquids).
My Dad is a radio man and has been around high-powered electronics for most of his career. He explained the pros and cons of liquid cooling this way:
Pros.
1. Theoretically very efficient.
2. Amazing equipment densities.
Cons.
1. Congratulations! You are now a plumber.
Even purified water is murderously destructive. He told me stories about replacing piping in a water cooled electronics room. Every few months they would have to shut down, drain the system and replace a pipe corner. The vortices of water going around a 90 degree bend caused enough cavitation to gouge out visible shapes on the inside of the pipes. After a few months the copper pipes would crack and leak on the electronics.
Immersion in inert liquids is fine so far as it goes, but you're still going to be a plumber.
It always made me wonder why liquid cooling wasn't more of a thing for datacenters.
Water has a massive amount of thermal capacity and can quickly and in bulk be cooled to optimal temperatures. You'd probably still need fans and AC to dissipate heat of non-liquid cooled parts, but for the big energy items like CPUs and GPUs/compute engines, you could ship out huge amounts of heat fairly quickly and directly.
I guess the complexity and risk of a leak would be a problem, but for amazon sized data centers that doesn't seem like a major concern.
reply