Enterprise costs are negotiable. I can easily imagine Google paying VMware several billion per year for a product they want. I can’t imagine Google using ESX — they want a product they fully control.
VMs and security sandboxing techniques are used to run external software by Google’s AppEngine (GAE) [38] and Google Compute Engine (GCE). We run each hosted VM in a KVM process [54] that runs as a Borg task.
There might be a few machines here and there running actual VMs on bare metal, but those are going to be very special cases. Using Infrastore, it's easy to find which teams are running which containers whose binaries were built before the new compiler defaults.
If I recall correctly, it took Google a while to actually offer the boring stuff. For a while, you could get a Google Compute Engine but you couldn't just get a dang VM image, because Google knows better than you and you should do things their way. They've fixed it now, but lost a lot of potential market share for that conceit.
> Every Google application, even search, runs on Kubernetes-managed containers.
I don't think this is true at all. They run on borg which is kubernetes spiritual predecessor. But a distinctly different, and much more mature, piece of software.
Google tried with GAE, which is much closer to the Borg model (“I don't care about VMs, just run this code somewhere for me, and make it scale, make it automatically have access to a database”), but the world wasn't ready for it.
The world wants VMs and virtual networks and firewalls and clickable web interfaces and to run big legacy enterprise apps that MUST NEVER DIE and to fuck around with Terraform and to SSH onto boxes. This sucks, though, and Google SREs know better than to use such terrible abstractions unless forced to. The internal software development ecosystem is eons ahead of anything you can get with any cloud provider, but that sort of alien technology doesn't sell.
Kubernetes/GKE is a good compromise between something that Google sells but Google also wants to use. That, and some of the GCP versions of internal tools (Spanner, BigTable, BigQuery, ...).
Google doesn't use VMs internally and they probably don't have a scalable SAN (EBS competitor) either, so they don't have as much of an advantage as you might think. Google bet big on a "legacy-free" stack (GFS, MapReduce, BigTable, etc.) and that has worked out great for their internal services but external customers just aren't willing to adopt it (see App Engine).
That's an anti-competitive position for an IaaS to take. "We already compete with your kind of software here, so we won't sell you the hardware"?
I work on some analytics software that includes GPU ML algorithms & non-ML GPU algorithms, neither of which Google makes. Our peers do a lot of visual computing in AWS. Really weird thing to hear from a Google rep.
Because cloud providers have started to build their own hardware to eke out performance advantages versus other cloud providers. It's not a far stretch to conclude that Google will prioritize its kernel's development for its proprietary hardware and not have to release any of the secret sauce to the public.
The idea with Google is that you don't need to worry about which instances you run, at which price, and in which region.
For example, with Google you don't need to get a specialized instance type to use it's monster-fast Local SSD - just use the same instance types. This alone should simplify use of Preemptible VMs.
reply