There was a thread here the other day on how DevOps has failed. And this should hopefully show everyone why DevOps is needed. Cloud infrastructure is complex and needs specialists, otherwise simple mistakes can be very costly. If I make a new lambda, you better believe I'm watching invocations.

in my experience developers are terrible at doing operations. (for good reason mind you).

Devops only really became possible because a lot of operational problems have been outsourced to cloud providers, but they still persist. Or do developers also want to build the networks which provide fault tolerance, redundancy and scalability while also building software?

Running a large, multi tenant system for different workloads takes a mind boggling amount of work and skill, not to mention deep understanding of how networks and systems operate in detail and at scale.

I was on the same boat until I've switched to a company where devs were also doing ops. With AWS, GitLab CI and Terraform, this works surprisingly well.

There are still ops persons for the hard parts, but most things can be done by the devs themselves.

Eh. Spinning up a new VM is wayyyyy easier than racking a new box. And deploying to an app-running service is generally way easier than managing a bare-metal/load-balancer/etc-interfacing deploy script.

A devops engineer building and maintaining that infrastructure is a split that makes sense to me, but generally doesn't obviate the need for systems/network admins and engineers, as you still have problems and issues with the underlying things often come up (sometimes it's outsourced to a cloud provider, though).

But this applies to everything. You also need Heroku or Kubernetes or whatever experience to maintain those systems, right?

Part of that shift is that prior to the cloud, the development environment and production environments were very similar. By that I mean if a programmer could set up his debian / redhat laptop from the command line, then it wasn't a big jump to set up a server running debian / redhat in a production environment.

But no one in their right mind runs k8s on their laptop just to support VSCode and a compiler. Worse they could not run AWS/GCP because they are proprietary, insanely large, complex and forever changing. I'm pretty sure becoming an expert engineer in those environments is a full time job. The end result is a new class of engineers have arisen that look after the production side, and the developers just throw their code over the wall once it's tested.

I'm sure the cloud providers love this, as they have created a cohort of engineers that have invested years in learning their product, now have their wages dependent on that product being used within their organisation, and a total monopoly over that product because it can never be reproduced by anyone else. But for the industry as a whole, it looks to be a backward step. Once the problem expands beyond what a single human mind can cope with, you need teams. The communication overhead of a team means they don't have a hope of being as productive as a single person.

The share amount of complexity introduced by these proprietary cloud frameworks looks to be unnecessary. Most of it smells like technical debit created by organic growth together with an insistence on backward compatibility (don't want to give those locked in customers an excuse to move). And desktop OS's are (Debian / Windows / ...) are insecure by design, at least when compared to their phone brethren. And, their phone brethren look somewhat like cloud designs now, Phones have isolated apps with private data areas. The apps communicate via channels provided by the OS, and the availability of those channels is controlled by permissions assigned by the user. In the cloud we isolated things for performance as they could run on distance machines, whereas in a personal device we did it for security. But the end result looks similar.

So maybe one day we will end up where we started, with a developer environment looking like the cloud we deploy too. I fear I'm too old to experience if / when it does happen, but it does seem like something we should aspire to.

If you ever need infrastructure/devops help, or even someone just to help with the little tasks you don't have time for, more than willing to help.

you can't just drop a sentence like that and not elaborate on it - unless you want to offend somebody.

Each resource hungry "DevOps" tool has its place and has its worth. They're just often misused. A small IT firm with <10 Hypervisors won't need openstack for example. Thats only worth its maintenance if you're working with tens or hundreds or hardware machines.

And a webapp which consists of <10 services probably doesn't need service discovery and advanced de-/commissioning of containers/vms. Administering hundreds of microservices without that is borderline impossible however.

That’s just half true. If something doesn’t work in production, guess who can get mysql production logs? Not the developers. Can you access production dbs (not that that’s the way to fix things, but I have seen that in many orgs)? Nop. And if there’s anything you can do yourself (like changing things via TF files), you’ll need to wait until the gatekeepers approve your request.

The gatekeeper are usually called platform engineers.

But in any case, I don’t want all that power (and responsibility) either if it does not come with an increased compensation.

When I read this though, I just think about how much time your developers are not actually developing because they're doing operational-side work.

I have the situation where my developers do this stuff, then things break or need debugging and they don't really know how to dig into any of this stack in any meaningful way, so the problems tend to compound. Meanwhile, they're not writing code. The cadence of development seems massively slower to me (coming from a traditional background where they're writing to a clear Ops-set target environment).

The logical outcome is to hire someone who is an expert in all this infrastructure stuff to help manage it - ostensibly, a "DevOps" person, but really, a classic Operations person, just for cloud.

I think that's what sold me on DevOps.

> You built it, you run it!

This adds too much responsibilities for devs but also makes it hard to find good enough developers who can also manage deployments and infrastructure. I have never seen happy and competent Dev+DevOps person. There is just too much cognitive load for a same person to do these two things right at the same time. The Hello World of Kubernetes deployment is easy on Cloud but anytime you need to do something a bit more complex, learning curve increases tremendously.

What seems to work is that each team having one or more dedicated DevOps person. Or I have seen a dedicated DevOps team in large orgs managing infrastructure for many other teams.

Bare metal infrastructure requires a lot more management at any given scale. I mean, you can run stuff that lets you do part of the management the same as cloud resources, but you also have to then manage that software and manage the hardware.

Has anyone done this in practice at significant scale? Every shop I know of with a big cloud deployment has armies of devops people managing the deployment, and further reserve armies on pager-duty standby in case something blows up. They're certainly doing different things than if you had an in-house datacenter (less hardware maintenance, more cloud orchestration), but I'm not convinced the sysadmin/devops headcount has actually gone down.

This has been my experience working with companies that use cloud services as well.

Another big waste of time is on application optimization especially around database usage. Cloud services tend to provide very low IOPS storage (and then charge exorbitant amounts for semi-decent performance) which forces spending a lot of wasted time on optimization which would never be an issue on dedicated hardware.

It's funny. I work for a company who has a fully Dockerized pipeline, and a relatively small operations/devops team. We're looking to expand the devops team while staying rather steady for the product development team.

Why?

Maintaining services, even ones installed over the top of PaaS and IaaS requires manpower and time. And the larger we grow, the more apps we produce, the more external services we consume... the more expertise and time is needed in operations. What could at one time be handled by AWS and Google on our behalf doesn't scale as we grow.

Someone needs to understand the AWS API and all of its lovely discrepancies. Someone needs to own CloudFormation template standards and IAM roles and secrets management. Someone needs to set up and maintain security packages and vulnerabilities and prop up the pipelines. Someone needs to own monitoring and the logging pipeline and be on call 24x7 to respond to issues that aren't, can't be automated.

Certainly not what management expects or even wants (especially since it did, at one time, just work), but production is still not free, even in this day and age.

> 95% of the budgets of most software companies goes towards writing code, not dev ops.

As a company grows, expect this number to be slashed dramatically. Not because of operations, but for sales and marketing and administration and legal and HR and CS and infosec and...

Which unfortunately is a problem in itself. A lot of core knowledge is lost and many people running infrastructure don't know how to read actual logs and debug outside of what the GUI shows. Just like you mention it's not even VPS these days it's a Dockerfile pushed to some cloud.

Copy/paste a Dockerfile, edit some yaml for the CI and claim you know DevOps. It's a shame really. I don't mind a nice minimal GUI when it makes sense, but it's important to understand the pieces below it.

Or the total opposite. People that not drink the cloud-narrative and the operational simplicity make devops no-brainer.

ie: You are truly made for the massive overenginering of cloud (because you ACTUALLY need that) or you keep things simple enough to fit in a single head.

What is problematic is when you try to do be first, or present to be second.

What decade are you from? No startups are hiring sysadmins to do any kind of work anymore. They're hiring "dev-ops" people, which seems to mean "Amateur $popularLanguage developer that deployed on AWS this one time."

That's the whole problem with the dev-ops ecosystem. None of these dev-ops people seem to have any ops experience.

Exactly. Which is why people go to Rackspace: not having to hire their own in-house DevOps guy (which doesn't scale as your servers grow) to take care of their servers.