> The more experience I have, the more I accept that code is complexity and cost and that it should be avoided.

At a previous company I worked for I proposed a solution which was considered to be too "complex" by the VP R&D. So he and another "Senior" turned around and rewrote it into something more "simple."

This "simplicity" slowly turned into a constantly patched clusterfuck of a mess and it stripped off a dozen different ways of customization. The "simplicity" was produced as a consequence of both of the Senior engineer's ignorance towards to the tools (language, frameworks) we were using and they just went with their gut experience on it.

I'm somewhat different to you I guess. As things go on, I realise that simplicity is usually a hallmark of developers who don't really understand the tools they're working with and can't really foresee where the business would be going with that feature in 12 to 24 months.

It's a symptom of "Jira culture," where people develop things exclusively to satisfy their ticket and ignore the bigger picture of what they're actually trying to achieve.

This is not to discount people over-engineering things, but over-engineering is very different from complexity.

I've been thinking about this in relation to the concept of incidental complexity lately. Every year there is more and more tooling and configuration to do before actually coding anything.

I have always considered myself a full stack developer, but each field gets more specialized, and when issues arise I'm just not on top of the suddenly very complex nature of the surrounding ecosystem, even though the goal is to eventually write the same code in the same language I normally use.

I don't think most people set out do do this, most complexity comes from attempting to simplify things. A develop will see two similar bit's of code and try to stuff the commonality into a base class. In their minds it's simplified because there is less code but then when someone else picks up the maintenance it's more complex because now the logic is distributed. Then the requirements of each diverge slowly and over time each simple change is hacked into more code paths of this distributed logic. Eventually most simple changes take time because you have to be sure you're not breaking other potential code paths.

On a more macro scale, complicated architecture is another symptom of this over optimistic pattern matching that humans are susceptible to.

I think you're overstating it. This is true of much of the industry, maybe even most in certain bubbles like San Francisco, but it's far from all.

> Technology-wise, we no longer use engineering as a means to an end to solve a business problem. Instead, engineering became its own thing and most companies encourage and reward those who opt for over-engineered solutions

I think there's more truth to this, although I still think it's far from all of the industry. There's also a little bit of a good reason for it: tech companies have figured out that when you let engineers work on things they find interesting or exciting, they tend to work much harder and longer. In some sense this isn't a bad deal for those programmers either, if they're getting to do something they love instead of something they hate.

What it does lead to is a weird situation where doing something in a way that's only 80% efficient in terms of effort because of over-engineering, may still end up being economical because it leads to 200% effort from the people working on it. I think this is the root of a lot of the technology decisions being made today. Which, again, is really weird and seems questionable, but it also seems to work out pretty well for all parties most of the time.

I worked in a top tier engineering environment (not SW). The problem was the same there. They valued the complexity of the solution, not the impact it had on the business for solving the problem.

I think this is generally a problem in large companies. In a tiny company the impact is much more apparent to the bottom line. Whereas in a big company, it's hard to see how much difference your solution made in the big picture because there are plenty of other bottlenecks in the pipeline (and out of your control).

The specs and the decisions surrounding the code are usually part of the over-engineering.

> Everything was built to spec.

Even with reasonable specs, I've worked with, and later managed, engineers who had a strong drive toward over-engineering solutions. I've also done it myself, and I suspect most people reading this have done the same.

The problem is that many of us engineers became developers because we enjoy building things and we enjoy solving complex problems. It can be exciting to imagine a particularly complex version of a problem and how elegant and fun a complex solution would be.

It can be as simple as a programmer who tries to write their own CMS when the spec could be solved with a simple blogging platform. If you're an ambitious programmer, writing a CMS is fun! Installing Wordpress is blech.

I didn't truly understand the ubiquity of the problem until managing ambitious engineers who wanted to do complex things. At times it felt like 1/4 of my job was just asking "If we wanted to simplify this solution, what would that look like?" Repeat for a few rounds and eventually the team has shed a lot of unnecessary complexity without sacrificing the spec at all.

> The general trend I see as I get older is that we are valuing the first to a solution rather than a more correct solution. Only to be stuck with a solution that requires constant care and work around.

This, 1000 times over. Right now I'm trying to figure out how to turn a project around that started without me and has gone down the path of using an inadequate (but mildly popular) open source tool instead of building something that's actually designed to solve our problem, not someone else's.

Again, this is dependent on your goals. If your goal is just to make a quick buck, any well-advertised shit will do, so any engineering you can cut out is waste.

But for my goals, I strongly disagree. The most widely-used front-end language lacks an integer type and has laughable security. MongoDB, a data store which requires replication not for scaling, but to maintain uptime when instances memory leak and crash, has a significant user base. The latest Macbook Pro sacrificed performance for a touch bar with questionable urility. Massive sections of our industry are built on sand. Over-engineering isn't the problem, under-engineering is.

> avoiding regular communication with management and avoiding budget discussions will both result in poor quality software

What do these contribute to high quality software?

Simplicity is a means to an end. There's no point in a "simpler" solution if it's more expensive and harder to maintain.

> Or is it that writing some code to glue together off-the-shelf parts is cheaper and easier than designing and manufacturing a giant machine?

Exactly.

Good engineering is about meeting the client's needs, not about showing off your own virtuosity. Too many on HN seem to forget that.

Not that I want to discourage you, but my view is that anything that makes software engineering simpler just leads us to tackling more complex problems until the complexity reaches the limit that people can handle.

So in that view, you can't succeed at making software engineering always simple. Instead, you can make previously intractable problems tractable.

It's worse than lazy, it's the new way of doing engineering that is glueing together ready made components.

I've felt alienated in some ways because of this, so i'm glad you put some light on it. I don't understand devs that so casually over-engineer or choose what's flashy for what works. I guess it's a matter of the tech mattering more than the product, and i've often held an inverse opinion.

It is different in the physical world where materials are a big factor and engineering will definitely not make free use of them. There are inefficiencies here and there but they're largely reduced. In the digital world of programming accidental and intentional complexity often slips through without anybody noticing and that is for many reasons. It's a relatively new field and is still quite inefficient. I worked in many places whose codebases are a giant maze designed with no clear architecture and sometimes I suspect this complicated mess of intentional moat building engineering.

Nope, we need to roll back everything that happened in the last ~10 years. We at the very least need to stop sticking JS and the web stack everywhere. Use the right tool for the job, NOT pick up a shiny tool and try to do literally everything with it. NOT stumble upon solution and start looking for problems to it. Finally realize that software engineering is actual engineering that, like other forms of engineering, has a sizable impact on other people's lives. Unlike in other forms of engineering, the cost of a mistake might feel diminutive enough, but people do suffer trying to use modern software products. I wish I was joking.

Honestly, I hate complex solutions. Even my own. What I like, and what impresses me, is the simple solution that takes a lot of work or math to prove it works. Like, emergent behavior type shit.

Control theory, for example, runs systems with very simple runtime behavior. A few open loops with some parameters. The analysis to chose those parameters is where the engineering happens.

Microservice practices are not yet that simple, but some day I hope they can be.

This is why I feel the software industry has too many problems with security, performance, etc.

I’d like to think that engineers care how good and efficient their code is. But too often, it’s up to managers or customers how quickly software needs to be completed.

This introduces bugs, incomplete features, and (in the case of Very Important Thigs to keep you Alive) potentially cause dangerous breaks.

It sounds like Big Corp was trying to push that laziness and lack of foresight into other engineering disciplines. If so, how many of our cars, planes, and other items that potentially directly affect our lives are affected by mechanical design flaws and software errors?

Hopefully the industry gets its shit together as a whole. As it stands, if I ever work on anything that affects a life, I’m damn well blowing a whistle if I feel like something is off.

I think this is unfair. Engineers usually have great (at least qualitative though often quantitative) insight into potential, cost, and complexity. The problem tends to be in getting that feedback back into "the room where it happens", the place where budgets are set, approvals are given, disputes are resolved, and performance is evaluated. In that room, there is usually a heavier presence by people who understand organizations, political concerns, messaging, and revenues. So when the cost and risk experts (the devs) are not in the room, of course we end up with overly complex, poorly understood products that chug along for a while and then become so massive that either:

a. the project collapses under its own weight

b. the project creates its own "gravitational pull" and starts pulling in resources to support itself; sometimes this is locked-in customers who cannot migrate away, sometimes this is locked-in enterprises who can't imagine how (and sometimes why) to kill it off

It really isn’t the case in most industries. There are a couple of industries with inherent complexity and where the cost of failure justifies some kind of over-engineering (you could add particle accelerators, medical imagery devices, aircrafts, and things like that). In these cases, you throw a lot of money and very good engineers at the problem. Most industries are not willing to do that, because dit is expensive. So they produce unreliable crap that breaks all the time.

In the vast majority of cases, over engineering means more parts that can fail in more “interesting” ways, and that are more difficult to diagnose because the thing is more complex than it needs to be. “Over-engineered” is really not a compliment applied to a car or a house, for example.

Engineers are not gods. Sometimes they are stupid just like any other human. They also need to stay humble and recognise that a system needs to be as complex as needed to fulfil its purpose, but not more.

> We seem to have a knack of taking positive attributes such as clever, and flipping them!

“Clever” is a compliment. But you can have too much of a good thing, as in “too clever by half”. If you depend on something, you don’t want it to be clever, you want it to be reliable first.

I’ve found the opposite. The young engineers are generally willing to listen to reason. The older Enterprise Architects are the ones that want to keep making things more complicated, or want to keep using suboptimal solutions because we’ve been using them for years.

Now that I write it down it’s kind of curious how on one hand it’s complicating things with stuff they already know, and on the other hand it’s absolute rejection of stuff they don’t.

Maybe I’m the same?