Except it never really worked that way, which was the entire problem with waterfall. Waterfall works best in a static environment with guaranteed funding because it doesn't react well to shifting priorities. In reality, requirements change and budgets get cut, which sets off a chain reaction of change management activity in a waterfall model. Agile is so much better -- you reassess requirement priorities every few weeks and that dictates what you work on over the next few weeks. In very few situations under which software is developed today is it even possible to state upfront what you need with any certainty. Waterfall assumes that requirements gathering and development take place serially, whereas agile recognizes the reality that they are more or less concurrent.

Waterfall still has its place - if I were developing a nuclear reactor or an airplane, waterfall is still the way to go. But for projects where 100% requirements completeness is unnecessary (and let's be real, nobody runs a full regression suite after every sprint), Agile will get you a better product faster and cheaper. Everything you said is true -- agile is a fertile ground for manipulators, but so was waterfall.

The manipulation is a byproduct of any reasonably large organizational structure, and if you want to stop it, it's got to be a cultural thing. Bad actors will exist under any development methodology, it's up to the leader of the engineering organization to stand up and refuse to allow his/her people to be abused.

This is rather my point. In "waterfall" shops I've worked at, the way it has never worked is that the application is completely designed up front, then implemented. The reality is that there has always been iteration involved, and there has always been refining and modification of design, and etc.

The only thing (aside from "ceremonies") I see agile bringing that didn't exist in my experiences with "waterfall" is the notion of bringing the customer in as a constant part of the development process -- which is a welcome addition.

The rest of Agile that I see can be thought of as refinements to waterfall, not as something entirely different.

Is anyone really doing Waterfall now? Maybe earlier some industries used a Waterfall like process. But actual descriptions of Waterfall seem to go back 50 years to Royce's paper, Managing the Development of Large Software Systems. That model was explicitly described as bad. I don't fully agree with that paper, but his suggested improvements are more iterative. Personally it seems like now Waterfall is mostly used as a strawman.

> but not usually for software where you learn more from users about what should be done and tweaked as you go along

I agree with that. I wonder if Agile is highly influenced by it's leaders being closely involved with consulting.

Well - yes, it was used, and still is: it's the default if you're not careful or if you don't think very hard or realistically, or are very naive. The confusion is that is was only given a name to disparage it: W. Winston Royce observed that the way most people managed software projects was completely unrealistic and didn't take into account changing requirements. He called it "waterfall" as a way to underscore how inflexible the default "write down all the requirements, then write down how long they're going to take, then do them in that amount of time" approach was.

Unfortunately, most people who adopt what they refer to as "agile" processes are still stuck in that same mindset; they think that, by having daily standups, putting in JIRA tickets, and referring to every two weeks as a "sprint", they'll somehow meet the project manager's pipe dream of 100% predictable software development schedules.

That's the primary failure mode of a straw man software development methodology called waterfall that Scrum practitioners like to use as a bogeyman to appeal to for rhetorical purposes during the inevitable arguments about niggly little details of how to implement Scrum during sprint retrospectives.

I've never actually done real waterfall, but I did study it way back when I was in college, and I have worked in government contracts, which tend to handle the work in a way that is similar to the waterfall I read about in school. One thing I distinctly remember is that it was designed to be a flexible and iterative methodology that would have been difficult to cram into a ticketing system in the first place, let alone cram into Jira while simultaneously carving the work into tiny pieces, up front, all at once.

FWIW, the only place I've ever been asked to do something like that is in ostensible Scrum shops. In fact, the only time I've ever seen anything that looks like the Waterfall of scrum lore is in shops that are trying to do Scrum. I am beginning to suspect that the Waterfall of Scrum lore isn't actually a thing that happens outside of Scrum at all. It's actually what naturally tends to emerge when you try to apply Scrum methodology in a situation where something along the lines of the real, textbook waterfall model would have been more appropriate.

As a concrete example, I currently work at a team that ostensibly uses Scrum, but where QA is a separate department with its own practices that the dev team does not control. They do still want some ability to anticipate work that's coming their way, though, so they're monitoring the scrum board for that purpose. This is the moment where it gets messy, because we're then asked to document things ahead of time, and it's a minor crisis if we keep it flexible during the sprint because any changes to the set of tickets becomes an inevitable hassle as they complain that we've screwed up work product that they generate based on those tickets. It's absolutely something straight out of waterfall horror stories from Scrum lore. But it's only happening because we're doggedly insisting on something that's at least cosmetically similar to scrum. If we weren't doing that, we'd be freer to choose modes of interaction and business artifacts that are better suited to the reality we occupy.

Why do people say that? The book "Rapid Development", written in the 1990s, comes straight out of the RAD movement, based out of Boehm spiral mode. "Rapid Development" emphasized that iterative development is an industry best practice, and that waterfall was only rarely appropriate.

It's like people want to believe in a mythology that we are the offspring of recent, visionary revolutionaries, instead of the reality that there's a long history behind what we do.

Incidentally, it turns out I'm a late comer to the "stop saying agile" point–an author of the Agile Manifesto was saying that 5 years ago https://pragdave.me/blog/2014/03/04/time-to-kill-agile.html

Yes it was.

> Think about it, how could you possibly code an entire app and only then debug it?

Which is why programmers hate waterfall so much, but you're absolutely wrong to think this isn't exactly what was being attempted. Time and time again, management in an attempt to cut the cost of programming time, thought the way to do it was to first build specification for everything and try and prototype the entire app up front, storyboard every screen, build out mountains of specs because naturally they think that's how you build things, with detailed blueprints so all the decisions have already been made. They couldn't be more wrong, but it's certainly the most natural way to think if you aren't a programmer and don't know better.

> So, the biggest innovation in Agile was the latter's orientation towards developer testing concurrently with coding. Which, of course, has had many other ramifications.

Agile in general had nothing to do with testing, the big change in agile was removing the big design up front, the specs and meetings and months spent planning something before being developed. Some agile methods like extreme programming certainly had testing as a big part of their process, but what differentiated agile from waterfall was introducing iterative programming where work was done in short week or two cycles and then delivered whereas waterfall wastes enormous time trying to nail down details that simply ended up being wrong come programming time.

tldr; agile is not about testing, it's about iterative development in short cycles with little planning and always has been and that's what made it different from waterfall; no "Big Design Up Front".

One persons anecdotal experience about agilefall doesn't mean it doesn't work. Don't fall for the trap of using analogies and anecdotal experiences as evidence. To top it off this guy didn't even measure the outcome of agilefall, all he did was declare that it violates some made up philosophical principles and demand change.

I've seen a project manager literally catch himself from doing something waterfall like it was the plague. Do people not realize that waterfall is the only technique used to build airplanes and bridges? I would not get on a plane built/being built using agile.

Note that I'm not criticizing agile. Agile works, waterfall works and no logic or evidence provided by that article says that combining what works for both is not good.

That is itself pretty naive. There is a reason why waterfall used to be popular, but it no longer is. Is it because people's ego's have ebbed? Doubtful. It's more likely that something else happened in the interim.

The obvious change is the proliferation of instant communications. If communicating is hard and expensive, it often makes more sense to change things as little as possible. If it's easy, then change becomes less expensive.

The internet gave us instant communication (e-mail, video conferencing, github, etc). Tools like these made agile more attractive. Humanity didn't just become enlightened.

It's mostly devolved to ritualistic bullshit performed dogmatically by corporate idiots. Every shitty third rate software sweat shop does agile/scrum these days. You need to look pretty hard to find companies that are not very eagerly proclaiming they do agile.It means absolutely nothing until you qualify what that actually means for you.

Those claiming they are doing it "right" or by the book are the worst because that means they read some stuff without grasping the meaning of what they read. Basically, the agile manifesto signers have come out arguing against that. That was the whole point of that manifesto. They basically provided us with tools, practices, and ways of working that we're supposed to adapt to our context as needed. They don't say: do this dogmatically and follow my instructions by the letter. They actually say don't do that but if you are getting started you might as well do this and then adapt as needed.

That's sort of the same thing that went wrong with waterfall. The article points out a few historical twists to historical context of that. I read the paper by Royce long time ago and found it to be quite modern and insightful. It doesn't mention waterfalls at all (as the article points out). It also points out that, waterfall as we know and love it, basically was institutionalized bureaucratic government organizations using it to sugar coat heavy processes that they came up with that got subsequently adopted as "the way to do things" by equally bureaucratic large software houses they worked with. It got a bad reputation for said corporations making obscene amounts of money with obscenely bad software delivered late and never on budget.

This pattern is something that is a constant and was referred to as the software crisis in the 1960s already and lead to Margaret Hamilton along with other famous pioneers in computing coining the notion of software engineering in the 1970s. She was of course working on the lunar program at Nasa (speaking of bureaucratic organizations). The software crisis is alive and kicking. Many scrum/agile organizations continue to peddle atrociously bad software developed. That doesn't mean that nothing has changed but you should beware of anyone claiming to know the one and only way to do stuff.

There's a lot of common sense that inspired agile and waterfall. I'd argue the modern take on this is to replace manual processes with automation to facilitate rapid flow of changes from inception to production. Another trend is to decouple decision points to facilitate asynchronous processes and communication. For example, you don't bundle features into releases. Instead you ship pull requests straight to CI and CD pipelines. That means that there are no central planning bottlenecks for delivering new features and they can be developed on their own schedules. This happens continually. Most big brand websites you use, update many times per day. Iterations and sprints are kind of obsolete as a way of orchestrating release schedules.

That practice is now creeping into the enterprise. If you are doing scrum and being told not to do this, that's something that you need to challenge.

No not really. It's called waterfall and we had it for decades.

And it was largely abandoned in favour of agile development because despite architects spending months designing something there was always something that was left out. Or the scope changed during those months. Or millions of other things that change whilst your hidden away from your customers/end users instead of delivering them new value every week.

For me personally the truth is somewhere between agile and waterfall. Some upfront design but not the slow code you refer to.

> Nope, this one’s in the bin, I’m afraid. It used to mean “not waterfall” and now means “waterfall with a status meeting every day and an internal demo every two weeks”. We have to find a new way to discuss the idea that maybe we focus on the working software and not on the organisational bureaucracy, and that way does not involve the word…

I've noticed that demos tend to lead to "demo driven development" model where software devs write code to impress other software devs, without any real connection to customers, and with the benefit of a decade of time most of those got dustbinned pretty quickly.

Because it works for them. Right tool for the right job maybe.

I see companies trying to switch to Agile without even considering that they're applying it to the wrong projects, like big infrastructure ones. It's a buzz word and they have to go for it to look good. That costs.

That is to be expected, because "waterfall" or pipeline is the general structure of work. At the very least you have design, production and verification steps.

Agile is not about removal of waterfall, but rather applying 1. waterfalls to small chunks of work 2. multiple "tiers" of waterfall. The mythical waterfall that many pseudo-agile proponents argue against is simply one giant waterfall applied to the whole project. If you start building image manipulation software you generally do not pivot to building a presentation software without redefining or scrapping the "master" waterfall. Stages of the whole project generally do have dependencies on other stages, there is no way around that.

As long as you have at least some visibility into the project you are going to have stages and checkpoints and in the end a "waterfall" will surface somewhere. Waterfall is at the core a way to represent staged/checkpointed work.