I like how he tries to supplant agile with something that doesn't even attempt to solve the same problems that agile (like waterfall before it) attempts to solve.
Both agile and waterfall methods attempt to give you a pattern by which you can predict when software will be delivered. That's the main business value. The suits don't care how the devs operate, provided they can get things on time.
Waterfall tried to do this by trying to understand the problem as fully as possible, to liken it as much to previously solved problems, and to involve gurus to say "that kind of problem will take X amount of time to resolve", and thereby create milestones and delivery dates.
Agile instead said "look, we don't know enough at the start of a project to do that. Let's instead keep track of everything we want to do. Let's try and estimate each set of tasks (stories), individually. And then let's rank them in priority. We can measure how good our estimates are, we can modify them, we can generate more data and determine what we'll have at a milestone, and then can either push back the due date, or at least recognize that we won't be able to ship the entire feature set at that time.
Continuous delivery can be done with either one of those (it almost never is in waterfall, but it -could- be). But by itself it offers a business nothing for planning purposes. All it does is allow an immediacy, a "as soon as it's done it's out in front of the users". This may or may not be a good thing to the business, but it doesn't solve the basic issue that business people want to know when they can expect a given set of features to be live.
> Waterfall seems like a discredited old system, and I don't love it either, but it has a huge advantage for developers. It requires people who will insist on and enforce deadlines to state upfront, in considerable detail, exactly what they want.
Except it never really worked that way, which was the entire problem with waterfall. Waterfall works best in a static environment with guaranteed funding because it doesn't react well to shifting priorities. In reality, requirements change and budgets get cut, which sets off a chain reaction of change management activity in a waterfall model. Agile is so much better -- you reassess requirement priorities every few weeks and that dictates what you work on over the next few weeks. In very few situations under which software is developed today is it even possible to state upfront what you need with any certainty. Waterfall assumes that requirements gathering and development take place serially, whereas agile recognizes the reality that they are more or less concurrent.
Waterfall still has its place - if I were developing a nuclear reactor or an airplane, waterfall is still the way to go. But for projects where 100% requirements completeness is unnecessary (and let's be real, nobody runs a full regression suite after every sprint), Agile will get you a better product faster and cheaper. Everything you said is true -- agile is a fertile ground for manipulators, but so was waterfall.
The manipulation is a byproduct of any reasonably large organizational structure, and if you want to stop it, it's got to be a cultural thing. Bad actors will exist under any development methodology, it's up to the leader of the engineering organization to stand up and refuse to allow his/her people to be abused.
Similar to you. There were several hints that the author didn't really understand software. The problem with waterfall wasn't that you had to do tasks in order, it was that you needed to know mostly everything in advance to get rid of issues at design time so they didn't bite you during coding.
Also, the rumblings of agile don't really point to anything to do with agile fundamentally, just the fact that people don't quite understand it or that management doesn't wholeheartedly embrace the mess.
"Agile meant that the team didn't really know what they were producing" - that is a Product Owner issue 100%.
So yeah, nothing's perfect but it's probably the least worse of the options we have!
The trouble with waterfall is that it tries to predict beyond the scope of a sprint, which just isn't valid.
This assertion is nonsense. There's nothing magical about a couple of weeks such that it forms a boundary outwith lie impossible predictions. The validity of predictions depend entirely on the understanding of the problem domain and the complexity of the solution space.
The single biggest benefit from agile in theory, IMO, is controlling risk by getting the customer in front of the software sooner, so it can be iterated based on feedback. The primary risk being controlled is building the wrong thing. But you wouldn't develop e.g. an autonomous driving subsystem for a car that way.
Agile (scrum, specifically) in practice is too often used simply to chop large tasks into bite-size stories to be fed on a conveyer belt to a team of more or less replaceable programming cogs; the sprint scope keeps blinkers on everybody so they don't look too far in the future, they just keep munching through stories.
And when agile is used in this way, not only can it be demoralizing, but also extremely inefficient: a focus on user stories typically encourages building small features that involve narrow vertical slices through the stack of an application. That's hard to parallelize effectively - related stories will affect the same bits of code and cause conflicts. If you can bundle a bunch of related features together based on how they are likely to be implemented, you can slice them up horizontally, and implement the different layers separately, using things like APIs and data models at the boundaries. This paralellizes quite well at the team level.
It's still not great for software design. It's still a very blinkered approach; you're not going to design an application-specific framework that makes implementing features easy. It doesn't allow any space for experienced developers who have foresight, and relies on refactoring to create reusable domain-specific abstractions. But refactoring isn't a user story, and a team munching on stories isn't in a good position to think holistically about a problem.
Nobody said anything about waterfall. A good agile process can and should have all of the things the parent commenter listed. If you eschew those things by using the "well we're agile, that means we don't plan ahead!" excuse, that's a sign of incompetence.
Waterfall was never that bad anyway. Agile doesn't magically make people more productive. It only solves the problem of overrunning deadlines by pretending they're not important. If anything it puts needless pressure on devs who ideally should be free to think of solutions at depth. Sprints only make sense in that a team should be hustling in the early days. As the software matures, I don't think it's healthy to strive for squeezing your devs for every last drop of productivity.
I wouldn't want to go back to waterfall, but my experiences with "on demand on schedule" (a very astute phrase, that) have caused me to realize it did have advantages.
Good software usually doesn't result from massive upfront requirements analysis, design, and scheduling, and the iterate and repeat approach has huge benefits (provided you've done some planning about where you're going).
However, waterfall did have one big advantage, in that it forces the business side to be clear about what they want. This is an impossible and unfair task, and of course the business side will need to change, amend, modify, limit and expand scope, and so forth. The advantage is that it's unfair to everyone. I know this will sound terribly cynical (and every time I say it, someone asks, not unreasonably, "why do you even work where you work!?"), but waterfall is a contract that puts everyone in breach. The all too common corruption of agile turns into a situation where only the developer is failing to make deadlines or deliver on promises, with a daily standup reminder of this fact.
I'm only partly cynical here. Software is a big, confusing, chaotic mess, and that's ok, it's the nature of the craft. However, people, especially on the outside, get frustrated. I think that the business side perceives the impossibility of truly estimating and planning software better when they try (and fail) to do waterfall. It puts them in the position of coming to the developer or technical team and saying "sorry, I know we sent you those specs, but it isn't working, we need to change our minds." With (corrupted) "agile", they are cleared of this responsibility, and they can just show up every day (or every few days), think about nothing but a very specific task, ask the dev for a time estimate, and then wonder why it hasn't gotten done (yet).
When agile works well, it's when the entire team is ok with measuring progress by the value of working software, released frequently. Those have been rare occasions for me, but that has been the best experience, the happiest one, for me. Corrupted agile has been the very worst. Waterfall is somewhere in the middle.
Sort of, the point of Agile is if it is bad, then you know much sooner than with waterfall.
But if everything is so bad no matter what you do, then it's not really anything to do with how to do software development, it's a whole different problem.
That's not waterfall as used in the conventional sense. A key element of waterfall was/is: "Thus the waterfall model maintains that one should move to a phase only when its preceding phase is reviewed and verified."[1]
>The details vary, to be sure, but the fundamentals remain the same.
I disagree. Agile is distincly different at a fundamental level from waterfall. If you walk into a modern shop doing mostly agile with continuous delivery, you'll find that there is almost no process at all that maps directly to waterfall. Literally, not a single one: not design, not coding, not testing, not building, not deploying. They're all fundamentally very different. Coding is probably the one that comes closest and if the site uses TDD or pair programming, it will be clear how drastically different even that is.
[1] https://en.wikipedia.org/wiki/Waterfall_model
> Most attacks from the agile community probably are coming from a place of never having actually seen or practiced the real thing
This is rather my point. In "waterfall" shops I've worked at, the way it has never worked is that the application is completely designed up front, then implemented. The reality is that there has always been iteration involved, and there has always been refining and modification of design, and etc.
The only thing (aside from "ceremonies") I see agile bringing that didn't exist in my experiences with "waterfall" is the notion of bringing the customer in as a constant part of the development process -- which is a welcome addition.
The rest of Agile that I see can be thought of as refinements to waterfall, not as something entirely different.
It may be that Agile developed in reaction to Waterfall, but that just says to me that the creators didn't look at all the other methodologies out there.
> It doesn't look like the waterfall model was ever used
Well - yes, it was used, and still is: it's the default if you're not careful or if you don't think very hard or realistically, or are very naive. The confusion is that is was only given a name to disparage it: W. Winston Royce observed that the way most people managed software projects was completely unrealistic and didn't take into account changing requirements. He called it "waterfall" as a way to underscore how inflexible the default "write down all the requirements, then write down how long they're going to take, then do them in that amount of time" approach was.
Unfortunately, most people who adopt what they refer to as "agile" processes are still stuck in that same mindset; they think that, by having daily standups, putting in JIRA tickets, and referring to every two weeks as a "sprint", they'll somehow meet the project manager's pipe dream of 100% predictable software development schedules.
> Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control...
> These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them.
A number of attacks on Agile I've read have made precisely the opposite claim: that Agile is "too generic" and not focused enough on the specifics of software development.
My experience is that an Agile approach can be effective at all the things you mention, and you don't particularly need to do anything different on the Agile side of things.
What is true is that Agile relies on being able to iterate quickly with the end customer (in the sense of "whoever is giving you this requirement"), and only adds value when the requirements change quickly and/or the customer isn't actually quite sure what they want.
In my experience this is a lot more common than you might think even in non-software, and creating the conditions for success can be easy once you try. E.g. if you ask your regulator whether you can take a more collaborative approach this time, have a frequent back-and-forth where you show them the current state of things and see whether they're getting what they want, or have any changes to what they want from you - you might be surprised how willing they are to try, and how happy both of you are with the resulting process.
But yeah, it is a substantial change, and I can believe there are domains where the costs outweigh the benefits. If you're working in a domain that's very well-understood, where the customer really can figure out the requirements up-front without even a prototype to go on and those requirements really are going to stay the same for however long it takes to implement them, then fair enough, maybe waterfall will work better. I do think very little of business actually works that way though; in my experience even "successful" waterfall projects that deliver all the requirements that were originally identified often turn out as something most stakeholders aren't very happy with. "It's just what we asked for, but not what we want" is very real.
That's your mistake; it's not a matter of competency and experience. You are not a solo venture, you are a member of a team, and part of that team needs to be a customer representative who can tell you, in real time, what the customer wants.
There is no room for a hero engineer in modern software development. That's not how it works anymore, and it may never have worked that way.
Your countless anecdotes of this working stack only a fraction as high as the countless anecdotes of what you're describing completely failing.
This battle was fought in the 90s and won in the early 2010s. Agile produces results, waterfall generally does not.
The much bemoaned waterfall is not a development methodology, it's a project management methodology. The developers had no process at all. That's an important point missed by developers who've never known anything other than agile development. From a developer's perspective it's not agile or waterfall, it's agile or nothing at all (typically). Agile, with its warts and all, is far better than nothing at all.
The typical waterfall assumptions are very overblown. We are doing ourselves a great disservice by throwing it away. Hardly anyone developing today has ever used waterfall much less agile.
The waterfall:bad, agile:good trope is really tiring. Like listening to a teenager that knows jack shit explain how the world works.
I think the parent post has, but for the reader who is a software engineer take 15 minutes out of your life to read WW Royce's "Managing the Development of Large Software Systems."
In this brief and engaging paper you will find the diagram used by many agile enthusiasts to describe the "waterfall method" and will be shocked to discover that it is held up as an example of a process that never actually works in reality.
You will then read quotes like this, which could have come out of an agile book:
"For some reason what a software design is going to do is subject to wide interpretation even after
previous agreement. It is important to involve the customer in a formal way so that he has committed
himself at earlier points before final delivery. To give the contractor free rein between requirement
definition and operation is inviting trouble."
Both agile and waterfall methods attempt to give you a pattern by which you can predict when software will be delivered. That's the main business value. The suits don't care how the devs operate, provided they can get things on time.
Waterfall tried to do this by trying to understand the problem as fully as possible, to liken it as much to previously solved problems, and to involve gurus to say "that kind of problem will take X amount of time to resolve", and thereby create milestones and delivery dates.
Agile instead said "look, we don't know enough at the start of a project to do that. Let's instead keep track of everything we want to do. Let's try and estimate each set of tasks (stories), individually. And then let's rank them in priority. We can measure how good our estimates are, we can modify them, we can generate more data and determine what we'll have at a milestone, and then can either push back the due date, or at least recognize that we won't be able to ship the entire feature set at that time.
Continuous delivery can be done with either one of those (it almost never is in waterfall, but it -could- be). But by itself it offers a business nothing for planning purposes. All it does is allow an immediacy, a "as soon as it's done it's out in front of the users". This may or may not be a good thing to the business, but it doesn't solve the basic issue that business people want to know when they can expect a given set of features to be live.
reply