How We Got Agile: An Origin Story
When I was a young man, a college student in the Computer Science program at Virginia Polytechnic Institute, we were assigned a book to read. It was called The Mythical Man-Month, by Frederick P. Brooks, Jr., and I still have my copy from the 1980s. The point of the book, which you might be able to glean from the title, is that you can’t simply measure work effort in “man-months,” on a scale such that you could conceivably get more work done by adding more people to a project. As an example, you couldn’t say that a project has a work effort of 120 man-months, meaning that with 10 men it will take 12 months to finish, and therefore with 20 men it will be done in 6 months.
If you had 10 men working on this hypothetical project, and added 10 more, you would not find that it completed 6 months sooner. It would, in fact, take longer than 12 months. The problem is, as you add more men (people) to a project, you need time to get new hires ramped up to where they understand the project well enough to be productive. You also multiply the lines of communication, which generates additional overhead keeping everyone in sync on specific information needed to make interacting components work together. In engineering, these pieces of information are called “specifications,” and they have to be tracked somehow. If you add more people to a technical project, you add more tracking effort. These complications are summarized in Brook’s law: “Adding manpower to a late software project makes it later”.
As a software engineer in the early 21st century, it fascinates me to read the author’s description of how specifications were tracked on the project he worked on – the IBM System/360 – in the 1950s and 60s. They had huge manuals kept in binders, and as changes were made, the responsible engineers would have to go in to the binders and update the appropriate pages – that is, take out the old pages and insert the new ones with the changed specs. This manual was the Bible of the system, and keeping it up to date was abolutely vital to the success of the project.
Modern day software engineers like me are not used to such meticulously maintained documentation. We consider ourselves lucky if there is any documentation at all for the software on which we are working. You’d think it would be easier, now that everything can be done online, but projects move too fast and the people working on them move around too much. No one is necessarily going to stay on top of documentation, and so long as software works as expected, that’s fine. It’s when it doesn’t work that you run into trouble.
Because personnel move around so frequently in the modern workforce, there is rarely anyone working on a software program who was there when it was originally programmed. But programmers still need to maintain it. Sometimes we are given requirements to modify existing software that has no documentation, with no one around who knows anything about it, and the only way to achieve that goal is through “reverse engineering.” This means poring over old code and documenting it from scratch, which is very time consuming. This underscores the point about the man-month: you can’t just insert a person into a project and expect them to get as much done in a given amount of time as a previous person on the project did. Certainly not if they are going to be reverse engineering the previous person’s work.
Since the start of the personal computing era and the long economic boom of the late 20th and early 21st centuries, computer software has been advancing at a faster pace than it did when Frederick P. Brooks worked as an engineer at IBM. The workforce has changed as well, with employees typically job hopping every few years, and often working as contractors through agencies rather than directly for the client that owns the software they are developing. So how do the software engineers of my generation handle project management in such a chaotic work environment?
The answer is “Agile” methodology, which came about around the start of this century. Agile is a lean or lightweight software development method that emphasizes individuals collaborating over plans and processes, and defines good software as software that works, not necessarily software that is well documented. At least, that’s the declaration in a famous “Manifesto for Agile Software Development” that was published in 2001.
The idea is that “Agile” is a mindset where you are focused as a team on communication and collaboration, continuous improvement, and responsiveness to change. In practice, it means breaking up the project work into short iterations called “sprints,” which typically last two weeks. Everyone’s tasks for the sprint are things that shouldn’t take more than a couple of weeks to finish. So right there the idea of a “man-month” is out; no one would work on one thing for a whole month!
Breaking the project work into chunks like this makes it easier to show progress, and to evaluate how effective the team is from sprint to sprint, and change processes and workflows as needed. It also makes it easier to accomodate personnel shifting around from project to project. It’s a way of coping with today’s volatile workplace, which makes long term planning harder to achieve. A whole panoply of “frameworks” and “ceremonies” has developed around the original concept since it was first elucidated.
If you are in a white collar profession (not even necessarily Information Technology) you might have experience with Agile-related frameworks in your career. I was first exposed to Agile in the late 2000s, and have been at positions where it is used comprehensively since 2018. Every company does it a little differently, but I have always found it to be a useful way to structure project work.
The way I see it, Agile came about because a new generation of software engineers needed to adapt to a faster pace of work than what the generation of Frederick P. Brooks experienced in their careers. They needed to find their own solution to the problem of how to get people to work effectively when they are added, out of the blue, to a new project. If you look at the signatories of the 2001 Agile Manifesto, you will see that they are almost entirely Baby Boomers and Gen Xers. Today’s Millennials and Gen Zers in the IT workforce have possibly never worked on a project that wasn’t using Agile.
I’ll have more to say about the different generations and Agile in a future post.
