What I Learned from a Waterfall Project Failure

In my most recent post on lessons learned in my career, I covered my time as an IT employee at Smith Tool. I learned so much in those years, and I need another few posts to cover some of the more interesting lessons.

By 1980, my work in manufacturing systems had been all in machining operations. Now Rodger gave me a new assignment, to develop a new system to support Smith’s forge plant. This opportunity took me upstream into the forge, which I was told was the largest forge in the United States west of the Mississippi¹.

As I noted in the previous post, I loved the nitty-gritty sights and sounds of the metalworking plant. But the forge was another whole level of physicality, almost violent. As you approached the forge, you could hear the hammer and feel the ground shake as the press hammered out parts². And outside the plant were pallets of newly forged parts, still red hot to the point where you could not stand closer than 20 or 30 yards without feeling the heat.

Here is a good video of a forge plant that gives a sense for what it’s like to be inside one. The press in this video is smaller than the two at Smith. Also, our forge plant was more modern and the parts being forged in the video are different, but the sights and sounds are the same.

Forging Operations Are Tricky to Schedule

Smith used a process called closed die forging. This means that the red-hot steel bar would be pressed into a die in the shape of the finished part. The tricky part is that the die would only be able to produce a given number of forgings before it had to be sent out for “resinking,” a sharpening operation, so that they could produce more forgings. The production scheduler used manual log books to keep track of how much life was left in each die and to know which dies had enough life left to fill an order. But if the log books were not updated correctly, the forge might not be able to meet its production schedule. This was the process the new system would automate, with the benefits of being able to better meet production schedules.

The new system was described in a company newsletter after the project was (supposedly) completed.

Simply stated, the Forge Die Tracking System keeps track of over 3,000 components that make up the 300 forge dies used to stamp out the various forgings to make Smith Tool’s products. Additionally, the system makes the die selections to fill each forge order according to which dies most closely match in wear and on the basis of how much life the dies have left. When the system notes that a die’s life is getting low, it will suggest to the scheduler that the die be used to make as many forging as is left in it to make, and then be sent out for resinking. So, while the order is being filled, the excess forgings being produced will go into inventory as safety stock.

I was assigned as project manager and lead analyst six months into the project, which had stalled for lack of someone who could develop the calculations for picking the best set of dies for a given order, suggesting the optimum production quantity to “run out” the die, and to track dies sent out for resinking.

I took it as a challenge. I remember distinctly that this was taking place during the run up to the US 1980 presidential election. So, I wrote my programming specifications with references to identifying candidate dies, nominating them, and then electing the best one. Although I had three programmers reporting to me for this project, I wrote some of that core scheduling logic myself.

The New System Installed but Not Implemented

So, what went wrong? There is a little hint of the problem in the final paragraph of that newsletter story on the project:

All the die structures will be on the system by the end of November. The history of all those components will be on by mid-December. The Forge personnel are enthusiastically anticipating the ease and efficiency the system will bring to the Forge operation. [Emphasis added]

Note the future tense.

Here’s how it went down. We put the system into production, and each week I would follow up with the director of forge operations to see how they were coming along with loading the die information. He had assigned this job to his administrative assistant, but the director told me she was too busy to get to it. After several weeks of follow up, it was clear that they had no intention of loading the master file data that the system would need to start scheduling the forge. The system became a very expensive piece of shelfware. I don’t recall how long they let the system continue to run in production without any data to process. But I was soon assigned to another development project.

So, how did this project get approved in the first place? Later I found out that the forge director felt the IT department was spending all its time developing systems for the machining plant, and that it was “his turn” for a new system. The IT steering committee complied.

Lesson Learned: Test User Commitment through Phased Implementation. This was my first real experience with the drawbacks of a waterfall development approach, where you define all the requirements up front, then design the entire system, then program it, test it, and deploy it. In this case, the users were happy to meet with us to provide their requirements and review our system design. But in terms of actually doing any real work, the users were off the hook until we put the system into production. At that point, they were not willing to let a low-level administrative assistant spend the time to do the necessary data entry, or hire a temp worker to backfill her regular duties so she could do so.

After that I vowed never again. What we should have done is build the database and then have the users enter the master file data before we invested more time programming the scheduling logic—the really difficult part, where the bulk of the development hours would go. That would have tested the users’ commitment and saved several months of wasted effort. This was 20 years before the Agile Manifesto, but my software engineering courses at UCI had already taught me about Barry Boehm’s spiral development methodology, which in many ways anticipated Agile. If only I had the foresight and permission to take this approach.

Postscript: In reviewing a draft of this post, the department manager at the time, Rodger Beard, has further recollections. He writes (lightly edited):

I felt at the time and actually still feel a sense of personal failure for allowing this project to unfold the way it did. Exactly the way you've described. A painful recognition, over many months, that good work was being completely wasted.

This was my first experience with having pure politics result in a significant waste of IT resources. However, like you, I learned from it. An aside, at the time, I also felt this system was not well conceived. If it were well deployed, it could have had an excellent ROI. But with the caveat that additional, easy-to-avoid ongoing human capital investment would be necessary to make it pay off. A red flag really.

I think you're spot on regarding how the requirement should have been addressed. This was definitely mine (especially) as well as [name withheld]’s leadership error. We knew it was an "it's my turn" situation. [Our leaders] had decided to throw the forge a bone to shut up the forge director. If only I had had the foresight to ask you to do what your article says should have been done. Sigh.

Thanks again, Rodger, for your confirmation in this series.   

---

End Notes

¹The forge was about 200 yards from the Smith Tool metalworking on Von Karmen Avenue in what used to be known as the Irvine Industrial Complex. Driving through that area today, now known as the Irvine Business Complex, it’s hard to believe there was such heavy manufacturing there into the 1980s. That part of Irvine is now mostly commercial offices and some distribution or light manufacturing facilities. The metalworking plant today is an Amazon distribution center. 

²The purpose of forging is to form the raw material into a part that has dimensions close to what is needed so that the first machining operation only needs to remove a minimum amount of metal. Forging also improves the metallurgical properties of the material. It is the same process as the ancient blacksmith employed with his hammer to make metal implements, such as horseshoes. In fact, Smith Tool began as a blacksmith shop in 1902 in Whittier, CA.

Photo Credit

Waterfall. The original uploader was PaulHoadley at English Wikipedia., CC BY-SA 2.5, via Wikimedia Commons