Wednesday, April 20, 2022

The Most Significant System Development Project of My Career

Drill rig
This post continues my series on lessons learned in my nearly half century in enterprise IT. We started in 1974 with my job at Macy’s headquarters in Manhattan, followed by my move to California in 1976 and my job at TRW Credit Data. I then took a job at Smith Tool in 1978, where I got thrown into the deep end with manufacturing systems. This led to several more important lessons learned, including the failure of a waterfall development project and my first encounter with shadow IT.

But there were more lessons to be learned at Smith Tool. 

Next would be the biggest and most important project of my career. Rolling off a series of manufacturing system development projects, I was now assigned to a task force to develop a strategic system to analyze the performance of Smith’s drill bits in the field. I would be the project manager for a small team of developers and the overall system architect. 

An Unspoken Objective

The first phase was to build a bit record database, which would become the foundation for several future systems. The database, which would ultimately contain millions of historical drilling records from around the world, would be used for preparing well proposals, evaluating product performance, conducting competitive analysis, and providing a feedback loop from the field to engineering to improve product quality. 

But there was another, unstated objective. Smith Tool had been sued for patent infringement by Hughes Tool (the business that made Howard Hughes his initial fortune). The patent was for a novel application of an O-ring, which sealed the lubricated bearing of the three roller cones from the harsh downhole conditions. O-rings (made famous for their failure in the space shuttle Challenger disaster) were in common use at the time, but Hughes had discovered that if you squeezed the O-ring a bit it actually extended the life of the seal. This was counter-intuitive, but it worked. The litigation had been dragging on for over a decade, starting with Smith getting a federal court in 1979 to invalidate the patent, and Hughes getting a federal appeals court in 1982 to reverse that decision. That was just before I was assigned to the development project, which would be an important element in Smith’s defense. 

The lawsuit, for about $1 billion, was at that time the largest patent infringement case in history. The lawsuit alleged that Smith’s use of the Hughes patent made Smith’s bits competitive with Hughes, earning Smith profits that it would otherwise not have earned. To defend against the Hughes claim, Smith would need a system to provide the data analysis. 

None of this was mentioned to me at the time. I only knew that the project was getting me a lot of attention from top management. In fact, my old manager, Rodger Beard, recently told me that at corporate headquarters they were talking about how my system would “save their bacon.” 

Lesson Learned: Immerse Yourself in the Business

Shortly after the project kick-off, I learned that there was a week-long training program about to start for new field sales people. I invited myself in and got to sit through detailed lessons on Smith’s products and how they were used by customers. I found the whole week fascinating. [1]

Halfway through the week, Dan Burtt, the IT director, noticed I was not at my desk and found out about the class. “Why is Frank taking sales training?” he asked. I managed to convince him to let me finish. 

Since I had been developing or maintaining many of Smith’s manufacturing systems, I already understood the engineering and production data that would be needed to correlate with field performance. What I lacked was an understanding of that field data. My degree in Geology helped, but all of this was mostly new information. 

There were also some thorny design problems, such as how to designate well locations in different parts of the world, using different coding schemes. I spent several hours at the UC Irvine library learning about various geographic location systems in use in the U.S. and around the world, such as the section-township-range system, originally proposed by Thomas Jefferson

In any new system development project, you have to start with a deep understanding of the business. It is not enough to have users tell you what they need. It’s more than gathering requirements. You have to have a sense of curiosity and immerse yourself in the industry and the business.

Lesson Learned: Take Advantage of Career Adversity

But the oil industry is notorious for booms and busts, and we were heading into a major bust. There was a massive company layoff, and the IT staff was not excluded. With fewer IT personnel, we didn’t need as many first level managers, so I was demoted back to project manager. Even worse, after I finished the requirements definition, my project was put on hold pending budget approval to move forward. This was the last straw. I resigned in August 1983 and returned to my retail industry roots, taking an IT manager position at Gemco, a now-defunct membership department store.

Beta Management Systems Logo

But, after a few months I got a call from Smith. The bit record project had been funded. Could I come back to lead it? I said yes, on one condition: I wanted to come back as a consultant, not an employee. I had been thinking for some time about a consulting career, and this was my opportunity. Smith agreed—I had so much knowledge of the project and the business requirements that it seemed like a small request. 

This launched my consulting career, as a sole proprietor doing business as Beta Management Systems. [2] [3]

Development, Implementation, and a Move into the Business

Now I was back at Smith, leading a small team of developers. I designed the system mostly as an online system (IBM’s CICS) but with a little batch programming to extract manufacturing and engineering data on a nightly basis. As usual, I wrote some of the most important code myself. The database was eventually going to hold millions of records, and it would be used for online analytical processing (OLAP), so it needed to be fast. I designed the database in IBM’s VSAM, and I set up alternate indices to provide quick access for the most common types of standard reporting. This was before the days of widespread use of relational databases, or at least before Smith had one. 

For the OLAP reporting, I used something new. The year before I had gotten trained in FOCUS, a fourth-generation language from Information Builders (acquired by TIBCO in 2021). This was an excellent tool for reporting and analysis, especially for ad-hoc inquires. This is how I would develop the OLAP reporting that would prove instrumental later on in supporting the patent litigation. 

Initial system development took less than a year. I still have a copy of the system user guide, dated November, 1984. Users began loading bit records in 1985. 

As soon as the system went into production, there was no more need for me in the IT department. But there was a huge need in the engineering department, where all that ad-hoc analysis would need to be done against the database. So, I left IT and went down the street to the “Hobie Cat Building” (the former owner) to begin as a consultant in the engineering group known as Product Evaluation. [4] 

Within Product Evaluation, I became part of a small team to develop the OLAP reporting for the bit record system. Looking back, this was the best experience I’ve ever had in a team. There was our manager, Jim Watson, who was a metallurgist by training and product failure analyst. Jim became a personal friend of mine over the years. There was Steve Steinke, a geologist, who provided the knowledge of the oil field. Rounding out the team was Joel Palmer, a statistician, who ensured that our analysis was statistically valid. Then there was me, the systems guy. 

Lesson Learned: Understand Basic Statistics

Textbook cover--Calculate Basic Statistics
Looking back, I now appreciate how the statistical validity of our analysis would be critical. This was important not only because we needed to ensure that the conclusions of our analysis were on a sound footing generally, but also because some of our analysis would be presented in court in Smith’s legal defense. 

I had started out as a math major at UPenn, but I’d never had a course in statistics. So, even though we had a statistician on our team, Smith brought in Dr. Mark Finkelstein, a mathematics professor from UC Irvine, to coach us once a week on basic statistics. He used his own text book, pictured nearby. We learned about descriptive statistics and inferential statistics, regression, correlation, and confidence intervals. 

The key point I learned was this: Just because a data set appears to show a correlation between two variables, it might not be statistically significant. For example, I might be asked to divide a sample of bit runs from a group of nearby wells into three groups according to some engineering parameter. My analysis might show that as the parameter increases, the bit performance improves. But that conclusion might be spurious. On more than one occasion I had to tell the requestor that, even though a graph might appear to support his theory, the statistics did not. 

Eventually, the Smith lawyers asked me to perform statistical analysis in support of the patent litigation. In response to the court ruling that Smith was infringing on the Hughes patent, Smith had redesigned its bits to use an older seal, called a Belleville seal, instead of the O-ring. Smith contended in court that the new seal provided performance equal to that of the O-ring, and my analysis supported that conclusion. But the new seal was more expensive than an O-ring, increasing the cost of a tricone bit by about $29. According to a Los Angeles Times account of the trial: 

According to [Judge] Hupp’s chronology of the events that led to Smith’s using Hughes’ patented device, Smith stopped manufacturing the Belleville-type seals in 1972, in part because they made the Smith device cost about $29.02, or an estimated 3.2% of the total purchase price, more than the competing Hughes product.

Smith’s attorneys argued, therefore, that the damages to be awarded Hughes should be calculated based on the difference in product cost for the half million infringing bits, or about $14.5 million, rather than the billion-plus that Hughes was claiming. 

Bottom line, as I was told: The judge agreed that the performance of the Belleville seal was equal to that of the O-ring but did not agree that damages should be based on the difference in cost. The judge assigned damages of just over $200 million. In other words, we won the battle that I was fighting, but lost the larger war. [5]

My appreciation of statistics would benefit me later in my career, when Dan Husiak and I acquired the IT research firm Computer Economics. I took over the research group, which collected and published metrics on IT spending and staffing. Many times, I was confronted with what appeared to be a correlation between IT spending and some other metric. My experience from Smith Tool taught me to be skeptical if the sample size was small. 

Postscript: Successor System Still Delivering Value

DRS Drilling Record System log in panel
The combination of the court judgment, a continuing downturn in the oil industry, and some poor business decisions was too much for Smith to overcome. The company filed for Chapter 11 bankruptcy protection, divested noncore businesses, and was able to come out of bankruptcy in the same year. I was still working as a contractor to Smith through this entire time, but at less than a full-time basis. This gave me time to develop business with other clients. 

So, what happened to the Bit Record Database? In 1988, while I winding down my work on the system, Steve and Jim delivered a presentation at the IADC/SPE Drilling Conference. They reported that the system contained 100,000 bit records. They also reported that the team had built an interface from the mainframe to PCs running dBase in field offices. This was how they were preparing bit programs for new wells. 

Then, in the mid-1990s, I got in touch again with Steve, who told me that Smith had migrated the system from the IBM mainframe to a personal computer running the Progress database. 

So, in writing this post, I got curious: Where is the Bit Record System today? Smith was acquired by Schlumberger in 2010, who rebranded the Smith Tool business as Smith Bits. A little digging uncovered a recent edition of the Smith Bits product catalog, and it has an interesting page on something called the “DRS drilling record system.” 

The Smith Bits DRS drilling record system is a collection of nearly 3 million bit runs from virtually every oil and gas field in the world. The database was initiated in May 1985, and since that time, records have been continuously added for oil, gas, and geothermal wells. With this detailed data and the capabilities of the IDEAS platform, Smith Bits engineers can simulate bit performance and make changes to their bit designs to optimize performance in a specific application. [Emphasis added]

With that date of May 1985, I have no doubt that this is the successor to the Bit Record Database. It is interesting that Schlumberger has renamed the system as the Drilling Record System. It may be because even in my original design the system included data on bottom hole assembly tools other than rock bits and other drilling data such as hydraulics. We called it the Bit Record Database because the form that the system was based on was commonly called a bit record. A DRS screen shot is shown below (click to enlarge). 

DRS Drilling Record System screen shot

Update, Aug. 13, 2022. I have now reconnected with my old teammate, Steve Steinke, who retired two years ago from Schlumberger's Smith Bits group. Steve worked with the DRS system over all those years since we were together. Steve confirmed my recollection of our discussion in the early 1990s that Smith converted the system to a single PC running the Progress database. The main motivation for this was to get off the mainframe. Then around 1999, Smith rewrote the system on an Oracle platform. At the same time, they greatly expanded its functionality to include records of other downhole tools besides rock bits. The team continued to expand the system to include records of other drilling equipment and systems as well. It now even includes geological data, such as formations encountered at various depths. Today it contains something like 1.5 million wells and is used by other Schlumberger business units in addition to Smith Bits. 

In an interesting side note, Steve confirms that the worldwide geographic location coding system I developed is still part of the system design. But Steve personally enhanced the design to automatically derive latitude-longitude from section-township-range, to more easily identify offset wells. 

In any event, I am proud that the system development work I did in the 1980s, over a period of about eight years, still continues to deliver value today. 


Footnotes

[1] The training sessions were not all technical. There were lessons on how to behave properly in the field, including advice such as, when driving through a gate on a cattle ranch, be sure to close the gate behind you.  Another lesson told us not to beg for business or claim that you’ll get fired if you don’t make the sale—unless that’s the only way to close the deal. There was another lesson with a pamphlet entitled, “How to Turn WAGs into SWAGs,” where a SWAG is a scientific WAG. It had something to do with using data in sales proposals. We also learned that in the early days, Smith was known as the “Whisky Bit,” because sales people would put a bottle of whisky in the pin of the bit. So, when the roughnecks would get thirsty, they’d say, “Let’s open one of them whisky bits.” 

[2] There was no significance to the word Beta. I didn’t have money to spend on a logo, so I figured I could get the printer to use the Greek letter beta in place of the normal font. That allowed me to use the business name as a logo. 

[3] Having at least a year of guaranteed contract work, maybe more, was a huge factor allowing me to break into consulting. A year earlier, our third child, Joanna, was born, and we had just bought our first home. Finances were tight. As it turned out, though, my work with Smith took me through most of the 1980s as I then began to add other clients, mostly through referrals from other “Smithereens” (people who had quit or left Smith during the rounds of layoffs). 

[4] Among other responsibilities, the Product Evaluation group provided post-mortem analysis of bits that failed in the field. They had a large room that they called the “morgue,” with bits that had failed, laid out in table top trays. The group included metallurgists and engineers that did root cause analysis to determine the causes of failures and make recommendations for changes in product design, manufacturing processes, and quality procedures.  

[5] This was a stressful time, with the Smith legal team often asking for additional ad-hoc analysis, sometimes just as I was about to leave for the day. But, to their credit, they did a good job keeping my name out of discovery so I wouldn’t have to be deposed. I think it helped that I was a contractor and not a Smith Tool employee. Not that we had anything to hide. But it wouldn’t have been a pleasant experience. Jim and Steve were deposed and testified in court. I got to see a trial transcript, and from what I read and what they told me, it was grueling. 

Photo Credit: Drill Rig, Pixabay