A legacy system rebuild gone wrong

This is one of the better articles on the site. Basically, a large company took their successful, but aging mainframe system, hired some right-out-of-college hotshots to build a new system in the latest language (VB.NET? Seriously?), and failed miserably. Great exercise in bad software engineering. Reminds us to be more careful when we brazenly think we can replace an existing system. Also a good reminder that the language and syntax doesn't matter, the architecture matters more.

If your business logic is all over every layer of your app (sound familiar?) then you're pretty much hosed. One of my recent projects has this trait to a small degree because of the ad-hock changes to business logic and inadequate development time allowed.

Daily WTF

Jurassic Programmers


by Alex Papadimoulis in Feature Articles

When Russ started at InsuraCorp (as I'll call it), one thing was immediately apparent: There were two classes of programmers. The "rock stars," who were recruited from top universities and given first-class accommodations, like windowed offices with brand new computers and dual 21-inch LCD monitors; and the "dinosaurs," who were cramped in dimly lit cubicles each about the size of a refrigerator box. The dinosaurs were lucky if they had a fully working keyboard for their Windows 98 workstations.

It hadn't always been that way. About six months earlier, the dinosaurs occupied the nicer offices. They were responsible for maintaining -- some, originally building -- InsuraCorp's 25-year-old cash-cow product. Though it worked very well, the system ran on the VMS operating system and was written in the now-forgotten Digital Interactive Business Oriented Language (DIBOL), rendering it accessible only through a text-based terminal emulator. In the age of the Web, and competitors with a more modern product, customers demanded more.

InsuraCorp vowed to hire a new team of sharp-minded programmers to completely rewrite the software as a Windows application using VisualBasic.NET. This was the project Russ was brought on for, albeit several months late. With years of Windows and .NET-based development experience, Russ would be a great fit for the rock stars. However, he had some background in COBOL, so his manager wanted him to "spend some time" with the dinosaurs.

Not in the Band
His association with the dinosaurs and his unwelcome suggestions (such as not focusing on developing a "cool-looking" control library) made him an outcast among the rock stars. They increasingly kept "forgetting" to invite him to meetings and to assign him tasks, pushing Russ' DIBOL responsibilities from "part time" to "full time."

Though he dreaded this at first, Russ became rather grateful as version 2 "progressed." After nearly a year of development, the rock stars had little more than a buggy control library to show.

More time passed, and version 2 degenerated into a hybrid of old and new technology. SQL Server had been abandoned for indexed files stored on VMS. Multi-tier programming was also abandoned, leaving business calculations everywhere from within DIBOL code on VMS to the paint routine of a Windows control. QA suffered as an inexperienced testing staff managed little more than "guerilla testing" on the constantly changing features.

Eventually, version 2 was ready. It took four years, 15 full-time developers (the rock stars), and cost nearly $10 million. There was just one problem: The customer that signed up to be the first conversion had changed its mind -- it was sticking with version 1. The second customer bailed as well. It had seen some of the beta software and was none too impressed.

Trial and Error
After a few months of cajoling, the higher-ups at InsuraCorp finally found another candidate. They convinced the client that version 2 ran circles around version 1.

Though version 2's data still resided on the VMS, it took a full weekend to migrate the data from version 1. The first week of version 2 was rough, to say the least. None of the accounts could be balanced, transactions would transfer odd amounts and exceptions would pop up from nowhere. All of the rock stars were flown out to the customer's main office to triage bugs and provide emergency fixes.

Things went downhill from there. By week five, the customer had become downright hostile. It was losing clients and demanded that InsuraCorp put it back on version 1 or face the wrath of its legal team.

Welcoming Back the Dinosaurs
With nowhere left to go, the higher-ups gave in. They vowed to dedicate all of their programming resources -- the dinosaurs and rock stars -- to "de-converting" the customer.

Easier said than done. There was now more than a month's worth of client data on version 2. The higher-ups relied on one of the dinosaurs to lead the deconversion. He had done many conversions from competitors in the past and was the most experienced DIBOL/VMS programmer.

The problem was, version 2's data structure was built by programmers fresh out of college, with no real-world experience and not a single architect to guide them. The rock stars had created a monstrosity of a database, consisting of hundreds of undocumented tables tied together with hundreds of undocumented relations, all glued together with copious amounts of XML.

The rock stars were demoralized and defeated. Some, sensing their impending termination, quit right away. Others stuck around to patch the still-live version 2 and help with the deconversion.

Three months later, the customer was finally deconverted to version 1. The following day, the rest of the rock stars were fired and the dinosaurs quickly moved back into their windowed offices. In the months since, the dinosaurs have shifted from "near-death" maintenance to full-on development. Russ, along with his colleagues, has added several new features to the product.

And although there are no immediate plans to try building version 2 again, the higher-ups have learned their lesson: Don't push the dinosaurs toward extinction -- work on evolution.


Jurassic Programmers was originally published in Alex's DevDisasters column in the January 1, 2008 issue of Redmond Developer News. RDN is a free magazine for influential readers that provides insight into Microsoft's plans, and news on the latest happenings and products in the Windows development marketplace.