Image may be NSFW.
Clik here to view.
The media for the past three years has failed to do its job as independent and skeptical investigative journalists with regards to Obamacare in general and the essential technology infrastructure in particular — not just the Healthcare.gov website itself but all the back-end systems and the state exchanges as well. Even when Healthcare.gov launched, most media outlets dutifully reported the problems as mere “glitches” and echoed the Obama Administration’s (false) statements that these problems were merely due to the website’s overwhelming popularity and that all would be well soon. A few intrepid reporters — such as Sheryl Attkisson at CBS News — felt the problems were deeper and kept digging, but most mainstream news outlets did not.
Now, however, the dam is collapsing. The true depth of problems with the ACA tech infrastructure is coming out, thanks largely to the Republicans in Congress fulfilling their oversight role, obtaining and releasing documents. This, in turn, has started the Great Finger Pointing of who is to blame for what will likely be the most public and visible US information technology failure to date, and all parties involved have turned to the media to air grievances and get out in front of the blame.
On Friday, Amy Goldstein and Juliet Ailperin of the Washington Post presented what was largely the Administration’s side:
At 9 a.m. on Aug. 22, a team of federal health officials sat down in a Baltimore conference room with at least a dozen employees of CGI Federal, the company with the main contract to build the online federal health insurance marketplace. For six weeks, the federal officials overseeing the project had become increasingly worried that CGI was missing deadlines, understaffing the work and overstating its progress. . . .
For that day and the next, CGI staff huddled with government officials in the semicircular conference room at the headquarters of the federal Centers for Medicare and Medicaid Services (CMS), the agency overseeing the project. They combed through 15 pages of spreadsheets they had brought, which spelled out the company’s level of confidence — high, medium or low — that individual components would be ready.
By the time HealthCare.gov launched 51/2 weeks later, many of those predictions proved wrong, according to internal documents obtained by The Washington Post and officials familiar with the project.
A final “pre-flight checklist” before the Web site’s Oct. 1 opening, compiled a week before by CMS, shows that 41 of 91 separate functions that CGI was responsible for finishing by the launch were still not working.
That same day (last Friday), Eric Lipton, Ian Austen, and Sharon LaFraniere of the New York Times presented the same sequence of events, even starting with the same August meetings, but gave more of the contractors’ point of view:
On a sultry day in late August, a dozen staff members of the Centers for Medicare and Medicaid Services gathered at the agency’s Baltimore headquarters with managers from the major contractors building HealthCare.gov to review numerous problems with President’s Obama’s online health insurance initiative. The mood was grim.
The prime contractor, CGI Federal, had long before concluded that the administration was blindly enamored of an unrealistic goal: creating a cutting-edge website that would use the latest technologies to dazzle consumers with its many features. Knowing how long it would take to complete and test the software, the company’s officials and other vendors believed that it was impossible to open a fully functioning exchange on Oct. 1.
Government officials, on the other hand, insisted that Oct. 1 was not negotiable. And they were fed up with what they saw as CGI’s pattern of excuses for missed deadlines. Michelle Snyder, the agency’s chief operating officer, was telling colleagues outright, “If we could fire them, we would.”
Interviews with current and former Obama administration officials and specialists involved in the project, as well as a review of hundreds of pages of government and contractor documents, offer new details into how tensions between the government and its contractors, questionable decisions and weak leadership within the Medicare agency turned the rollout of the president’s signature program into a major humiliation.
The online exchange was crippled, people involved with building it said in recent interviews, because of a huge gap between the administration’s grand hopes and the practicalities of building a website that could function on opening day.
Conor Friedsdorf over at the Atlantic looks that New York Times article and asks the important question: Why did President Obama say the Healthcare.gov website would work?
There’s something I don’t understand about the rollout of Healthcare.gov: that President Obama expressed unqualified confidence about it shortly before it was launched.
Why?
After the president acknowledged at a November 14 news conference that the website “was failing the most basic tests internally,” CBS reporter Major Garrett noted that the White House was aware of its failings “and yet a decision was made to launch the website on October 1st.” In response, Obama seemed to recognize everyone’s confusion.
Here’s what he said:
OK. On the website, I was not informed directly that the website would not be working as—the way it was supposed to. Has I been informed, I wouldn’t be going out saying, boy, this is going to be great. You know, I’m accused of a lot of things, but I don’t think I’m stupid enough to go around saying, this is going to be like shopping on Amazon or Travelocity, a week before the website opens, if I thought that it wasn’t going to work.
So, clearly, we and I did not have enough awareness about the problems in the website. Even a week into it, the thinking was that these were some glitches that would be fixed with patches, as opposed to some broader systemic problems that took much longer to fix and we’re still working on them.
So you know, that doesn’t excuse the fact that they just don’t work, but I think it’s fair to say, no, Major, we—we would not have rolled out something knowing very well that it wasn’t going to work the way it was supposed to, given all the scrutiny that we knew was going to be on—on the website.
You know what? That reasoning probably sounded persuasive to most people listening. It would be stupid to tout something knowing it’s about to be proven not to work. And for good reason, no one thinks that Barack Obama is a stupid man. If he knew the website would fail, his actions wouldn’t seem to make any sense.
There’s just one problem: The idea that he didn’t know it would fail doesn’t seem to make any sense either [in light of what the New York Times article reports].
Go read all three articles and watch as the dam behind which all this information has been pent up for the past few years falls apart.
I’ve written about the various root causes and symptoms behind the Obamacare failure over the past two months, starting before Healthcare.gov even went live. Here’s a brief summary; compare what I wrote in each one with what came out last Friday (November 22) in the Washington Post and New York Times articles:
September 26: Obamacare and the Thermocline of Truth — The top people (up to and including President Obama) probably don’t know how bad things are due to the ‘thermocline of truth’ effect (a term I coined in 2008), but even if they do, they may launch the website anyway, with disastrous results.
(On October 4th and again on October 9th, I noted in two post-launch posts the reports coming out from various sources about the problems with Healthcare.gov and made some observations based on my professional experience with failing IT projects. )
October 21: Obamacare, IT, and Magic Thinking — The Healthcare.gov project appears to have profound issues in architecture, design, and implementation, but those involved appear to believe that the political importance of this project and the October 1st dte somehow supersedes the reality of software engineering. As I wrote:
This is, of course, feeble-minded crap. Software is very, very unforgiving, and if you haven’t taken the right approach to ensure proper functionality, performance, and quality, no amount of wishing and hoping and planning and dreaming is going to make it work right.
October 23: Obamacare and the Project of Doom — I describe the essential quality aspects of a successful IT system – Reliability, Performance, Functionality, Compatibility, Lifespan, Deployment, Support, and Cost — and note that Healthcare.gov appears to have failed or be failing on just about every single one. I also list six different patterns of IT project failure I identified in a white paper I published back in 2000 while at PricewaterhouseCoopers and note that the Healthcare.gov project appears to match three, maybe four of them. I cite — a few days before Jeffrey Zients shows up and proclaims that all will be well by the end of November — one of Fred Brooks’s dictums on late projects: “Take no small slips.” I also predict that the website (at least, the application section) will be shut down at some point for a number of months.
October 25: Obamacare and the Aluminum Falcon — I seize upon a wonderful Star Wars analogy from Jim Geraghty of National Review, who writes:
They [the Obama Administration] need to heave Hail Mary passes from here on out, and hope the thing suddenly and miraculously starts working like the hyperdrive of the Millennium Falcon at the end of The Empire Strikes Back.
I then note that, to the contrary:
The problem with large, complex software is that each bug you fix usually just uncovers some number of new bugs that you weren’t encountering before. (This is also the problem with performance improvement on large systems: when you discover and fix the worst performance bottleneck, you merely uncover the next performance bottleneck.)
But wait! It gets worse! It is well- and long-known in software engineering circles that any change to a program’s source code runs a measurable and not-insignificant risk of introducing new bugs (or re-introducing old ones). This is why thorough regression testing — in effect, re-running all your existing test scripts again after each code change — is so important, particularly in the late stages of system development. Without rigorous regression testing (and source code change control, limiting who can make changes to code and when they are allowed into the system), you can in effect be chasing your tail for months without achieving a stable acceptable system.: the cumulative expense of developing, deploying, upgrading, and maintaining the system must appear to be justified in the eyes of the client.
I also link to a couple of Robot Chicken Star Wars clips that turn out to be very appropriate in retrospect.
October 26: Obamacare and the Long Bomb — Jeffrey Zients shows up in DC, takes charge, and proclaims that “By the end of November, HealthCare.gov will work smoothly for the vast majority of users. . . .Let me be clear: HealthCare.gov is fixable.” I am dubious (at best), and I predict that we’re likely to see the Obama Administration have end users contact insurance companies directly instead of going through Healthcare.gov — which is exactly what happens a few weeks later.
October 28: Obamacare and the Oscillation Overthruster — Failing IT project often enter a period of oscillation, where they are perpetually appear to be several weeks or a few months away from stabilizing and working well; I suspect this is what Healthcare.gov is going through and will continue to go through. I then note:
By an odd coincidence, the end of November — the new projected “all is well” date for Healthcare.gov — happens to be Thanksgiving weekend.
Expect a bad news dump on Black Friday.
October 29: Obamacare and the Three Errors — I take a step back and note three fundamental logical/conceptual errors with the Left’s push for healthcare reform:
- Doing anything is better than doing nothing.
- A comprehensive “big-bang” solution is better than a cautious piecemeal solution.
- Intent can produce results contrary to reality.
October 31: Obamacare and the Testing Gap — By now, news has been leaking out about the lack of successful performance and stress testing on Healthcare.gov just weeks before it went live. I quote from my own decade-old professional review of a $500M struggling IT project of the need for rigorous quality assurance. What I wrote about 10 years ago appears to apply to the Healthcare.gov project:
The key to this cycle is the ‘gatekeeping’ function—that is, deciding whether the deliverable being created or modified has sufficient quality to pass along to someone else. When that gatekeeping function is ignored, overridden, weakened, or simply not present, the whole software development process gradually breaks down, because the quality of the system in development becomes more and more constrained by the (poor) quality of its components. One cannot build a safe, enduring house with incomplete blueprints, untested construction techniques, shoddy materials, and a sandy foundation. The same is true of software.
November 3 — Obamacare and the Unforgiving Gauntlet — I continue to respond to what are increasingly appalling reports of the lack of software quality assurance — and testing in particular — with the Healthcare.gov project. Again, I cite from my review a decade ago of a massive failing IT project, listing all the various tests that need to be done in a large-scale project and noting the consequences of failing to do so. They all seem quite applicable in retrospect with the on-going problems, not just with the website, but with the state exchanges and various back-end systems as well.
November 4 — Healthcare.gov: what to expect next – Over at Ace of Spades, I predict what we are likely to see with Healthcare.gov going forward: either a shutdown for some period of time, or the ‘oscillation’ mode I wrote about back on October 28th. Among other things, I write:
Now, Sen. Dianne Feinstein (D-CA) and Rep. Mike Rogers (R-MI) gave Obama lots of cover on the Sunday talk shows by recommending that Healthcare.gov be taken down for an extended period (Rogers said six months) until at least the security problems are worked out.
I’m not sure that’s going to happen, at least not yet. At the least, I think that the Healthcare.gov effort will push ahead until Thanksgiving weekend (end of November, remember?), and then they will go off-line, burying the news on the ultimate Friday-for-news-dumps.
At that point, they will have two choices: try to fix the existing systems (or major sections thereof), or start a new, highly simplified system from scratch, with manual support, and they slowly graft it into the necessary back-end systems. This will take months; frankly, it could actually take a year or two.
If, however, they do not pull the plug then (or before then), then expect to see the oscillation continue: some modest improvements, accompanied by a rash of new problems (or old ones resurfacing). Usage numbers for the website will steadily drop — actual non-Medicaid enrollment will continue to be very low — and the Administration with its enablers and flacks will continue to try to find a way to blame this disaster on anyone but themselves. Ultimately, the site will either persist in low functionality or will be halted altogether.
November 13 — Obamacare and the Unstopped Project — I paraphrase William F. Buckley and state:
The most valuable IT consultant is someone who stands athwart a failing software project, yelling Stop, at a time when no one is inclined to do so, or to have much patience with those who so urge it.
What has come out since then is that there were people within the project who felt October 1st was an impossible date, but top Administration people said things such as “October 1st in non-negotiable” and “Failure is not an option.” Of course, anyone who had done large-scale IT management and design knows that failure is not only always an option, but it is the most likely and common outcome unless you take extraordinary pains in all aspects of the project, including giving it enough time to be done right.
November 16 — Obamacare and the Potemkin Website — I now conclude that the Obama Administration — rather than taking the rational, professional course and shutting down the website until it is properly working — will instead continue to define success downward:
That’s why I now suspect that Healthcare.gov as of December 1, 2013, to be a Potemkin website: that is, the same glossy interface, but significantly constrained and unreliable functionality, performance, and quality behind the facade. I expect a number of the state exchanges to likewise be struggling, both for their own reasons and because of on-going problems with the Federal Data Hub.
November 19 — Obamacare and the Three Heads – On that day, I wrote:
In a large-scale, well-managed IT project, there are three key roles that need to be filled by three different people who are each talented, qualified, tough-nosed, and given commensurate authority. Actual titles may vary a bit, but the roles themselves are well-known: Project Manager…Chief Architect…Director of Quality. . . .
As far as I can tell — from the documents that have come out in the seven weeks since its disastrous launch — none of these three roles existed (or, at least, were filled by competent and qualified people) in the Healthcare.gov project.
The Washington Post and New York Times articles cited above confirm and underscore that conclusion.
November 20 — Obamacare and the 90% Solution — In response to Henry Chao’s testimony before Congress that 30-40% of the overall Healthcare.gov system still remains to be written, I cite what I like to call the Metric Law of 90s:
The first 90% of a software project takes 90% of the schedule. The remaining 10% takes the other 90% of the schedule.
I note again that this project, if not shut down, is likely to go into ‘oscillation’ mode.
November 21 — Why Obamacare is different — Writing again over at Ace of Spades (as ‘Fritzworth’, my guest-blogging nom de plume), I step back from the technical issues and note why the problems with Obamacare itself are likely to have serious political repercussions:
8) How many of you know someone (including yourself) who had their existing health insurance policy cancelled as a result of Obamacare, and who are now looking at options that are more expensive and have higher deductibles? (My own hand goes up.)
9) How many of you know someone (including yourself) who works for a business whose health insurance coverage is either going to be eliminated or become more expensive as a result of Obamacare? . . .
…between now and November 2014, I am willing to bet that the vast majority of the voting population will answer yes to questions 8 and 9. And that is why, in spite of the Left’s desperate attempts to find some equivalents between the burgeoning Obamacare debacle and the sins of the Right (as they define them), it’s really no contest.
And that takes us up to this post. In effect, what is now coming out in the media is what I’ve been saying for two months: not because I have inside sources (I don’t), but because the root causes of IT project failure are well-known and well-documented, and Healthcare.gov exhibits all of them. Again, note that “failure” doesn’t mean the website doesn’t work at all; it means that it is not achieving the quality attributes I spoke about back on October 23rd: Reliability, Performance, Functionality, Compatibility, Lifespan, Deployment, Support, and Cost. There are plenty of failed or failing IT projects that limp along in product for months or years without ever achieving the desired benefits or goals.
As I and many other have noted, the past few weeks have seen the Obama Administrations continuing to lower expectations for the December 1st date. As I have said for a month now, I suspect this weekend will be a major decision point. It will be interesting to see what happens. ..bruce..
P. S. Click here to see all my Obamacare posts.
P. P. S. Made some minor edits and reformatting, and corrected one mistake.