Identity

The individual components of KP-IT maintained different work procedures that reflected deeper differences of identity.Footnote ⁷ The web development team (Web Development) focused on meeting the needs of their customers—specifically, pharmacists waiting for a new prescription application in KP Online and the KP members waiting for replies to their e-mails. As a development group with a project focus, Development felt driven by deadlines with relatively short-term time horizons. When problems emerged, project members strived to resolve them as quickly as possible using expedient, ad hoc methods. In contrast, the national operations team (Operations) focused on keeping their technology up and running according to standard operation procedures that had been learned and developed through years of experience in the mainframe environment. Operational uptime and established service level agreements, not project deadlines, drove their perception of urgency. Thus, Operations’ approach to problem solving strictly followed standard procedures, with formal reviews and networks of accountability.

Ongoing Events

The KP Online accident unfolded as one of a series of cascading errors, accidents, and breaches in the context of major technical, management, and organizational transitions.Footnote ⁸ KP had recently completed integrating its regional IT infrastructures into a national enterprise information architecture. As illustrated by KP Online itself, Kaiser was beginning to experiment with early web-enabled technologies, a process conducted largely without input from traditional mainframe Operations, its seat of historical IT experience and memory. In the middle of the KP Online development process, the KP enterprise architecture planning group (IT Planning) imposed a new web development platform on Web Development. As Web Development struggled with the new technology, it also experienced local organizational and managerial changes. With little experience in web technologies, KP-IT engaged outside contractors into the Web Development group. At the time of the breach, a new manager had assumed responsibility for the Web Development group but was still in transition. On the day of the breach, the contract project manager left town for the weekend and the current Web Development team leader was out with an illness. At multiple levels, the KP Online project existed betwixt and between—a situation ripe for the disruption of organizational sense making and failure.Footnote ⁹

Cues and Retrospective Sense Making

The unanticipated and unintended consequences of the cascading accident functioned as cues that various KP-IT processes had gone awry. During the cascading accident itself, each cue emerged somewhat independently of the others and reflected the compartmentalized sense making characteristic of the various components of KP-IT at the time. These cues became starting points for the retrospective analysis of the KP Online incident management team as it began its investigation. This process culminated in two “root cause” analyses, one focusing on the late deployment of Rx Refill and the other focusing on the KP Online security breach. The root cause analysis of the failure to meet the Rx Refill deadline recommended, among other things, that all KP-IT problems be handled henceforth through the online Enterprise Support System, thereby documenting and attempting to replace the common practice of ad hoc problem-solving methods. The root cause analysis of the KP Online security breach made several recommendations with the aim of formalizing procedures in the Web Development unit according to established KP-IT principles.

These efforts to restore organizational sense making in the face of nonsense largely succeeded. Even before formulating a detailed picture of why the breach occurred, the KP Online incident response team developed and implemented a plan to call all affected members, explain what had happened, and suggest certain reparations. With few exceptions, the KP members accepted the call as evidence of Kaiser's good faith and basic competence. No lawsuits resulted from the case. The public press praised Kaiser's forthright response even as they reported the incident. The root cause analysis identified the immediate cause of the problem: flawed coding. KP disciplined no one because, like its members and the public press, it recognized the error as an expression of the programmers’ commitment to their mission, not as evidence of incompetence or sloppy work. In conjunction with the recommendations of the root cause analysis, these actions rendered nonsense (the breach) sensible, reaffirmed core KP values and transitioned the KP-IT program to a new stage in its organizational maturity.Footnote ¹⁰

Embedded Ethics

Embedded ethics are cultural values of the organization that can be observed by the way an organization conducts its activities. KP's response to the breach of KP Online included activities that demonstrate three specific embedded ethical values, namely concern for member well-being, aggressive internal organizational communication, and commitment to open business practices.

Concern for member well-being—KP employees are preoccupied with assuring the well-being of their members. Prompt, formal response to member e-mail malfunction: Recognizing the importance of prompt communication with members, KP-IT and KP Online business team had established a service level agreement requiring transmittal of messages within 24 hours. On July 24, 2000, it was noted that member questions sent via the KP Online member e-mail messaging function were not getting to the customer services center. Upon verification of the delay, KP-IT immediately assigned responsibility for resolving this malfunction. The KP-IT staff documented and monitored each step in the assessment and resolution of this malfunction using the online Enterprise Support System. Senior management focused attention on this effort because of its assigned high severity level.

Notification of all members affected by the breach of confidentiality: As documented in the memo from counsel, on the day that a member reported the breach of confidentiality, KP created a critical situation management team. This team decided to determine the identity of, notify, and develop a process to stay in touch with each individual member affected by the breach of confidentiality of personal health information.

Standardized process for communicating with affected members: Senior KP legal and KP Online business administrators developed standardized telephone and e-mail scripts to ensure that communication with members was accurate, complete, and responsive. The scripts acknowledged that KP Online breached the confidentiality of their personal health information (PHI) through a technical error. The scripts offered an apology for the error and expressed KP's wish to understand and address their concerns. An 800 toll-free number was established for members to respond to messages left on answering machines.

Senior staff made calls: The critical situation management team asked senior administrators, nurses, and physicians to contact members affected by the breach depending on the sensitivity of the PHI involved.

Aggressive internal communications—KP employees aggressively raise the alarm upon discovering a problem. Member services call center alerted KP-IT of PHI breach response: Upon receiving the member's report, the member services call center immediately recognized the KP Online breach as a significant information technology issue and promptly notified KP-IT by telephone. This communication occurred even though the call center had never before received a call of this nature. They usually received requests to schedule or cancel appointments or to take messages for providers.

KP-IT staff sustained the chain of communication: Immediately on being notified of the breach of member PHI confidentiality, KP-IT notified the KP Online business team by telephone. In its turn, the KP Online business team immediately contacted the CEOs of KP and the Permanente Federation. As described at the beginning of this article, senior counsel chartered and documented the work of the KP Online Situation Management Team through a memo to its members with copies to the CEOs of KP and the Permanente Federation. This pattern of open communication persisted throughout the crisis.

KP-IT uses an automated “trouble ticket” information system: KP-IT used an online Enterprise Support System for internal communication, management, and tracking of efforts to address technical aspects of all its computer and network operations problems. As described above, this practice supports member well-being through prompt response to member-related issues. It also demonstrates a commitment to open communication across the enterprise because this software was available to all KP-IT staff on any workstation attached to the KP network. The recommendations flowing from review of this incident included requiring the Web Development division of KP-IT to begin using the online system to track all “trouble tickets.”

Open business practices—In order to maintain trust with its members and the public, KP communicates openly about its business practices, including its errors. KP took three steps to communicate the KP Online breach to outside audiences.

Prompt notification of affected members: The critical situation team immediately recognized the potential for the KP Online breach to harm members and negatively affect the reputation of KP. To mitigate this potential their initial concern was the notification of the members of the breach of their PHI. As explained in the memo from counsel, they organized a comprehensive plan to call, support, and respond to all members implicated in the breach of PHI.

Prompt reporting of such situations to regulatory agencies: The critical situation team also reported the breach to all relevant regulatory agencies. Although the draft HIPAA security rules were under public review in 2000, KP made this disclosure to regulatory agencies on its own volition, not in response to a regulatory mandate.

Prompt disclosure of the incidents to the public media: The critical situation team also held a press conference to publicly acknowledge the KP Online breach. Reports of the breach appeared in major newspapers across the United States, including the front page of the Washington Post. Footnote ¹¹

Resource Individuals

An organization's repertoire includes individuals with technical and cultural expertise from whom to recruit assistance in managing nonsensical situations.

KP mobilized relevant technical experts as dictated by the problem. The technical issues involved in both the malfunction of the KP Online e-mail failure and restoration of the e-mail messages from the dead letter file posed complex infrastructure, application, and programming questions and, thus, required technical expertise from several different sections of KP-IT. KP-IT division managers assigned individuals with the most appropriate technical skills and experience to investigate and resolve these problems. In the case of the e-mail failure, these teams included technical staff who routinely worked together. Clearing the dead letter file, however, required assigning programmers from separate divisions who had never worked as a team before. They had to pool their efforts because no KP-IT staff member or division had the complete knowledge necessary to fix the problem.

KP mobilized cultural and organizational expertise when managing technical failures. The KP Online security breach occurred as part of a cascading system accident that included multiple technical and organizational failures.Footnote ¹² Kaiser acknowledged the multidimensional nature of these failures by assigning staff with cultural as well as technical expertise to their investigation. The KP Online critical situation response team consisted of over 15 members including the KP online business manager, several attorneys, a physician, healthcare operations administrators, and information technology professionals. Some members of this team displayed technical and programmatic knowledge of KP Online. Others demonstrated cultural expertise through their understanding of Kaiser's values with respect to its members, clinical practice, crisis management, and public relations. KP-IT assigned a senior information technology architect to conduct a “postimplementation review” of the delayed implementation of the new prescription refill functionality for KP Online, a delay that contributed to the e-mail malfunction and security breach. The postimplementation review team identified the “root technical cause” of the delays but also highlighted several even more significant problems of business process and organizational structure among the KP-IT divisions that put the entire KP Online program at risk if not corrected. The team documented lessons learned and developed an action plan for resolving the technical and cultural problems.

Experienced legal counsel guided response. As documented in the memo from counsel described above, KP assigned legal counsel with prior experience in managing unexpected events (crises) to guide the response to this incident. Although well versed in the law, counsel functioned broadly as a senior expert in Kaiser embedded ethics and values, resource individuals, and practiced procedures—all of which are expressed in the chartering memo.

Practiced Procedures

In its years of experience managing large data centers based on mainframe computer technology, Kaiser learned many lessons and practiced many procedures for addressing critical incidents. Our informants about the KP Online breach repeatedly observed that the Web Development team applied few of these lessons while trying to incorporate innovative web technology into Kaiser's work. Yet, we observe that, when faced with nonsense emanating from the Web Development group's work, KP-IT invoked well-practiced procedures, in particular its mobilization of critical incident management teams, and began applying them to this anomalous area. Before the end of the work day on which the KP Online security breach became known, Kaiser had created a critical incident response management team. As we have already observed, this team expressed embedded Kaiser ethics, recruited technical and cultural experts as members, and implemented various specific procedures that Kaiser routinely used to manage comparable incidents. To conduct the root cause analysis of technical failures underlying the breach and delayed Rx Refill implementation, Kaiser invoked a robust postimplementation review process that bears great resemblance to the critical situation response team. The Kaiser organizational repertoire certainly included practiced incident management procedures even if the KP Web Development team ignored disciplined development procedures.

Segregation

Declaration of two KP Online-related crisis management teams established the situation and their work as outside the ordinary flow of bureaucratic life at Kaiser, particularly for the team managing the security breach. As soon as news of the KP Online breach began to disseminate throughout the organization, KP assembled a crisis management team of expert and responsible individuals, who suspended their normal work and refocused all their attention on the situation. These individuals did not routinely work together. A few but not most of the crisis team members actually worked on the KP Online project. They did not represent people to whom KP assigned routine crisis management responsibilities. Some made a forced return from leave, expressing a sense of urgency uncharacteristic of ordinary project life. All worked over the weekend to develop a response. The nonsensicality of the breach created an organizational hiatus out of ordinary bureaucratic time and space that KP peopled with the crisis management team who thus left their own ordinary temporal and spatial niches. In a similar if less dramatic fashion, members of the Rx refill postimplementation review team also interrupted their everyday activities to reflect on the reasons for the application's late launch.

Liminality

The security breach and late delivery of the Rx Refill application clashed dramatically with KP's expected mode of operations. The crisis management teams examined and proposed action to reduce these inconsistencies between the real and the ideal during their deliberations. These deliberations about the ethical foundations of Kaiser life among resource individuals mobilized in the practiced procedure of crisis management occurred “betwixt and between” undesirable nonsensical failures and restored organization sense making. Their deliberations addressed the structural conditions along the four dimensions of compartmentalized sense making then prevailing in KP-IT, including the following.

Identity: The postimplementation review team directly addressed the problem of the differing work procedures and identities among the various components of KP-IT in its “Summary of Lessons Learned.” Four lessons bear special notice. Lesson 1 reads: “Effective QA and stress testing prior to production requires that development, QA and production environments be as similar as possible and gaps well-understood.” Lesson 3 reads: “Effective response to unanticipated problems requires close teamwork between KP-IT internal organizations.” Lesson 7 reads: “Disagreements within KP-IT over standards must be resolved early (i.e., Weblogic versus Websphere).” Lessons 8 reads: “New technology requires special attention across the organization.” Taken individually and together, these lessons articulate and condemn the effects of Web Development, Operations, the E-mail group, and the IT Planning group each maintaining its own separate identity, work procedures, and priorities. Thanks to these clashes, KP-IT failed to meet its internal customers’ needs for timely delivery of a new application and harmed patient members through a breach of their privacy, both violations of core-embedded KP ethical values. The essential recommendation follows that the various components of KP-IT must surrender their individuality and work as a team.

Ongoing events: These and other lessons also addressed the major technical, management, and organizational transitions that were occurring in KP-IT at the time of the failures. Lessons 6 reads: “KP-IT must become self-sufficient in supporting complex, multi-vendor applications.” This lesson refers to the fact that KP-IT at the time had an overreliance on third-party consultants in web application development. As development platforms changed, the third-party consultants also changed, yielding general uncertainty and incidental but important errors such as untimely expiration of contracts and the failure of support at critical moments. The night of the KP Online security breach, no manager with requisite knowledge or experience was available to help either from KP-IT or the third-party vendor. The essential recommendation of this lesson follows: Install mature system development processes with appropriate testing.

Cues and retrospective sense making: The KP Online security breach and late delivery functioned as cues to the disruption of sense making in the KP Online development process and, thus, became starting points for the retrospective analyses of the crisis management teams. They articulated many lessons learned. Some were new, such as the lessons about web development explained above. Others reaffirmed embedded Kaiser values. In a study of Kaiser's response to the security brief, a student in a Masters in Public Health program outlined these key lessons learned. In managing a breach, an organization should

• • Be honest with its customers when a mistake is made.

• • Keep top management informed so that they fully support the strategy.

• • Communicate within the organization.

• • Consider a proactive approach with the media.

• • Conduct a root-cause analysis.

• • Fully support the people who work the crisis.

We see all these “lessons learned” as elements in the organizational repertoire of Kaiser Permanente mobilized during its response to the KP Online security breach and late delivery. By the time the teams had finished their deliberation, they recognized the fundamentally organizational basis for KP Online's difficulties. Although they acknowledged that certain individuals made errors, they refused to blame them. Rather, they recognized that a failure of teamwork and process among the KP-IT units during a time of technical, managerial, and organizational transition as the true origin of the problems.

Reaggregation

Reflecting on events later, two persons intimately involved with the response to the breach made separate but comparable assessments. The KP Online program manager stated, “We had no mitigation plan because this was the adolescence of web development. People thought you could abandon everything you knew in conventional IT because this was the web” (personal communication). The person who assumed responsibility for Web Development just after the security breach described the event as an organizational epiphany and turning point. She noted that, thanks to the security breach, they became aware of KP Online's value and risk. Quoting from the postimplementation review team's report, she observed how they had to end web development group's isolation from the rest of KP-IT by imposing standard corporate IT discipline on its work process, hiring more people, and working on building trust between the Web Development and Operations staff. The problem, she argued, was KP-IT's overall maturity with respect to web technology and patient information. She said, “The IT organization was not mature with respect to web technology. We did not realize the brevity of our support. If we had, we'd have had a disaster recovery plan. To create a disaster recovery plan document, you had to pull away from working on items in the queue. That's a perennial challenge for IT: Development competes with production. If you don't have processes in place, you will fail” (personal communication). Thankfully, they both noted, KP-IT had instituted an incident management process, on-call procedures, and other contingency plans. The KP Online program manager stated, “Today this wouldn't happen because we have an on-call system. We are always evolving. We keep reviewing. We now have a whole new system with split calls and subspecialists in incident response” (personal communication). Both persons use the language of growth and development. The Web Development team was “adolescent,” “immature,” and indifferent to the lessons of its elders. The breach taught them the error of their immature ways. Having realized their state of immaturity, KP-IT passed through an organizational life crisis to the next stage in its maturity.

Benediction. Reflecting on her experience during the KP Online crisis, a senior KP-IT engineer told us

Like a priest at the end of mass, the Kaiser CEO offered a benediction to his faithful staff upon closing the transition from disrupted to restored organizational sense making in the KP Online program. Through his words, he reaffirmed the core organizational value of using technological innovation for the benefit of Kaiser's members. As a source and embodiment of Kaiser's cultural universe, he validated his staff's identities as vital and effective members of that universe. As long as they kept the faith, practiced sound doctrine, and continued the struggle for progress, he exonerated them from blame for errors. With these words, he sanctified their efforts and enabled them to return to work revitalized as sensible members of the community.