Our article in the November issue of 24×7 presented a case report of a system failure that illustrates a growing concern—the hazards associated with the integration of biomedical technologies into information systems. We described how an ECG archiving system crash—possibly initiated by a virus attack and exacerbated by several organizational and technical vulnerabilities-propagated into a system failure that impacted a hospital’s ability to provide vital patient information to caregivers. The complexity of the crash was such that repair was lengthy and to this day incomplete. There were many root causes of this event. We will mention those briefly, but will concentrate primarily on a series of suggestions for how biomedical and information system organizations can collaborate better. The purpose of that collaboration is to create barriers to such failures and opportunities to better serve our health care systems by designing robust, integrated networks that effectively and safely move data, signals, and information; and provide the connectivity demanded in modern health care. We suggest that the reader review the previous article to appreciate the source of these suggestions.
We have not been able to identify the exact steps by which the system failure occurred. But, in reviewing the event, we ascertained weaknesses in system design, thorough documentation, and our own incomplete definition of communication pathways between the biomedical and information systems organizations. Lack of a backup system for the main archiving hardware hampered the recovery process. Capital budget constraints on hospital equipment can drive purchases without inclusion of such systems. Also contributing to the failure process was that this system was at the end of its life cycle yet still entirely functional. Funding for its replacement had been approved, but was delayed for several reasons, not the least of which was challenges in managing legacy ECG carts requiring connectivity. Before the replacement could be implemented, the system spiraled into its failure. The system was running on the Windows NT platform for which security patches were no longer available, leaving it vulnerable to attack. The manufacturer’s slower than desired response time in validating urgent security patches frequently left the system at risk of virus infection and may have contributed to its initial vulnerability. Perhaps most critical was the custom interface that had been developed to enable improved workflow and billing: a turnover of personnel who had developed the interface combined with a lack of complete documentation contributed to the delayed resolution.
Deeper root causes also existed. The lack of industry standards that define connectivity requirements for medical devices to health care information systems makes new interface development more challenging. And, the differing cultures within biomedical and information systems organizations likely contributed to the communication difficulties.
In many ways, managing integrated biomedical technology and information technology systems is much more challenging than managing either alone. In our discussions, based on the responses from readers of the article and our own analysis, the following themes and suggestions emerged:
- Organizations should create a biomedical engineering (BME)—information systems (IS) board for the purpose of evaluating and approving new clinical technologies and medical device systems such as those that have medical devices connected to hospital networks. The primary responsibilities of this board would be to evaluate the technical and clinical appropriateness of any new systems that have an impact on clinical care, and to identify the proper groups and individuals who need to be involved in the decision-making.
- This group should also help delineate responsibilities in dealing with special-case systems, for example clinical applications on vendor-provided hardware delivered as part of the equipment purchase that require connections to the IS network in addition to the biomedical network.
- Organizations should consider how to organize the reporting and communication structures of the BME and IS departments to ensure effective joint planning and communication. In some instances, organizations have decided to merge these departments. In other instances, the convergence has been accomplished through liaison roles and better communication and planning.
- Hospitals and/or health care systems should identify individual(s) who are responsible for the administration of integrated device-IT systems. These are likely best managed by the BME organization, but that will vary depending upon the size and capabilities of the IT and BME organizations. The responsibility should be concentrated rather than diffused.
- BME and IS professional organizations should actively work with device manufacturers and the FDA toward ensuring the rapid deployment of security fixes and software updates to the underlying operating systems. These updates would protect both the biomedical systems and the networks they reside on from crashes and intrusions.
- Appropriate service-level agreements should be established with vendors, inclusive of penalty clauses, that address how software patches will be applied, by whom, and in what time frame. Performance measures should also be included that address not only the equipment performance but also the associated software and interfaces.
- BME and IS should insist on the necessary investments to maintain adequate test environments and the necessary equipment for disaster recovery.
- Configuration management of BME software should be held to the same standard and processes employed by the IS organization.
- BME and IS should work to formalize their technical expectations of manufacturers in an attempt to manage the rapid rate of change of products, instead of being forced to continually adapt and react to the latest offerings.
- BME and IS should address the situation of medical device systems requiring access to the IS network and determine whether establishing a set of general technical requirements can alleviate these situations in the future.
- BME and IS should prepare for the possibility that the FDA will regulate either or both disciplines serving in the roles of technology integrators and network administrators. We should operate with the expectation that regulation will happen, and investigate what likely needs to be done to comply with future regulations.
- BME and IS should identify a comprehensive set of drawing standards and symbols, based on existing standards, to facilitate technical communication within and between the two groups.
- Each BME system should be documented with a full system diagram with an emphasis on interfaces and triage, testing, and disaster recovery procedures.
- Incident response procedures should include access to on-call lists for BME, IS, and vendor contact information among one another. Escalation procedures should be documented and as well understood for interconnected systems as they are for other individual BME or IS systems.
See the related Networking article: “System Crashes: A Case Report.”
Creating a secure, available, and reliable biomedical equipment and systems environment will require intense and robust cooperation among BME, IS, vendors, and the FDA. We players on the field of device/network integration should not operate in silos. We must approach our challenges as a cohesive team whose goal is to ensure that integrated biomedical systems are designed, implemented, and supported with an understanding of the inherent risks of any new technology. It is only in that way that we will achieve the promise of connectivity without compromising safety.
James Noga, BS, MS, is the chief information officer, Massachusetts General Hospital, Boston; Patricia Volpe, BSBE, MBA, is director of biomedical engineering, Massachusetts General Hospital; and Jeffrey B. Cooper, PhD, is director of biomedical engineering, Partners Healthcare System, and professor of anaesthesia, Harvard Medical School, Boston. For more information, contact .
A Reader’s View
In 24×7‘s November “Networking” article, “System Crashes: A Case Report,” the authors—Jeffrey B. Cooper, Patricia Volpe, and James Noga—asked readers to send their recommendations regarding the topic. The following is a response by 24×7 reader Barry Kohler:
That was a very relevant article. It highlights what will probably be a major responsibility of clinical engineering departments if they want to stay relevant. There are two issues that need to be resolved at the manufacturer level:
- The ability to accommodate the latest security fixes; and
- The ability to accommodate 3- to 4-year planned obsolescence from all players in general computing.
There has been some motion in the past few years concerning antivirus software compatibility. I hope that all your readers are pursuing this and asking their vendors to do likewise. Not all vendors share that information willingly, and some don’t look into it at all. Our IT department recently changed antivirus vendors to one that most of our medical equipment manufacturers have not certified. As for planned obsolescence, it would be nice if we could have performance metrics for hardware that the FDA would accept as being justification for a comparable or superior component. As for the OS, I think that gorilla is way too big to expect any changes soon.
In our hospital, we have had some major organizational changes in the last few years that acknowledge the overlap of skill sets and responsibilities. The most important was to have the biomed department answer to the CIO before the CEO. With fewer people to go through before a common administrator, it is much easier to balance department skills and priorities with those of the clinical staff.
There have been successes and setbacks toward a better working relationship between clinical engineering and IT. I would recommend to anyone who can, try to rotate people between departments so they can become more conversant with each other and work more like a team toward a common goal.
I recommend keeping all of our medical equipment on completely separate networks due to security lapses that cannot be easily overcome. The number of interfaces should be limited. Many vendors advertise that they can use the hospital backbone to transmit information between devices. It saves some upfront infrastructure cost and time to implement, but, unfortunately, many manufacturers tend to add options that need a separate interface like one for HL7 and another for a Web server. I also prefer to avoid generic PC-based equipment simply because of the issues above. Again, I would recommend that your readers take the time to emphasize their priorities the next time they see a salesperson in their building.
Besides passing recommendations to salespeople, it would be nice if committees within AAMI, application vendors, and perhaps ECRI would assemble to work out the discrepancies between cost containment, high reliability expected by the FDA, and rapid deployment of innovation that pervades the general computing field.