Evaluating medical devices and systems is becoming increasingly complex. Not only must the usability of a device itself be evaluated, but also the software-based accessories and information systems that will be connected to the product under evaluation. Because of the variability among hospital environments, such as pre-existing systems and equipment, and different policies and procedures, it is nearly impossible to identify any one vendor’s solution as being the best for everyone.
The vendor selection for products that are advancing the state of the art present a particular challenge to providers. When contemplating a capability that has never existed before, how do you best define such a product or evaluate the most appropriate solution? For example, the optimal way to incorporate smart pumps into a closed-loop medication administration system may not be known for another 3 to 5 years—until the industry has had sufficient real-world experience with this new and complex application.
Under the conventional equipment evaluation process, a multidisciplinary team discusses requirements and participates in the vendor selection process. The typical process can include requests for information, requests for proposals, numerous sales calls, and vendor site visits, and it usually ends with on-site evaluations of the top one or two vendors under consideration.
The on-site evaluation is becoming a weak link in the conventional equipment evaluation process when complex systems are under consideration. An on-site evaluation can certainly reveal how effective the medical device itself may be in a specific clinical environment. This evaluation tool breaks down with purchases that require additional peripherals like personal data assistants or computers on wheels, application software installations, and, especially, systems integration with other hospital information systems. All of these components—that extend beyond the actual medical device—come at considerable expense and time required for systems integration. The result is frequently an on-site pilot that represents only a small fraction of the total system under consideration, leaving the buyer in the dark as to how the complete solution would perform in their environment.
At some hospitals, human factors engineering is also used to determine the superiority of a prospective system. Using sophisticated tools like contextual inquiry, cognitive walk-through, heuristic analysis, and other tools, human factors engineering can be very effective at evaluating a product’s usability and inherent safety.
Where both the conventional equipment evaluation process and human factors engineering fall down is in the consideration of workflow.
The Importance of Workflow
Workflow is a description of the steps taken to complete a task. These steps can be manual, automated, or a combination of both. Depending on the scope of the task at hand, a workflow can include many important subtasks. It frequently becomes such a normal part of the user’s environment that it becomes invisible, like the air we breathe. But, as soon as something changes the workflow, what was once a comfortable and automatic process is now interrupted and workflow can become a major issue. Workflow is so important that in some cases it can determine the ultimate success of a new acquisition by users.
In addition to the design and operation of a medical system itself, many other important factors influence workflow, such as the use of accessories, disposables, other products from different vendors, and the system’s location. External systems or processes that seemingly have nothing to do with the workflow under consideration may exert considerable influence.
In response to the growing impact workflow is having on the ultimate acceptance of equipment purchases, many hospital biomedical/clinical engineering departments have an opportunity to step in and leverage their clinical focus to better account for workflow in vendor selections. An appreciation of the value of workflow is growing. Hospitals are increasingly interested in improving workflow through process reengineering methodologies like Lean and Six Sigma.
There are two sides to workflow: the current workflow based on using existing products, policies, and procedures; and the new workflow that will result from the vendor selection decision. Workflow impact extends beyond the product and will include new policies and procedures necessitated by changes from the status quo by the new product.
While the multidisciplinary vendor selection team has an opportunity to gain a deep appreciation for why a particular solution is chosen, the users of that new solution will use their current workflow as the benchmark against which they will rank any new solution.
Documenting current workflows through use cases is critical to identifying the potential solution that best matches an organization’s needs. When the system under consideration is purchased and installed, users will compare how things worked before. The only way to ensure that the new solution entails fewer steps, improved usability, and productivity is to document in detail the user’s current workflow.
Defining Use Cases
Perhaps the easiest and most effective means to capture workflow is the use case method. Created through direct observation and interviews with users, use cases capture a process within “a day in the life” of the target users. Use cases provide a straightforward way to capture workflow in the form of a narrative, or day-in-the-life description of activities around a specific activity. The result captures a sequence of events described in a terminology-free manner. The goal is to describe the process that is used by the actors (people or systems) to achieve the desired outcome.
Use case development can establish a baseline against which users will judge the quality of the winner of the equipment evaluation process. Creating use cases can also provide a common basis for stakeholders to consider what is being implemented.
Effective tools in helping a facility better understand its own requirements, use cases also provide important new selection criteria; help communicate requirements to prospective vendors; provide structure for systems analysis and configuration during installation; help develop verification test plans to ensure the resulting purchase is in fact installed and configured properly; and are easy to create and use because no expert knowledge is required, unlike some human factors engineering methods.
How to Create Use Cases
Use cases fall into two general categories: informal and formal. Informal use cases are based on narrative descriptions that may be only a few pages long. The formal category is also a narrative, but it is organized in a much more structured way. There are several components to use cases, each used to organize the narrative as information is collected.
Actors are the people and systems that participate in the workflow. They each have a specific role and designation such as physicians, technicians, nurses, and unit clerks. Include on-site vendors that will participate in the workflow under evaluation. Actors can be played by different types of people, so be sure to focus on role and function, rather than job title.
Triggers are the events that cause the initiation of a use case. These events can be external, temporal, or internal. Using the smart pump example, triggers could include a new order (external), a specific time or time period for therapy adjustment (temporal), or simply running out of fluid and needing a new bag (internal). Preconditions are the conditions that must be true before a trigger can cause the initiation of the use case. These could be events like a diagnosis resulting in an admission or the criteria that triggers a discharge.
The use case should describe the basic course of events, the 30,000-foot view of a process. This high-level narrative becomes the framework for the resulting use case.
Care delivery can be fast-paced and rapidly changing. Based on situations that arise, alternate paths in the use case may require capture and documentation. These alternate paths include exceptions or when things go wrong. Depending on the scope of an alternate path, it may be incorporated into the overall use case, or require a separate use case of its own.
Post conditions describe the change in state after the use case completes. Post conditions should be captured for those key actors that are affected by the use case. Patients, clinicians, and support staff are obvious actors for which post conditions should be described. Some information systems may also entail an important post condition, like an entry in an electronic medication administration record that an infusion therapy has been completed.
The last major component of use cases are business rules. These are the policies and procedures that can trigger the initiation or end of a use case, as well as cause the creation of alternate paths. In addition to being written policies and procedures, business rules can also be “rules of thumb,” or undocumented operating procedures. Some business rules will consistently apply across the entire use case, while others apply only to certain alternate paths. Be sure to capture all of the business rules because whatever vendor’s system is selected, it will have to support all of the facility’s current business rules or provide acceptable alternatives.
Use cases sound great on paper, but like life, there are always exceptions that do not fit within the use case framework. Be sure to note these exceptions where and when they occur in the use case. Footnotes make an excellent way to capture these types of variances.
How to Apply Use Cases
Once the use cases for a specific equipment evaluation are captured, they represent a baseline against which to gauge the workflows supported by the vendors under consideration. Use cases can also provide excellent requirements for requests for information and requests for proposals.
Many requests for information, and especially requests for proposals, are based on methodologies developed prior to the advent of sophisticated medical device systems that may also integrate with hospital information systems. These evaluation tools are oriented to determining the product with the best features and specifications, and provide little or no information regarding which system will best match the workflow needs of an institution.
Implementing use cases presents a challenge in most hospitals because staffing in both biomedical engineering and nursing is usually very tight. The IT department typically is the only hospital department with resources dedicated to project management. So while IT may have the resources for such an effort, it typically lacks the clinical background necessary to capture good use cases.
If the resources are available, clinical/biomedical engineering represents a natural department to support this activity. Biomeds are both technical and analytical like IT, but they also have the familiarity and understanding of clinical environments that IT usually lacks.
Read about human factors engineering in “Engineering Detectives,” September 2008.
Nursing may have its own nursing IT resources, clinical nurse specialists that cover the area targeted for the system being evaluated, who may be interested in developing use cases. A small team of two to three people can be very effective in developing use cases.
An evaluation of past projects, user acceptance issues with new systems, or unanticipated process reengineering to accommodate new systems are all indicators that use case development might make a meaningful contribution to your institution.
Tim Gee is a connectologist and principal at Medical Connectivity Consulting, Beaverton, Ore, www.medicalconnectivity.com. For more information, contact .
Create a Use Case
Getting started with a use case is not as hard as it seems. The sample “day in the life” high-level narrative that follows lays the groundwork for the development of a formal use case for a nurse report at a shift change, which can be valuable in evaluating point-of-care applications like EMRs, smart pumps, nurse call, and alarm notification systems.
- Day shift charge nurse Nancy notes that her shift ends in 2 hours. She reminds her staff to complete their shift-change worksheets for their patients.
- Tom Smith, charge nurse during the next shift, arrives 45 minutes early and reviews patient status, problems, and upcoming tasks with Nancy. Tom then reviews his nurse to patient assignments from yesterday and makes adjustments for admissions, discharges, and changes in staff or patient status. The resulting assignments are given to the unit clerk, who posts them on a marker board.
- As the various nurses arrive, they read their assignments from the marker board, seek out the nurse(s) caring for the patients they will have on the next shift, and “take report.” The nurses review the shift report together.
- Upon the completion of reviewing patients with nurses for the next shift, each nurse may sign out.