Search Terms: Effects, compliance with EMD protocol, emergency medical dispatchers, emergency medical dispatch priority reference system (EMDPRS), ongoing quality assurance, extensive case review, continuing dispatch education (CDE), dispatch protocol, prehospital care, dispatcher's tools, training processes, curriculum, dispatcher's compliance, quality management systems, test center, pre-arrival instructions, Advanced medical priority dispatch system (AMPDS), study, compliance, performance, prioritize calls, ALPHA-drift, dispatch entropy, Annals of Emergency Medicine, 1998, protocols
Annals of Emergency Medicine, November 1998
Effect of a Comprehensive Quality Management Process on Compliance With Protocol in an Emergency Medical Dispatch Center
Study objective; Modern emergency medical dispatch provides appropriate resource responses with the use of an emergency medical dispatch priority reference system (EMDPRS). The EMD-PRS is a systematic protocol for all aspects of the dispatch process, including interrogating the caller, matching responses with severity, and providing pre-arrival care. We tested the hypothesis that appropriate performance feedback would increase dispatcher compliance with the protocol.
Methods: We examined how emergency medical dispatchers complied with the protocols contained in the Advanced Medical Priority Dispatch System, a commercially available EMDPRS. Six key areas and overall compliance were studied. Dispatchers performed for 2 months without feedback and for a further 2 months with performance feedback. We used statistical methods to compare the dispatchers‚ compliance with the protocols each month/
Results: The mean overall compliance score improved from 76.4% +/- 10.2% (mean +/-SD) in the absence of performance feedback to 96.2% +/- 4.0% (n=217; P<.001) when performance feedback was provided. Five of 6 key areas showed similar improvements.
Conclusion: Providing emergency medical dispatchers with regular and objective feedback regarding their performance dramatically improves how rigorously they follow a systematized dispatch protocol.
The American Society for Testing and Materials (ASTM) issued guidelines for emergency medical dispatch (EMD), EMD training and certification, and EMD management in designations F1258-95, F1552-94, and F1560-94 (1-3). In these, the ASTM recommended not only that all emergency medical dispatchers use a medically approved emergency medical dispatch priority reference system (EMDPRS) but also that ongoing quality assurance activities include extensive care review and regular feedback to the emergency medical dispatcher. We hypothesized that appropriate feedback to emergency medical dispatchers, combined with continuing dispatch education (CDE), would lead to consistently high levels of compliance (defined as "use exactly as written or directed") with the dispatch protocol, resulting in coordinately high reliability and consistency in response accuracy.
Beginning in many places as a clerical-level position with little involvement in patient evaluation or care, the EMD professional has changed dramatically since the mid-1970s and now plays a pivotal role in prehospital care (4). This development has been paralleled by partial maturation of both the dispatcher's tools (protocols) (3) and training processes (curriculum) (1,4). Discussions regarding the emergency medical dispatcher's ability to effectively manage the deployment of mobile EMS resources and to provide pre-arrival care of the patient have been increasingly prevalent in the EMS and public safety literature (5-14).
Paradoxically, there is limited validation regarding design, process, treatment, or outcomes related to EMD in the medical literature. Many studies (12-14) purport to test whether various protocols and algorithms assist the dispatcher in making better resource response and treatment decisions. However, these studies have generally failed to measure or report dispatcher's compliance with the EMD protocol, without which it is impossible to determine whether the favorable outcomes of a study were the result of the protocol or of the decision making ability of the dispatchers. In varying degrees, failure to quantify compliance with the protocol reduces the investigators‚ ability to correlate decision outcomes with the protocol: the dispatcher's "interpretation" and modification of the formal protocol introduces an uncontrolled variable that significantly reduces the validity of many reported findings. That many studies fail to quantify compliance with the protocol is surprising given that several regulatory agencies and professional organizations have subsequently published recommendations for the quantification of dispatcher compliance with the policies and procedures that govern the effective use of EMD medical triage systems (2,15-18).
To ensure optimal delivery of medical care, delivery systems must meet industry-established benchmarks; this is a goal that can be attained only through the ongoing use of appropriate quality management (QM) systems. Any EMD QM process must include case review by advanced life support (ALS)-level emergency medical providers and other medical professionals and the use of other prospective, concurrent, and retrospective QM tools (2). Unlike field providers, who record events only after they have assisted the patients, the performance of emergency medical dispatchers is audiotaped during the call. The dispatcher's performance therefore can be measured precisely by comparing it with a systematized or scripted protocol. Quantifying how dispatchers are using an EMD system is essential to the study of the system. Only through consistently high-if not absolute-compliance with a protocol, or to the specific component being studied, can the results obtained be directly attributed to the protocol or component itself. We proposed that a properly designed and uniformly applied QM process, which includes quantification of protocol compliance and feedback to dispatchers, would result in improved compliance with the protocol. Better compliance allows more rigorous evaluation of the protocol itself, and when emergency medical dispatchers follow a valid protocol precisely, medical decision accuracy and efficient utilization of resources are maximized.
The test center, a combined police-fire-EMS dispatch center in the US Pacific Northwest, internally evaluated its medical dispatch process. Having practiced a type of protocol-driven EMD since 1984, the test center initially believed that its performance was adequate. However, when we interviewed the test center's communications staff, both management and dispatch support staff acknowledged that actual use of the protocol was "rare and sporadic." The test center's administration determined that dispatcher's compliance with the protocol needed to be improved in the general areas of caller interrogation, clinical coding, and pre-arrival instructions. The test center decided to upgrade its EMD protocol and selected a commercially available EMDPRS, the Advanced Medical Priority Dispatch System (AMPDS) (maintained by the NAEMD and available from Medical Priority Consultants); they additionally decided to implement a comprehensive QM program. This study examined the compliance-with-protocol performance of a group of dispatchers during the 5 months following implementation of a protocol-driven EMD system, the AMPDS.
Materials and Methods
Training on the use of AMPDS was completed by the test center in July 1993. After a 1-month gap to allow the dispatchers to adjust completely to the new system, we began formal monitoring of their compliance with the protocol. During September and October 1993, we evaluated dispatchers' performance in the absence of any feedback. At the end of October, a performance-feedback and CDE program was started; monitoring continued until the end of December to evaluate its effects.
The AMPDS uses scripted caller interrogation protocols to provide symptom-based (rather the diagnosis) information with which to prioritize calls and allocate resources. It features the following:
The AMPDS also facilitates the QM process through its automated case-review software, which calculated dispatcher compliance with the protocol in these 6 key areas:
The "total quality management" process adopted by the test center included the following components:
After the test center had installed the new EMD system, we monitored all aspects of their QM program. The center established an organized EMD management and oversight committee structure before starting dispatcher training. Training included all the goals and learning objectives described in the ASTM standards document (1) and required by the accreditation guidelines of the National Academy of Emergency Medical Dispatch (NAEMD) (19). The center established operational policies that clearly defined management‚s position on compliance and set acceptable lower performance limits of 90% or 95%, depending on the protocol component being assessed. Percentages were calculated using a scoring formula approved by the Board of Accreditation of the NAEMD and contained in AQUA-a specially designed, commercially available case-review database (Medical Priority Consultants).
The case-review process is central to this study. Case review consisted of evaluation of EMD compliance with the protocol for each of its 6 key areas and calculation of an overall compliance score. An ALS-level paramedic was selected to serve in the quality improvement unit as an EMD case reviewer. The case reviewer was further trained in case-review procedures, which included audiotape review, use of a grading standard, critical listening skills, and use of case review scoring software. The reviewer compared the dispatcher's interrogation of the caller with the scripted series of questions listed for each of the 32 chief complaint interrogation protocols. The dispatcher's interpretation of the caller's responses was assessed for agreement with the reviewer's interpretation. To ensure review impartiality, an additional reviewer from the NAEMD was selected to re-review 10 randomly selected cases per month.
The reviewer reviewed a random selection of approximately 10% of each day's cases as well as every request for service that required the provision of scripted pre-arrival instructions (ie, those calls involving respiratory and cardiac arrest, choking, or emergency childbirth). For purposes of review, cases were identified by incident number only. During September and October, the review information was not shared with the individual dispatcher, but at the end of October, November, and December, direct written and one-on-one verbal feedback concerning compliance-with-protocol performance scores were provided to the dispatch staff individually. The dispatchers were encouraged to compare their own performance with the NAEMD accreditation requirements and with the center and shift average scores.
In October, the test center also initiated serial CDE, which included review of the AMPDS and the policies and procedures that governed its use within the communications environment. These activities were specific to the use of the protocol and therefore were not viewed as confounding factors in assessing the results of the QM activities.
Results from the study were compiled as monthly reports containing the number of cases reviewed for each dispatcher and scores for the 6 key areas and for total compliance. Owing to the random nature of the case selection, some of the 32 dispatchers at the test center were not reviewed in every month and others received multiple case reviews. We inspected the data to ensure that no bias was introduced as a result of multiple reviews of a small subset of the dispatch staff. We then prepared an average score for the entire dispatch staff for each of the 6 compliance scores and an overall score for each month of the study. Using the null hypothesis that there would be no differences in scores from month to month, we performed 2-tailed z-score tests on the mean scores for pair-wise monthly combinations.
Because proportional data are not usually normally distributed, we repeated the z-score test using an angular transformation of the data. We also compared the median scores of the first and last months‚ results using Wilcoxon‚s rank-sum test. Finally, because some dispatchers had cases reviewed in each of the 4 months of the study, their data could be viewed as partially matched. We examined the scores for just those dispatchers who were evaluated in all 4 months of the study, using paired Student‚s t tests. Statistical tests were performed with Microsoft Excel 97 software.
All 32 communications center dispatchers successfully completed the EMD course and received NAEMD certification on the use of the AMPDS. Approximately 100 cases were reviewed each month, including 25 to 29 dispatchers with an average of approximately 4 cases per dispatcher. The additional (NAEMD) reviewer did not identify any discrepancies between the in-house reviewer‚s interpretation of the re-reviewed cases and the grading standard. The scores for those dispatchers who received multiple reviews did not bias the overall scores for the center.
Compliance with the protocol, calculated using the AQUA case-review database, provided a percentage score for each reviewed case. In this calculation, the dispatcher began each case with a score of 100% in each of the 6 key areas and lost points for deviations from the protocol. Full details of this procedure are available on request from NAEMD. The overall compliance score was calculated as the simple average of the 6 key area percentage scores. In total, 440 randomly selected cases were analyzed for statistical significance, using a 2-tailed z-score test of sample means for each month of the study period.
After initiation of feedback and CDE, overall compliance increased from 76.4% +/- 10.2% (mean +/-SD) in September to 96.2% +/- 4.0% (n=217; P<.001) in December. Each of the protocol categories was tested separately for statistical significance (Table). Of the 6 key-area scores, all except chief complaint selection increased significantly. An angular transformation of the data resulted in distributions that were less skewed; repetition of the z-score test with these data revealed very similar trends (not shown). The paired t tests also confirmed the dramatic improvement in compliance with the protocol that occurred in December (not shown).
Comparison of the median scores using Wilcoxon's rank-sum test also revealed dramatic improvements in compliance with the protocol after feedback and CDE. The median of the overall compliance score increased from 75.8% in September to 97.5% in December (P<.001). Scores for case-entry interrogation, key question interrogation, determinant code selection, and post-dispatch instructions all improved dramatically.
This study evaluated how rigorously a group of 32 emergency medical dispatchers complied with an EMD protocol. During the first 2 months of the study, the dispatchers did not receive CDE or feedback on their levels of performance; during the last 3 months, they were given feedback concerning their previous performance and were provided with CDE.
This study examined overall compliance with the protocol and compliance in 6 key areas (which together covered all aspects of the dispatch process). These key areas are described as follows:
Case-entry Once a caller reaches the dispatcher, the protocol immediately directs the dispatcher to determine the incident location, type of presenting condition, and likelihood of life-threatening conditions (primary survey).
Chief complaint selection The dispatcher selects a chief complaint subprotocol routine from among the 32 chief complaint categories in the protocol (preliminary working diagnosis).
Key questions The key questions are 4 or 5 closed-ended questions for the selected chief complaint (secondary survey).
Determinant code selection The determinant code selection (which permits prioritization of response of response resources) is driven by information obtained through key questions evaluation.
Post-dispatch instructions Once the dispatcher has initiated a response, the protocol directs the dispatcher to provide generalized instructions (basic first aid, cautionary statements, and verifications) related to the chief complaint.
Pre-arrival instructions The pre-arrival instructions are scripted instructions for life-threatening incidents that cover verification and management of respiratory and cardiac arrest, choking, emergency childbirth, and problems with airway, breathing and circulation.
Overall and key area dispatcher compliance data were analyzed for the study period (Table). Mean compliance with the protocol improved significantly from the beginning to the end of the study in 5 of the 6 key areas and overall. The sixth key area, chief complaint selection, did not improve significantly.
In this study, overall compliance was at about the 80% level 2 and 3 months after training in the absence of any feedback and CDE were provided (November) and further improved to more than 96% during the second month of feedback (December). Of the 6 key areas, 3 (key questions, determinant code selection, and post-dispatch instructions ) were at unacceptably low compliance levels in the initial 3 months after training (65.7%, 78.8%, and 41.2%, respectively, in September). Once feedback and continued training were provided, better than 90% compliance was achieved in all these areas. Compliance in the other 3 key areas (case-entry, chief complaint selection, and pre-arrival instructions) remained at about 90% during the first 3 months of the study. After feedback and CDE were provided, further, statistically significant improvements to almost 100% compliance were obtained in 2 of these areas, case-entry and post-dispatch instructions. The only key area in which significant improvement did not occur during the study period was chief complaint selection (i.e., identification of the correct chief complaint).
The absence of significant improvement in the selection of the correct chief complaint could have resulted from 2 influencing factors. First, the compliance score for this area was relatively high (89.0%) at the beginning of the study. The experience and training of dispatchers influences their ability to select the chief complaint. The dispatchers at the test center had preexisting experience with EMD, and the complaint types in their previous protocol were similar to the complaint categories in the AMPDS. This experience appeared to enhance their ability to interpret the presenting complaint. Second, the ability to correctly select the chief complaint is less affected by QM feedback because it appears to be a more interpretive skill than the other components of the protocol. Because the scoring for this area is based on whether the dispatcher identified the correct chief complaint rather than on how the dispatcher reached a conclusion, the dispatcher‚s interpretive skills almost certainly biased this area of the study. QM feedback influences only how rigorously the dispatcher complies with the protocol, so it is not surprising that there were smaller effects after feedback in this key area.
Before implementation of QM processes within the medical dispatch environment, an old dispatch axiom unfortunately held true. Question; "How do dispatchers know when they're doing a good job?" Answer: "Nobody says anything!" But the improvements in compliance with the protocol observed in this study validate an equally true modern management axiom: most employees want to do a good job. In the great majority of communications centers, however, dispatch personnel are not provided with specific information regarding their performance or with the support necessary to improve.
We have observed a phenomenon referred to as "ALPHA drift," which appears shortly after implementation of an EMDPRS. ALPHA is the lowest response assignment in the AMPDS, corresponding to a "cold" (non-lights-and-siren) response by a basic life support (BLS) unit. Between 30 and 60 days after implementation of the EMDPRS, in the absence of QM feedback, the proportion of ALPHA assignments begins to drop; conversely, the proportion of BRAVO, CHARLIE, and DELTA (ALS and/or hot) assignments begins to rise. It is speculated that, in the absence of QM feedback, dispatchers lose confidence in the EMDPRS and begin to select higher responses than are necessary. This phenomenon has also been called "dispatch entropy," because dispatchers progressively comply less stringently with the protocol.
The absence of QM processes, which often results from management fears regarding employee response to objective measures of performance, allows phenomena such as ALPHA drift to continue unchecked. Before starting case review and feedback, management teams are concerned that dispatchers will be reluctant to accept QM feedback because performance review is perceived as disciplinary action. This attitude reflects the fact that in most dispatch centers case review is sporadic and usually occurs in response to problematic cases-often resulting in negative feedback to the dispatcher or in some form of punitive action. To be effective in modifying dispatcher performance and to change the perception of its use as a punitive action, feedback must be objective, be nonarbitrary in the selection of reviewed cases, and occur on a continuous and scheduled basis.
This study strongly suggest that objectivity concerning dispatcher performance and provision of performance information as meaningful feedback result in dramatically improved dispatcher compliance with the protocol. This is a QM process that closely resembles those found in industrial settings, where statistical process controls are used to improve quality. The reliable quantification of human activity can serve the organization in a variety of ways, but in today's changing health care environment it has significant value because of its ability to provide validation of organizational processes. As health care providers continue to struggle with price-quality issues, this study illustrates how health care activities can be quantified and how those data can be used to improve health care delivery.
The results of this study support the hypothesis that appropriate QM processes result in improved dispatcher compliance with EMD protocols. Even without statistical analysis, the improvements in the average scores in 5 of the 6 key areas and in total compliance are striking. However, we recognize that there are some potential weaknesses in this study.
Statistical analysis of proportional data is hampered by the data not being normally distributed. Transformation of the raw data to a form that has a more nearly normal distribution, followed by repetition of the analysis with statistical tests that are less sensitive to skewed data, lends additional weight to our argument that the improvements in mean compliance scores were dramatic and significant. Second, the use of a review process that depends on case reviewers in potentially subject to reviewer error. In an effort to limit this possibility, we employed a member of the College of Fellows of the NAEMD to re-review a selection of the cases. Because no disagreements were reported between the 2 reviewers, we believe that the chosen case-review process was justifiable. Also worth mentioning is the possibility that some deviations from the protocol were actually appropriate under the circumstances. Although this is possible, the AMPDS is a well-established protocol, so appropriate deviations will be very rare occurrences. Finally, the effect of CDE could be considered a confounding factor in this study. Although we cannot rule out a contribution from CDE, our observations at other centers, in which there was ongoing CDE but no performance feedback, weaken this claim.
In the test center, overall compliance percentages, and compliance scores in 5 of the 6 key areas of the protocol improved significantly during the study. Performance feedback appeared to play a significant role in this improvement. To date, the center (an NAEMD-accredited "center of excellence") continues to achieve compliance-to-protocol scores consistent with those observed at the end of the study period; in December 1997, overall compliance was reported to be greater than 99%. The results of this study lead us to conclude that a properly designed and uniformly applied QM process-one that includes quantification of compliance with the protocol and direct feedback to trained dispatchers-results in improved overall compliance with all components of an EMD protocol. Improved compliance with the EMD protocol results in accurate and appropriate emergency medical resource allocation.
We acknowledge the invaluable assistance of Doug Smith-Lee, Emergency Medical Services Manager, and Thera Bradshaw, Director, Clark Regional Communications Agency, and of Lynn Wittwer, MD, Director, and Marc Muhr, Assistant Director, Clark County Medical Program, Vancouver, Washington, for sharing with us details of the quality improvement program and the compliance-with-protocol performances of the Clark County dispatchers.