Abstract
Clinical audit has an important role as an indicator of the clinical practice in a given community. The European Respiratory Society (ERS) chronic obstructive pulmonary disease (COPD) audit was designed as a pilot study to evaluate clinical practice variability as well as clinical and organisational factors related to outcomes for COPD hospital admissions across Europe.
The study was designed as a prospective observational noninterventional cohort trial, in which 422 hospitals from 13 European countries participated. There were two databases: one for hospital’s resources and organisation and one for clinical information.
The study was comprised of an initial 8-week phase during which all consecutive cases admitted to hospital due to an exacerbation of COPD were identified and information on clinical practice was gathered. During the 90-day second phase, mortality and readmissions were recorded. Patient data were anonymised and encrypted through a multi-lingual web-tool. As there is no pan-European Ethics Committee for audits, all partners accepted the general ethical rules of the ERS and ensured compliance with their own national ethical requirements.
This paper describes the methodological issues encountered in organising and delivering a multi-national European audit, highlighting goals, barriers and achievements, and provides valuable information for those interested in developing clinical audits.
Chronic obstructive pulmonary disease (COPD) is a significant cause of morbidity and mortality in Europe and a major consumer of resources in both primary and secondary healthcare [1, 2]. The disease has an increasingly high profile with health authorities, health insurance companies and healthcare providers. In this regard, clinical audit is potentially a vital tool in assessing clinical practice in this chronic debilitating disease. Audits of patient care have been extended to measure the organisation of care and the resources dedicated to COPD care in acute units. Over the last 10 yrs, several audits have been carried out in individual countries highlighting important information about the delivery of care to COPD patients and the structure of the hospitals serving them. The first national COPD audit was developed in the UK in 1997 [3]. Others have followed, including Spain [4], Scandinavian countries [5] and Australia [6].
These audits provide growing evidence that the quality of COPD patient care varies widely between different hospitals and between different countries, and is frequently not consistent with published guidelines. Additionally, the organisation and resource provision for COPD varies considerably from unit to unit, and there is no mechanism for either identifying or disseminating examples of high-quality care or innovation in service delivery. In this context, it remains unknown which national systems deliver the best practice for different aspects of patients care. In all likelihood, we can all improve the care of COPD patients if we have better knowledge of our performance and gain understanding of the factors associated with better patient outcomes. However, there is not yet a culture of participation in audits across and within most European countries to provide a basis for such comparisons. Notwithstanding, the technology to facilitate an audit of this kind is available and large quantities of relevant data could be collected and reported to clinicians.
Being aware of this scenario, the European Respiratory Society (ERS) developed the first European COPD Audit as a pilot study to evaluate clinical practice as well as clinical and organisational factors related to outcomes for COPD admissions across Europe. In the present article, we describe the methodology used to perform this audit and the challenges of auditing across differing healthcare systems and countries.
METHODS
The audit was designed as a prospective, observational, non-interventional cohort trial over a defined time period, in which 13 European countries participated.
Governance
The ERS managed the audit, named a steering committee, which reported to the ERS executive committee, to oversee the process and assigned a project manager to help the steering committee develop the audit. The steering committee was formed by three respiratory physicians with expertise in COPD and clinical audits from three different countries: C.M. Roberts (UK), S. Hartl (Austria) and J.L. López-Campos (Spain). Each national society named one or two national experts to coordinate the audit in that particular country and to represent their views. Altogether, the steering committee, the project manager and all national experts formed the expert panel, an operational group that was responsible for ensuring the success of the data collection and which provided feedback on the process and suggested improvements through regular face-to-face meetings and teleconferences. Within each national participating society a number of investigators were appointed, each operating at an individual hospital level and responsible for local data collection on patients and organisation of care (fig. 1).
Governance of the audit. ERS: European Respiratory Society.
Funding
Central funding of the project was entirely granted by the ERS covering all costs at a European level. Expenses at a national level were not covered. The project gave freedom to national experts to raise funds to cover expenditure to develop the project in their own country according to their ethical regulations.
Selection of the participant countries
During 2009, contact with National Respiratory Societies across Europe was established by the steering committee with a proposal to participate. Those interested attended a meeting that took place during the 2009 ERS Annual Congress in Vienna, Austria. Subsequently, the steering committee provided information about the project to all members through the Forum of European Respiratory Societies. Two major conditions were set for national societies to participate: 1) the logistic capacity to provide the administrative structure for a national organisation of local investigators; and 2) the financial resource to support the national audit process. The national societies of 13 countries agreed to participate in the audit (table 1). A further 11 national societies that expressed an interest but were not in a position to participate were invited to meetings as observers. All participants were not-for-profit organisations with a legal status without profitable interests and were independent of industry, commercial and business or other conflicting interests. Participant national societies held the responsibility to organise data collection within their own territory following the decisions made by the expert panel. Each national society was responsible for selecting the participant hospitals in that particular country.
Data item selection
Items for the organisational database were selected by the steering committee based upon those that had previously been used and validated in the Spanish and UK national audits (table 2) [3, 5]. A number of terms had to be consensually defined, as there was no shared meaning across Europe and were considered key concepts for the audit.
Items for the clinical database were selected by the expert panel through a two-round modified Delphi process. The steering committee circulated a list of potential variables based upon items that had previously been used in the Spanish and UK national audits [3, 5], and included process issues matched to guidelines and those considered relevant by national experts. The aim was to select a relatively small group of items that were relevant to clinical practice and were easy to collect across the participating healthcare systems. All potential clinical variables were organised in a spreadsheet and sent to national experts for their evaluation. Each national expert graded items using one of three following options according to their importance: high relevance (five points), medium relevance (three points) or low relevance (one point). Thus, each item could be scored from 10 to 50 points. Those with >30 points were selected for the second round of the Delphi process. In the second round, the initial scoring was fed back to participants as a group mean allowing for a second voting round before a final list was determined (table 3). Items that scored >30 points in the second round were selected as variables for the clinical audit tool.
Web-tool development
A software company (IDCode, Lausanne Switzerland) was commissioned to design a web-based collection tool encompassing both the organisation and clinical databases. Data were entered remotely at each participant site to a centrally controlled server. Patient data were anonymised and encrypted. The web-tool was established as a multilingual database to allow each country to document the data in their own language.
The web-tool was organised as a hierarchical tool with different levels of responsibilities and rights to process data. Only the ERS and steering committee had full access to all data and the right to process them. At the national level there was a hierarchy for country administrators down to the local level hospital managers and doctors or research nurses coordinating the local data collection.
Each participating country received training from the steering committee on COPD audit practice through teleconferences, local meetings and workshops before starting the definite data collection. Subsequently, each country trained hospital managers and doctors/nurses responsible for the data collection.
Inclusion–exclusion criteria
As there is no operational European definition for a COPD exacerbation case admission, the Global Initiative for Chronic Obstructive Lung Disease recommendation for diagnosing an exacerbation exclusively on the clinical presentation of the patient was adopted [7]. Thus, two inclusion criteria based on clinical grounds were established. 1) Patients who were admitted to hospital for ≥12 h with a senior clinician-made diagnosis of COPD exacerbation or any other synonym, confirmed at discharge as judged by the investigator/audit lead. 2) Patients who were admitted to hospital for ≥12 h with a respiratory cause of admission as indicated by the discharge report and a history compatible with COPD.
Exclusion criteria were defined to distinguish other primary conditions that might produce symptoms similar to those of a COPD exacerbation where COPD exacerbation was not the primary cause for admission (table 4). Critically, patients admitted with a senior clinician-made diagnosis of COPD exacerbation and treated as such were included within the audit, regardless of the findings on the chest radiograph.
Protocol of study
The study was comprised of two phases. During the first, all consecutive cases admitted to hospital due to an exacerbation of COPD were identified during an 8-week period and registered in the clinical database. Local investigators had to identify all COPD admissions on a daily basis according to the local hospital protocol. There was no intervention by the audit team to the care provided by the medical staff in charge of the patient. At discharge, the local investigator accessed the discharge report and evaluated whether COPD exacerbation remained the cause of admission. If this was the case, all clinical data were extracted from the medical record, uploaded to the database and the case was entered to the second phase.
During the second phase, data on patient outcomes of death and readmission at 90 days were sought from various sources, including hospital records, primary care practitioners or from the patient and carers. The date of death was recorded and the cause of death or readmission was classified as either COPD-related or not COPD-related.
The 8-week collection period was planned to start on November 2010, but a number of countries asked for a later start date either because of their milder climate or because of delays in obtaining ethical approval for data collection. Thus, there were two inclusion periods, as follows. Group 1 (Austria, UK, Slovakia and Poland), begun on October 25, 2010 and ended on December 19, 2010; and group 2 (Belgium, Greece, Spain, Switzerland, Croatia, Romania, Malta, Turkey and Ireland) recorded clinical cases from January 3, 2011 until February 27, 2011. The 90-day follow-up period ended on March 18, 2011 and May 28, 2011 for groups 1 and 2, respectively. After these periods, investigators had an extended time to complete the databases of the clinical cases that were pending for a few weeks more. Final closing of the databases for both groups was on June 22, 2011.
Ethics
The European audit followed the European ethical requirements for scientific studies. All partners of the project accepted the general ethical rules of the ERS, particularly the rules on conflict of interests and relationships with the tobacco industry, which was an exclusion criterion for individual participation as a national representative. As there is no European Ethics Committee for audits, national societies ensured compliance with European and national ethical requirements. Some countries needed complex ethics agreements. Grants at the national level to support the audit had to be given as unrestricted grants to the national society without any further influence or interference of the sponsor on future results. The Steering Committee produced a standard patient consent form and an outline ethics application proforma for those countries that required a formal ethics research application. In the case of ethical dilemmas, the Ethics Committee of the ERS was consulted.
Statistical analysis
Statistical computations were performed by a data analysis team located in Seville, Valencia and Madrid, Spain. A preliminary data description was made to identify extreme values and inconsistencies. Thus, the database entered a data cleaning process starting on June 22, 2011. Those values considered extreme or found to have inconsistencies with other related variables were sent to local investigators to check and sent back the correct value. Once the database was completed, reports at a national and hospital level were created for the national experts, with their national information benchmarked against the rest of the countries or hospitals and the European average value. Median and interquartile range was used for quantitative variables, and the absolute and relative frequencies were used for qualitative ones using SAS 9.2 software (SAS Institute Inc., Cary, NC, USA). A multilevel multivariate analysis controlling for national and hospital clustering of cases will subsequently be performed.
DISCUSSION
There is increasing evidence and awareness that patients with various health problems do not consistently receive recommended care despite the proliferation of clinical practice guidelines. In the USA, it has been reported that only 33% of hospitalised patients with COPD receive guidelines-specified care [8]. Given the actual burden of COPD in both the population and healthcare systems worldwide, the failure to apply managed care guidelines is a major concern for respiratory professional societies [9]. Thus, there is a growing interest of managers in the development of specific measures on the performance of clinicians to improve healthcare. In this context, clinical audits have an important role as a reference of the quality of clinical practice in a given community.
Although the design of this audit is similar to previous ones [3–6], different factors, such as the varied health systems, the different provision of material and human resources, and the fact that there is no clinical audit system in most of Europe, make the present audit unique. The key component of a clinical audit is habitual performance, providing a framework to enable improvements to be made [10]. In this regard, it is of outstanding importance to supportively use the data obtained to bring about clinical practice improvement rather than to criticise the practice of one particular hospital or country.
The potential for the European COPD Audit Project is to raise the profile of COPD, provide an opportunity to promote respiratory medicine across Europe, inform the next COPD management guideline with the addition of recommendations about organisation of care, and develop educational resources to support improved clinical practice in areas identified as both good and poor practice. Additionally, a European audit may allow formal documentation of where management practice differs from evidence-based best practice guidelines and thereby identify areas of need for national and international improvement strategies.
Clinical features of severity of the disease are key factors influencing outcomes [11, 12]. However, there are also organisational aspects that may influence outcomes. The UK COPD audit described how resources may be of importance for the outcome from an admission [13–15]. A recent retrospective observational study evaluated the impact of nurse staffing on in-hospital mortality [16]. These authors found that lower levels of nurse staffing were associated with increased mortality. Furthermore, in Ontario, Canada, higher spending has been recently associated to better clinical outcomes [17]. Together, these articles together strongly suggest hospital resources as an important factor influencing in-hospital outcomes.
Although our project was prospective in nature, the data gathering was retrospective, as all the information was extracted from the medical records. This has two main consequences. First, this could lead to missing values in some cases. Secondly, the project relied on the extraction of data from different types of hospitals with different data extractors, and different types of documents were checked. The use of electronic health records, a potential solution to these difficulties, is now much debated [18]. For these reasons, our database will have to undergo a process of data cleaning to ensure data accuracy.
The variability of the population included in the study from every participant country was a source of discussion. As this was a pilot study an estimation of the sample size for a representative homogeneous distribution of the population screened was not calculated. Relying on the experience from the UK and Spain audits, we initially intended to include 50 patients per centre of at least 10 participating centres in the country in order to gather a comparable sample size between countries. However, during the training workshops it became clear that the size of hospitals varies considerably in the participating European countries and that this could become a limitation for some countries with small service populations covered by smaller units. Therefore, the steering committee gave freedom to national experts to select as many hospitals as they could recruit in their own countries, to include all cases during an 8-week period, and to compare the catchment population of all centres with the total population of the country to evaluate the representative value. Consequently, this led to a differential contribution of the two countries with an audit history (UK and Spain) and a predominance of larger specialist hospitals taking part. As a consequence, patients are clustered within hospitals/countries and data at national and European levels will have to be interpreted in the context of these clusters.
In accordance with current guidelines [7], COPD exacerbations were defined as diagnosed by a senior clinician on clinical grounds. Subsequent analysis of included cases against spirometry will be used to assess the accuracy of diagnosis within the audit.
Two options were discussed to define the inclusion period. First, a time-fixed inclusion criterion for all cases admitted during a certain period would potentially under-represent smaller hospitals, and thus include fewer cases. Secondly, a fixed number of cases over a variable time period would reduce the impact of hospital size but limit recruitment. In our case, time was fixed during two different periods and the number of cases was open.
Not all interested countries were able to take part. In some cases, this was due to a lack of the national funding needed to establish the administrative structure, and in others this was due to barriers, including reservations about contribution of data to an external source. Although no country was excluded for ethical issues, ethical permission became a key factor as there is no clinical audit system in most of the participant countries with which to approve this kind of study and there is no mechanism for a pan-European ethical consent. There is also no comprehensive documentation of patient outcomes beyond discharge from hospital in many countries. To obtain this information in these cases, patients or caregivers had to be contacted, prompting a major ethical concern that needed formal ethical consideration.
Accuracy of mortality figures was thoroughly evaluated by the expert panel. In-hospital mortality was assumed to be accurate, but the post-discharge mortality registration method varied depending on the country. Some countries, such as the UK, have a system of notifying hospitals when a patient dies. In other countries, data on patient outcomes such as death and readmission were sought from various sources, including hospital records, primary care practitioners or from the patient and caregivers at 90 days after the admission.
The European audit has the potential to gather large-scale data on numerous patients from differing healthcare systems. The pilot study has proven feasibility although revealing a number of difficulties and weaknesses to overcome across different healthcare systems. Clinicians working across national boundaries will be able to promote better processes of care that are difficult to identity in smaller scale audit programmes and will improve the future system of audit by their feedback.
Acknowledgments
The European COPD Audit team consists of the following members. The steering committee: C.M. Roberts (Barts and The London School of Medicine and Dentistry Queen Mary University of London, London, UK), S. Hartl (Ludwig Boltzmann Institute of COPD and Respiratory Epidemiology, Vienna, Austria) and J.L. López-Campos (Hospital Universitario Virgen del Rocío, Instituto de Biomedicina de Sevilla (IBiS), Sevilla, and CIBER de Enfermedades Respiratorias (CIBERES), Instituto de Salud Carlos III, Madrid, Spain). The data analysis team: F. Pozo-Rodríguez (Hospital 12 de Octubre, Instituto de Investigación i+12, Madrid, Spain), J.L. López-Campos (see previous entry), A. Castro-Acosta (Hospital 12 de Octubre, Instituto de Investigación i+12, Madrid, Spain), V. Abraira-Santos (Hospital Universitario Ramón y Cajal, IRYCIS, and Centre for Biomedical Research on Epidemiology and Public Health (CIBERESP), Instituto de Salud Carlos III, Madrid, Spain), A. López-Quilez (Department of Statistics and Operational Research, Universidad de Castellón, Castellón, Spain) and J. Dorado (Consultoría estadística y de investigación clínica PERTICA). The national experts (by alphabetical order of country): O. Burghuber and R. Kohansal (Otto-Wagner Hospital, Vienna, Austria), W. Janssens (Universitaire Ziekenhuizen, Leuven, Belgium), V. Heinen (Centre Hospitalier Universitaire de Liège, Liège, Belgium), N. Miculinic and H. Puretic (Clinic for Respiratory Diseases, University Hospital “Zagreb”, Croatia), N. Tzanakis (University Hospital of Heraklion, Dept of Thoracic Medicine, Medical School, University of Crete, Greece), E. Nontas Kosmas (Respiratory Unit, Metropolitan General Hospital, Athens, Greece), S. McCormack (Irish Thoracic Society, Ireland), T. McDonell (St Vincent’s University Hospital, Dublin, Ireland), C. Farrugia Jones (Mater Dei Hospital, Malta), J. Chorostowska-Wynimko (Institute of Tuberculosis and Lung Diseases, Warsaw, Poland), F. Mihaltan and I. Munteanu (Institutul De Pneumoftiziologie "Marius Nasta", Bucharest, Romania), M. Alexandru Bogdan ("Carol Davila" University of Medicine and Pharmacy, Bucharest, Romania), I. Solovic (National Institute for TB, Lung Diseases and Thoracic Surgery, Slovakia), R. Tkacova (University Hospital Kosice Pneumolog. Clinic, Solvakia), F. Pozo-Rodríguez (Hospital 12 de Octubre, Instituto de Investigación i+12, Madrid, Spain), J. Ancochea (Hospital Universitario de La Princesa, Madrid, Spain), D. Stolz (Universitätsspital Basel, Basel, Switzerland), M. Polatli (Adnan Menderes Üniversitesi Tıp Fakültesi Göğüs Hastalıkları AD, Aydın, Turkey), E. Şen (Ankara Üniversitesi Tıp Fakültesi Göğüs Hastalıkları AD, Ankara, Turkey), C. Bucknall (Stobhill General Hospital, Glasgow, UK), and S. Welham and C. Routh (British Thoracic Society, London, UK). Project managers: M. Haan and M. Zarelli (ERS), E. Lechat and R.J. Buckingham (Royal College of Physicians of London, UK). The ERS COPD Audit liaison officer is G. Joos.
Footnotes
Statement of Interest
A statement of interest for S. Hartl can be found at www.erj.ersjournals.com/site/misc/statements.xhtml
- Received February 4, 2012.
- Accepted March 23, 2012.
- ©ERS 2013
REFERENCES
