EMR Analysis
Preliminary research of the current prevalence and management patterns of speech and language disorders is necessary before more in-depth epidemiologic analyses may be undertaken. One source of data for this type of study is electronic medical records (EMR) extracted from primary care clinics themselves. . EMRs provide patient histories in the form of diagnoses, prescriptions, billing information, referrals laboratory testing and other information recorded by the attending physician. Thus, EMRs present a rich and comprehensive source of data, through which an accurate and precise understanding of primary care medicine may be gained. According to Griever et al. (2013), 49% of family medicine practices in Canada used EMR software to record
…show more content…
patient visit information as of 2010 (347). The current body of literature regarding EMR-derived data, focuses on the validity and availability of such data (Virnig and McBean, 2001). A review of the literature also highlights the differences between countries with regards to clinical data and how health system configuration impacts data.
A recurring finding in the Canadian literature is the potential for inaccuracy in clinical datasets due to the multiplicity of EMR vendors in our primary care system Tu et al).
In their 2010 retrospective cohort study, Harris et al. determined that a lack of “consult letters [and] visit notes” limited the utility of EMRs for creating a diabetes case definition (351). Birtwhistle et al. (2009) also identified the absence of often critical sources of EMR-derived data, in addition to several other noteworthy challenges such as “dirty data…[including] misspelled words, extra words, inconsistent word strings…; inconsistent data…[as in] diagnoses stored in several different places…[and an absence of] standardization” (418). Tu et al. (2014), who performed a retrospective comparison of several administrative databases in Ontario, supported Birtwhistle et al. (2009)’s assertion that “lack of standardization” (418) creates significant problems for researchers using EMR-derived data: “the quality of the data captured and the ability to identify discrete data elements in the EMR may not be the same across EMR software packages from different vendors” (Tu et al., 2014, …show more content…
e19-e20).
While Canada’s current system of employing different EMR vendors to record patient information creates considerable difficulties at the system-level, there are also strengths and limitations inherent in EMRs as a data source. Virnig and McBean (2001) note that clinical datasets are a low-cost alternative to study-specific data collection and is more easily acquired by researchers. Additionally, data linkage creates the possibility of using data from various types of administrative sources which is not possible with traditional data collection methods (Virnig and McBean, 2001). However, Harris et al. (2010) argue that such data may not be truly “population-based” and is biased in favour of those who choose to or are able to access health services (351).
In light of these issues in the use of EMR-derived data for research purposes, the Canadian Primary Care Sentinel Surveillance Network (CPCSSN) offers a unique source of primary care, EMR-derived data.
CPCSSN is a network functioning across Canada to survey chronic diseases and improve quality of data for research purposes as well as care for patients (CPCSSN, 2013). Primary care clinics participating in CPCSSN provide consent to have health information on patients served by their practices collected and used for academic research (CPCSSN, 2013). CPCSSN currently extracts data from 600 primary care physicians across Canada for 750,000 patients. EMR software products included in the CPCSSN database include Accuro, Bell, DaVinci, Healthscreen, Jonoke, Med Access, Nightingale, Oscar, Practice Solutions, Purkinje, Telin Mediplan and Wolff. In the Southern Alberta Primary Care Research Network, a regional network within CPCSSN specifically, extractions are made from Med Access, Wolff and Telin Mediplan. The CPCSSN database currently tracks the prevalence of eight major conditions: diabetes, osteoarthritis, hypertension, chronic obstructive pulmonary disease (COPD), Parkinson’s disease (PD), epilepsy, depression and dementia (CPCSSN, 2013). A major issue with utilizing EMR-extracted data for research is that the data is used for a purpose other than that for which it was originally recorded, however CPCSSN has made significant efforts to extract EMR records and convert them
into data which is reliable and useable for epidemiologic research. Data extracted from EMRs from multiple vendors, and primary care clinics is standardized using data mapping (Birtwhistle et al. 2009; Williamson et al., 2013). Data mapping is defined as a “broad technical function” which “ ‘matches’ between a source and a target…[which] enables software and systems to meaningfully exchange patient information, reimbursement claims, outcomes reporting and other data” (McBride et al. 2006, 44). Furthermore data which contains issues such as “misspelled words, extra words in [the] field, inconsistent strings”, etc. is cleaned utilizing various algorithms (Birtwhistle et al. 2009, 418).
Databases such as CPCSSN employ case definitions for conditions of interest in the form of diagnostic algorithms. These algorithms comprise a “set of criteria” which can indicate ‘caseness’ when applied to a data set through a combination of free text items, billing codes (ICD-9), medication codes and laboratory test results (Chubak et al., 2012, 344). While diagnostic algorithms have the ability to facilitate data collection easily and accurately, there are issues inherent in their use (Chubak et al., 2012). Chubak et al. (2012) illustrate several ways in which EMR data may be of limited accuracy: firstly, patients may see a physician other than their usual family physician and data linkage between physicians in separate practices (or even health systems) is extremely uncommon; physicians may not record details of the visit factually or with regularity; ICD-9/ICD-10 codes may not effectively convey the patient’s diagnosis (particularly lifestyle, risk factors and socioeconomic status) are not typically recorded. Therefore, in order to generate data of sufficient quality for epidemiologic research, case definitions utilized for epidemiology must be evaluated for validity and accuracy.
In a systematic review of validation methods in the United Kingdom, Herrett et al. (2010) determined that the “majority of validations were external [i.e. a questionnaire or record request to GP or comparison of rates of disease incidence or prevalence]” and are used to obtain a measure of positive predictive value (8). Hassey et al. (2001) adds that positive predictive value is a useful measure when used in conjunction with sensitivity and likelihood ratio(s). Nicholson et al. (2011) suggests the main issue from the perspective of the literature is “ambiguity” over the term ‘validation’ (322). The authors explain that an ‘internal validation’ asks “Did the GP think that the patient had this condition?” whereas an ‘external validation’ questions “[whether] the GP was correct?” (322). Validation studies take both forms, often without clearly differentiating which question is the primary focus of the work. Therefore inappropriateness and incompleteness of methodology and analysis further complicates the current body of literature. In order to address the paucity of validity data for case definitions applied to EMR data (as outlined by Herrett et al. (2010) and Nicholson et al. (2011)), CPCSSN has undertaken two previous validation studies with the goal of increasing the validity of common conditions with the database and increasing the ease with which these conditions may be researched using CPCSSN data. Kadhim-Saleh et al. (2013) was a limited pilot study which validated five case definitions (diabetes, hypertension, osteoarthritis, COPD, depression) using data from a single contributing network (the Kingston-based PBRN-CPCSSN network). The authors noted that while the case definition algorithms that CPCSSN initially employed for osteoarthritis, COPD and depression had high specificity, low sensitivity was a recurring issue. Thus, prevalence may have been higher than reported for those conditions prior to implementing the current (higher sensitivity) algorithms (Kadim-Saleh et al., 2013). In their study, Kadim-Saleh et al. (2013) reported high specificities for all five conditions as well as high sensitivities for diabetes (100%), and hypertension (83%). Case definitions for three conditions (osteoarthritis, depression and COPD) continued to report low sensitivities ranging from 39%-45%.
Williamson et al. (2014) followed Kadim-Saleh et al. (2013) (having found the validation methodology to be appropriate), but included data from 6 CPCSSN networks across Canada and validated case definitions for the original five conditions, in addition to epilepsy, Parkinsonism and dementia. This case definition validation not only included a significantly larger sample size (n=1920) than previous validation studies, it assessed all eight conditions using the same gold standard and included both case positive and case negative patients (Williamson et al. 2013, 371). The validation study resulted in high sensitivity (77.8 – 98.8%), specificity (93.5-99.0%), positive predictive value (72.1-92.9%) and negative predictive value (86.0-99.9%) for all eight conditions
In terms of methodology, Williamson et al. (2013) used an age-stratified approach to assess patients who were overwhelmingly older adults (90%) (368). Chart reviewers were trained to appraise ‘caseness’ in patients and were blinded to the diagnosis of each case by the algorithm (368). Uncertain cases were reviewed subsequently by an epidemiologist and physician. The case definitions within this study are “specifically developed for use in primary care contexts” (Williamson et al., 2013, 368) and are thus more focused on symptoms, diagnostic patterns, etc. commonly found in primary care than more broad or general case definitions.
Therefore, when undertaking a study employing EMR-derived data, it is necessary to understand and account for the challenges such data may present (particularly for less common diseases in the primary care context). While certain aspects of EMR-derived data are inherent to the source, many of issues explored above can and have been addressed by CPCSSN, resulting in data of high quality and reliability. The methodology developed for case definition validation using CPCSSN data is both rigorous and sufficiently adaptable to a variety of possible conditions.
In order to develop a case definition which could accurately assess prevalence, it was first necessary to determine what factors constitute a case in the EMR context. Compiling a case definition for speech and language disorders is challenging due to the subjectivity of symptoms and variations within disorders themselves. For example, Speech-Language & Audiology Canada (2013) define aphasia as “a language disorder…resulting in difficulty formulating, expressing, and/or understanding language” (par.4) The Merck Manual of Diagnosis and Therapy differentiates aphasia into two categories (Wernicke and Broca) and delves further into symptomology: “Wernicke aphasia [patients]…speak normal words fluently…but do not know meaning or relationships [resulting in]…a jumble of words or “word salad”; “Broca aphasia…patients can comprehend and conceptualize relatively well, but their ability to form words is impaired…[which] affects speech production and writing…[and] may include anomia (inability to name objects) and impaired prosody” (par. 9-12). While speech and language disorders are typically diagnosed through observing the communication of a patient and “screening” for abnormalities in speech and comprehension (S-L&AC 2016, 2), there are different manifestations of speech and language problems both within individual speech and language disorders as well as between speech and language disorders. Thus, in order to account for the symptom-based diagnosis of speech and language disorders and clinical variation, the case definition was intended to be as broad and inclusive as possible.
patient visit information as of 2010 (347). The current body of literature regarding EMR-derived data, focuses on the validity and availability of such data (Virnig and McBean, 2001). A review of the literature also highlights the differences between countries with regards to clinical data and how health system configuration impacts data.
A recurring finding in the Canadian literature is the potential for inaccuracy in clinical datasets due to the multiplicity of EMR vendors in our primary care system Tu et al).
In their 2010 retrospective cohort study, Harris et al. determined that a lack of “consult letters [and] visit notes” limited the utility of EMRs for creating a diabetes case definition (351). Birtwhistle et al. (2009) also identified the absence of often critical sources of EMR-derived data, in addition to several other noteworthy challenges such as “dirty data…[including] misspelled words, extra words, inconsistent word strings…; inconsistent data…[as in] diagnoses stored in several different places…[and an absence of] standardization” (418). Tu et al. (2014), who performed a retrospective comparison of several administrative databases in Ontario, supported Birtwhistle et al. (2009)’s assertion that “lack of standardization” (418) creates significant problems for researchers using EMR-derived data: “the quality of the data captured and the ability to identify discrete data elements in the EMR may not be the same across EMR software packages from different vendors” (Tu et al., 2014, …show more content…
e19-e20).
While Canada’s current system of employing different EMR vendors to record patient information creates considerable difficulties at the system-level, there are also strengths and limitations inherent in EMRs as a data source. Virnig and McBean (2001) note that clinical datasets are a low-cost alternative to study-specific data collection and is more easily acquired by researchers. Additionally, data linkage creates the possibility of using data from various types of administrative sources which is not possible with traditional data collection methods (Virnig and McBean, 2001). However, Harris et al. (2010) argue that such data may not be truly “population-based” and is biased in favour of those who choose to or are able to access health services (351).
In light of these issues in the use of EMR-derived data for research purposes, the Canadian Primary Care Sentinel Surveillance Network (CPCSSN) offers a unique source of primary care, EMR-derived data.
CPCSSN is a network functioning across Canada to survey chronic diseases and improve quality of data for research purposes as well as care for patients (CPCSSN, 2013). Primary care clinics participating in CPCSSN provide consent to have health information on patients served by their practices collected and used for academic research (CPCSSN, 2013). CPCSSN currently extracts data from 600 primary care physicians across Canada for 750,000 patients. EMR software products included in the CPCSSN database include Accuro, Bell, DaVinci, Healthscreen, Jonoke, Med Access, Nightingale, Oscar, Practice Solutions, Purkinje, Telin Mediplan and Wolff. In the Southern Alberta Primary Care Research Network, a regional network within CPCSSN specifically, extractions are made from Med Access, Wolff and Telin Mediplan. The CPCSSN database currently tracks the prevalence of eight major conditions: diabetes, osteoarthritis, hypertension, chronic obstructive pulmonary disease (COPD), Parkinson’s disease (PD), epilepsy, depression and dementia (CPCSSN, 2013). A major issue with utilizing EMR-extracted data for research is that the data is used for a purpose other than that for which it was originally recorded, however CPCSSN has made significant efforts to extract EMR records and convert them
into data which is reliable and useable for epidemiologic research. Data extracted from EMRs from multiple vendors, and primary care clinics is standardized using data mapping (Birtwhistle et al. 2009; Williamson et al., 2013). Data mapping is defined as a “broad technical function” which “ ‘matches’ between a source and a target…[which] enables software and systems to meaningfully exchange patient information, reimbursement claims, outcomes reporting and other data” (McBride et al. 2006, 44). Furthermore data which contains issues such as “misspelled words, extra words in [the] field, inconsistent strings”, etc. is cleaned utilizing various algorithms (Birtwhistle et al. 2009, 418).
Databases such as CPCSSN employ case definitions for conditions of interest in the form of diagnostic algorithms. These algorithms comprise a “set of criteria” which can indicate ‘caseness’ when applied to a data set through a combination of free text items, billing codes (ICD-9), medication codes and laboratory test results (Chubak et al., 2012, 344). While diagnostic algorithms have the ability to facilitate data collection easily and accurately, there are issues inherent in their use (Chubak et al., 2012). Chubak et al. (2012) illustrate several ways in which EMR data may be of limited accuracy: firstly, patients may see a physician other than their usual family physician and data linkage between physicians in separate practices (or even health systems) is extremely uncommon; physicians may not record details of the visit factually or with regularity; ICD-9/ICD-10 codes may not effectively convey the patient’s diagnosis (particularly lifestyle, risk factors and socioeconomic status) are not typically recorded. Therefore, in order to generate data of sufficient quality for epidemiologic research, case definitions utilized for epidemiology must be evaluated for validity and accuracy.
In a systematic review of validation methods in the United Kingdom, Herrett et al. (2010) determined that the “majority of validations were external [i.e. a questionnaire or record request to GP or comparison of rates of disease incidence or prevalence]” and are used to obtain a measure of positive predictive value (8). Hassey et al. (2001) adds that positive predictive value is a useful measure when used in conjunction with sensitivity and likelihood ratio(s). Nicholson et al. (2011) suggests the main issue from the perspective of the literature is “ambiguity” over the term ‘validation’ (322). The authors explain that an ‘internal validation’ asks “Did the GP think that the patient had this condition?” whereas an ‘external validation’ questions “[whether] the GP was correct?” (322). Validation studies take both forms, often without clearly differentiating which question is the primary focus of the work. Therefore inappropriateness and incompleteness of methodology and analysis further complicates the current body of literature. In order to address the paucity of validity data for case definitions applied to EMR data (as outlined by Herrett et al. (2010) and Nicholson et al. (2011)), CPCSSN has undertaken two previous validation studies with the goal of increasing the validity of common conditions with the database and increasing the ease with which these conditions may be researched using CPCSSN data. Kadhim-Saleh et al. (2013) was a limited pilot study which validated five case definitions (diabetes, hypertension, osteoarthritis, COPD, depression) using data from a single contributing network (the Kingston-based PBRN-CPCSSN network). The authors noted that while the case definition algorithms that CPCSSN initially employed for osteoarthritis, COPD and depression had high specificity, low sensitivity was a recurring issue. Thus, prevalence may have been higher than reported for those conditions prior to implementing the current (higher sensitivity) algorithms (Kadim-Saleh et al., 2013). In their study, Kadim-Saleh et al. (2013) reported high specificities for all five conditions as well as high sensitivities for diabetes (100%), and hypertension (83%). Case definitions for three conditions (osteoarthritis, depression and COPD) continued to report low sensitivities ranging from 39%-45%.
Williamson et al. (2014) followed Kadim-Saleh et al. (2013) (having found the validation methodology to be appropriate), but included data from 6 CPCSSN networks across Canada and validated case definitions for the original five conditions, in addition to epilepsy, Parkinsonism and dementia. This case definition validation not only included a significantly larger sample size (n=1920) than previous validation studies, it assessed all eight conditions using the same gold standard and included both case positive and case negative patients (Williamson et al. 2013, 371). The validation study resulted in high sensitivity (77.8 – 98.8%), specificity (93.5-99.0%), positive predictive value (72.1-92.9%) and negative predictive value (86.0-99.9%) for all eight conditions
In terms of methodology, Williamson et al. (2013) used an age-stratified approach to assess patients who were overwhelmingly older adults (90%) (368). Chart reviewers were trained to appraise ‘caseness’ in patients and were blinded to the diagnosis of each case by the algorithm (368). Uncertain cases were reviewed subsequently by an epidemiologist and physician. The case definitions within this study are “specifically developed for use in primary care contexts” (Williamson et al., 2013, 368) and are thus more focused on symptoms, diagnostic patterns, etc. commonly found in primary care than more broad or general case definitions.
Therefore, when undertaking a study employing EMR-derived data, it is necessary to understand and account for the challenges such data may present (particularly for less common diseases in the primary care context). While certain aspects of EMR-derived data are inherent to the source, many of issues explored above can and have been addressed by CPCSSN, resulting in data of high quality and reliability. The methodology developed for case definition validation using CPCSSN data is both rigorous and sufficiently adaptable to a variety of possible conditions.
In order to develop a case definition which could accurately assess prevalence, it was first necessary to determine what factors constitute a case in the EMR context. Compiling a case definition for speech and language disorders is challenging due to the subjectivity of symptoms and variations within disorders themselves. For example, Speech-Language & Audiology Canada (2013) define aphasia as “a language disorder…resulting in difficulty formulating, expressing, and/or understanding language” (par.4) The Merck Manual of Diagnosis and Therapy differentiates aphasia into two categories (Wernicke and Broca) and delves further into symptomology: “Wernicke aphasia [patients]…speak normal words fluently…but do not know meaning or relationships [resulting in]…a jumble of words or “word salad”; “Broca aphasia…patients can comprehend and conceptualize relatively well, but their ability to form words is impaired…[which] affects speech production and writing…[and] may include anomia (inability to name objects) and impaired prosody” (par. 9-12). While speech and language disorders are typically diagnosed through observing the communication of a patient and “screening” for abnormalities in speech and comprehension (S-L&AC 2016, 2), there are different manifestations of speech and language problems both within individual speech and language disorders as well as between speech and language disorders. Thus, in order to account for the symptom-based diagnosis of speech and language disorders and clinical variation, the case definition was intended to be as broad and inclusive as possible.