Qualitative Paper Sample
How to critically appraise a paper
How to review a paper 1 Checklist for a qualitative research paper 3 Checklist for material provided by a pharmaceutical company representative 4 Checklist for a paper that claims to validate a diagnostic or screening test 4 Checklist for a set of clinical guidelines 4 Checklist for a systematic review or meta-analysis 4 Checklist for an economic analysis 5 Checklist for health care organisations working towards an evidence based culture for clinical and purchasing decisions 5 Checklist for searching Medline or the Cochrane library 6
How to review a paper
Determine what a paper is about
• Why was the study done (what clinical question did it examine)? • Is …show more content…
there a clear description of the problem • Is the study useful and relevant to General Practice • If the study is conducted in a hospital environment, can the results be interpreted in a general practice situation • Was the study design appropriate to the broad field of research examined (therapy, diagnosis, screening, prognosis, causation)? • What type of study was done? - Primary research (experiment, randomised controlled trial, other controlled clinical trial, cohort study, case-control study, cross sectional survey, longitudinal survey, case report, or case series)? - Secondary research (simple overview, systematic review, meta-analysis, decision analysis, guideline development, economic analysis)? • Was the study ethical? - Are there are any ethical objections to the design or reporting of the study • Is there a review of the literature • Is the writing style easily understood • Is the paper was well laid out and easy to follow
Paper methodology
• The design of the study is consistent with the aims - Observational studies - qualitative, by interviewing - Observational studies - quantitative, obtain baseline values - Retrospective studies - information from past events - Prospective studies - following events as they happen - Experimental Studies – e.g. randomized control trial • Was the design of the study sensible? • Was the study original? • Who is the study about? • The sample of the study is representative of the population as a whole • How were subjects recruited? • Are controls needed in the study - If a cohort, case-control, or other non-randomised comparative study were the controls appropriate? • Who was included in and who was excluded from the study? • The methods of selecting cases and controls is defined well • Were the subjects studied in “real life” circumstances? • What intervention or other manoeuvre was being considered? • Details of the study such as numbers, time intervals, statistical test used are clear and appropriate • The questionnaire and proformas are appropriate and relevant to the study • Was the study adequately controlled? • If a “randomised trial” was randomisation truly random? • Were the groups comparable in all important aspects except for the variable being studied? • What outcome(s) were measured and how? • Was assessment of outcome (or, in a case-control study, allocation of caseness) “blind”? • Are there sources of bias in the setting of the subjects • Have confounding influences and multiple influences been removed • Was the study large enough and continued for long enough, and was follow up complete enough, to make the results credible?
Statistical aspects of a paper
• Have the authors set the scene correctly? ❖ Have they determined whether their groups are comparable and, if necessary, adjusted for baseline differences? ❖ What sort of data have they got and have they used appropriate statistical tests? ❖ If the statistical tests in the paper are obscure why have the authors chosen to use them? ❖ Have the data been analysed according to the original study protocol? • Paired data, tails, and outliers: ❖ Have the appropriate tests been used e.g. t-test for distributions, chi square tests for dichotomous values ❖ Were paired tests performed on paired data? ❖ Was a two tailed test performed whenever the effect of an intervention could conceivably be a negative one? ❖ Were outliers analysed with both common sense and appropriate statistical adjustments? • Correlation, regression and causation: ❖ Has correlation been distinguished from regression and has the correlation coefficient (r value) been calculated and interpreted correctly? ❖ Have assumptions been made about the nature and direction of causality? • Probability and confidence and degree of significance been interpreted correctly: ❖ Have P values been calculated and interpreted appropriately? ❖ Have confidence intervals been calculated and do the authors’ conclusions reflect them? • Have the authors expressed their results in terms of the likely harm or benefit that an individual patient can expect, eg: ❖ Relative risk reduction ❖ Absolute risk reduction ❖ Number needed to treat ❖ Odds ratio.
Result aspects of a paper
• Missing data such as drop outs, non-responders are accounted for • Details of the results such as the numbers , statistics, are accurate and clear • If statistics are appropriate to the study, then they were well used • The sample size is of a significant amount • Are the results clearly presented • Have they used bar charts, tables appropriately used
Discussion aspects of a paper
• The study is discussed critically • The results are discussed with reference to other important literature • The discussion and conclusions do not speculate too far beyond what has been shown in the study.
Checklist for a qualitative research paper
• Did the article describe an important clinical problem examined via a clearly formulated question? • Was the qualitative approach appropriate? • How were the setting and the subjects selected? • What was the researcher’s perspective and has this been taken into account? • What methods did the researcher use for collecting data—and are these described in enough detail? • What methods did the researcher use to analyse the data—and what quality control measures were implemented? • Are the results credible and if so are they clinically important? • What conclusions were drawn and are they justified by the results? • Are the findings of the study transferable to other clinical settings?
Checklist for material provided by a pharmaceutical company representative
• Does this material cover a subject that interests me and is clinically important in my practice? • Has this material been published in independent peer reviewed journals? Has any significant evidence been omitted from this presentation or withheld from publication? • Does the material include high level evidence such as systematic reviews, meta-analyses, or double blind randomised controlled trials against the drug’s closest competitor given at optimal dosage? • Have the trials or reviews examined a clearly focused, important and answerable clinical question that reflects a problem of relevance to patients?
Do they provide evidence on safety, tolerability, efficacy, and price? • Has each trial or meta-analysis defined the condition to be treated, the patients to be included, the interventions to be compared, and the outcomes to be examined? • Does the material provide direct evidence that the drug will help my patients live a longer, healthier, more productive, and symptom free life? • If a surrogate outcome measure has been used, what is the evidence that it is reliable, reproducible, sensitive, specific, a true predictor of disease, and rapidly reflects the response to therapy? • Do trial results indicate whether (and how) the effectiveness of the treatments differed and whether there was a difference in the type of incidence of adverse reactions? Are the results expressed in terms of numbers needed to treat, and are they clinically as well as statistically significant? • If large amounts of material have been provided by the representative, which three papers provide the strongest evidence for the company’s …show more content…
claims?
Checklist for a paper that claims to validate a diagnostic or screening test
• Is this test potentially relevant to my practice? • Has the test been compared with a true gold standard? • Did this validation study include an appropriate spectrum of subjects? • Has work up bias been avoided? • Has observer bias been avoided? • Was the test shown to be reproducible both within and between observers? • What are the features of the test as derived from this validation study? • Were confidence intervals given for sensitivity, specificity, and other features of the test? • Has a sensible “normal range” been derived from these results?
• Has this test been placed in the context of other potential tests in the diagnostic sequence for the condition?
Checklist for a set of clinical guidelines
• Did the preparation and publication of these guidelines involve a significant conflict of interest? • Are the guidelines concerned with an appropriate topic, and do they state clearly the goal of ideal treatment in terms of health and/or cost outcome? • Was the guideline development panel headed by a leading expert in the field (ideally it should not be) and was a specialist in the methodology of secondary research (for example, meta-analyst, health economist) involved? • Have all the relevant data been scrutinised and do the guidelines’ conclusions seem to be in keeping with the data? • Do they cover variations in medical practice and other controversial areas (for example, optimum care in response to genuine or perceived underfunding)? • Are the guidelines valid and reliable? • Are they clinically relevant, comprehensive, and flexible? • Do they take into account what is acceptable to, affordable by, and practically possible for
patients? • Do they include recommendations for their own dissemination, implementation, and periodic review?
Checklist for a systematic review or meta-analysis
• Did the review examine an important clinical question? • Was a thorough search done of the appropriate database(s) and were other potentially important sources explored? • Was methodological quality assessed and the trials weighted accordingly? • How sensitive are the results to the way the review has been done? • Have the numerical results been interpreted with common sense and due regard to the broader aspects of the problem?
Checklist for an economic analysis
• Is the analysis based on a study that answers a clearly defined clinical question about an economically important issue? • Whose viewpoint are costs and benefits considered from? • Have the interventions being compared been shown to be clinically effective? • Are the interventions sensible and workable in the settings where they are likely to be applied? • Which method of economic analysis was used and was this appropriate? ❖ If the interventions produced identical outcomes ( cost minimisation analysis ❖ If the important outcome is unidimensional ( cost-effectiveness analysis ❖ If the important outcome is multidimensional ( cost-utility analysis ❖ If the cost-benefit equation for this condition needs to be compared with cost-benefit equations for different conditions ( cost-benefit analysis ❖ If a cost-benefit analysis would otherwise be appropriate but the preference values given to different health states are disputed or likely to change ( cost-consequences analysis. • How were costs and benefits measured? • Were incremental rather than absolute benefits compared? • Was health status in the “here and now” given precedence over health status in the distant future? • Was sensitivity analysis performed? • Were “bottom line” aggregate scores overused?
Checklist for health care organisations working towards an evidence based culture for clinical and purchasing decisions
• Leadership—How often has information on effectiveness or evidence based medicine been discussed at board meetings in the past 12 months? Has the board taken time out to learn about developments in clinical and cost effectiveness? • Investment—What resources are the organisation investing in finding and using clinical effectiveness information? Is there a planned approach to promoting evidence based medicine that is properly resourced and staffed? • Using available resources—What action has been taken by the organisation in response to EL(93) 115 (Improving Clinical Effectiveness) and EL(94)74 (Improving the Effectiveness of the NHS)? What has changed in the organisation as a result? • Implementation—Who is responsible for receiving, acting on, and monitoring the implementation of Effective Health Care bulletins? What action has been taken on each of the bulletins issued to date? • Clinical guidelines—Who is responsible for receiving, acting on, and monitoring clinical practice guidelines? Do those arrangements ensure that both managers and clinicians play their part in guideline development and implementation? • Training—Has any training been provided to staff within the organisation (both clinical and non-clinical) on appraising and using evidence of effectiveness to influence clinical practice? • Contracts—How often does clinical and cost-effectiveness information form an important part of contract negotiation and agreement? How many contracts contain terms that set out how effectiveness information is to be used? • Incentives—What incentives—both individual and organisational—exist to encourage the practice of evidence based medicine? What disincentives exist to discourage inappropriate practice and unjustified variations in clinical decision making? • Information systems—Is the potential of existing information systems to monitor clinical effectiveness being used to the full? Is there a business case for new information systems to deal with the task, and is this issue being considered when purchasing decisions about information technology are made? • Clinical audit—Is there an effective clinical audit programme throughout the organisation, capable of examining issues of clinical effectiveness and bringing about appropriate changes in practice?
Checklist for searching Medline or the Cochrane library
• To look for an article you know exists, search by textwords (in title, abstract, or both) or use field suffixes for author, title, institution, journal, and publication year. • For a maximally sensitive search on a subject, search under both MeSH headings (exploded) and textwords (title and abstract), then combine the two by using the Boolean operator “or” • For a focused (specific) search on a clear cut topic, perform two or more sensitive searches as in (2), and combine them by using the Boolean operator “and”. • To find articles that are likely to be of high methodological quality, insert an evidence based quality filter for therapeutic interventions, aetiology, diagnostic procedures, or epidemiology and/or use maximally sensitive search strategies for randomised trials, systematic reviews, and meta-analyses. • Refine your search as you go along—for example, to exclude irrelevant material, use the Boolean operator “not”. • Use subheadings only when this is the only practicable way of limiting your search as manual indexers are fallible and misclassifications are common. • When limiting a large set, browse through the last 50 or so abstracts yourself rather than expecting the software to pick the best half dozen.
How to review a paper 1 Checklist for a qualitative research paper 3 Checklist for material provided by a pharmaceutical company representative 4 Checklist for a paper that claims to validate a diagnostic or screening test 4 Checklist for a set of clinical guidelines 4 Checklist for a systematic review or meta-analysis 4 Checklist for an economic analysis 5 Checklist for health care organisations working towards an evidence based culture for clinical and purchasing decisions 5 Checklist for searching Medline or the Cochrane library 6
How to review a paper
Determine what a paper is about
• Why was the study done (what clinical question did it examine)? • Is …show more content…
there a clear description of the problem • Is the study useful and relevant to General Practice • If the study is conducted in a hospital environment, can the results be interpreted in a general practice situation • Was the study design appropriate to the broad field of research examined (therapy, diagnosis, screening, prognosis, causation)? • What type of study was done? - Primary research (experiment, randomised controlled trial, other controlled clinical trial, cohort study, case-control study, cross sectional survey, longitudinal survey, case report, or case series)? - Secondary research (simple overview, systematic review, meta-analysis, decision analysis, guideline development, economic analysis)? • Was the study ethical? - Are there are any ethical objections to the design or reporting of the study • Is there a review of the literature • Is the writing style easily understood • Is the paper was well laid out and easy to follow
Paper methodology
• The design of the study is consistent with the aims - Observational studies - qualitative, by interviewing - Observational studies - quantitative, obtain baseline values - Retrospective studies - information from past events - Prospective studies - following events as they happen - Experimental Studies – e.g. randomized control trial • Was the design of the study sensible? • Was the study original? • Who is the study about? • The sample of the study is representative of the population as a whole • How were subjects recruited? • Are controls needed in the study - If a cohort, case-control, or other non-randomised comparative study were the controls appropriate? • Who was included in and who was excluded from the study? • The methods of selecting cases and controls is defined well • Were the subjects studied in “real life” circumstances? • What intervention or other manoeuvre was being considered? • Details of the study such as numbers, time intervals, statistical test used are clear and appropriate • The questionnaire and proformas are appropriate and relevant to the study • Was the study adequately controlled? • If a “randomised trial” was randomisation truly random? • Were the groups comparable in all important aspects except for the variable being studied? • What outcome(s) were measured and how? • Was assessment of outcome (or, in a case-control study, allocation of caseness) “blind”? • Are there sources of bias in the setting of the subjects • Have confounding influences and multiple influences been removed • Was the study large enough and continued for long enough, and was follow up complete enough, to make the results credible?
Statistical aspects of a paper
• Have the authors set the scene correctly? ❖ Have they determined whether their groups are comparable and, if necessary, adjusted for baseline differences? ❖ What sort of data have they got and have they used appropriate statistical tests? ❖ If the statistical tests in the paper are obscure why have the authors chosen to use them? ❖ Have the data been analysed according to the original study protocol? • Paired data, tails, and outliers: ❖ Have the appropriate tests been used e.g. t-test for distributions, chi square tests for dichotomous values ❖ Were paired tests performed on paired data? ❖ Was a two tailed test performed whenever the effect of an intervention could conceivably be a negative one? ❖ Were outliers analysed with both common sense and appropriate statistical adjustments? • Correlation, regression and causation: ❖ Has correlation been distinguished from regression and has the correlation coefficient (r value) been calculated and interpreted correctly? ❖ Have assumptions been made about the nature and direction of causality? • Probability and confidence and degree of significance been interpreted correctly: ❖ Have P values been calculated and interpreted appropriately? ❖ Have confidence intervals been calculated and do the authors’ conclusions reflect them? • Have the authors expressed their results in terms of the likely harm or benefit that an individual patient can expect, eg: ❖ Relative risk reduction ❖ Absolute risk reduction ❖ Number needed to treat ❖ Odds ratio.
Result aspects of a paper
• Missing data such as drop outs, non-responders are accounted for • Details of the results such as the numbers , statistics, are accurate and clear • If statistics are appropriate to the study, then they were well used • The sample size is of a significant amount • Are the results clearly presented • Have they used bar charts, tables appropriately used
Discussion aspects of a paper
• The study is discussed critically • The results are discussed with reference to other important literature • The discussion and conclusions do not speculate too far beyond what has been shown in the study.
Checklist for a qualitative research paper
• Did the article describe an important clinical problem examined via a clearly formulated question? • Was the qualitative approach appropriate? • How were the setting and the subjects selected? • What was the researcher’s perspective and has this been taken into account? • What methods did the researcher use for collecting data—and are these described in enough detail? • What methods did the researcher use to analyse the data—and what quality control measures were implemented? • Are the results credible and if so are they clinically important? • What conclusions were drawn and are they justified by the results? • Are the findings of the study transferable to other clinical settings?
Checklist for material provided by a pharmaceutical company representative
• Does this material cover a subject that interests me and is clinically important in my practice? • Has this material been published in independent peer reviewed journals? Has any significant evidence been omitted from this presentation or withheld from publication? • Does the material include high level evidence such as systematic reviews, meta-analyses, or double blind randomised controlled trials against the drug’s closest competitor given at optimal dosage? • Have the trials or reviews examined a clearly focused, important and answerable clinical question that reflects a problem of relevance to patients?
Do they provide evidence on safety, tolerability, efficacy, and price? • Has each trial or meta-analysis defined the condition to be treated, the patients to be included, the interventions to be compared, and the outcomes to be examined? • Does the material provide direct evidence that the drug will help my patients live a longer, healthier, more productive, and symptom free life? • If a surrogate outcome measure has been used, what is the evidence that it is reliable, reproducible, sensitive, specific, a true predictor of disease, and rapidly reflects the response to therapy? • Do trial results indicate whether (and how) the effectiveness of the treatments differed and whether there was a difference in the type of incidence of adverse reactions? Are the results expressed in terms of numbers needed to treat, and are they clinically as well as statistically significant? • If large amounts of material have been provided by the representative, which three papers provide the strongest evidence for the company’s …show more content…
claims?
Checklist for a paper that claims to validate a diagnostic or screening test
• Is this test potentially relevant to my practice? • Has the test been compared with a true gold standard? • Did this validation study include an appropriate spectrum of subjects? • Has work up bias been avoided? • Has observer bias been avoided? • Was the test shown to be reproducible both within and between observers? • What are the features of the test as derived from this validation study? • Were confidence intervals given for sensitivity, specificity, and other features of the test? • Has a sensible “normal range” been derived from these results?
• Has this test been placed in the context of other potential tests in the diagnostic sequence for the condition?
Checklist for a set of clinical guidelines
• Did the preparation and publication of these guidelines involve a significant conflict of interest? • Are the guidelines concerned with an appropriate topic, and do they state clearly the goal of ideal treatment in terms of health and/or cost outcome? • Was the guideline development panel headed by a leading expert in the field (ideally it should not be) and was a specialist in the methodology of secondary research (for example, meta-analyst, health economist) involved? • Have all the relevant data been scrutinised and do the guidelines’ conclusions seem to be in keeping with the data? • Do they cover variations in medical practice and other controversial areas (for example, optimum care in response to genuine or perceived underfunding)? • Are the guidelines valid and reliable? • Are they clinically relevant, comprehensive, and flexible? • Do they take into account what is acceptable to, affordable by, and practically possible for
patients? • Do they include recommendations for their own dissemination, implementation, and periodic review?
Checklist for a systematic review or meta-analysis
• Did the review examine an important clinical question? • Was a thorough search done of the appropriate database(s) and were other potentially important sources explored? • Was methodological quality assessed and the trials weighted accordingly? • How sensitive are the results to the way the review has been done? • Have the numerical results been interpreted with common sense and due regard to the broader aspects of the problem?
Checklist for an economic analysis
• Is the analysis based on a study that answers a clearly defined clinical question about an economically important issue? • Whose viewpoint are costs and benefits considered from? • Have the interventions being compared been shown to be clinically effective? • Are the interventions sensible and workable in the settings where they are likely to be applied? • Which method of economic analysis was used and was this appropriate? ❖ If the interventions produced identical outcomes ( cost minimisation analysis ❖ If the important outcome is unidimensional ( cost-effectiveness analysis ❖ If the important outcome is multidimensional ( cost-utility analysis ❖ If the cost-benefit equation for this condition needs to be compared with cost-benefit equations for different conditions ( cost-benefit analysis ❖ If a cost-benefit analysis would otherwise be appropriate but the preference values given to different health states are disputed or likely to change ( cost-consequences analysis. • How were costs and benefits measured? • Were incremental rather than absolute benefits compared? • Was health status in the “here and now” given precedence over health status in the distant future? • Was sensitivity analysis performed? • Were “bottom line” aggregate scores overused?
Checklist for health care organisations working towards an evidence based culture for clinical and purchasing decisions
• Leadership—How often has information on effectiveness or evidence based medicine been discussed at board meetings in the past 12 months? Has the board taken time out to learn about developments in clinical and cost effectiveness? • Investment—What resources are the organisation investing in finding and using clinical effectiveness information? Is there a planned approach to promoting evidence based medicine that is properly resourced and staffed? • Using available resources—What action has been taken by the organisation in response to EL(93) 115 (Improving Clinical Effectiveness) and EL(94)74 (Improving the Effectiveness of the NHS)? What has changed in the organisation as a result? • Implementation—Who is responsible for receiving, acting on, and monitoring the implementation of Effective Health Care bulletins? What action has been taken on each of the bulletins issued to date? • Clinical guidelines—Who is responsible for receiving, acting on, and monitoring clinical practice guidelines? Do those arrangements ensure that both managers and clinicians play their part in guideline development and implementation? • Training—Has any training been provided to staff within the organisation (both clinical and non-clinical) on appraising and using evidence of effectiveness to influence clinical practice? • Contracts—How often does clinical and cost-effectiveness information form an important part of contract negotiation and agreement? How many contracts contain terms that set out how effectiveness information is to be used? • Incentives—What incentives—both individual and organisational—exist to encourage the practice of evidence based medicine? What disincentives exist to discourage inappropriate practice and unjustified variations in clinical decision making? • Information systems—Is the potential of existing information systems to monitor clinical effectiveness being used to the full? Is there a business case for new information systems to deal with the task, and is this issue being considered when purchasing decisions about information technology are made? • Clinical audit—Is there an effective clinical audit programme throughout the organisation, capable of examining issues of clinical effectiveness and bringing about appropriate changes in practice?
Checklist for searching Medline or the Cochrane library
• To look for an article you know exists, search by textwords (in title, abstract, or both) or use field suffixes for author, title, institution, journal, and publication year. • For a maximally sensitive search on a subject, search under both MeSH headings (exploded) and textwords (title and abstract), then combine the two by using the Boolean operator “or” • For a focused (specific) search on a clear cut topic, perform two or more sensitive searches as in (2), and combine them by using the Boolean operator “and”. • To find articles that are likely to be of high methodological quality, insert an evidence based quality filter for therapeutic interventions, aetiology, diagnostic procedures, or epidemiology and/or use maximally sensitive search strategies for randomised trials, systematic reviews, and meta-analyses. • Refine your search as you go along—for example, to exclude irrelevant material, use the Boolean operator “not”. • Use subheadings only when this is the only practicable way of limiting your search as manual indexers are fallible and misclassifications are common. • When limiting a large set, browse through the last 50 or so abstracts yourself rather than expecting the software to pick the best half dozen.