Foundations of Knowledge
FOUNDATIONS OF KNOWLEDGE
BECOMING A CRITICAL READER OF QUANTITATIVE RESEARCH
RE-SEARCH
methodical investigation to seek answers that involve explanation and understanding Positivism – falsification and replication all research has flaws limited resources & ‘the least worst option’ is research ‘fit for purpose?’ research is presented as if ‘fit for purpose’ facts, findings and critical reading
Tim Hartford – but there are more Ben Goldacre (Guardian), Seife (Proofiness)
WHY BE A CRITICAL READER?
The seduction & authority of numbers “Proofiness” (Siefe 2010)
the dark arts of mathematical deception Potemkin numbers dis-estimation fruit packing & cherry picking
WHY BE A CRITICAL READER?
numbers can clarify but also confuse or misdirect?
Examples 1. Fish oil mothers depression and child intelligence
2. School-age drinking and social networking sites http://www.straightstatistics.org/article/seeing-doubleover-school-age-drinking-wales http://www.straightstatistics.org/article/fishing-significance
3. Improvements in re-offending
http://www.straightstatistics.org/article/bent-statisticsgoing-straight
? Some possibly deliberately falsify?
e.g. Sir Cyril Burt’s study of intelligence in twins & 2 tiered education system & 11+
BE A CRITICAL READER…..
Look for potential flaws in the statistical analysis
But do not assume that all studies are equal (research design)
Randomised control trails (experiments) Longitudinal studies One-time cross sectional surveys
e.g. Fish oil mothers depression and child intelligence http://www.straightstatistics.org/article/fishingsignificance
Several reports * The Times. The Daily Telegraph, the Guardian, the Daily Mail, The Sun, the Independent and the Daily Mirror • Lancet published 2007 • eat less than 340g fish a week & children in lowest quartile One report only in British Newspapers (daily telegraph) * randomised control trail *Journal of the American Medical Association, October 2010
CRITICAL READING (NOT JUST BEING NEGATIVE)
What are the arguments? Where do they come from? What data have they got to support them? plus data transparency
What is the data? Is it fit for purpose? i.e. measured well? How was the data collected? i.e. who is it from (sample) What is the research design & analysis – it is reported in detail? Can you evaluate it? • What is not reported? • • • •
Note: You will need to demonstrate transparency in your dissertation studies – so start practicing and evaluating what others studies do.
QUANTITATIVE METHODS
Usually associated with philosophy of positivism Associated with ‘survey’, standardised information, & large scale (sample)
OFFICE OF NATIONAL STATISTICS (ONS) SURVEYS
PHILOSOPHY-OF-SCIENCE ISSUES
• Positivism
– There is a world ‘out there’ that exists prior to, and independent of the research and the researcher – It is possible to discover, know or find out something about this world (through certain types of research practice) – It is possible to discover causal relationships in social phenomena – Theories about the world must be tested using evidence in order for them to be accepted or trusted
CAN RESEARCHERS BE ‘NEUTRAL’?
• to be as objective and value-free as possible. • to find out how the world really is, value judgements about whether these findings are good or bad is for others (you) to debate and decide • researchers need to ensure their research has
– Validity – we need to be as sure as possible that a research method actually measures (without bias) what it claims to measure – Reliability – we need to ensure and check that results are stable and consistent
QUANTITATIVE METHODS
Descriptive or Explanatory Sometimes exploratory
Expressed numerically & analysed statistically
BUT - Is a number always a number?
COUNTING NUMERICALLY
• How old are you? 18-24 years 25-30years 31-35 years 36-40 years 41-50 years 51-60 years
How many cigarettes
did you smoke yesterday?
None 1-4 cigarettes 5-10 cigarettes 11-20 cigarettes 20+ cigarettes
COUNTING NUMERICALLY
• How old are you? 28 (please specify)
How many cigarettes did you
smoke yesterday? (please specify)
4
DATA
Facts and figures collected, statistically manipulated and reported Also known as ‘variables’ or ‘questions in survey’ or ‘observation’
Cross sectional or time series
Statistically manipulated to produce the research findings
DATA
A critical reader needs to evaluate the data that has been collected The GIGO problem with quant methods
DATA
Garbage in – garbage out
The key is Measurement and Research Design
• Research design begins (and ends) with the research questions • RQ’s drive the measurement of data, data collection and data analysis • even if the question in the questionnaire is measured well poor research design will result in poor, inaccurate or off the wall data
SOME QUESTIONS?
What is the average cost of a wedding? Is marriage getting more popular? Where do people celebrate their marriages? What do guests like about weddings? Why do married men get paid 20% more than single men? How many people watched the royal wedding?
BUT WHAT HAS BEEN MEASURED?
Measurement and content validity Measuring how many people watched the royal wedding? The marriage premium & proxy variables Is wages a good way to measure productivity? Sampling and Zombie statistics
DATA – LEVEL OF MEASUREMENT
• • • • How to know which statistical tools can be used? Levels of measurement Classifies the character of data Four possible levels of measurement
• • • • Nominal Ordinal Internal Ratio
NOMINAL (OR CATEGORICAL) DATA
has values which have no numerical value classify data into categories this process involves labelling categories and then counting frequencies of occurrence there is no order or sequence in the values of nominal variables values must be mutually exclusive can be dichotomous or have several values
• Gender, occupation
ORDINAL DATA
• values whose order is significant, but on which no meaningful arithmetic-like operations can be performed.
• greatly dislike > moderately dislike, but • indifferent / moderately like = ? • quite useful for subjective assessment of 'quality’ and ‘preferences’
INTERVAL DATA
• An ordinal variable with the additional property that the magnitudes of the differences between two values are meaningful. • Thus the order of data is known as well as the precise numeric distance between data points
• Analyze the actual percentage scores of the essays (assuming they are given by the instructor). • Time 8 PM > 6AM but 10 PM * 2 hrs = ?
RATIO DATA
• A variable with the features of interval variable and, additionally, whose any two values have meaningful ratio, making the operations of multiplication and division meaningful.
Type of Variable
Is there a true zero point?
Are distances between categories equal?
Can the categories be ranked or ordered?
Ratio
Interval
Yes
No
Yes
Yes
Yes
Yes
Ordinal
Nominal
No
No
No
No
Yes
No
Summary –we will come back to this levels of measurement determine appropriate analysis
increasing level of sophistication
Discrete (Non-metric) Non-parametric statistics small sample sizes large sample sizes
Continuous (Metric) Parametric statistics
non-numerical/ categorical non-numerical no rank order mutually exclusive no equal intervals ordered data no fixed zero no equal intervals
Nominal
Ordinal
Interval numerical & ordered data no fixed zero point equal intervals
Ratio
numerical & ordered data fixed & known zero point
IF YOU ENJOYED THIS, TRY THESE…
BBC Radio 4 programme “More or Less” http://www.bbc.co.uk/programmes/b006qshd Royal Statistics Society videos http://scijourntraining.wordpress.com/2011/07/11/behind -the-numbers-video/
XKCD - A webcomic of romance,sarcasm, math, and language http://xkcd.com/
BECOMING A CRITICAL READER OF QUANTITATIVE RESEARCH
RE-SEARCH
methodical investigation to seek answers that involve explanation and understanding Positivism – falsification and replication all research has flaws limited resources & ‘the least worst option’ is research ‘fit for purpose?’ research is presented as if ‘fit for purpose’ facts, findings and critical reading
Tim Hartford – but there are more Ben Goldacre (Guardian), Seife (Proofiness)
WHY BE A CRITICAL READER?
The seduction & authority of numbers “Proofiness” (Siefe 2010)
the dark arts of mathematical deception Potemkin numbers dis-estimation fruit packing & cherry picking
WHY BE A CRITICAL READER?
numbers can clarify but also confuse or misdirect?
Examples 1. Fish oil mothers depression and child intelligence
2. School-age drinking and social networking sites http://www.straightstatistics.org/article/seeing-doubleover-school-age-drinking-wales http://www.straightstatistics.org/article/fishing-significance
3. Improvements in re-offending
http://www.straightstatistics.org/article/bent-statisticsgoing-straight
? Some possibly deliberately falsify?
e.g. Sir Cyril Burt’s study of intelligence in twins & 2 tiered education system & 11+
BE A CRITICAL READER…..
Look for potential flaws in the statistical analysis
But do not assume that all studies are equal (research design)
Randomised control trails (experiments) Longitudinal studies One-time cross sectional surveys
e.g. Fish oil mothers depression and child intelligence http://www.straightstatistics.org/article/fishingsignificance
Several reports * The Times. The Daily Telegraph, the Guardian, the Daily Mail, The Sun, the Independent and the Daily Mirror • Lancet published 2007 • eat less than 340g fish a week & children in lowest quartile One report only in British Newspapers (daily telegraph) * randomised control trail *Journal of the American Medical Association, October 2010
CRITICAL READING (NOT JUST BEING NEGATIVE)
What are the arguments? Where do they come from? What data have they got to support them? plus data transparency
What is the data? Is it fit for purpose? i.e. measured well? How was the data collected? i.e. who is it from (sample) What is the research design & analysis – it is reported in detail? Can you evaluate it? • What is not reported? • • • •
Note: You will need to demonstrate transparency in your dissertation studies – so start practicing and evaluating what others studies do.
QUANTITATIVE METHODS
Usually associated with philosophy of positivism Associated with ‘survey’, standardised information, & large scale (sample)
OFFICE OF NATIONAL STATISTICS (ONS) SURVEYS
PHILOSOPHY-OF-SCIENCE ISSUES
• Positivism
– There is a world ‘out there’ that exists prior to, and independent of the research and the researcher – It is possible to discover, know or find out something about this world (through certain types of research practice) – It is possible to discover causal relationships in social phenomena – Theories about the world must be tested using evidence in order for them to be accepted or trusted
CAN RESEARCHERS BE ‘NEUTRAL’?
• to be as objective and value-free as possible. • to find out how the world really is, value judgements about whether these findings are good or bad is for others (you) to debate and decide • researchers need to ensure their research has
– Validity – we need to be as sure as possible that a research method actually measures (without bias) what it claims to measure – Reliability – we need to ensure and check that results are stable and consistent
QUANTITATIVE METHODS
Descriptive or Explanatory Sometimes exploratory
Expressed numerically & analysed statistically
BUT - Is a number always a number?
COUNTING NUMERICALLY
• How old are you? 18-24 years 25-30years 31-35 years 36-40 years 41-50 years 51-60 years
How many cigarettes
did you smoke yesterday?
None 1-4 cigarettes 5-10 cigarettes 11-20 cigarettes 20+ cigarettes
COUNTING NUMERICALLY
• How old are you? 28 (please specify)
How many cigarettes did you
smoke yesterday? (please specify)
4
DATA
Facts and figures collected, statistically manipulated and reported Also known as ‘variables’ or ‘questions in survey’ or ‘observation’
Cross sectional or time series
Statistically manipulated to produce the research findings
DATA
A critical reader needs to evaluate the data that has been collected The GIGO problem with quant methods
DATA
Garbage in – garbage out
The key is Measurement and Research Design
• Research design begins (and ends) with the research questions • RQ’s drive the measurement of data, data collection and data analysis • even if the question in the questionnaire is measured well poor research design will result in poor, inaccurate or off the wall data
SOME QUESTIONS?
What is the average cost of a wedding? Is marriage getting more popular? Where do people celebrate their marriages? What do guests like about weddings? Why do married men get paid 20% more than single men? How many people watched the royal wedding?
BUT WHAT HAS BEEN MEASURED?
Measurement and content validity Measuring how many people watched the royal wedding? The marriage premium & proxy variables Is wages a good way to measure productivity? Sampling and Zombie statistics
DATA – LEVEL OF MEASUREMENT
• • • • How to know which statistical tools can be used? Levels of measurement Classifies the character of data Four possible levels of measurement
• • • • Nominal Ordinal Internal Ratio
NOMINAL (OR CATEGORICAL) DATA
has values which have no numerical value classify data into categories this process involves labelling categories and then counting frequencies of occurrence there is no order or sequence in the values of nominal variables values must be mutually exclusive can be dichotomous or have several values
• Gender, occupation
ORDINAL DATA
• values whose order is significant, but on which no meaningful arithmetic-like operations can be performed.
• greatly dislike > moderately dislike, but • indifferent / moderately like = ? • quite useful for subjective assessment of 'quality’ and ‘preferences’
INTERVAL DATA
• An ordinal variable with the additional property that the magnitudes of the differences between two values are meaningful. • Thus the order of data is known as well as the precise numeric distance between data points
• Analyze the actual percentage scores of the essays (assuming they are given by the instructor). • Time 8 PM > 6AM but 10 PM * 2 hrs = ?
RATIO DATA
• A variable with the features of interval variable and, additionally, whose any two values have meaningful ratio, making the operations of multiplication and division meaningful.
Type of Variable
Is there a true zero point?
Are distances between categories equal?
Can the categories be ranked or ordered?
Ratio
Interval
Yes
No
Yes
Yes
Yes
Yes
Ordinal
Nominal
No
No
No
No
Yes
No
Summary –we will come back to this levels of measurement determine appropriate analysis
increasing level of sophistication
Discrete (Non-metric) Non-parametric statistics small sample sizes large sample sizes
Continuous (Metric) Parametric statistics
non-numerical/ categorical non-numerical no rank order mutually exclusive no equal intervals ordered data no fixed zero no equal intervals
Nominal
Ordinal
Interval numerical & ordered data no fixed zero point equal intervals
Ratio
numerical & ordered data fixed & known zero point
IF YOU ENJOYED THIS, TRY THESE…
BBC Radio 4 programme “More or Less” http://www.bbc.co.uk/programmes/b006qshd Royal Statistics Society videos http://scijourntraining.wordpress.com/2011/07/11/behind -the-numbers-video/
XKCD - A webcomic of romance,sarcasm, math, and language http://xkcd.com/