Internal Validity In Research
Internal validity - means that the conclusion which comes from an experiment is valid for the respondents who participated in the experiment. For any experimental design, we always require interval validity, without which experimental results are not interpretable.
Threats to Internal Validity
Selection - Subjects bring with them into the investigation unique characteristics, some learned and some inherent. Examples include sex, height, weight, color, attitude, personality, motor ability, and mental ability. If assigning subjects to comparison groups results in unequal distribution of these subject-related variables, then there is a possible threat to internal validity.
History - the specific events occurring between …show more content…
the first and second measurement in addition to the experimental variable. Suppose that the dependent variable is measured twice for a group of subjects, once at Time A and later at Time B, and that the independent variable is introduced in the interim. Suppose also that Event A occurs between Time A and Time B. If scores on the dependent measure differ at these two times, the discrepancy may be due to the independent variable or to Event A.
Maturation - processes within the respondents operating as a function of the passage of time per se (not specific to the particular events), including growing older, growing hungrier, growing more tired/bored, and the like.
Some of these changes are permanent (e.g., biological growth), while others are temporary (e.g., fatigue). Suppose that the dependent variable is measured twice for a group of subjects, once at Time A and later at Time B, and that the independent variable is introduced in the interim. If scores on the dependent measure differ at these two times, the discrepancy may be due to the independent variable or to naturally occurring developmental …show more content…
processes.
Testing - the effects of taking a test upon the scores of a second testing (like soc. desirability). Suppose that the dependent variable is recorded twice for a group of subjects, once at Time A and later at Time B, and that the independent variable is introduced in the interim. If scores on the dependent measure differ at these two times, the discrepancy may be due to the independent variable or to the procedure involved in measuring the dependent variable at Time A.
Instrumentation in which changes in the calibration of a measuring instrument or changes in the observers or scorers used may produce changes in the obtained measurements. Examples include changes in the calibration of a mechanical measuring device as well as the proficiency of a human observer or interviewer. Suppose that the dependent variable is measured twice for a group of subjects, once at Time A and later at Time B, and that the independent variable is introduced in the interim. Suppose also that the ability of a recording device to detect instances of the target behavior improves (declines) as the experiment progresses. If scores on the dependent measure differ at these two times, the discrepancy may be due to the independent variable or to more/less sensitive recordings of the target behavior at Time B relative to at Time A.
Regression - -operating where groups have been selected on the basis of their extreme scores (e.g., bootcamps for troubled teenagers). Take any dependent measure that is repeatedly sampled, move along it as in a time dimension, and pick a point that is the "highest/lowest so far. On the average, the next point will be lower/higher, nearer the general trend." Suppose that the dependent variable is measured twice for a group of subjects, once at Time A and later at Time B, and that the independent variable is introduced in the interim. Suppose also that value observed for subjects at Time A is considerably higher/lower than would typically be the case. If scores on the dependent measure differ at these two times, it may be due to the independent variable or to a regression artifact.
Mortality - or differential loss of respondents from the comparison groups. Suppose that subjects in two comparison groups differ with respect to the independent variable. Suppose also that subjects in one group are more likely to discontinue their participation part way through an experiment than subjects in another group and that dependent variable is measured at the end of the experiment. If scores on the dependent measure differ between those subjects remaining in the two groups, the discrepancy may be due to the independent variable or to a unique characteristic of subjects able to endure a particular condition, a subject-related variable that would be disproportionately present in each group.
Selection - Biases resulting in differential selection of respondents for the comparison groups.Subject-related variables and time-related variables may interact. Suppose that subjects in two comparison groups differ with respect to the independent variable and a subject-related variable such as age. Suppose also that the dependent variable is measured twice for each group, once at Time A and later at Time B, and that the independent variable is introduced in the interim. If the change in scores on the dependent measure from Time A to Time B differs between the two groups, this discrepancy may be due to the independent variable or to distinctive naturally occurring developmental processes for the two age categories that comprise the two comparison groups.
External Validity - asks the question of generalizability: to what extent can the experimental results be generalized? Many factors can jeopardize external validity or representativeness. The main ones are discussed as follows. When you have strong external validity, you can generalize to other people and situations with confidence. Public opinion surveys typically place considerable emphasis on defining the population of interest and drawing good samples from that population. On the other hand, laboratory experiments often employ "convenience samples," such as intact college classes taught by a friend. As a result, we may not know whom the subjects represent.
Threats to External Validity - the reactive effect of testing, in which a pretest might increase or decrease the respondent's sensitivity or responsiveness to the experimental variable and thus make the results obtained for a pretested population unrepresentative of the effects of the experimental variable for the unpretested universe from which the experimental respondents were selected. The interaction effects of selection biases and the experimental variable. Reactive effects of experimental arrangements, which would preclude generalization about the effect of the experimental variable upon persons being exposed to it in no experimental settings. A threat to external validity is an explanation of how you might be wrong in making a generalization. For instance, you conclude that the results of your study (which was done in a specific place, with certain types of people, and at a specific time) can be generalized to another context (for instance, another place, with slightly different people, at a slightly later time). There are three major threats to external validity because there are three ways you could be wrong -- people, places or times. Your critics could come along, for example, and argue that the results of your study are due to the unusual type of people who were in the study.
Question #2
Differences between online and traditional focus groups
Abstract
The purpose of my research is to identify differences between online and traditional focus groups. Personally, I have participated in Traditional Focus group 6 months ago. We discussed alternatives to a Bank (popular in Kazakhstan) video and billboard advertisements. Finally, I saw a billboard in Almaty, the image and content was exactly the same that we chose. When I took part in traditional Focus group, I didn't think about online focus group exist. That's why this topic is very interesting to discuss.
Focus groups First of all, I would like to define the meaning of focus group. Focus groups are a method of group interviewing in which the interaction between the moderator and the group, as well as the interaction between group members, serves to elicit information and insights in response to carefully designed questions. The dynamic nature of the questions asked by the moderator and the group process, produces a level of insight that is rarely derived from 'unidirectional' information collection devises such as observation, surveys and less interactional interview techniques. Methods of recording and analyzing information gathered during focus groups, and strategies for collecting unbiased information have helped focus group research to gain credibility as an accurate and useful source of information collection.
Focus group methods gained popularity in marketing research. In the 1980s social scientists recognized the value of focus groups for qualitative research and adapted the techniques accordingly. In the 1990s focus group strategies have become widely researched and used in social sciences and human service organizations.
Advantages and Disadvantages of Focus Groups The decision of whether to use focus groups for a given evaluation project depends on the strengths and limitations of focus groups in contrast to other evaluation techniques. Below, are three ways of collecting information for program evaluation and how the process and results might differ from focus groups.
1) Naturalistic observation has some advantages over focus groups. Focus groups are conducted in an unnatural social setting. The presence and direction of the moderator may influence responses that might be different in a more natural setting. While naturalistic observation allows for observation of a broader range of information and potentially a more open discussion, focus groups set an agenda and use questioning strategies that influence the group process.
2) Individual interviews are more efficient that focus groups and interviewers are typically able to cover more ground interviewing one person versus a group. While focus groups may actually get at less information that a one hour individual interview, the dynamic interchange between the group members may result in more in depth and unbiased information concerning a particular topic. A potential weakness of focus groups may occur when members do not express their personal opinions and conform to a popular opinion or acquiesce to a particular group member. Strategies for increasing an open exchange of ideas will be discussed below. Focus groups can be used in tandem with individual interviews. Evaluators may use focus groups to initially explore issues and then seek expanded (perhaps private) information through individual interviews.
3) Questionnaires compared to focus groups are relatively easy and inexpensive to create, analyze and communicate the findings. Questionnaires may be administered to the masses while focus groups typically elicit information from only 8 to 24 people who hopefully represent the population being investigated. Questionnaires may include as many questions as the evaluator thinks the respondents will complete, while focus group moderators have to crack the whip in order to get responses to five or so key questions. So why use a focus group?? In some cases evaluators want to gain a deeper understanding of the issues. Focus groups are able to delve much deeper into issues than questionnaires.
Many evaluators use both focus groups and surveys. Surveys may indicate areas that require further probing through focus groups. Focus groups may also be used in the creation of surveys. By discussing the area in need of investigation, focus groups can illuminate key points about the topic so that the survey is comprehensive enough. Focus groups can also clarify terminology used to describe what is being investigated. Maximizing knowledge about key concerns and effective communication with the potential focus group participants helps developers of surveys create questions that are understood in the same way by all respondents. The human race has always loved offering advice and opinion. These days, many companies organize focus groups in order to receive specific feedback. A group of people who have an interest in the company’s product or service are gathered together, and are asked specific questions on topics including the direction of the company, what new products should be offered, and possible changes. Here is an outline of the most important considerations that a marketing professional should consider when making a decision between using online or traditional focus groups. Specifically, we would like to mention some of the key reasons why traditional focus groups are effective, and online version could not overcome its problems by offering somewhat lower costs and faster timing.
* The authority role of the moderator is one of the most important reasons why traditional focus groups are so important. An experienced moderator is in complete charge of the group activities and is able to ensure that everyone participates and that the focus of the discussion remains on target.
It is virtually impossible to establish authority from behind a computer screen.
* One of the major benefits of traditional focus groups is the interaction among the various participants. A well conducted focus group utilizes this interaction to explore topics in more detail and to draw out the feelings of each of the participants based on their reactions to what others in the room have said.
This is not viable in an Internet environment.
* A competent focus group moderator will use non-verbal cues from participants to direct the discussion in the room. Often the non-verbal inputs can be as important as the verbal in determining the reactions to various ideas.
It is impossible to address non-verbal reactions in an online focus group.
Security
* There is dramatically more security in a traditional focus group than in the online version. All participants in the traditional focus group are re-screened at the facility with photo Ids. Thus it is almost impossible to have someone in the session who should not be there.
In the online environment, however, it is impossible to tell who is behind the computer screen. Moreover, it is possible for stimuli from an online session to be left in cyberspace for others to find and utilize to their benefit.
* Online focus groups also lose the very significant benefit of client involvement with the end user. In traditional focus groups, with client personnel viewing the sessions live behind a one-way mirror (or via remote broadcast locations), the communications of the group are much more convincing than in an online environment. With the client in the back room, it is easy for them to become part of the overall process as they can talk to the moderator during the session and make suggestions relative to areas that need to be discussed.
* Finally, it is much easier and more effective to show stimuli to the participants in a live setting than in an on-line focus group. While it definitely is possible to send stimuli, such as pictures or a concept statement, to participants in cyber groups, the full impact of this material is often lost in the two dimensional environment of the computer screen. The vast differences in the quality of screens and the ability of users to download the materials further exacerbate this situation. It is essential that the marketing community not become too enthralled with using the Internet to conduct qualitative research. While there may be a role for this vehicle in the very earliest stages of the process, it clearly is not a viable substitute for well-run, traditional focus groups. Online focus groups often can be accomplished faster than traditional groups because respondents are recruited from online panel members who are often qualified to match research
criteria. Taken into account all cons and pros of both traditional and online focus groups each method is good in a particular situation. Finally, I would like to notice that if there is no need to involve people outside the country or city then it is more effective to use traditional focus groups. The reason is that you interact with people in reality and you can see body language and the fairness of a person. However, if there is a need to organize focus group with people around the world it's effective and efficient to conduct online focus group.
Threats to Internal Validity
Selection - Subjects bring with them into the investigation unique characteristics, some learned and some inherent. Examples include sex, height, weight, color, attitude, personality, motor ability, and mental ability. If assigning subjects to comparison groups results in unequal distribution of these subject-related variables, then there is a possible threat to internal validity.
History - the specific events occurring between …show more content…
the first and second measurement in addition to the experimental variable. Suppose that the dependent variable is measured twice for a group of subjects, once at Time A and later at Time B, and that the independent variable is introduced in the interim. Suppose also that Event A occurs between Time A and Time B. If scores on the dependent measure differ at these two times, the discrepancy may be due to the independent variable or to Event A.
Maturation - processes within the respondents operating as a function of the passage of time per se (not specific to the particular events), including growing older, growing hungrier, growing more tired/bored, and the like.
Some of these changes are permanent (e.g., biological growth), while others are temporary (e.g., fatigue). Suppose that the dependent variable is measured twice for a group of subjects, once at Time A and later at Time B, and that the independent variable is introduced in the interim. If scores on the dependent measure differ at these two times, the discrepancy may be due to the independent variable or to naturally occurring developmental …show more content…
processes.
Testing - the effects of taking a test upon the scores of a second testing (like soc. desirability). Suppose that the dependent variable is recorded twice for a group of subjects, once at Time A and later at Time B, and that the independent variable is introduced in the interim. If scores on the dependent measure differ at these two times, the discrepancy may be due to the independent variable or to the procedure involved in measuring the dependent variable at Time A.
Instrumentation in which changes in the calibration of a measuring instrument or changes in the observers or scorers used may produce changes in the obtained measurements. Examples include changes in the calibration of a mechanical measuring device as well as the proficiency of a human observer or interviewer. Suppose that the dependent variable is measured twice for a group of subjects, once at Time A and later at Time B, and that the independent variable is introduced in the interim. Suppose also that the ability of a recording device to detect instances of the target behavior improves (declines) as the experiment progresses. If scores on the dependent measure differ at these two times, the discrepancy may be due to the independent variable or to more/less sensitive recordings of the target behavior at Time B relative to at Time A.
Regression - -operating where groups have been selected on the basis of their extreme scores (e.g., bootcamps for troubled teenagers). Take any dependent measure that is repeatedly sampled, move along it as in a time dimension, and pick a point that is the "highest/lowest so far. On the average, the next point will be lower/higher, nearer the general trend." Suppose that the dependent variable is measured twice for a group of subjects, once at Time A and later at Time B, and that the independent variable is introduced in the interim. Suppose also that value observed for subjects at Time A is considerably higher/lower than would typically be the case. If scores on the dependent measure differ at these two times, it may be due to the independent variable or to a regression artifact.
Mortality - or differential loss of respondents from the comparison groups. Suppose that subjects in two comparison groups differ with respect to the independent variable. Suppose also that subjects in one group are more likely to discontinue their participation part way through an experiment than subjects in another group and that dependent variable is measured at the end of the experiment. If scores on the dependent measure differ between those subjects remaining in the two groups, the discrepancy may be due to the independent variable or to a unique characteristic of subjects able to endure a particular condition, a subject-related variable that would be disproportionately present in each group.
Selection - Biases resulting in differential selection of respondents for the comparison groups.Subject-related variables and time-related variables may interact. Suppose that subjects in two comparison groups differ with respect to the independent variable and a subject-related variable such as age. Suppose also that the dependent variable is measured twice for each group, once at Time A and later at Time B, and that the independent variable is introduced in the interim. If the change in scores on the dependent measure from Time A to Time B differs between the two groups, this discrepancy may be due to the independent variable or to distinctive naturally occurring developmental processes for the two age categories that comprise the two comparison groups.
External Validity - asks the question of generalizability: to what extent can the experimental results be generalized? Many factors can jeopardize external validity or representativeness. The main ones are discussed as follows. When you have strong external validity, you can generalize to other people and situations with confidence. Public opinion surveys typically place considerable emphasis on defining the population of interest and drawing good samples from that population. On the other hand, laboratory experiments often employ "convenience samples," such as intact college classes taught by a friend. As a result, we may not know whom the subjects represent.
Threats to External Validity - the reactive effect of testing, in which a pretest might increase or decrease the respondent's sensitivity or responsiveness to the experimental variable and thus make the results obtained for a pretested population unrepresentative of the effects of the experimental variable for the unpretested universe from which the experimental respondents were selected. The interaction effects of selection biases and the experimental variable. Reactive effects of experimental arrangements, which would preclude generalization about the effect of the experimental variable upon persons being exposed to it in no experimental settings. A threat to external validity is an explanation of how you might be wrong in making a generalization. For instance, you conclude that the results of your study (which was done in a specific place, with certain types of people, and at a specific time) can be generalized to another context (for instance, another place, with slightly different people, at a slightly later time). There are three major threats to external validity because there are three ways you could be wrong -- people, places or times. Your critics could come along, for example, and argue that the results of your study are due to the unusual type of people who were in the study.
Question #2
Differences between online and traditional focus groups
Abstract
The purpose of my research is to identify differences between online and traditional focus groups. Personally, I have participated in Traditional Focus group 6 months ago. We discussed alternatives to a Bank (popular in Kazakhstan) video and billboard advertisements. Finally, I saw a billboard in Almaty, the image and content was exactly the same that we chose. When I took part in traditional Focus group, I didn't think about online focus group exist. That's why this topic is very interesting to discuss.
Focus groups First of all, I would like to define the meaning of focus group. Focus groups are a method of group interviewing in which the interaction between the moderator and the group, as well as the interaction between group members, serves to elicit information and insights in response to carefully designed questions. The dynamic nature of the questions asked by the moderator and the group process, produces a level of insight that is rarely derived from 'unidirectional' information collection devises such as observation, surveys and less interactional interview techniques. Methods of recording and analyzing information gathered during focus groups, and strategies for collecting unbiased information have helped focus group research to gain credibility as an accurate and useful source of information collection.
Focus group methods gained popularity in marketing research. In the 1980s social scientists recognized the value of focus groups for qualitative research and adapted the techniques accordingly. In the 1990s focus group strategies have become widely researched and used in social sciences and human service organizations.
Advantages and Disadvantages of Focus Groups The decision of whether to use focus groups for a given evaluation project depends on the strengths and limitations of focus groups in contrast to other evaluation techniques. Below, are three ways of collecting information for program evaluation and how the process and results might differ from focus groups.
1) Naturalistic observation has some advantages over focus groups. Focus groups are conducted in an unnatural social setting. The presence and direction of the moderator may influence responses that might be different in a more natural setting. While naturalistic observation allows for observation of a broader range of information and potentially a more open discussion, focus groups set an agenda and use questioning strategies that influence the group process.
2) Individual interviews are more efficient that focus groups and interviewers are typically able to cover more ground interviewing one person versus a group. While focus groups may actually get at less information that a one hour individual interview, the dynamic interchange between the group members may result in more in depth and unbiased information concerning a particular topic. A potential weakness of focus groups may occur when members do not express their personal opinions and conform to a popular opinion or acquiesce to a particular group member. Strategies for increasing an open exchange of ideas will be discussed below. Focus groups can be used in tandem with individual interviews. Evaluators may use focus groups to initially explore issues and then seek expanded (perhaps private) information through individual interviews.
3) Questionnaires compared to focus groups are relatively easy and inexpensive to create, analyze and communicate the findings. Questionnaires may be administered to the masses while focus groups typically elicit information from only 8 to 24 people who hopefully represent the population being investigated. Questionnaires may include as many questions as the evaluator thinks the respondents will complete, while focus group moderators have to crack the whip in order to get responses to five or so key questions. So why use a focus group?? In some cases evaluators want to gain a deeper understanding of the issues. Focus groups are able to delve much deeper into issues than questionnaires.
Many evaluators use both focus groups and surveys. Surveys may indicate areas that require further probing through focus groups. Focus groups may also be used in the creation of surveys. By discussing the area in need of investigation, focus groups can illuminate key points about the topic so that the survey is comprehensive enough. Focus groups can also clarify terminology used to describe what is being investigated. Maximizing knowledge about key concerns and effective communication with the potential focus group participants helps developers of surveys create questions that are understood in the same way by all respondents. The human race has always loved offering advice and opinion. These days, many companies organize focus groups in order to receive specific feedback. A group of people who have an interest in the company’s product or service are gathered together, and are asked specific questions on topics including the direction of the company, what new products should be offered, and possible changes. Here is an outline of the most important considerations that a marketing professional should consider when making a decision between using online or traditional focus groups. Specifically, we would like to mention some of the key reasons why traditional focus groups are effective, and online version could not overcome its problems by offering somewhat lower costs and faster timing.
* The authority role of the moderator is one of the most important reasons why traditional focus groups are so important. An experienced moderator is in complete charge of the group activities and is able to ensure that everyone participates and that the focus of the discussion remains on target.
It is virtually impossible to establish authority from behind a computer screen.
* One of the major benefits of traditional focus groups is the interaction among the various participants. A well conducted focus group utilizes this interaction to explore topics in more detail and to draw out the feelings of each of the participants based on their reactions to what others in the room have said.
This is not viable in an Internet environment.
* A competent focus group moderator will use non-verbal cues from participants to direct the discussion in the room. Often the non-verbal inputs can be as important as the verbal in determining the reactions to various ideas.
It is impossible to address non-verbal reactions in an online focus group.
Security
* There is dramatically more security in a traditional focus group than in the online version. All participants in the traditional focus group are re-screened at the facility with photo Ids. Thus it is almost impossible to have someone in the session who should not be there.
In the online environment, however, it is impossible to tell who is behind the computer screen. Moreover, it is possible for stimuli from an online session to be left in cyberspace for others to find and utilize to their benefit.
* Online focus groups also lose the very significant benefit of client involvement with the end user. In traditional focus groups, with client personnel viewing the sessions live behind a one-way mirror (or via remote broadcast locations), the communications of the group are much more convincing than in an online environment. With the client in the back room, it is easy for them to become part of the overall process as they can talk to the moderator during the session and make suggestions relative to areas that need to be discussed.
* Finally, it is much easier and more effective to show stimuli to the participants in a live setting than in an on-line focus group. While it definitely is possible to send stimuli, such as pictures or a concept statement, to participants in cyber groups, the full impact of this material is often lost in the two dimensional environment of the computer screen. The vast differences in the quality of screens and the ability of users to download the materials further exacerbate this situation. It is essential that the marketing community not become too enthralled with using the Internet to conduct qualitative research. While there may be a role for this vehicle in the very earliest stages of the process, it clearly is not a viable substitute for well-run, traditional focus groups. Online focus groups often can be accomplished faster than traditional groups because respondents are recruited from online panel members who are often qualified to match research
criteria. Taken into account all cons and pros of both traditional and online focus groups each method is good in a particular situation. Finally, I would like to notice that if there is no need to involve people outside the country or city then it is more effective to use traditional focus groups. The reason is that you interact with people in reality and you can see body language and the fairness of a person. However, if there is a need to organize focus group with people around the world it's effective and efficient to conduct online focus group.