I-130 Immigration Benefit Fraud Assessment
The discussion topics for the I-130 Immigration Benefit Fraud Assessment (IBFA) before the Technical Working Group (TWG) are: 1) the sample size, 2) the operational definitional of fraud, and 3) the cut-off date for case inclusion. OP&S is dedicated to ensuring USCIS stakeholders and senior leaderships have relevant information in order to develop realistic expectations of the study. The role of the TWG is to discuss and provide consolidated recommendations to USCIS Senior Leadership so that they can make informed decisions for the IBFA study. Each of these issues has the potential to limit the usefulness of the estimated fraud rate and should be discussed thoroughly prior to making any recommendations to Senior Leadership.
1. Sample Size …show more content…
The goal of the I-130 marriage-based IBFA is to generate a defensible fraud rate for the I-130 marriage-based petition. IFRET recognizes the need to select a sample size large enough for the study but small enough so that it is operationally feasible for our operational staff to complete cases for the study.
The calculated sample size for a research study is a product of two factors: the estimated fraud rate and the level of precision (often discussed as the “margin of error” ). The previous I-130 marriage-based BFCA conducted by FDNS in 2005, which was completed in 2008, estimated a fraud rate of 17 percent. Using that assumed fraud rate, IFRET calculated a sample size of 217 cases with a margin of error of 5 percent. If, however, IFRET assumes a fraud rate of 50 percent (a standard assumption if a rate is unknown ), IFRET would need to construct a sample of 384 to achieve the same margin of error of 5 percent.
IFRET recommended USCIS to draw a sample size of 217 cases for two reasons. First, it is standard practice in research to rely on previous research findings to calculate estimated sample size for a research study. Although there were some methodological problems with the previous BFCA, these problems do not necessarily impugn the fraud estimate. Second, the benefit of drawing a sample size of 217 is that this sample size will not unduly interfere with production at the Field Office level. Table 1 provides the differing assumed fraud rates and the resulting sample size based on a 5% margin of error.
Table 1: Differing Fraud Rates and Resulting Sample Size Based on 5% Margin of Error
Assumed Fraud Rate Sample Size
5% 73
10% 138
17% 217
25% 288
50% 384
At the recent TWG meeting, some TWG member raised concerns about the sample size and suggested that it should be larger; however, increasing the sample will impact production and nearly double the estimate for manpower hours needed to complete the study. The cost for an increase in sample size, therefore, is an increase in the work load for production staff and the project will be at risk of not being completed in a timely manner. In addition, if a larger sample size is desired, FOD will necessarily need to adjust operations in such a way that normal production may be impacted. The benefit of having a larger sample size is that should the estimated fraud rate falls below the assumed fraud rate of 50 percent, the margin of error will be smaller, and the estimated fraud rate will be more precise.
IFRET believes that a national fraud rate can be estimated using the smaller sample size if USCIS is prepared to have a wider confidence interval (lower precision) for the estimates should the fraud rate be higher than 17 percent. If the estimated fraud rate is higher than the assumed 17 percent, the margin of error increases. Even if the fraud rate is estimated at 50 percent, a sample size of 217 will provide an estimated fraud rate with a margin of error of 7 percent (see Figure 1). A slight decrease in precision will result in a dramatic reduction in required sample size of 384 to 217. Figure 1 illustrates the trade-off between sample size and precision.
An additional risk for using the 17 percent fraud rate from a prior unreleased study, USCIS should be prepared to handle any inquiries as to why that fraud rate was used and prepare a response to any questions regarding why the previous study was not released to the public. In addition, once the unreleased study is referenced in the new IBFA and is made known to the public, USCIS needs to understand that the former BFCA may be FOIAed.
Figure 1 Sample Size by Margin of Error Varying Fraud Rates
2. Operational Definition of Fraud
USCIS Senior Leadership determined that the definition of a fraud will be a case in which the adjudicator will deny based on fraud.
However, IFRET’s previous research (Qualitative Research on Marriage-Based Immigration Benefit Fraud Detection) identified a number of strategies employed by ISOs to stop fraudulent cases from continuing through the adjudication process including denying petitions for other reasons (in which case, the “fraud” is not captured in USCIS data systems) and issuing a Notice of Intent to Deny (NOID). Each of these strategies demonstrate different thresholds for the concept of “fraud” with case denial for a reason other than fraud being the lowest threshold and the Agency definition of a fraud finding the highest threshold. IFRET has been tasked to estimate the “true” fraud rate, meaning IFRET constructed a research design that would capture these lower thresholds for fraud. In this way, IFRET believes the study will be able to calculate a confidence interval for the “true” fraud rate by capturing all instances of fraud irrespective of a formal Agency finding. IFRET agreed, however, to compromise and use the mid-level measure of fraud: the issuance of a NOID. It should be noted that this definition will generate a fraud rate that may be unrealistically low as not all fraudulent cases can be denied for fraud due to the stringent criteria set up by the immigration law and
policies.
Previous IFRET research on fraud rates based on a finding of fraud calculated a fraud rate of 2.5%. This number is based on the normal adjudicative process wherein only cases that are suspected of fraud are referred to FDNS for closer scrutiny and different investigative procedures. IFRET’s IBFA research design is predicated on the approach of viewing all cases as suspected fraud; consequently, all cases will be subject to the same procedures utilized when a case is referred for fraud to FDNS. While IFRET expects that the study will show a higher fraud rate than 2.5%, the restriction of using an adjudicated denial for fraud as the metric for operationally measuring fraud is likely to produce a downward biased “true” fraud rate. For example, even if the fraud rate IFRET calculates doubles to 5% (from the above 2.5% rate), when placing that estimate within the confidence interval, IFRET could identify that the “true” fraud rate lies between 0% and 10%. Thus, it is possible that IFRET’s estimate of the “true” fraud rate could be 0% or even become negative based on the restricted definition of fraud.
3. Artificial Cut-Off Date for Case Inclusion
Due to the above guidance by USCIS Senior Leadership on fraud definition, Senior Leadership requested that cases be followed through to their final adjudication and recorded in the study accordingly. Due to such request, USCIS Senior Leadership wanted OP&S to recommend a time threshold by which a case will be excluded from the study because it has not received a final adjudicative outcome. They further request that the sample should be adjusted to allow for such attrition without unduly sacrificing the accuracy of the resulting estimates.
The desire to select a cut-off date to keep selected cases in the sample while removing those cases that took longer than expected to complete the fraud investigation is problematic. ISOs and IOs specifically noted in IFRET’s previous research (Qualitative Research on Marriage-Based Immigration Benefit Fraud Detection) that fraudulent petitions take significantly more time than non-fraudulent petitions. The length of time is directly related to the need to identify sufficient evidence that can be articulated in such a way that an Agency finding of fraud survives administrative or judicial scrutiny. Removing cases that have a higher probability of being fraudulent from the sample (called selection bias) will bias the fraud estimate downward by undercounting cases that may be more likely to be fraud cases. This introduces measurement error into the study. IFRET can only adjust for these excluded cases if there is reliable empirical information on the probability that a petition is fraudulent given the length of time between filing and final adjudication. IFRET has no such reliable empirical data. Consequently, this bias cannot be countered through statistical manipulation. Given the fraud estimate is reported with a confidence interval, there is an unacceptable risk that IFRET will produce a confidence interval that includes the value “0”(e.g., if the estimated fraud rate for the study is 5%, the confidence interval could range between 0% and 10%). Producing a study that suggests the fraud rate is 0% creates a risk of heightened public and Congressional scrutiny resulting in questions regarding the credibility of this first IBFA and the Agency. This particular risk cannot be reduced through increasing the sample size as sampling error and selection bias are not related.
SUMMARY OF IFRET’S POSITION
In summary, IFRET has the following positions regard the issues identified above.
1. With respect to the sample size, IFRET is able to estimate a fraud rate that is a scientifically-defensible with a sample size of 217 or 384. However, given the issues raised at TWG, IFRET suggests that the sample size for the I-130 IBFA be decided by Senior Leadership with input from FOD on how an increasing sample size would impact production. Further, IFRET suggests that FOD and Senior Leadership make the determination of sample size based on 1) their estimates of the true fraud rate and 2) their estimate of impact to production.
2. With respect to the operational definition of fraud for this study, IFRET argues for using the issuance of a NOID for fraud as the IBFA’s metric to count a case as fraud. However, IFRET is aware that there are Agency concerns regarding labeling cases as “fraud” that have not been adjudicated as fraud through an administrative or judicial process. The higher threshold of defining “fraud” will necessarily result in reducing the estimated fraud rate by excluding fraudulent cases that have not been found fraudulent by judicial process. These types of findings would strain credulity and invite negative attention for USCIS from both the public and Congress.
3. With respect to the cut-off date for inclusion, IFRET cannot conduct a scientifically-defensible IBFA if cases are subjected to an artificial cut-off date. This introduces selection bias into the study. This type of bias will bias the fraud rate downward. Selection bias cannot be compensated for through statistical manipulation. We recommend reporting fraud rates based on a case that will be denied for fraud (NOID issued) and exclude reporting the final outcomes of all cases to eliminate the selection bias of a cut-off date.
Discussion Questions for TWG / Discussion Goals
1. Please identify the potential impact for the study, in particular, for FOD, if the sample size increases to 384. FOD should lead this discussion.
2. The TWG, following the discussion of impact on production, make a recommendation for a sample size.
3. What is the current range of processing time for current cases? What is a reasonable cut-off dates for cases if we use the current definition of fraud?
4. The TWG should discuss the different thresholds for fraud and the impact on the IBFA timeline.
1. Sample Size …show more content…
The goal of the I-130 marriage-based IBFA is to generate a defensible fraud rate for the I-130 marriage-based petition. IFRET recognizes the need to select a sample size large enough for the study but small enough so that it is operationally feasible for our operational staff to complete cases for the study.
The calculated sample size for a research study is a product of two factors: the estimated fraud rate and the level of precision (often discussed as the “margin of error” ). The previous I-130 marriage-based BFCA conducted by FDNS in 2005, which was completed in 2008, estimated a fraud rate of 17 percent. Using that assumed fraud rate, IFRET calculated a sample size of 217 cases with a margin of error of 5 percent. If, however, IFRET assumes a fraud rate of 50 percent (a standard assumption if a rate is unknown ), IFRET would need to construct a sample of 384 to achieve the same margin of error of 5 percent.
IFRET recommended USCIS to draw a sample size of 217 cases for two reasons. First, it is standard practice in research to rely on previous research findings to calculate estimated sample size for a research study. Although there were some methodological problems with the previous BFCA, these problems do not necessarily impugn the fraud estimate. Second, the benefit of drawing a sample size of 217 is that this sample size will not unduly interfere with production at the Field Office level. Table 1 provides the differing assumed fraud rates and the resulting sample size based on a 5% margin of error.
Table 1: Differing Fraud Rates and Resulting Sample Size Based on 5% Margin of Error
Assumed Fraud Rate Sample Size
5% 73
10% 138
17% 217
25% 288
50% 384
At the recent TWG meeting, some TWG member raised concerns about the sample size and suggested that it should be larger; however, increasing the sample will impact production and nearly double the estimate for manpower hours needed to complete the study. The cost for an increase in sample size, therefore, is an increase in the work load for production staff and the project will be at risk of not being completed in a timely manner. In addition, if a larger sample size is desired, FOD will necessarily need to adjust operations in such a way that normal production may be impacted. The benefit of having a larger sample size is that should the estimated fraud rate falls below the assumed fraud rate of 50 percent, the margin of error will be smaller, and the estimated fraud rate will be more precise.
IFRET believes that a national fraud rate can be estimated using the smaller sample size if USCIS is prepared to have a wider confidence interval (lower precision) for the estimates should the fraud rate be higher than 17 percent. If the estimated fraud rate is higher than the assumed 17 percent, the margin of error increases. Even if the fraud rate is estimated at 50 percent, a sample size of 217 will provide an estimated fraud rate with a margin of error of 7 percent (see Figure 1). A slight decrease in precision will result in a dramatic reduction in required sample size of 384 to 217. Figure 1 illustrates the trade-off between sample size and precision.
An additional risk for using the 17 percent fraud rate from a prior unreleased study, USCIS should be prepared to handle any inquiries as to why that fraud rate was used and prepare a response to any questions regarding why the previous study was not released to the public. In addition, once the unreleased study is referenced in the new IBFA and is made known to the public, USCIS needs to understand that the former BFCA may be FOIAed.
Figure 1 Sample Size by Margin of Error Varying Fraud Rates
2. Operational Definition of Fraud
USCIS Senior Leadership determined that the definition of a fraud will be a case in which the adjudicator will deny based on fraud.
However, IFRET’s previous research (Qualitative Research on Marriage-Based Immigration Benefit Fraud Detection) identified a number of strategies employed by ISOs to stop fraudulent cases from continuing through the adjudication process including denying petitions for other reasons (in which case, the “fraud” is not captured in USCIS data systems) and issuing a Notice of Intent to Deny (NOID). Each of these strategies demonstrate different thresholds for the concept of “fraud” with case denial for a reason other than fraud being the lowest threshold and the Agency definition of a fraud finding the highest threshold. IFRET has been tasked to estimate the “true” fraud rate, meaning IFRET constructed a research design that would capture these lower thresholds for fraud. In this way, IFRET believes the study will be able to calculate a confidence interval for the “true” fraud rate by capturing all instances of fraud irrespective of a formal Agency finding. IFRET agreed, however, to compromise and use the mid-level measure of fraud: the issuance of a NOID. It should be noted that this definition will generate a fraud rate that may be unrealistically low as not all fraudulent cases can be denied for fraud due to the stringent criteria set up by the immigration law and
policies.
Previous IFRET research on fraud rates based on a finding of fraud calculated a fraud rate of 2.5%. This number is based on the normal adjudicative process wherein only cases that are suspected of fraud are referred to FDNS for closer scrutiny and different investigative procedures. IFRET’s IBFA research design is predicated on the approach of viewing all cases as suspected fraud; consequently, all cases will be subject to the same procedures utilized when a case is referred for fraud to FDNS. While IFRET expects that the study will show a higher fraud rate than 2.5%, the restriction of using an adjudicated denial for fraud as the metric for operationally measuring fraud is likely to produce a downward biased “true” fraud rate. For example, even if the fraud rate IFRET calculates doubles to 5% (from the above 2.5% rate), when placing that estimate within the confidence interval, IFRET could identify that the “true” fraud rate lies between 0% and 10%. Thus, it is possible that IFRET’s estimate of the “true” fraud rate could be 0% or even become negative based on the restricted definition of fraud.
3. Artificial Cut-Off Date for Case Inclusion
Due to the above guidance by USCIS Senior Leadership on fraud definition, Senior Leadership requested that cases be followed through to their final adjudication and recorded in the study accordingly. Due to such request, USCIS Senior Leadership wanted OP&S to recommend a time threshold by which a case will be excluded from the study because it has not received a final adjudicative outcome. They further request that the sample should be adjusted to allow for such attrition without unduly sacrificing the accuracy of the resulting estimates.
The desire to select a cut-off date to keep selected cases in the sample while removing those cases that took longer than expected to complete the fraud investigation is problematic. ISOs and IOs specifically noted in IFRET’s previous research (Qualitative Research on Marriage-Based Immigration Benefit Fraud Detection) that fraudulent petitions take significantly more time than non-fraudulent petitions. The length of time is directly related to the need to identify sufficient evidence that can be articulated in such a way that an Agency finding of fraud survives administrative or judicial scrutiny. Removing cases that have a higher probability of being fraudulent from the sample (called selection bias) will bias the fraud estimate downward by undercounting cases that may be more likely to be fraud cases. This introduces measurement error into the study. IFRET can only adjust for these excluded cases if there is reliable empirical information on the probability that a petition is fraudulent given the length of time between filing and final adjudication. IFRET has no such reliable empirical data. Consequently, this bias cannot be countered through statistical manipulation. Given the fraud estimate is reported with a confidence interval, there is an unacceptable risk that IFRET will produce a confidence interval that includes the value “0”(e.g., if the estimated fraud rate for the study is 5%, the confidence interval could range between 0% and 10%). Producing a study that suggests the fraud rate is 0% creates a risk of heightened public and Congressional scrutiny resulting in questions regarding the credibility of this first IBFA and the Agency. This particular risk cannot be reduced through increasing the sample size as sampling error and selection bias are not related.
SUMMARY OF IFRET’S POSITION
In summary, IFRET has the following positions regard the issues identified above.
1. With respect to the sample size, IFRET is able to estimate a fraud rate that is a scientifically-defensible with a sample size of 217 or 384. However, given the issues raised at TWG, IFRET suggests that the sample size for the I-130 IBFA be decided by Senior Leadership with input from FOD on how an increasing sample size would impact production. Further, IFRET suggests that FOD and Senior Leadership make the determination of sample size based on 1) their estimates of the true fraud rate and 2) their estimate of impact to production.
2. With respect to the operational definition of fraud for this study, IFRET argues for using the issuance of a NOID for fraud as the IBFA’s metric to count a case as fraud. However, IFRET is aware that there are Agency concerns regarding labeling cases as “fraud” that have not been adjudicated as fraud through an administrative or judicial process. The higher threshold of defining “fraud” will necessarily result in reducing the estimated fraud rate by excluding fraudulent cases that have not been found fraudulent by judicial process. These types of findings would strain credulity and invite negative attention for USCIS from both the public and Congress.
3. With respect to the cut-off date for inclusion, IFRET cannot conduct a scientifically-defensible IBFA if cases are subjected to an artificial cut-off date. This introduces selection bias into the study. This type of bias will bias the fraud rate downward. Selection bias cannot be compensated for through statistical manipulation. We recommend reporting fraud rates based on a case that will be denied for fraud (NOID issued) and exclude reporting the final outcomes of all cases to eliminate the selection bias of a cut-off date.
Discussion Questions for TWG / Discussion Goals
1. Please identify the potential impact for the study, in particular, for FOD, if the sample size increases to 384. FOD should lead this discussion.
2. The TWG, following the discussion of impact on production, make a recommendation for a sample size.
3. What is the current range of processing time for current cases? What is a reasonable cut-off dates for cases if we use the current definition of fraud?
4. The TWG should discuss the different thresholds for fraud and the impact on the IBFA timeline.