Examining Best Practices for Sampling and Weighting Dual ...

Examining Best Practices for Sampling and Weighting Dual ...

Examining Best Practices for Sampling and Weighting Dual Frame Surveys Liz Kantor Advised by Dr. David Redlawsk Rutgers, The State University of New Jersey PANJAAPOR 2015 Spring Event We know we need to include cell phones in telephone RDD samples and weight accordingly but we dont agree on how PANJAAPOR 2015 Spring Event Considerations Sampling What % of the sample

should be cell phones? Landlines? Weighting Base weight? Frame overlap adjustment? PANJAAPOR 2015 Spring Event Calibration adjustments? Trimming? Low variance! Data and Methods Data Rutgers-Eagleton Poll Methods Demographics: Chi-square Sampling analysis: weight C & LL samples

to make up varying proportions of overall sample Weighting: 17 weighting & trimming combinations PANJAAPOR 2015 Spring Event Weight Calculations Base weight: sample selection probability multiplied by household selection probability Frame adjustment: Compositing, Effective Sample Size PANJAAPOR 2015 Spring Event From Kennedy (2012) Weight Calculations Calibration

Basic Sample Balancing one manual iteration of each demographic variable Raking multiple iterations of each demographic variable; SPSSINC RAKE Extension Trimming Hard cap: .2 and 5 Percentile cap: 2nd and 98th percentiles PANJAAPOR 2015 Spring Event Weighting Combinations Taking a cue from Kennedy (2012): COMBINATION 0 A B C D E

F G H I J K L M N O P BASE WEIGHT? No Yes Yes Yes Yes Yes Yes Yes

Yes No No No No No No No No PANJAAPOR 2015 Spring Event FRAME OVERLAP N/A 1/2 Compositing 1/2 Compositing 1/2 Compositing 1/2 Compositing Effective Sample Size Effective Sample Size

Effective Sample Size Effective Sample Size 1/2 Compositing 1/2 Compositing 1/2 Compositing 1/2 Compositing Effective Sample Size Effective Sample Size Effective Sample Size Effective Sample Size CALIBRATION TRIMMING METHOD Basic Sample Balancing Hard Cap, .3 & 3 Basic Sample Balancing Hard Cap Basic Sample Balancing Percentile Cap Raking Hard Cap

Raking Percentile Cap Basic Sample Balancing Hard Cap Basic Sample Balancing Percentile Cap Raking Hard Cap Raking Percentile Cap Basic Sample Balancing Hard Cap Basic Sample Balancing Percentile Cap Raking Hard Cap Raking Percentile Cap Basic Sample Balancing Hard Cap Basic Sample Balancing

Percentile Cap Raking Hard Cap Raking Percentile Cap *0 = Rutgers-Eagleton Poll practices at the time of research Hypotheses I. Cell-only respondents will follow patterns in the literature of demographic distinctiveness and geographic mobility II. Dual users from the cell frame will differ demographically from dual users from the landline frame III. A higher % of cell phones than used in the Rutgers-Eagleton Poll will be required in the sample to match NHIS PANJAAPOR 2015 Spring

Eventestimates Hypotheses IV. Half compositing will be less efficient (higher DEFF) than effective sample size V. No difference in efficiency between hard cap and percentile cap; cant know a priori which will increase variance more PANJAAPOR 2015 Spring Event Demographics Statistically significant differences at = .001 C Only vs. Dual Dual from cell Gender Age Race

Party ID Educatio n Income PANJAAPOR 2015 Spring Event vs. LL only vs. Dual from LL

* * = .01 Demographics Differences in gender composition Rutgers-Eagleton Poll PANJAAPOR 2015 Spring Event Geographic Mobility Match between FIPS code (county) from sample provider and from respondent Rutgers-Eagleton Poll PANJAAPOR 2015 Spring Event

Simulated Sample Composition CELLLANDLIN CELL- MOSTL DUAL ELANDLIN ONLY Y USE MOSTLY E-ONLY NHIS (NJ) 19.4% 30% C 70% LL 10.4% 40% C 60% LL 13.9% 50% C

50% LL 17.4% 60% C 40% LL2015 Spring 20.8% PANJAAPOR 70% C Event 25.7% 31.1% 15.2% 6.9% 33.7%

26.6% 23.3% 6.0% 34.1% 25.8% 21.1% 5.1% 34.4% 24.9% 19.0% 4.3%

34.8% 24.1% Rutgers-Eagleton Poll 16.9% 3.4% Design Effects Rutgers-Eagleton Poll PANJAAPOR 2015 Spring Event Margins of Error Rutgers-Eagleton Poll

*Calculated using Langer Research Associates MoE Machine PANJAAPOR 2015 Spring Event Average deviation from NHIS estimates Rutgers-Eagleton Poll NHIS (NJ) CELLCELLONLY MOSTLY 19.4% 25.7% PANJAAPOR 2015 Spring Event DUAL LANDLINE LANDLINE USE

-MOSTLY -ONLY 31.1% 15.2% 6.9% * indicates that the weighting combination came within 1% of the cell-only estimate Discussion Support for Hypotheses I, II, and III: Significant demographic differences among cell only, dual users, and landline only Estimated geographic mobility much greater for Rs from cell sample vs. Rs from landline sample Significant demographic differences

between dual users from cell and landline samples More PANJAAPOR 2015cell Springphone interviews (~50%) are Event Discussion No support for Hypothesis IV Compositing was not less efficient than Effective Sample Size Little support for Hypothesis V Percentage cap was more efficient than the hard cap PANJAAPOR 2015 Spring Event Weighting

Recommendations Ultimately, no obvious best practices Criteria for choosing weighting and trimming combinations: Includes a frame adjustment (avoid biased estimators) Inclusion of base weight Brings percentage of cell-only respondents within 1% of NHIS estimates Percentile cap more efficient PANJAAPOR 2015 Spring Event Weighting Recommendations Under these criteria, good weighting and trimming combinations are: B: Base Weight, Compositing, Basic

Sample Balancing, Percentile Cap D: Base Weight, Compositing, Raking, Percentile Cap F: Base Weight, Effective Sample Size, Basic Sample Balancing, Percentile Cap PANJAAPOR 2015 Spring Event Future Research Replicate using other data Strengthen the theoretical basis of the criteria used to recommend weighting combinations Experiment with more aggressive percentile caps (e.g. 5% and 95%) PANJAAPOR 2015 Spring Event Thank You [email protected]

PANJAAPOR 2015 Spring Event

Recently Viewed Presentations

  • Sedimentologi Kamal Roslan Mohamed DELTA INTRODUCTION The mouths

    Sedimentologi Kamal Roslan Mohamed DELTA INTRODUCTION The mouths

    A delta building out into a region with strong tides will be modified into a pattern that is different to both river- and wave- dominated deltas. The delta-top channel(s) are subject to tidal influence with reverses of flow and/or periods...
  • Diapositiva 1 - ComexPerú

    Diapositiva 1 - ComexPerú

    Paintgs,drawgs&pastels executd by hand exc hd 4906&h-p&h-d mfd art '902119. Orthopaedic or fracture appliances, nes '920993. Parts and accessories for the musical instruments of heading No 92.03 '911280. Clock cases&cases of a similar type,for oth goods of this Chapter,nes '910899....
  • Social Studies Administrator Update

    Social Studies Administrator Update

    Workshop Model in AP SS. AP classes may be differentiated away from traditional workshop model even more. Work Time might be a socratic seminar. Openings may be shorter . practicing a skill already introduced . taking a quiz that mimics...
  • Chapter 7: Conformity

    Chapter 7: Conformity

    Compliance Make direct requests of us & hope we comply Differs from conformity: Sometimes we mindlessly comply Langer's experiment w/xerox machine… Strategies for compliance? Reciprocity norm - repay someone a favor Sequential Request: Foot-in-the-door Door-in-the-face Obedience Compliance through direct orders.
  • Patterns within the periodic table: Completing the Figure

    Patterns within the periodic table: Completing the Figure

    Which pair of particles will experience the stronger coulombic attraction? +2. Coulomb's Law: Certain atomic properties change predictably across, and up and down, the periodic table! Oppositely charged particles will be attracted .
  • LG Matrix - LOGIN Asia

    LG Matrix - LOGIN Asia

    Disaggregated SDG data (and M&E) recognized from the beginning (reduce inequalities within countries) MDGs in Asia Pacific - significant progress yet significant deficits remain. Asia-Pacific region is expected to meet 13 of 21 targets and made significant progress on two...
  • Prezentācijas tēmas nosaukums

    Prezentācijas tēmas nosaukums

    Iepirkuma līguma priekšmets (būvdarbi) nav iekļauts MK noteikumu Nr.353 „Prasības zaļajam publiskajam iepirkumam un to piemērošanas kārtība" (Noteikumi) 1.pielikumā, bet līgums paredz iekštelpu apgaismojuma ierīkošanu, uz kuru attiecas 1.pielikums. ... PRINCE2, PMP, PMI vai ...
  • IEEAF Press Conference 6-30-04

    IEEAF Press Conference 6-30-04

    The Internet Educational Equal Access Foundation IEEAF Global Quilt High speed links to Western Africa's coastal countries International Workshop on African Research & Education Networking