Abstract

This study aims to assess the data collection mode effects (face-to-face vs. web surveys) on abortion-related attitudinal questions, using the 2012 American National Election Studies. The results show that face-to-face respondents are more likely to provide opinionated and informative answers, including which issues they favor and oppose. Also, web respondents are more likely than face-to-face respondents to answer “neither agree nor disagree.” In addition, the item nonresponse rate for a Web survey is higher than for a face-to-face survey with three questions. The analysis shows that mode effects exist in both substantive responses and item nonresponses for the abortion questions.

Keywords

Face-to-face survey, web survey, abortion, mode effect

Introduction

Public opinion on abortion has been changing over time (Shaw, 2003). The attitudes toward abortion are very complex around the world and in the U.S., it is largely polarized. The debate over pro-choice versus pro-life has always been a central topic for discussion (Medoff, 2013) and the public opinion has becoming even more divided on this. For example, Shaw (2003) showed that the in 1992 35% of the public were pro-life and 59% were pro-choice. In 2003, the difference became much smaller (45% pro-life versus 48% pro-choice). The acceptance of abortion varied depending on a several factors, such as pre-adulthood factors (Pacheco & Kreitzer, 2016), demographic (Woodhams, Hill, Fabiyi, & Gilliam, 2016), media consumption (Altshuler, Gerns Storey, & Prager, 2015), and occupation (Begun, Kattari, McKay, Winter, & O’Neill, 2017; Sjöström, Essén, Sydén, Gemzell-Danielsson, & Klingberg-Allvin, 2014). Along with the change of public opinion on abortion, in the U.S. the estimated abortion rate per 1000 women aged 15 to 44 years also declined from 19.4 to 14.6 between 2008 and 2014, a 25% change (Jones & Jerman, 2017). However, the rate of decline varied depending on the demographics of women, such as age, race and ethnicity, and income.

It is important to note that both public opinion research toward abortion and population estimates of the abortion are often based on survey data. This type of studies often carries a profound policy making impact when policies are informed by findings of public opinion survey research. Given that, research on the measurement of the relevant questions will benefit the understanding of public opinion as it relates to abortion. More specifically, a solid understanding of mode effect on abortion attitudes can have implication on how researchers, practitioners and policy makers measure the public opinion of this important topic. To date, survey researchers have spent limited efforts in examining the survey methodology aspect of the abortion-related attitudinal question. An early study on measuring abortion in surveys of U.S. women analyzed three major surveys – National Survey of Family Growth, National Surveys of Young Women and National Longitudinal Surveys of Work Experience of Youth – and found that the self-reported abortion was highly deficient, especially among nonwhite women (Jones & Forrest, 1992). In a survey conducted in Mexico, Lara, Strickler, Olavarrieta, & Ellertson (2004) tested four data collection modes, namely face-to-face, audio computer-assisted self-interview, paper-based self-administered questionnaire and a random-response technique. They found that the random-response technique yielded the highest self-report abortion attempts. A meta-analysis also showed that the random-response technique increased the self-report to sensitive question, including abortion, compared to other data collection mode (Lensvelt-Mulders, Hox, Van der Heijden, & Maas, 2005). A recent study by Singer and Couper (2014) compared using “baby” versus “fetus” when asking about attitudes toward abortion and found that there was no significant difference on abortion preferences but the preference for a prenatal testing for genetic defects differed by the question wording (Singer & Couper, 2014). The choice of data collection mode is a crucial factor when it comes to measuring attitudes and opinions toward abortion. The literature shows that survey responses vary, depending on the survey mode, especially for sensitive questions like the ones examined in this study (Kreuter, Presser, & Tourangeau, 2008).

There are some literatures on the mode effect between face-to-face and Web surveys, and some consistent findings can be drawn from previous studies. First, the response rates tend to be higher for face-to-face than Web surveys. This is possibly due to different levels of interviewer contacts between these two modes (Christensen, Ekholm, Glümer, & Juel, 2014; Heerwegh & Loosveldt, 2008; Manfreda et al., 2008; Revilla & Saris, 2012). Interviewers in the face-to-face surveys recruit respondents, introduce the survey, address their concerns, and persuade them to participate, while in Web surveys, respondents are usually recruited by email or mail, and they initiate the survey themselves. Very limited direct inter-personal interaction with respondents from the survey organization exists in a Web survey. Second, face-to-face surveys suffer from a higher level of social desirability bias than Web surveys, especially when asking sensitive questions (Heerwegh, 2009). Social desirability bias refers to the phenomenon of overreporting socially desirable attitudes and behaviors while underreporting socially undesirable ones (Callegaro, 2008). Social desirability bias is most prevalent when respondents answer questions asking for sensitive information or questions with a potentially socially desirable response (Christensen et al., 2014; Duffy, Smith, Terhanian, & Bremer, 2005). Face-to-face respondents are more likely to provide a socially desirable answer to present themselves in a favorable image or to avoid tensions and negative judgments from the interviewer or when other people are present during the survey. The relatively higher level of anonymity and confidentially in a self-administered Web survey can increase the disclosure of undesirable responses. Third, the data quality for these two modes is mixed. While Web surveys tend to show a higher level of item nonresponse in general and more non-differentiation for rating scales, face-to-face surveys are susceptible to more extreme response bias (Beukenhorst et al., 2014; Goldenbeld & de Craen, 2013; Heerwegh, 2009; Heerwegh & Loosveldt, 2008). Given the mixed findings, more comparative research is pressing to examine the mode effects on measurement and data quality between face-to-face and Web surveys.

This current study intends to expand the existing literature on the measurement of abortion questions by examining the data collection mode effect on attitudes toward abortion. In particular, this study examines the responses to eight attitudinal questions on abortion from two nationally representative surveys, one through face-to-face and one through Web. More specifically, this study compares the substantive responses and item nonresponse rates of abortion questions between these two modes of data collection. The eight questions asked respondent’s opinion about abortion in eight scenarios, including nonfatal health risk of pregnant women, fatal health risk of pregnant women, incest, rape, birth defect of fetus, financial hardship, child will not be the sex woman wants it to be, and woman’s choice (see Appendix for exact wordings). Why should we expect that a mode effect exists in abortion questions? On the one hand, attitudinal questions on abortion are typically seen as sensitive and the differential levels of social desirability bias between face-to-face and Web surveys are likely to result in different patterns of response. Specifically, there are two possibilities. First, respondents hide their real attitudes and provide more acceptable albeit untruthful responses (Christensen et al., 2014; Duffy et al., 2005; Liu & Wang, 2015). In this case, we should expect to observe a different response pattern to the abortion questions. Second, respondents can also withhold their opinion completely by not offering a substantive response (Christensen et al., 2014). This will result in item nonresponse, including “don’t know” and “refusal” responses. The interviewer involvement in the face-to-face survey is likely to increase the social desirability bias, and hence face-to-face respondents are more likely to provide socially acceptable responses or not provide answers at all. On the other hand, the higher motivation resulting from the interviewer involvement in the face-to-face interviews is likely to result in more thoughtful and conscientious responses compared to Web surveys (Beukenhorst et al., 2014; Goldenbeld & de Craen, 2013; Heerwegh, 2009; Heerwegh & Loosveldt, 2008). Therefore, face-to-face respondents will provide answers that are less ambiguous answers (i.e., the middle option) and less non-substantive answers (i.e., item nonresponse). This study will also examine whether the survey mode effect differs by gender. Research has shown gender differences on attitudes toward abortion (Finlay, 1981; Lohan, Cruise, O’Halloran, Alderdice, & Hyde, 2011; Schwandt et al., 2013). Women hold more liberal attitudes and a higher approval of women’s autonomy in abortion decisions than men (Patel & Johns, 2009). It is possible that when a topic is more related to the respondents, as abortion is to women, respondents are likely to possess a well-formed attitude and be less susceptible to the impact of the survey mode. By contrast, men’s attitudes toward abortion may not be as solid and hence they are more likely to edit their responses based on a sense of privacy, anonymity and confidentially of the survey mode.

Methods

Study population and data collection
This study examines the data collection mode effect using the 2012 American National Election Studies (ANES). The 2012 ANES is a national representative survey examining the general population’s electoral participation, voting behavior, and public opinion. The target population is U.S. citizens aged 18 or older as of the 2012 Election Day. The 2012 ANES includes two waves of data collection, namely a pre-election study and a post-election study, and the same respondents were interviewed twice. The field period for the pre-election study was between September and November 2012, and the post-election study was between November 2012 and January 2013. The 2012 ANES innovatively conducted two parallel surveys, one through a face-to-face mode and one through the Web, using two independent national samples and one identical questionnaire.

The Web survey was conducted using GfK KnowledgePanel, which is a nationally representative online panel. The panelists were recruited through address-based sampling and random-digit dialing. All household members were enumerated at the recruiting stage and demographic information was collected before any survey. Respondents for the 2012 ANES were selected from this probability-based GfK KnowledgePanel. The face-to-face survey used an address-based, stratified, multi-stage cluster sample. The first stage of sampling consisted of stratifying the 48 contiguous states and the District of Columbia into nine regions corresponding to Census Divisions, which constitute this study’s strata. Within each of the nine regions, census tracts were then randomly selected proportionally to the region’s proportion of the U.S. adult population. In the second stage, residential addresses within each tract were randomly selected. In the third stage, one eligible person per household was randomly selected. The sample included a main sample and two oversamples for African Americans and Hispanic Americans, respectively. Within each household, random selection was performed, and one person was selected for the survey. Since these two probability samples both target the same U.S. general population, they should have comparable coverage¹.

In total, the 2012 ANES finished 5,914 pre-election interviews (2,054 through face-to-face interviews) and 5,510 post-election interviews (1,929 through face-to-face interviews). The pre-election response rates (AAPOR RR1) were 38% and 2%, respectively, for face-to-face and Web surveys, while the post-election re-interview rates were 94% and 93%, respectively.

Measures
As mentioned, eight abortion questions were asked about a third into the post-election survey (see Appendix A). The order of the first seven questions was randomized, and the last question on women’s choice was always asked last. Three response options, namely favor, oppose, or neither favor nor oppose, were provided. Face-to-face respondents can answer “don’t know” or refuse to answer any question and it is coded as item nonresponse. Web respondents can skip a question, which is coded as item nonresponse. In the analysis, I calculated the percentage of item nonresponse for each of the eight questions and compared them across the two modes.

Analytical approach
The analyses contain two parts. First, the percentages of item nonresponse are compared between face-to-face and Web surveys using a chi-square test. Second, the response distribution of each question is compared between the two modes using a chi-square test. Both analyses are performed for the whole samples, and for males and females separately. Considering the quasi-experimental nature of the survey data, I analyzed the data through propensity score weighting technique. Specifically, I first conducted a propensity model to predict the participation in face-to-face versus Web survey with variables that were potentially correlated with the response propensity for both data collection modes. The variables used in the propensity model included respondent’s gender, age, marital status, education level, employment status, belonging to social class, race and ethnicity, number of children, home internet access, household income, home ownership, and years lived in the current address. Taking the predicted probability of the participating to face-to-face versus Web survey, I created a propensity weight for each respondent. Last, I calculated the weighted distribution for both the substance responses and item nonresponse, and performed the weighted statistical test accordingly. All the results showed in this paper were adjusted with the propensity weight. All analyses were conducted in R.

Results

Demographic distributions
The unweighted demographic distributions differ significantly between face-to-face and Web (Table 1). Once weights are applied, the demographic distributions become not significant between modes. For both surveys, the weighted analyses show over 70% of the respondents are non-Hispanic white; over 53% of the respondents are married; and half of the respondents’ household income falls less than $50,000.

Table 1. Demographic distribution by mode of data collection, 2012 American National Election Studies

	Unweighted						Weighted
	Face-to-face (n=1929)		Web (n=3581)		χ2	p-value	Face-to-face (n=1929)		Web (n=3581)		χ2	p-value
	Mean	S.E.	Mean	S.E.			Mean	S.E.	Mean	S.E.
Gender
Male	42.9	1.1	51.6	0.8	37.6	<.0001	48.2	1.6	48.4	1.1	0.0	0.88
Female	57.1	1.1	48.4	0.8			51.8	1.6	51.6	1.1
Age
<30	22.2	1.0	12.4	0.6	208.0	<.0001	21.1	1.3	20.6	1.0	3.6	0.61
30~39	19.6	0.9	12.0	0.5			15.4	1.1	14.9	0.8
40~49	16.7	0.9	16.3	0.6			17.7	1.2	17.0	0.8
50~59	19.5	0.9	23.9	0.7			18.7	1.2	20.1	0.8
60~69	13.0	0.8	22.3	0.7			14.5	1.2	16.3	0.7
70+	9.0	0.7	13.2	0.6			12.6	1.1	11.1	0.6
Race and ethnicity
White (non-Hispanic)	45.0	1.1	67.3	0.8	269.9	<.0001	71.1	1.3	71.1	1.0	0.6	0.90
Black (non-Hispanic)	25.1	1.0	13.3	0.6			12.1	0.8	11.9	0.8
Hispanic	22.8	1.0	13.5	0.6			10.4	0.7	11.0	0.7
Other (non-Hispanic)	7.1	0.6	5.9	0.4			6.4	0.7	5.9	0.5
Education attainment
Less than high school credential	15.5	0.8	7.7	0.4	123.8	<.0001	9.9	0.8	10.2	0.7	0.9	0.93
High school credential	26.1	1.0	23.9	0.7			30.4	1.5	29.9	1.1
Some post-high-school	34.1	1.1	33.3	0.8			30.1	1.4	30.5	1.0
Bachelor's degree	15.9	0.8	20.8	0.7			19.4	1.3	18.4	0.8
Graduate degree	8.5	0.6	14.2	0.6			10.2	1.0	11.0	0.6
Marital status
Married	39.9	1.1	54.9	0.8	130.8	<.0001	53.4	1.6	53.2	1.1	1.1	0.90
Widowed	6.6	0.6	6.6	0.4			5.8	0.7	5.7	0.5
Divorced	16.7	0.9	14.1	0.6			12.9	0.9	12.9	0.7
Separated	4.8	0.5	2.5	0.3			2.6	0.4	2.1	0.3
Never married	31.8	1.1	22.0	0.7			25.2	1.3	26.1	1.0
Household income
$49,999 or less	64.9	1.1	48.9	0.8	145.5	<.0001	49.7	1.6	49.1	1.1	1.8	0.61
$50,000-$99,999	24.4	1.0	30.0	0.8			30.1	1.5	32.0	1.1
$100,000-$149,999	6.3	0.6	12.4	0.6			11.8	1.2	11.7	0.7
$150,000 or more	4.4	0.5	8.7	0.5			8.3	1.0	7.1	0.5

Item nonresponse
As a first step to assess data quality for these two modes, I calculated the percentage of item nonresponse for each of the eight questions and compared them across the two modes. Across both mode and all questions, the item nonresponse rate is between 8.2% and 10.6%. For six out of the eight questions, Web survey has slightly more missing data than face-to-face survey although none of the difference is statistically significant. Similarly, when comparing the item nonresponse rate between the two data collection modes for male and female separately, there is no statistically significant difference for any of the questions. This suggests a lack of mode effect on item nonresponse for abortion related questions. Respondents are neither more or less likely to provide a non-substantive response, including “don’t know” and “refuse,” to these three questions in either modes.

Table 2. Item nonresponse to abortion questions by mode of data collection, 2012 American National Election Studies (weighted results)

	Whole sample					Male					Female
	Face-to-face (n=1929)		Web (n=3581)			Face-to-face (n=828)		Web (n=1847)			Face-to-face (n=1101)		Web (n=1734)
	%	S.E.	%	S.E.	χ2 , df, p	%	S.E.	%	S.E.	χ2 , df, p	%	S.E.	%	S.E.	χ2 , df, p
1) Nonfatal health risk	8.7	0.6	9.4	0.9	0.94, 1, 0.50	9.6	1.0	9.2	1.6	0.13, 1, 0.83	1.4	0.3	1.1	0.4	2.35, 1, 0.24
2) Fatal health risk	9.3	0.6	9.4	0.9	0.01, 1, 0.92	10.2	1.0	9.1	1.5	0.89, 1, 0.57	1.9	0.4	1.0	0.4	0.89, 1, 0.47
3) Incest	10.6	0.7	9.5	0.9	1.95, 1, 0.34	10.0	1.0	9.1	1.5	0.70, 1, 0.62	4.1	0.7	1.2	0.4	1.24, 1, 0.39
4) Rape	8.2	0.6	9.5	0.9	2.90, 1, 0.24	8.8	1.0	9.3	1.6	0.21, 1, 0.78	1.2	0.3	1.1	0.4	3.25, 1, 0.16
5) Birth defect	9.8	0.7	9.5	0.9	0.10, 1, 0.83	9.9	1.0	9.4	1.6	0.25, 1, 0.76	3.1	0.6	1.2	0.4	0.00, 1, 0.99
6) Financial hardship	8.3	0.6	9.4	0.9	2.33, 1, 0.29	8.6	0.9	9.1	1.5	0.23, 1, 0.77	2.0	0.5	1.2	0.4	2.39, 1, 0.23
7) Child gender wrong	8.2	0.6	9.5	0.9	2.80, 1, 0.25	8.6	0.9	9.1	1.5	0.23, 1, 0.77	2.0	0.6	1.2	0.4	3.03, 1, 0.18
8) Woman's choice	8.5	0.6	9.6	0.9	2.12, 1, 0.31	9.4	1.0	9.5	1.6	0.01, 1, 0.96	1.7	0.4	1.0	0.4	3.24, 1, 0.16

^* p <.05, ^** p<.001, ^*** p<.0001

Attitudes toward abortion
I next present the results of the mode effect on substantive responses for the eight questions in Table 3. For each abortion question, I compared the percentages of favor, oppose, and neither favor nor oppose under each mode. The distribution comparisons of the substantive responses between the two modes reveal a fairly consistent pattern. In particular, the chi-square tests for these eight questions across the two modes indicate that significant mode effects exist on all questions at p<.0001 level.

For the “oppose” response option, seven out of eight questions indicate a higher percentage for this negative category under face-to-face interviews than Web interviews, whereas the only remaining question (Rape) show no difference between face-to-face and web surveys. A close look at the distribution comparison shows that the oppositions are high and quite disparate for child gender wrong (face-to-face 86% oppose vs. Web 74% oppose), financial hardship (face-to-face 61% oppose vs. Web 50% oppose) and nonfatal health risk (face-to-face 40% oppose vs. Web 26% oppose) scenarios. The percentages of opposing are middling for both woman’s choice (face-to-face 42% oppose vs. Web 37% oppose), incest (face-to-face 32% oppose vs. Web 23% oppose) and birth defect (face-to-face 29% oppose vs. Web 22% oppose) scenarios, although the differences between face-to-face and Web surveys are smaller compared to the first three questions. Fatal health risk and rape scenarios both receive low opposition and the differences between the two survey modes are relatively small and negligible.

For the “neither favor nor oppose” response, which is also regarded as the neutral option, all the questions show a higher percentage for Web than face-to-face surveys. The questions about birth defect (Web 29% vs. face-to-face 15%), incest (Web 28% vs. face-to-face 14%), nonfatal health risk (Web 28% vs. face-to-face 17%) and financial hardship (Web 26% vs. face-to-face 12%) scenarios received substantially more “neither favor nor oppose” answers from Web respondents than from face-to-face respondents. The higher percentages of the two end points of the rating scales in the face-to-face survey indicate that face-to-face respondents tend to provide more divided attitudes toward abortion than Web respondents. Web respondents, in contrast, provided answers that are more neutral or ambiguous, suggesting that Web surveys tend to elicit less clear-cut opinions toward abortion than face-to-face surveys do.

I also conducted the analyses separately by the respondent’s gender (Tables 2 and 3). For the item nonresponse analysis, similar to the whole sample analysis, the differences between face-to-face and Web survey for male and for female respondents are small and not statistically significant. For substantive responses, the patterns of response differences between face-to-face and Web for both genders are similar to the combined whole sample. All differences are statistically significant. Both male and female face-to-face respondents are more in favor of abortion for six out of eight scenarios (nonfatal health risk and child gender wrong scenarios show little to no difference between modes) while more Web respondents are selecting the middle option. The patterns of the level of favoring/opposing across all the items for both genders are also similar to the combined whole samples.

Table 3. Attitudes toward abortion by mode of data collection, 2012 American National Election Studies (weighted results)

	Whole sample					Male					Female
	Face-to-face (n=1929)		Web (n=3581)			Face-to-face (n=828)		Web (n=1847)			Face-to-face (n=1101)		Web (n=1734)
	Mean	S.E.	Mean	S.E.	χ2 , df, p	Mean	S.E.	Mean	S.E.	χ2 , df, p	Mean	S.E.	Mean	S.E.	χ2 , df, p
1) Nonfatal health risk
Favor	44	1.2	46	1.4	170.1, 2, <.001	46	1.8	46	1.9	56.1, 2, <.001	43	1.5	45	2.0	112.1, 2, <.001
Oppose	40	1.1	26	1.2		37	1.7	27	1.8		41	1.5	26	1.7
Neither favor nor oppose	17	0.9	28	1.4		17	1.3	27	1.8		16	1.1	29	2.0
2) Fatal health risk
Favor	80	0.9	70	1.4	138.3, 2, <.001	81	1.4	68	1.9	85.5, 2, <.001	79	1.3	71	1.9	62.5, 2, <.001
Oppose	11	0.7	10	0.8		10	1.1	12	1.1		11	1.0	9	1.0
Neither favor nor oppose	10	0.7	20	1.3		9	1.0	21	1.9		10	0.9	20	1.8
3) Incest
Favor	54	1.2	49	1.4	173.2, 2, <.001	54	1.8	48	1.9	84.6, 2, <.001	53	1.6	50	2.1	90.9, 2, <.001
Oppose	32	1.1	23	1.2		31	1.7	24	1.7		32	1.5	22	1.6
Neither favor nor oppose	14	0.8	28	1.4		14	1.2	28	1.7		14	1.1	27	2.0
4) Rape
Favor	76	1.0	66	1.5	135.2, 2, <.001	77	1.5	66	1.9	84.8, 2, <.001	76	1.3	67	2.1	56.1, 2, <.001
Oppose	15	0.8	15	1.2		15	1.3	15	1.4		16	1.1	16	1.8
Neither favor nor oppose	8	0.6	18	1.2		8	0.9	19	1.8		9	0.9	18	1.6
5) Birth defect
Favor	55	1.2	49	1.4	165.2, 2, <.001	56	1.8	50	1.9	65.0, 2, <.001	55	1.5	49	2.0	98.1, 2, <.001
Oppose	29	1.1	22	1.1		29	1.6	23	1.6		29	1.4	21	1.5
Neither favor nor oppose	15	0.8	29	1.4		15	1.3	27	1.8		16	1.1	30	2.1
6) Financial hardship
Favor	27	1.0	25	1.2	200.5, 2, <.001	28	1.6	24	1.5	95.2, 2, <.001	27	1.4	25	1.7	106.9, 2, <.001
Oppose	61	1.1	50	1.4		60	1.7	51	1.9		62	1.5	49	2.1
Neither favor nor oppose	12	0.7	26	1.3		11	1.1	26	1.7		12	1.0	26	1.8
7) Child gender wrong
Favor	8	0.6	8	0.9	184.8, 2, <.001	10	1.0	10	1.5	79.5, 2, <.001	7	0.8	7	1.0	103.7, 2, <.001
Oppose	86	0.8	74	1.4		84	1.3	73	1.8		87	1.0	75	2.0
Neither favor nor oppose	6	0.6	17	1.3		6	0.9	17	1.4		6	0.7	18	1.9
8) Woman's choice
Favor	45	1.2	39	1.4	120.1, 2, <.001	45	1.8	38	1.8	44.2, 2, <.001	45	1.5	40	2.0	76.6, 2, <.001
Oppose	42	1.1	37	1.4		41	1.7	39	1.9		42	1.5	36	1.9
Neither favor nor oppose	13	0.8	24	1.2		14	1.2	23	1.7		12	1.0	24	1.8

Discussion

This study set out to examine the abortion attitude difference between face-to-face and Web surveys through two independent national samples. The questions under study are sensitive which can lead to social desirability bias. When facing abortion-related attitudinal questions, if the respondent feels that his or her real underlying attitude is not in accordance with the social norm, one of the choices is to withhold his or her opinion, which results in item nonresponse. Considering the relatively higher level of anonymity and confidentially in a self-administered Web survey, I expected lower item nonresponse in Web survey than face-to-face survey. In other words, I expected the Web survey respondents to be more forthcoming when responding to sensitive questions. However, the results showed that the item nonresponse rates between the two modes were similar and not statistically significant. This suggests that respondents in face-to-face survey mode were no more likely to withhold their opinions in face of the abortion-related questions than Web survey respondents.

The responses to eight abortion questions in this survey show significant mode effects. Web respondents are more likely to choose the neither favor nor oppose option than face-to-face respondents for all of the questions. On the other hand, face-to-face respondents are more likely to choose favor or oppose than Web respondents. This is in line with the previous study, which reports that face-to-face respondents select more extreme answers from ordinal rating scales than Web respondents (Goldenbeld & de Craen, 2013). These findings suggest that mode effects exist in terms of people’s responses toward abortion-related attitudinal questions, and the estimates on abortion attitudes drawn from face-to-face interviews and Web interviews are not entirely comparable. One possible explanation for the mode effect is the respondent’s motivation. The presence of an interviewer in a face-to-face survey is likely to enhance the motivation of the respondents, and consequently, respondents are likely to take the survey more seriously and hence give more informative answers (such as whether they favor or oppose abortion) in comparison with the Web respondents(Christensen et al., 2014; Heerwegh, 2009). The middle options are more difficult to interpret in comparison to the other two options that clearly show where the respondents stand on the topic. More Web respondents endorse the middle options, possibly due to the lack of motivation in the self-administered interview. Social desirability bias may also contribute to the response difference. However, both directions of the responses could be seen as socially desirable, depending on what the respondents think that the interviewer or society in general perceive as the norm. Another possibility is the narrow nature of the response options, which may have forced respondents who feel mildly in favor of or oppose the statement to choose the middle option, since otherwise they will risk seeming hard-line. The specific question topic may also contribute to the response differences between face-to-face and Web respondents. For the scenarios where abortion is more acceptable, such as fatal health risk, the oppose rate is low overall and no difference exists between the surveys. For the less socially acceptable scenarios, such as incorrect fetus gender, most respondents disapprove of abortion and relatively large differences exist between surveys. One may also argue that Web panel respondents have more survey experience than cross-sectional face-to-face respondents and that may contribute to the difference. However, the literature shows that the survey experience has little impact on survey responses (Toepoel, Das, & Soest, 2008).

Although the results show significant differences for many items, the absolute differences are not large for many of them. In addition, when ranking the scenarios from the most favoring to the least favoring, the patterns for both face-to-face and Web surveys are almost identical. In many situations, such as policy making, the general degree of favoring/opposing, rather than the exact number, is of the most interest. In that case, there is very little mode difference in the impression gained regarding public attitudes to the comparative acceptability for the various reasons for abortion.

Future work is definitely needed to further examine the mode effect on abortion questions, using other data sources. When asking abortion-related attitudinal questions, I encourage future researchers to explore other modes. Since both interviewer motivation and privacy are the potential factors contributing to the measurement bias, one should consider a combination of modes that can maximize the effectiveness of both. For example, computer-assisted self-interviews could be a worthwhile research effort. Similarly, a leave-behind self-administered questionnaire for sensitive questions after a face-to-face survey is also a potential approach.

One major limitation of the 2012 ANES is the quasi-experimental nature of the survey and low response rates for both modes of data collection, and for Web surveys in particular. The low response rate is potentially correlated with higher nonresponse bias. Selection bias is another possible source of error for the observed mode difference, given the response rate difference between the two surveys (Vannieuwenhuyze & Revilla, 2013). The data at hand do not allow us to tease apart the mode difference from the selection bias. Therefore, the real mode difference may be smaller or even nonexistent. Future surveys should consider a strict randomized experiment for studying the face-to-face vs. Web differences, and use techniques to improve the response rate and make them comparable between the modes under comparison. Another limitation is the limited scale type analyzed in the study. It is entirely possible that mode effect interacts with the rating scale characteristics and variations in scales is necessary to make conclusions that are more general about mode effect on abortion questions. As previous research shows, the way people respond to answer scales can differ by the data collection mode (Liu, Conrad, & Lee, 2016; Weijters, Schillewaert, & Geuens, 2008). Therefore, future research should a variety of scales on the same topic between modes to see if the results reported here still hold. Last, survey is just one of the methodologies for collecting public opinion on abortion. In fact, there is a long running debate between quantitative and qualitative research methods on issues like the one studied here (for examples, see Jayaratne & Stewart, 1991; Lawson, 1995; Westmarland, 2001). Whether different survey modes will draw similar or different conclusion from qualitative studies is unknown but worth exploring.

Regardless of the limitations, this is the first study that reports the potential abortion attitude differences between national face-to-face and Web surveys. There are slightly more face-to-face respondents who did not provide an answer to the questions than Web respondents. Also, larger responses differences exist between modes observed from scenarios that are less socially acceptable while smaller differences exist for scenarios that are more acceptable. The relative degrees of favoring/opposing for various scenarios are quite similar between the two survey modes. For one thing, the differences between these two modes urge that caution be taken when directly comparing results collected through these two different modes. For another, the mode differences do not impose a serious threat on measuring the general population’s opinions toward abortion.

¹For more information on sample design, see http://www.electionstudies.org/studypages/anes_timeseries_2012/anes_timeseries_2012_userguidecodebook.pdf.

Appendix

Question wordings used in the analysis.

Do you favor, oppose, or neither favor nor oppose abortion being legal if staying pregnant would hurt the woman’s health but is very unlikely to cause her to die.

Do you favor, oppose, or neither favor nor oppose abortion being legal if staying pregnant could cause the woman to die.

Do you favor, oppose, or neither favor nor oppose abortion being legal if the pregnancy was caused by the woman having sex with a blood relative.

Do you favor, oppose, or neither favor nor oppose abortion being legal if the pregnancy was caused by the woman being raped.

Do you favor, oppose, or neither favor nor oppose abortion being legal if the fetus will be born with a serious birth defect.

Do you favor, oppose, or neither favor nor oppose abortion being legal if having the child would be extremely difficult for the woman financially.

Do you favor, oppose, or neither favor nor oppose abortion being legal if the child will not be the sex the woman wants it to be.

Do you favor, oppose, or neither favor nor oppose abortion being legal if the woman chooses to have one?

References

• Altshuler, A. L., Gerns Storey, H. L., & Prager, S. W. (2015). Exploring abortion attitudes of US adolescents and young adults using social media. Contraception, 91(3), 226–233. https://doi.org/10.1016/j.contraception.2014.11.009
• Begun, S., Kattari, S. K., McKay, K., Winter, V. R., & O’Neill, E. (2017). Exploring U.S. Social Work Students’ Sexual Attitudes and Abortion Viewpoints. The Journal of Sex Research, 54(6), 752–763. https://doi.org/10.1080/00224499.2016.1186586
• Beukenhorst, D., Buelens, B., Engelen, F., van der Laan, J., Meertens, V., & Schouten, B. (2014). The impact of Survey item characteristics on mode-specific measurement bias in the Crime Victimisation Survey. Retrieved from http://www.cbs.nl/NR/rdonlyres/639072AA-6903-468A-950C-4BFA590C3CDD/0/201416x10pub.pdf
• Callegaro, M. (2008). Social desirability. Encyclopedia of Survey Research Methods, 825–826.
• Christensen, A. I., Ekholm, O., Glümer, C., & Juel, K. (2014). Effect of survey mode on response patterns: comparison of face-to-face and self-administered modes in health surveys. The European Journal of Public Health, 24(2), 327–332. https://doi.org/10.1093/eurpub/ckt067
• Duffy, B., Smith, K., Terhanian, G., & Bremer, J. (2005). Comparing data from online and face-to-face surveys. International Journal of Market Research, 47(6), 615–639.
• Finlay, B. A. (1981). Sex differences in correlates of abortion attitudes among college students. Journal of Marriage and the Family, 571–582.
• Goldenbeld, C., & de Craen, S. (2013). The comparison of road safety survey answers between web-panel and face-to-face; Dutch results of SARTRE-4 survey. Journal of Safety Research, 46, 13–20. https://doi.org/10.1016/j.jsr.2013.03.004
• Heerwegh, D. (2009). Mode Differences Between Face-to-Face and Web Surveys: An Experimental Investigation of Data Quality and Social Desirability Effects. International Journal of Public Opinion Research, 21 (1), 111–121. https://doi.org/10.1093/ijpor/edn054
• Heerwegh, D., & Loosveldt, G. (2008). Face-to-Face versus Web Surveying in a High-Internet-Coverage Population Differences in Response Quality. Public Opinion Quarterly, 72(5), 836–846. https://doi.org/10.1093/poq/nfn045
• Jayaratne, T. E., & Stewart, A. J. (1991). Quantitative and qualitative methods in the social sciences. Beyond Methodology: Feminist Scholarship as Lived Research, 85–106.
• Jones, E. F., & Forrest, J. D. (1992). Underreporting of abortion in surveys of US women: 1976 to 1988. Demography, 29(1), 113–126.
• Jones, R. K., & Jerman, J. (2017). Population Group Abortion Rates and Lifetime Incidence of Abortion: United States, 2008–2014. American Journal of Public Health, 107(12), 1904–1909. https://doi.org/10.2105/AJPH.2017.304042
• Kreuter, F., Presser, S., & Tourangeau, R. (2008). Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity. Public Opinion Quarterly, 72(5), 847–865. https://doi.org/10.1093/poq/nfn063
• Lara, D., Strickler, J., Olavarrieta, C. D., & Ellertson, C. (2004). Measuring induced abortion in Mexico: a comparison of four methodologies. Sociological Methods & Research, 32(4), 529–558.
• Lawson, V. (1995). The Politics of Difference: Examining the Quantitative/Qualitative Dualism in Post-Structuralist Feminist Research*. The Professional Geographer, 47(4), 449–457. https://doi.org/10.1111/j.0033-0124.1995.00449.x
• Lensvelt-Mulders, G. J., Hox, J. J., Van der Heijden, P. G., & Maas, C. J. (2005). Meta-analysis of randomized response research: Thirty-five years of validation. Sociological Methods & Research, 33(3), 319–348.
• Liu, M., Conrad, F. G., & Lee, S. (2016). Comparing acquiescent and extreme response styles in face-to-face and web surveys. Quality & Quantity. https://doi.org/10.1007/s11135-016-0320-7
• Liu, M., & Wang, Y. (2015). Data collection mode effect on feeling thermometer questions: A comparison of face-to-face and Web surveys. Computers in Human Behavior, 48(7), 212–218. https://doi.org/10.1016/j.chb.2015.01.057
• Lohan, M., Cruise, S., O’Halloran, P., Alderdice, F., & Hyde, A. (2011). Adolescent men’s attitudes and decision-making in relation to an unplanned pregnancy. Responses to an interactive video drama. Social Science & Medicine, 72(9), 1507–1514.
• Manfreda, K. L., Bosnjak, M., Berzelak, J., Haas, I., Vehovar, V., & Berzelak, N. (2008). Web surveys versus other survey modes: A meta-analysis comparing response rates. Journal of the Market Research Society, 50(1), 79.
• Medoff, M. (2013). Pro-Choice Versus Pro-Life: The Relationship Between State Abortion Policy and Child Well-Being in the United States. Health Care for Women International, (ahead-of-print). https://doi.org/10.1080/07399332.2013.841699
• Pacheco, J., & Kreitzer, R. (2016). Adolescent Determinants of Abortion AttitudesEvidence from the Children of the National Longitudinal Survey of Youth. Public Opinion Quarterly, 80(1), 66–89. https://doi.org/10.1093/poq/nfv050
• Patel, C. J., & Johns, L. (2009). Gender role attitudes and attitudes to abortion: Are there gender differences? The Social Science Journal, 46(3), 493–505.
• Revilla, M. A., & Saris, W. E. (2012). A Comparison of the Quality of Questions in a Face-to-face and a Web Survey. International Journal of Public Opinion Research, eds007.
• Schwandt, H. M., Creanga, A. A., Adanu, R. M., Danso, K. A., Agbenyega, T., & Hindin, M. J. (2013). Pathways to unsafe abortion in Ghana: the role of male partners, women and health care providers. Contraception, 88(4), 509–517.
• Shaw, G. M. (2003). Trends: Abortion. Public Opinion Quarterly, 67(3), 407–429. https://doi.org/10.1086/377459
• Singer, E., & Couper, M. P. (2014). The Effect of Question Wording on Attitudes toward Prenatal Testing and Abortion. Public Opinion Quarterly, 78(3), 751–760. https://doi.org/10.1093/poq/nfu028
• Sjöström, S., Essén, B., Sydén, F., Gemzell-Danielsson, K., & Klingberg-Allvin, M. (2014). Medical students’ attitudes and perceptions on abortion: a cross-sectional survey among medical interns in Maharastra, India. Contraception, 90(1), 42–46. https://doi.org/10.1016/j.contraception.2014.02.005
• Toepoel, V., Das, M., & Soest, A. V. (2008). Effects of Design in Web Surveys Comparing Trained and Fresh Respondents. Public Opinion Quarterly, 72(5), 985–1007. https://doi.org/10.1093/poq/nfn060
• Vannieuwenhuyze, J. T. A., & Revilla, M. (2013). Relative Mode Effects on Data Quality in Mixed-Mode Surveys by an Instrumental Variable. Survey Research Methods, 7(3), 157–168. https://doi.org/10.18148/srm/2013.v7i3.5137
• Weijters, B., Schillewaert, N., & Geuens, M. (2008). Assessing response styles across modes of data collection. Journal of the Academy of Marketing Science, 36(3), 409–422. https://doi.org/10.1007/s11747-007-0077-6
• Westmarland, N. (2001). The Quantitative/Qualitative Debate and Feminist Research: A Subjective View of Objectivity. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research , 2(1). Retrieved from http://www.qualitative-research.net/index.php/fqs/article/view/974
• Woodhams, E. J., Hill, B. J., Fabiyi, C., & Gilliam, M. (2016). Describing abortion attitudes among young, African American men. Contraception, 94(2), 134–136. https://doi.org/10.1016/j.contraception.2016.03.008