We used a cloud-based, medical knowledge voluntary closed survey regarding public perceptions of miscarriage via Amazon’s Web service MTurk (Multimedia Appendix 1). MTurk is an online labor market in which “requesters” have access to recruit a large number of “workers” to complete tasks. Typical tasks include transcribing audio, categorizing data, algorithm training, classifying images, reviewing databases for key information, performing online searches, tagging photos with preset words, and other tasks that are difficult to automate [24].

MTurk has templates for basic surveys that can be customized by the requester via their basic user interface (UI). The UI functions as a low-end word processing software and automatically creates a Hyper Text Markup Language (HTML) based code. The survey was tested in MTurk sandbox, which mimics a live release and allows the writers to see results in real-time without cost. Requesters can take their own survey and see the results using the same interface and server as the live version. After testing in sandbox, we ran a 100 respondent test-run that took 3 hours to complete. This allowed us to tweak any questions that were either unclear or gave ambiguous results.

Workers log in and browse Human Intelligence Tasks (HITs) by title, reward, requester or keyword to find a topic of interest. For our survey, we listed the keywords “answers, survey, experiment, medicine, questionnaire, miscarriage, simple, quick, fun, money, and pregnancy”. When a worker clicks on a requester’s work, the user sees a description of the work prior to accepting the HIT. In the description, we informed workers that they would not be paid for repeating the survey. There is no way to determine the percentage of workers who viewed but did not accept our HIT, but we were able to record how many people accepted our HIT but did not submit.

The system allows requesters to set qualifications for “MTurkers”. Since this was a study regarding the US population, we limited respondents to only those based in the United States. Another qualification we used is the MTurker “HIT approval rate”. The approval rating qualification allows us to select workers who have proven to perform quality work on other tasks. We set the approval rating at 85%, which means that 85% or more of their previous work on various HITs was approved. This is important as it allows us to select higher “quality” workers. We required that workers had finished at least 50 other HITs to increase the likelihood of quality data collection.

HITs are listed in the order they are posted; at any given time there can be thousands of HITs available. We released the surveys in batches of 50. When a worker completes his or her task, the results are automatically sent to the requester for approval and payment. As part of the data received, there is a unique worker ID and an approval rating. The workers’ approval ratings show the percentage of work approved by the requester in the last 7 days, 30 days, and lifetime of the worker. That percentage allows one to filter out anyone who has taken the survey previously and reject his or her work. This method allowed us to control for workers repeating the survey, which would skew the data.

We used an attention check question (ACQ) of “Have you had a fatal heart attack while watching TV” as well as a time filter. If the respondent answered yes or maybe to the ACQ or if it took a respondent less than 60 seconds to finish the 23-item survey, they were excluded from the study. Once we finished screening respondents based on the ACQ and time filters, we were able to approve or reject the results. Upon acceptance, respondents were immediately paid US $0.25 for their work. Rejected work was immediately reposted to allow for proper completion. After the survey was completed, the data was downloaded into a Microsoft Excel file for analysis. The study was approved by the Albert Einstein College of medicine institution review board (IRB).

Respondents were aware they were taking a survey regarding pregnancy, however, were unaware of any relationship with an investigator or university. The data was collected over 3 days in January of 2013. The items were not randomized and adaptive questioning was used based on respondents answer to whether they had a miscarriage. If they answered yes to miscarriage, further questions regarding their feelings after a miscarriage were added. The 23-item survey (with additional 10 added if they answered yes to the question regarding miscarriage) appeared on a single HTML page. A completeness check was not required prior to submitting and respondents were able to review their answers prior to submission. We were unable to provide a view rate, as we are unable to determine the number of people who viewed our survey but did not choose to accept. All accepted surveys were completed with a 99% item completeness rate.

All survey data were preserved in the original format for analysis and continuous attributes were compared using t tests. All significance values were calculated for 2-sided 95% CIs or P values less than .05. Microsoft Excel (2016, Redmond, California) was used for analysis.

A total of 1147 responses were collected in 2 days 14 hours and 42 minutes. On average, this translated to one survey collected every 3.3 minutes. It took 93 minutes to obtain the first 100 responses and 316 minutes to obtain the first 300 responses. The next 845 responses took 57 hours and 26 minutes to collect (Figure 1).

Of the respondents, 57 repeated the survey, 6 participants answered yes to the ACQ, and no one completed the survey in less than 60 seconds. This left us with a total of 1084 usable responses (usable response rate of 94%).

We found that if we posted batches larger than 50, the batch would take longer to finish and we would cancel the remaining surveys in that particular batch. There were no significant differences in the time to batch completion between posting a batch of 25 (156 min) versus 50 surveys (209 min) (P=0.5). Batches posted between 11:00 pm and 8:30 am (Eastern Standard Time, EST) took the longest to complete. With each subsequent batch posted, the average time to completion increased. The shortest time to finish a batch was 26 minutes, and the longest was 591 minutes (Figure 2).

The mean age of the respondents was 31, with the majority of the sample between 18 to 34 years (Table 1). Of the total number participants, 44.74% (485/1084) were male and 54.80% (594/1084) female. In addition, 53.60% (580/1082) were never married, with 37.80% (409/1082) currently married. Most of our respondents were white (82.87%, 895/1080) with 6.29% (68/1080) Hispanic and 5.55% (60/1080) black. Over half the respondents (51.40%, 548/1066) reported a religious affiliation. The majority of religious people identified themselves as Christian (44.93%, 479/1066). The majority of our respondents either attended some college (39.11%, 422/1079) or graduated college (36.79%, 397/1079). As well, 10.19% (110/1079) of the respondents reported graduating high school as their highest degree earned.

The respondents represented a diverse socioeconomic status with 17.14% (17/1079) earning less than US $19,000 a year. The majority of the respondents earned less than US $60,000 (70.06%, 756/1079) with 9.73% (105/1079) earning over US $100,000 a year. We collected responses from 49 of the 50 states. California (9.45%, 102/1079), New York (6.85%, 74/1079), Florida (5.37%, 58/1079), and Pennsylvania (5.28%, 57/1079) were the four largest contributors. Arkansas, Montana and North Dakota had the fewest respondents with each contributing 1 (0.09%, 1/1079).

We reimbursed survey respondents US $0.25. We paid a total of US $271.00 to 1084 respondents for completing the survey. The average effective hourly rate for the respondents came out to US $3.97. Since Amazon charges a 10% fee of what is paid to the workers, our total cost for running the survey was US $298.10 or US $0.275 per respondent.

Our study utilized Amazon’s MTurk to effectively and efficiently obtain survey data from a large national pool of both men and women. We were able to obtain both quantitative and qualitative information regarding participants’ knowledge of and experience with miscarriage. What makes this data unique is that we obtained quality data from over 1000 respondents over a 3-day window for under US $300. Obtaining 1000 respondents with other survey methods would cost between 4 to 69 times as much as our method depending on the survey type [4,12]. In addition, it would have taken on average weeks to months to complete the requisite number of responses [5-7].

We found that posting batches larger than 50 led to longer batch completion times, and that the best time to post was between 8:30 am to 8:30 pm EST. Based on our data, researchers should post batches during the hours in which their targeted population is awake. For example, researchers on the east coast who are targeting a west coast population should adjust their posting schedules to match the sleep/wake schedules of their expected participants.

Previously, complex technical computer skills were required to create online surveys [25]. MTurk’s UI is simple to use. For those without technical skills, MTurk can be set to function similar to simple word processing programs. For those with more advanced programming skills, other more powerful programming languages may be added. MTurk, therefore, allows basic and advanced users of computer technology to easily create and distribute surveys.

Limitations to the study include the potential for non-responder bias, as it is not possible to determine how many people previewed our survey without completing it. It is possible that those who responded felt more strongly about issues related to miscarriage. However, these limitations may exist regardless of method of survey distribution.

In addition, previous studies have suggested that the demographics of MTurk participants differed from national demographics; a paper by Paolacci et al suggested that only 47% of MTurk workers were in the United States [18]. However, our study included only those who were US based, thereby eliminating bias from international responders. As prior studies using MTurk have shown [14], race and ethnicity are not proportionately represented, with an underrepresentation of blacks and Hispanics and an overrepresentation of Asians. This may limit the generalizability of our study to the United States as a whole. In addition, respondents had attained a higher level of education than the general public, which may have led to under emphasis of misperceptions found in the general public [26]. However, participants were from 49 of 50 states with no one region over or underrepresented, and overall, the sociodemographic distribution across gender, age, religion, geographic location, and household income was consistent with 2010 national census statistics [27].

To our knowledge, our study is the first to examine perceptions and understanding of miscarriage among the US national population, and the first to use crowdsourced surveys to examine patient satisfaction with health care providers. Other studies have been able to obtain information from a national sample, however, data collection was costly as well as labor and time intensive. The results of our study demonstrate that MTurk is a safe, cost-effective, and time-efficient way to confidentially obtain important, sensitive information on reproductive health from a large, diverse participant population. Our study also shows that it is also possible to obtain rapid data for general health questions. While many previous studies have assessed MTurk’s validity within psychological and behavioral research, our survey utilized MTurk to conduct medical research, the results of which may impact the future health care of both men and women.