HHS Public Access
Author manuscript
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Published in final edited form as:
Epidemiology. 2019 July ; 30(4): 597–608. doi:10.1097/EDE.0000000000001028.
Development of the Initial Surveys for the All of Us Research 
Program
Robert M. Cronin, MD, MS1,2, Rebecca N. Jerome, MLIS, MPH3, Brandy Mapes, MLIS3, 
Regina Andrade, MS3, Rebecca Johnston, MS3, Jennifer Ayala, PhD3, David Schlundt, 
PhD4, Kemberlee Bonnet, MA4, Sunil Kripalani, MD, MSc5,6,7, Kathryn Goggins, MPH5,6,7, 
Kenneth A. Wallston, PhD8, Mick P. Couper, PhD9,10, Michael R. Ellitt, PhD11,12, Paul Harris, 
PhD3, Mark Begale13, Fatima Munoz, MD, MPH14, Maria Lopez-Class, PhD, MPH15, David 
Cella, PhD16, David Condon, PhD16, Mona AuYoung, MS, MPH17, Kathleen M. Mazor, EdD18, 
Steve Mikita, J.D.19, Michael Manganiello, MPA20, Nicholas Borselli, MPP20, Stephanie 
Fowler, PhD, MPH15, Joni L. Rutter, PhD15, Joshua C. Denny, MD, MS1, Elizabeth W. 
Karlson, MD, MPH21, Brian K. Ahmedani, MSW, PhD22, Chris O’Donnell, MD, MPH23,24, and 
the Vanderbilt University Medical Center Pilot Team, and the Participant Provided 
Information Committee
1Department of Biomedical Informatics and Internal Medicine, Vanderbilt University Medical 
Center, Nashville, Tennessee, USA
2Department of Pediatrics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
3Vanderbilt Institute for Clinical and Translational Research, Vanderbilt University Medical Center, 
Nashville, Tennessee, USA
4Department of Psychology, Vanderbilt University, Nashville, TN, USA
5Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee, USA
6Center for Clinical Quality and Implementation Research, Vanderbilt University Medical Center, 
Nashville, Tennessee, USA
7Center for Effective Health Communication, Vanderbilt University Medical Center, Nashville, 
Tennessee, USA
8Institute for Medicine and Public Health, Vanderbilt University Medical Center, Nashville, 
Tennessee, USA
9Survey Research Center, University of Michigan. Ann Arbor, MI, USA
10Joint Program in Survey Methodology, University of Maryland, College Park, MD, USA
Corresponding author: Robert M. Cronin, Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West 
End Blvd., Suite 1475, Nashville, TN, 37232, USA, robert.cronin@vanderbilt.edu. 
Conflicts of interest: The authors have no conflicts of interest
Data availability: All relevant data are within the manuscript and its Supporting Information files or are available from http://
researchallofus.org.
Research Ethics and Informed Consent:
The research assessments described in this manuscript were approved by the Institutional Review Board (IRB) of the All of Us 
Research Program. Those participants that completed cognitive interview gave oral consent to do so.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 2
11Department of Biostatistics, School of Public Health, University of Michigan, Ann Arbor, MI, USA
12Survey Research Center, Institute for Social Research, University of Michigan, Ann Arbor, MI 
USA
13Vibrent Health, Fairfax, Virginia, USA
14Department of Research and Health Promotion, San Ysidro Health, San Diego, California, USA
15National Institutes of Health, Office of the Director, Bethesda, Maryland, USA
16Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, 
Chicago, IL, USA
17Scripps Whittier Diabetes Institute, Scripps Health, San Diego, California, United States
18Meyers Primary Care Institute, Worcester, MA, USA
19Spinal Muscular Atrophy Foundation, New York, New York, United States of America
20HCM Strategists, Washington, DC, United States of America
21Department of Medicine, Division of Rheumatology, Allergy, and Immunology, Section of Clinical 
Sciences, Brigham and Women’s Hospital, Boston, Massachusetts, USA
22Center for Health Policy & Health Services Research, Henry Ford Health System, Detroit, MI, 
USA
23Cardiology Section, Department of Medicine, Veterans Affairs Boston Healthcare System, 
Boston, Massachusetts, USA
24Cardiovascular Medicine Division, Department of Medicine, Brigham and Women’s Hospital, 
Harvard Medical School, Boston, Massachusetts, USA
Abstract
Background—The All of Us Research Program is building a national longitudinal cohort and 
collecting data from multiple information sources (e.g., biospecimens, electronic health records 
(EHRs), and mobile/wearable technologies) to advance precision medicine. Participant-provided 
information, collected via surveys, will complement and augment these information sources. We 
report the process used to develop and refine the initial three surveys for this program.
Methods—The All of Us survey development process included: (1) prioritization of domains for 
scientific needs, (2) examination of existing validated instruments, (3) content creation, (4) 
evaluation and refinement via cognitive interviews and online testing, (5) content review by key 
stakeholders, and (6) launch in the All of Us electronic participant portal. All content was 
translated into Spanish.
Results—We conducted cognitive interviews in English and Spanish with 169 participants, and 
573 individuals completed online testing. Feedback led to over 40 item content changes. Lessons 
learned included: (1) validated survey instruments performed well in diverse populations reflective 
of All of Us; (2) parallel evaluation of multiple languages can ensure optimal survey deployment; 
(3) recruitment challenges in diverse populations required multiple strategies; and (4) key 
stakeholders improved integration of surveys into larger Program context.
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 3
Conclusions—This efficient, iterative process led to successful testing, refinement, and launch 
of three All of Us surveys. Reuse of All of Us surveys, available at http://researchallofus.org, may 
facilitate large consortia targeting diverse populations in English and Spanish to capture 
participant-provided information to supplement other data, such as genetic, physical 
measurements, or data from EHRs.
Keywords
Health surveys; Precision medicine; Questionnaires; Cohort studies; Cognitive interviews; Online 
surveys
Introduction
Precision medicine is an approach to identifying risk factors, etiology, treatment, and 
prevention of disease emphasizing variability in an individual’s genes, environment, and 
lifestyle. Precision medicine research has yielded numerous discoveries regarding genomic 
influences on diseases and drug responses1–4, which has resulted in improved patient 
outcomes5,6. These studies have historically leveraged phenotypic information from 
physiologic measurements, health surveys, and bioassays. Surveys have been an important 
part of observational research for decades. Prominent studies such as the National Health 
and Nutrition Examination Survey7, the National Health Interview Survey8, and UK 
Biobank9,10, have added substantially to biomedical knowledge in part through survey-based 
capture of exposures and outcomes. Research has also demonstrated the power of routine 
clinical information extracted from the electronic health record (EHR); others are also 
exploring the utility of newer modalities such as wearable biosensors and environmental 
data11–23. Collectively, these strategies hold promise for more precise ways to identify 
patterns in large datasets and, ultimately, understanding of factors that contribute to health 
and disease.
The All of Us Research Program (All of Us) has the goal of enrolling a longitudinal cohort 
of at least one million participants reflecting the rich diversity of the U.S. population24–26, 
prioritizing groups historically underrepresented in biomedical research27–29. A broad range 
of data will be collected to describe each participant, including surveys, EHRs, 
biospecimens, physical exams, wearable technologies, and geospatial and environmental 
sources. These data will enable creation of a robust research resource to facilitate deep 
exploration of biologic, clinical, social, environmental, and behavioral determinants of 
health and disease.
All of Us will use surveys to collect data directly from a diverse cohort of participants across 
the socio-economic spectrum with different racial/ethnic backgrounds and for whom English 
may not be their primary language. These surveys are primarily designed to complement 
information collected from other sources such as EHRs. EHRs do not routinely collect data 
on many social, environmental, or behavioral determinants of health, or these variables may 
be difficult to find within the text of EHR clinical notes30, thus the additional data from 
surveys have the potential to enable researchers to test hypotheses at a greater scope and 
level of precision than before.
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 4
Surveys also offer an opportunity to engage participants, explicitly recognizing the 
importance of the information that individuals can share over time about their own socio-
demographic characteristics, health, and other factors. In All of Us, participants answer 
survey questions via an online participant portal, available on both computers and mobile 
devices, which guide participants through surveys to be completed, while keeping track of 
their progress and engaging them throughout the life of the Program. This paper describes 
the rigorous process for rapid development, testing, and refinement of the initial three 
surveys included in the launch of All of Us in 2018.
Methods
The All of Us survey development process comprised a multidisciplinary collaboration and 
included: 1) preparatory work such as choosing the scientific domains of focus and 
examining existing validated instruments to create content; 2) content testing and 
refinement; and 3) stakeholder agreement on final versions for program deployment (Figures 
1 and 2).
Drafting initial surveys
This process began in October 2015 with an NIH All of Us Protocol Working Group 
convened by the National Institutes of Health (NIH) Office of the Director, consisting of 
approximately 25 NIH staff and non-NIH scientists from a variety of disciplines. The 
working group drafted a list of desired survey domains to be included in All of Us. Priorities 
of the domains were determined based on a set of criteria (eTable 1). This group identified 
and reviewed existing validated survey instruments for use in the All of Us Research 
Program from October 2015 to March 2016, including 19 instruments from large research 
endeavors (Table 1). Instrument questions were chosen based on validation evidence in 
diverse populations, usage in other national studies, and access to their use (lack of 
copyright).
The All of Us Pilot team was formed in early 2016 to continue survey development. The 
Pilot team employed qualitative methods to test and refine survey content to ensure optimal 
implementation, described below. This interdisciplinary team consisted of approximately 20 
experts representing a range of fields: (1) cognitive sciences and electronic/non-electronic 
survey design and related methodology; (2) health literacy and effective health 
communication; (3) engagement and inclusion of underrepresented minority populations, 
such as African Americans and Hispanics/Latinos, Lesbian, Gay, Bisexual, Transgender, 
Queer or Questioning, and Intersex (LGBTQI), and rural residents; (4) information science, 
for evidence review and synthesis; (5) racially and ethnically diverse bilingual research staff 
proficient in English and Spanish; and (6) biomedical informatics, for guidance around data 
collection methods, data harmonization, and quality and quantity of information.
The Pilot team built upon the materials originally drafted by the Working Group. The three 
surveys prioritized by the Program for initial testing and included as part of the national 
launch of All of Us recruitment, guided by the information most typically collected at time 
of enrollment in large cohort studies included: The Basics (sociodemographic questions), 
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 5
Overall Health (general health overview), and Lifestyle (tobacco, alcohol, and drug 
questions).
The team created a standardized process, described below, to test each survey’s readability, 
response format and options, participants’ confidence in their ability to answer the questions 
accurately, and other key issues, such as redundancy and sensitivity.
As part of this phase of work, the Pilot team also collated all source instruments from which 
content was drawn to create a metadata database. The metadata database was used to ensure 
that principles of survey development were adhered to, including: (1) maintenance of 
scientific validity of the original survey instruments, (2) conversion of interviewer-led 
surveys to an online participant-completed format, and (3) usage of validated items widely 
used in the same diverse populations recruited to the Program. The Pilot team reviewed the 
metadata for gaps related to these guiding principles.
As the Program grew, survey creation became the responsibility of the Participant-Provided 
Information Committee. This committee is a group of approximately 10–15 experts tasked 
with overseeing the entire survey development cycle, comprised of the types of domain 
experts mentioned above, as well as representatives from participant recruitment sites. This 
committee has been instrumental in refining content as well as determining new areas of 
focus. Table 2 describes prioritized domains and current development status.
Testing sample
The Pilot team developed an approach for qualitative testing, including cognitive 
interviewing and web-based testing. Participants were recruited from a robust and diverse 
pool of ~5,000 individuals who had previously expressed interest in helping to develop All 
of Us content., as well as through additional methods such as in-person events with 
community partners. These recruitment approaches enabled the Pilot team to reach minority 
populations and those with limited access to the internet, as well as rural and urban areas.
Qualitative and quantitative testing
Cognitive interviewing employed a “think aloud” approach, exploring participants’ 
understanding about each question and response option. Probes, including interviewer 
prompts and follow-up questions, were used as needed to elicit detailed discussion of each 
item31–33. Cognitive interviews were conducted in person or by Skype31,33,34 based on 
participant preference. Transcripts of interview audio recordings were analyzed qualitatively. 
A hierarchical coding system was developed to thematically assess the qualitative data. 
Quotes supporting the different codes were recorded as part of the analysis. Interviews 
continued until saturation was achieved.
Online qualitative and quantitative testing, done in parallel with cognitive interviews, 
intended to augment the data from cognitive interviews by engaging a larger number of 
diverse participants. Online testing employed the Research Electronic Data Capture 
(REDCap)35 system hosted in a secure online environment, which required that each 
participant login using unique account credentials to help ensure only the proper participant 
completed the survey. The online qualitative testing allowed participants to first answer the 
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 6
question, then share feedback about the clarity, understandability, and sensitivity of each 
question and its response options (Figure 3); in quantitative testing, used to estimate the time 
to complete each survey, participants only answered questions without a feedback option. 
Where appropriate, analysis of these results included exploratory factor analysis using 
principal components extraction and varimax rotation, frequency statistics, and assessment 
of internal consistency using Cronbach’s alpha36.
Translation and testing in Spanish
The surveys were developed and tested in both English and Spanish (Figures 1 & 2). Prior to 
testing, the English surveys were translated into Spanish. If the English language questions 
had an equivalent Spanish version from the original source instrument, then that Spanish 
equivalent was used as the primary translated version. However, if the English version did 
not have an existing Spanish translation, those questions went through a multi-step 
translation process developed by the Pilot team. This included primary translation by a third-
party company (MotionPoint), during which a certified professional translator translated the 
content into Spanish, followed by secondary review of the translation by community 
reviewers to ensure the content was easily understandable to a wide range of Spanish 
speakers with diverse educational attainment and country of origin. This secondary review 
also identified opportunities for improving readability.
For the first Spanish language iteration of the initial surveys (The Basics, Overall Health and 
Lifestyle), secondary translation review was conducted by a small group of Program 
stakeholders representing diverse geographical native Spanish speakers ranging from 
Mexico, Spain, Venezuela, and El Salvador to ensure that the Spanish language review 
captured wide ethno-geographic and cultural differences. This group also reviewed the final 
recommendations for the translated Spanish materials used in the Program.
Testing of the Spanish-language materials primarily included recruiting diverse Spanish-
speaking participants across the U.S. at in-person events and connecting with community 
organizations embedded within Latino communities. The approach for identifying any 
critical changes in the Spanish-language materials followed the same process outlined for 
the English versions.
Review by key Program stakeholders
The Pilot team reviewed and proposed further refinements to the materials based on review 
of the original validated instruments and analyses from the cognitive interviews and online 
testing. The survey materials and their accompanying proposed list of changes were then 
reviewed by key Program leadership, including the Participant-Provided Information and 
Steering Committees. Feedback from these groups was used to inform a refined iteration of 
each survey, leading to a second round of testing (Figures 1 & 2, eTable 2). After this second 
round of testing, the Pilot team provided recommendations that led to a final review and 
approval by key Program leadership.
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 7
Implementation in participant portal
In the final step of development, the three surveys were incorporated into the All of Us 
participant portal, the mechanism for survey self-administration in the program. These 
surveys are available to participants once they sign into the portal, allowing for completion 
in their own time.
Results
Process overview
The testing and refinement process successfully implemented for the first three modules in 
the program included two rounds of cognitive interviews and online testing, complemented 
by expert review and input after each round of testing. The results of each of the 
development phases is described in further detail below.
First round of testing and revisions
The first round of testing focused on both qualitative and quantitative testing of the English 
survey materials. Cognitive interviews were conducted with 74 participants. A total of 337 
individuals provided online feedback on the English surveys, of which 225 provided 
qualitative feedback, and 112 provided quantitative feedback (Table 3). Qualitative testing 
revealed major themes that demonstrated issues with clarity, understandability, and 
sensitivity (eTable 3). A summary of the estimated completion time data calculated based on 
this round of testing is included in Table 4.
Second round of testing and revisions
The second round of cognitive interviewing focused on expanding participant diversity and 
testing both English (n = 48) and Spanish (n = 47) versions (total n = 95). Online testing in 
English was completed with 236 participants (qualitative testing n=159, quantitative testing 
n=77) (Table 3). There were no new substantial changes in items that emerged from online 
qualitative testing in the second round. See Table 4 for estimated completions times. Review 
of the data led to very minor recommendations involving changing wording and adding 
clarifying language to some of the questions.
The yield of this development process: final revisions to survey content
A summary of the over 40 recommended changes based on English language testing is 
included in Table 5. Minor changes consisted of small edits in phrasing of a question or 
response options (e.g., converting questions that were originally administered by 
interviewers in the parent survey to a format appropriate for self-administration). Major 
changes included more substantive modifications to a question and/or response options when 
noteworthy concerns were expressed by participants during qualitative testing. For example, 
the questions about race, ethnicity, and gender that were included in the first round of 
qualitative testing were modified and retested in the second round of testing based on input 
from participants and key Program stakeholders. These modifications led to refinement of 
our approach to race and ethnicity. We ultimately leveraged extensive testing completed by 
the US Census Bureau, which found that a combined question for capturing participant-
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 8
reported race and ethnicity was the strongest approach for gathering complete and accurate 
data37. The recommendations for changes in the Spanish language version included minor 
changes that implemented more simplified and common terminology. In addition, testing 
Spanish-language materials resulted in feedback and recommendations that went beyond 
translation, and included conceptual changes to questions to make them more 
understandable and accessible to their specific communities.
We explored concerns from members of the Program, such as about asking potentially 
sensitive questions regarding topics like gender identity and sexual orientation. Our testing 
did not confirm anticipated concerns but instead found appreciation of being asked these 
questions among participants, including those self-identifying as sexual and gender 
minorities.
All questions were finalized by the committees and are summarized with their original 
instruments in eDocuments 1–6. The summary of iterative changes for both English and 
Spanish are summarized in eTable 4.
Exploratory factor analysis
While much of the data in the three surveys were not amenable to further statistical analysis, 
an exploratory factor analysis was appropriate for two subcomponents (i.e. PROMIS, Brief 
Health Literacy Screen) of the English version of Overall Health. This factor analysis 
showed three distinct factors with coefficients alpha being 0.92 for General Physical Health, 
0.81 for Emotional Health, and 0.55 for Health Literacy (eTable 5).
Discussion
We created an iterative process leveraging diverse experts to develop and refine materials for 
collection of participant-provided information for All of Us that is applicable to a diverse 
audience, leverages existing validated surveys, and supports English and Spanish. We 
initially launched three surveys and are following this model for ongoing development of 
future surveys in All of Us.
Lessons learned
Other large consortia such as the Million Veterans Program38 and the U.K. Biobank9,10 have 
included survey materials as a core data component; however, approaches for combining 
survey items from multiple sources in the context of a large research program are sparse. We 
learned several lessons useful for future Program development, as well as others undertaking 
similar work. First, we discovered generally minor issues with clarity and sensitivity for 
some module questions drawn from previously validated survey instruments. While 
questions from certain validated instruments, such as the PROMIS Global Health Scale39 
and the Brief Health Literacy Screen40, performed well in our testing (eTable 3), we 
implemented minor modifications for other items (eTable 4). Modifications included adding 
examples to clarify a question (e.g., providing the number of cigarettes in a pack); new 
response options (e.g., e-cigarettes); and explanatory text before some items. As many of the 
module questions are derived from existing national surveys, we wanted to avoid substantial 
wording revisions, as this would compromise established scientific validity. Instead, we 
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 9
targeted all newly created explanatory text to the 5th grade level and focused on identifying 
areas within the question text where minor revisions could lead to major improvement in 
readability. Second, we explored potential issues of concern to All of Us. In fact, similar to 
other findings41, participants expressed appreciation for the Program’s recognition of the 
importance of asking about gender identity and sexual orientation, which initially concerned 
members of the Program. Third, we successfully leveraged prior extensive participant testing 
within national programs, such as the US Census37, as our findings echoed the Census 
Bureau’s observations that a combined approach for querying race and ethnicity is more 
aligned with the way participants identify themselves and, thus, allows gathering more 
granular data. Fourth, we found that many validated questions from studies that may have 
been developed for a specific population, such as the California Teachers Study42, only 
required minor wording changes, but otherwise performed well in a more diverse 
population. Finally, collaborating with consortium experts led to substantial improvements 
in integrating survey materials into the larger context of the Program. This collaboration led 
to a volume of survey items reasonable to ask participants to complete at enrollment, within 
the larger scope of enrollment activities such as consent and physical exam.
Our experience emphasizes the value of systematically vetting multiple languages to ensure 
optimum survey deployment, as well as the importance of testing with various dialects of a 
specific language. Creating and assessing surveys in both English and Spanish led to 
improved clarity, while maintaining concordance between translations. Testing in both 
languages ensured alignment in understability and accessibility for the English and Spanish 
versions. This experience continues to inform the consortium’s work regarding the 
complexities of testing surveys in different languages. The translation process included 
review by members from multiple Spanish speaking regions to develop surveys understood 
by Spanish speakers from different regions. Instituting multiple versions of Spanish surveys 
is worth consideration, however, we did not find evidence within our interviews with 
participants that indicated this need.
We learned multiple lessons about cognitive interviewing strategies in this population. First, 
this testing process yielded valuable lessons learned regarding strategies for overcoming 
challenges in recruiting under-represented or hard to reach populations. Over time, the Pilot 
team expanded recruitment to include methods such as in-person recruitment within the 
community, facilitating increased enrollment of harder-to-reach populations such as those 
without internet access, those with lower educational attainment, racial/ethnic minority 
populations, and Spanish-speaking participants. This face-to-face community interaction 
was an effective method to recruit certain populations that were more likely to engage in 
research opportunities through a trusted and familiar entity such as an established 
community organization. As echoed by others these methods require more time, but are 
critical to ensure inclusion of diverse populations43–45. Second, cognitive interview probes 
authored by content experts allowed a deeper understanding of what needed to be explored 
to ensure accurate answers. Third, using a web application to perform interviews over the 
internet provided a cost-effective way to interview people who were not located locally. 
Fourth, our sample size was large, and we achieved saturation prior to reaching our full 
sample size. Because enrollment of underrepresented populations is a major aim of the All 
of Us Research Program, we wanted to be thorough in our attempts to include key 
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 10
populations in this component of the program planning. Future efforts of this type will likely 
include smaller enrollment targets. However, based on our experiences, we feel that the size 
and demographic characteristics of the sample are important and should be tailored to best 
fit the scope and goals of the project. For example, researchers interested in exploring 
readability and comprehension alone may reach saturation with a small sample. In instances 
where sensitivity and individual perspectives are also being sought, researchers may want to 
consider a larger and more diverse sample size.
Limitations
We acknowledge several limitations: (1) Our testing did not include all populations 
underrepresented in biomedical research. While we included very important populations, 
including sexual and gender minorities, those of low socioeconomic status, and Spanish-
only speakers, gaps may require additional testing with other populations.(2) The speed at 
which initial survey development work proceeded limited our ability initially to include 
individuals from certain key groups, such as lower educational attainment. However, this 
was addressed during online testing and out second round of cognitive interviews. (3) The 
validity of combining questions from existing instruments was not thoroughly tested. To 
mitigate this, we intentionally minimized changes from existing instruments and executed 
entire scales or sets of questions about a topic area from a single instrument. (4) Email 
communication was the main method of contact for study recruitment, potentially limiting 
our reach of those less comfortable with technology. (5) Small sample size and potential lack 
of representativeness limits generalization of our exploratory factor analysis. (6) Online 
surveying poses an inherent risk, however small, of receiving a response from someone other 
than the intended participant. Finally, (7) we tested primarily on a computer platform and 
not on different technology platforms such as smartphones. Our plan will be to test future 
surveys on different platforms.
Deployment and Future Directions
To achieve the ambitious scope and scale of All of Us, surveys will need to be modular to 
not overwhelm participants, accessible on a digital platform so surveys can be completed in 
a variety of settings, available to be completed at a participant’s own pace, and engaging so 
participants continue to contribute after initial enrollment in the Program.
Building upon this successful process for development of these initial materials, All of Us is 
developing other surveys (Table 2). Future areas of interest for the program will include roll 
out of these surveys, engaging participants to complete these additional surveys, and repeat 
administration of some surveys to reflect participant changes over time. As the Program 
evolves, survey development will continue to be a core activity. As described above, the 
marriage of scientific value, engagement, and participant experience will remain an 
important consideration for future work in this area. Future testing and integration with other 
sources of information, such as EHR data, mobile sensors, and a range of technology 
platforms, genetics, and physical measurements, will help enhance the value and 
completeness of All of Us data for future hypothesis exploration. Further validation of our 
findings related to the surveys in the launch of the Program will be needed to ensure what 
we found in our initial testing holds true for the larger cohort. Translation into other 
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 11
languages and the testing of these translations will also be an important area of activity as 
the Program grows.
Participant-provided information is a critically important part of the data that will power All 
of Us. The survey questions and response options must be carefully documented and 
communicated to researchers in a way that makes these data accessible and easy to integrate 
into external aspects of clinical research. An All of Us survey codebook is under 
development and is being designed to map All of Us survey items to standard vocabularies 
that can help align these data with EHR data where possible (e.g., for diagnoses). Finally, 
All of Us surveys reflect a mixture of questions from validated instruments in the public 
domain, accompanied by supplementary questions that are not currently publicly available. 
All of Us will publish these surveys and their metadata, as they become available, at http://
researchallofus.org.
Conclusions
This flexible process combined multidisciplinary expertise with Program leadership input 
and proven methods to create and refine surveys that are appropriate for use in the diverse 
participant population of All of Us. The process, which was generalizable across multiple 
survey domains, formed a firm roadmap for the development and testing of future materials. 
Other large consortia that target a diverse population in multiple languages could employ 
this process to create surveys that supplement other data sources, such as genetics and EHR 
data.
Prioritization and generation of All of Us surveys will continue in close collaboration with 
representatives across the NIH Institutes and Centers to ensure item integrity and scientific 
validity and help ensure that questions reflect the mission of the NIH to improve health 
outcomes broadly. Other surveys may be driven by participant interest. Participant 
engagement and experience will also be important components of future development. 
Gathering and incorporating participant-provided information in a systematic way will 
enhance the scientific validity and breadth of information obtained from All of Us, leading 
to exciting new advances in the era of precision medicine.
Supplementary Material
Refer to Web version on PubMed Central for supplementary material.
Acknowledgements:
We wish to thank our participants who have joined All of Us and contributed to participant-provided information; 
helped refine early materials; engaged in the development and evaluation of the surveys, and provided other 
ongoing feedback. We thank the countless other co-investigators and staff across all awardees and partners without 
which All of Us would not have achieved our current goals. All of Us is supported by grants through the National 
Institutes of Health Office of the Director: Regional Medical Centers: 1 OT2 OD026549; 1 OT2 OD026554; 1 OT2 
OD026557; 1 OT2 OD026556; 1 OT2 OD026550; 1 OT2 OD 026552; 1 OT2 OD026553; 1 OT2 OD026548; 1 
OT2 OD026551; 1 OT2 OD026555; IAA #: AOD 16037; Federally Qualified Health Centers: HHSN 
263201600085U; Data and Research Center: 5 U2C OD023196; Biobank: 1 U24 OD023121; The Participant 
Center: U24 OD023176; Participant Technology Systems Center: 1 U24 OD023163; Communications and 
Engagement: 3 OT2 OD023205; 3 OT2 OD023206; Community Partners: 1 OT2 OD025277; 3 OT2 OD025315; 1 
OT2 OD025337; 1 OT2 OD025276. Pilot Team: OT2 OD023132, OT2 OD023132–02S1, K23HL141447.
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 12
Original NIH Protocol Working Group Members: Teri Manolio, Rebekah Rasooly, Josephine Briggs, Rory Collins, 
Caroline Dilworth, Montserrat Garcia-Closas, Germaine Buck Lewis, Daniel Masys, Jean Olson, Tim Peakman, 
Bill Riley, Joni Rutter, Paul Sorlie, Elizabeth Wagner, Debbie Winn, Dana Wolff, Kathleen Meister, Luenda 
Charles, Michael Gaziano, Emily Harris, Carolyn Hutter, Sue Krebs-Smith, Sara Luckhaupt, Steven Moore, 
Kathleen Merikangas, Dale Sandler, Amy Subar, Jennifer Thornton, Gordon Willis, Ellen O’Donnell
Pilot Team Members: Charles Mouton, Katherine Donato, Taneya Koonce, Sheila Kusnoor, Marcia Epelbaum, 
Mallory Blasingame, Robert Cronin, Rebecca Jerome, Brandy Mapes, Regina Andrade, Rebecca Johnston, David 
Schlundt, Kemberlee Bonnet, Sunil Kripalani, Kathryn Goggins, Ken Wallston, Mick Couper, Michael Elliott
All of Us Participant Provided Committee Members: Paul Harris, Christopher/Chris O’Donnell, Stephanie Fowler, 
James McClain, Brian Ahmedani, Regina Andrade, Maria Argos, Mona AuYoung, Mark Begale, Bartali Benedetta, 
Pedro Rey Biel, Louise Bier, Marnie Bloom, Nicolas Borselli, Clinton Brawner, Beth Calhoun, David Cella, 
Carmen Chinea, David Condon, Rob Cronin, Julien Dedier, Olivier Elemento, Chris Foster, David Garcia, Holly 
Garriock, John Jackicic, Rebecca Jerome, Daozhong Jin, Christine Johnson, Christine Joseph, Elizabeth Karlson, 
Mike Kellen, Michelle Kienholz, Andrea LaCroix, Elizabeth Lee, Maria Lopez-Class, Michael Manganiello, 
Brandy Mapes, Heather Marino, Fernando Martin-Sanchez, Kathy Mazor, Wanda Montalvo, Fatima Munoz, 
Jyotishman Pathak, Susan Redline, Carolina Rodriguez-Cook, Heather Sansbury, David Schlundt, August Slater, 
Vicki Smith, Carolina Stamoulous, Susan Tirhi, Rhonda Trousdale, Febe Wallace, Joyce Winkler, Jennifer 
Worthington, Jennifer Yttriz, Alvaro Alonso
Spanish Translation Committee: James McClain, Michael Stokes, Regina Andrade, Oscar Beita, Dianne Beltran, 
Amaris Castellanos, Carmen Chinea, Rene Covarrubias, Alfredo Ramirez Clark, Alvaro Donayre, Angelica 
Espinoza, Iliana Faries, Sergio Fernandez-Gonzales, Pablo Gejman, Viridiana Johnson, Maria Lopez-Class, 
Elizabeth Lugo, Fernando Martin-Sanchez, Rima Matsumoto, Daniel Mompoint, Jorge Navarrete, Vijay Rayanker, 
Carolina Rodriguez-Cook, Carolina Stamoulous, Gregory Talavera, Solomon Torres, Sujel Valentin, Emma Viera, 
Carmen Zaldivar, Alejandra Zapien-Hidalgo, Flor McKinley, Ilse Salinas, Janence Ortiz, Janisse Mercado, Jose 
Guadalupe Martinez Lopez, Marcela Gaitán, Marcia Lobos
We also wish to thank All of Us Research Program Direct Eric Dishman as well as our partners as Verily, Vibrent, 
Scripps, and Leidos.
“Precision Medicine Initiative, PMI, All of Us, the All of Us logo, and The Future of Health Begins with You are 
service marks of the U.S. Department of Health and Human Services.”
Sources of financial support: This work was funded by the National Institutes of Health (https://allofus.nih.gov/) 
OT2OD023132 (RMC, RNJ, BMM, RA, RJ, JA, DGS, KRB, SK, KG, KAW, MPC, MRE, PAH, and JCD), 
U2COD023196 (JCD), R01AG0532641U24OD023163–01 (MJB), HHSM500201200008I/HHSN263201800208U 
(FAM), U2CCA186878 (DC), U24 OD023176 (MA), 3OTOD 024610–01S1 (KMM), OT2OD023206 (MM, NB), 
1OT2OD026553 (EWK), and OT2OD026550 (BKA). Additional support included the National Heart, Blood, and 
Lung Institute (https://www.nhlbi.nih.gov/) K23HL141447 (RMC), United States Veteran’s Administration 
Healthcare System (CJO), the National Institute on Aging (https://www.nia.nih.gov/) U2C AG060426–01 (DMC). 
Program staff of the funding organization, National Institutes of Health, co-contributed to this work through 
conceptualization, methodology, validation, project administration and review and editing of this manuscript.
References
1. MacArthur J, Bowler E, Cerezo M, Gil L, Hall P, Hastings E, Junkins H, McMahon A, Milano A, 
Morales J, Pendlington ZM, Welter D, Burdett T, Hindorff L, Flicek P, Cunningham F, Parkinson H. 
The new NHGRI-EBI Catalog of published genome-wide association studies (GWAS Catalog). 
Nucleic Acids Res 2017;45(D1):D896–D901. [PubMed: 27899670] 
2. Delaney JT, Ramirez AH, Bowton E, Pulley JM, Basford MA, Schildcrout JS, Shi Y, Zink R, 
Oetjens M, Xu H, Cleator JH, Jahangir E, Ritchie MD, Masys DR, Roden DM, Crawford DC, 
Denny JC. Predicting clopidogrel response using DNA samples linked to an electronic health 
record. Clin Pharmacol Ther 2012;91(2):257–63. [PubMed: 22190063] 
3. Schildcrout JS, Denny JC, Bowton E, Gregg W, Pulley JM, Basford MA, Cowan JD, Xu H, Ramirez 
AH, Crawford DC, Ritchie MD, Peterson JF, Masys DR, Wilke RA, Roden DM. Optimizing drug 
outcomes through pharmacogenetics: a case for preemptive genotyping. Clin Pharmacol Ther 
2012;92(2):235–42. [PubMed: 22739144] 
4. Wilke RA, Ramsey LB, Johnson SG, Maxwell WD, McLeod HL, Voora D, Krauss RM, Roden DM, 
Feng Q, Cooper-Dehoff RM, Gong L, Klein TE, Wadelius M, Niemi M, Clinical 
Pharmacogenomics Implementation C. The clinical pharmacogenomics implementation consortium: 
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 13
CPIC guideline for SLCO1B1 and simvastatin-induced myopathy. Clin Pharmacol Ther 2012;92(1):
112–7. [PubMed: 22617227] 
5. Mallal S, Phillips E, Carosi G, Molina JM, Workman C, Tomazic J, Jagel-Guedes E, Rugina S, 
Kozyrev O, Cid JF, Hay P, Nolan D, Hughes S, Hughes A, Ryan S, Fitch N, Thorborn D, Benbow 
A, Team P-S. HLA-B*5701 screening for hypersensitivity to abacavir. N Engl J Med 2008;358(6):
568–79. [PubMed: 18256392] 
6. Cavallari LH, Lee CR, Beitelshees AL, Cooper-DeHoff RM, Duarte JD, Voora D, Kimmel SE, 
McDonough CW, Gong Y, Dave CV, Pratt VM, Alestock TD, Anderson RD, Alsip J, Ardati AK, 
Brott BC, Brown L, Chumnumwat S, Clare-Salzler MJ, Coons JC, Denny JC, Dillon C, Elsey AR, 
Hamadeh IS, Harada S, Hillegass WB, Hines L, Horenstein RB, Howell LA, Jeng LJB, Kelemen 
MD, Lee YM, Magvanjav O, Montasser M, Nelson DR, Nutescu EA, Nwaba DC, Pakyz RE, 
Palmer K, Peterson JF, Pollin TI, Quinn AH, Robinson SW, Schub J, Skaar TC, Smith DM, 
Sriramoju VB, Starostik P, Stys TP, Stevenson JM, Varunok N, Vesely MR, Wake DT, Weck KE, 
Weitzel KW, Wilke RA, Willig J, Zhao RY, Kreutz RP, Stouffer GA, Empey PE, Limdi NA, 
Shuldiner AR, Winterstein AG, Johnson JA, Network I. Multisite Investigation of Outcomes With 
Implementation of CYP2C19 Genotype-Guided Antiplatelet Therapy After Percutaneous Coronary 
Intervention. JACC Cardiovasc Interv 2018;11(2):181–191. [PubMed: 29102571] 
7. Massey JT, Moore TF, Tadros W, Parsons V. Design and estimation for the National Health 
Interview Survey 1985–94. VITAL AND HEALTH STATISTICS. SERIES 2: DATA 
EVALUATION AND METHODS RESEARCH 1989(110):1–33.
8. Parsons VL, Moriarity CL, Jonas K, Moore TF, Davis KE, Tompkins L. Design and estimation for 
the national health interview survey, 2006–2015. 2014.
9. Sudlow C, Gallacher J, Allen N, Beral V, Burton P, Danesh J, Downey P, Elliott P, Green J, Landray 
M. UK biobank: an open access resource for identifying the causes of a wide range of complex 
diseases of middle and old age. PLoS medicine 2015;12(3):e1001779. [PubMed: 25826379] 
10. Sudlow C, Gallacher J, Allen N, Beral V, Burton P, Danesh J, Downey P, Elliott P, Green J, 
Landray M, Liu B, Matthews P, Ong G, Pell J, Silman A, Young A, Sprosen T, Peakman T, Collins 
R. UK biobank: an open access resource for identifying the causes of a wide range of complex 
diseases of middle and old age. PLoS Med 2015;12(3):e1001779. [PubMed: 25826379] 
11. Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better research 
applications and clinical care. Nat Rev Genet 2012;13(6):395–405. [PubMed: 22549152] 
12. Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics 
to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014;33(7):1123–
31. [PubMed: 25006137] 
13. Hripcsak G, Albers DJ. Next-generation phenotyping of electronic health records. J Am Med 
Inform Assoc 2013;20(1):117–21. [PubMed: 22955496] 
14. Martin-Sanchez F, Verspoor K. Big data in medicine is driving big changes. Yearb Med Inform 
2014;9:14–20. [PubMed: 25123716] 
15. Baig MM, GholamHosseini H, Moqeem AA, Mirza F, Linden M. A Systematic Review of 
Wearable Patient Monitoring Systems - Current Challenges and Opportunities for Clinical 
Adoption. J Med Syst 2017;41(7):115. [PubMed: 28631139] 
16. Uddin MZ, Khaksar W, Torresen J. Ambient Sensors for Elderly Care and Independent Living: A 
Survey. Sensors (Basel) 2018;18(7).
17. Rucco R, Sorriso A, Liparoti M, Ferraioli G, Sorrentino P, Ambrosanio M, Baselice F. Type and 
Location of Wearable Sensors for Monitoring Falls during Static and Dynamic Tasks in Healthy 
Elderly: A Review. Sensors (Basel) 2018;18(5).
18. Camomilla V, Bergamini E, Fantozzi S, Vannozzi G. Trends Supporting the In-Field Use of 
Wearable Inertial Sensors for Sport Performance Evaluation: A Systematic Review. Sensors 
(Basel) 2018;18(3).
19. Johansson D, Malmgren K, Alt Murphy M. Wearable sensors for clinical applications in epilepsy, 
Parkinson’s disease, and stroke: a mixed-methods systematic review. J Neurol 2018;265(8):1740–
1752. [PubMed: 29427026] 
20. Papi E, Koh WS, McGregor AH. Wearable technology for spine movement assessment: A 
systematic review. J Biomech 2017;64:186–197. [PubMed: 29102267] 
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 14
21. Rovini E, Maremmani C, Cavallo F. How Wearable Sensors Can Support Parkinson’s Disease 
Diagnosis and Treatment: A Systematic Review. Front Neurosci 2017;11:555. [PubMed: 
29056899] 
22. Kozitsina AN, Svalova TS, Malysheva NN, Okhokhonin AV, Vidrevich MB, Brainina KZ. Sensors 
Based on Bio and Biomimetic Receptors in Medical Diagnostic, Environment, and Food Analysis. 
Biosensors (Basel) 2018;8(2).
23. Mukhopadhyay SC. Wearable sensors for human activity monitoring: A review. IEEE sensors 
journal 2015;15(3):1321–1330.
24. Hudson K, Lifton R, Patrick-Lake B. The precision medicine initiative cohort program-building a 
research foundation for 21st century medicine. Precision Medicine Initiative (PMI) Working 
Group Report to the Advisory Committee to the Director, ed 2015.
25. Health NIo. Program Overview - All of Us | National Institutes of Health. https://
www.ncbi.nlm.nih.gov/pubmed/.
26. Investigators AoURP. The All of Us Research Program – Building a Foundation for 21st Century 
Precision Health. In: Personal Communication Cronin R, ed, 2018.
27. Ford JG, Howerton MW, Lai GY, Gary TL, Bolen S, Gibbons MC, Tilburt J, Baffi C, 
Tanpitukpongse TP, Wilson RF, Powe NR, Bass EB. Barriers to recruiting underrepresented 
populations to cancer clinical trials: a systematic review. Cancer 2008;112(2):228–42. [PubMed: 
18008363] 
28. Heller C, Balls-Berry JE, Nery JD, Erwin PJ, Littleton D, Kim M, Kuo WP. Strategies addressing 
barriers to clinical trial enrollment of underrepresented populations: a systematic review. Contemp 
Clin Trials 2014;39(2):169–82. [PubMed: 25131812] 
29. National Academy of Sciences NAoE, Institute of Medicine Committee on Underrepresented G, 
the Expansion of the S, Engineering Workforce P. The National Academies Collection: Reports 
funded by National Institutes of Health Expanding Underrepresented Minority Participation. 
Washington (DC): National Academies Press (US) National Academy of Sciences, 2011.
30. Hripcsak G, Forrest CB, Brennan PF, Stead WW. Informatics to support the IOM social and 
behavioral domains and measures. Journal of the American Medical Informatics Association : 
JAMIA 2015;22(4):921–924. [PubMed: 25914098] 
31. Beatty PC, Willis GB. Research synthesis: The practice of cognitive interviewing. Public Opinion 
Quarterly 2007;71(2):287–311.
32. DeMuro CJ, Lewis SA, DiBenedetti DB, Price MA, Fehnel SE. Successful implementation of 
cognitive interviews in special populations. Expert review of pharmacoeconomics & outcomes 
research 2012;12(2):181–187. [PubMed: 22458619] 
33. Willis GB. Cognitive interviewing: A tool for improving questionnaire design Sage Publications, 
2004.
34. Dean E, Head B, Swicegood J. Virtual cognitive interviewing using Skype and Second Life. Social 
Media, Sociality, and Survey Research 2013:107–132.
35. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture 
(REDCap)--a metadata-driven methodology and workflow process for providing translational 
research informatics support. J Biomed Inform 2009;42(2):377–81. [PubMed: 18929686] 
36. Cronbach L Coefficient alpha and the internal structure of tests. psychometrika 1951;16(3):297–
334.
37. Jones NA. Update on the US Census Bureau’s race and ethnic research for the 2020 Census. 2017.
38. Gaziano JM, Concato J, Brophy M, Fiore L, Pyarajan S, Breeling J, Whitbourne S, Deen J, 
Shannon C, Humphries D. Million Veteran Program: a mega-biobank to study genetic influences 
on health and disease. Journal of clinical epidemiology 2016;70:214–223. [PubMed: 26441289] 
39. Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, Amtmann D, Bode R, Buysse D, Choi 
S. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and 
tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of 
clinical epidemiology 2010;63(11):1179–1194. [PubMed: 20685078] 
40. Wallston KA, Cawthon C, McNaughton CD, Rothman RL, Osborn CY, Kripalani S. Psychometric 
properties of the brief health literacy screen in clinical practice. Journal of general internal 
medicine 2014;29(1):119–126. [PubMed: 23918160] 
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 15
41. Haider AH, Schneider EB, Kodadek LM, Adler RR, Ranjit A, Torain M, Shields RY, Snyder C, 
Schuur JD, Vail L, German D, Peterson S, Lau BD. Emergency Department Query for Patient-
Centered Approaches to Sexual Orientation and Gender Identity : The EQUALITY Study. JAMA 
Intern Med 2017;177(6):819–828. [PubMed: 28437523] 
42. Parikh-Patel A, Allen M, Wright WE, California Teachers Study Steering C. Validation of self-
reported cancers in the California Teachers Study. Am J Epidemiol 2003;157(6):539–45. 
[PubMed: 12631544] 
43. Alvarez RA, Vasquez E, Mayorga CC, Feaster DJ, Mitrani VB. Increasing minority research 
participation through community organization outreach. West J Nurs Res 2006;28(5):541–60; 
discussion 561–3. [PubMed: 16829637] 
44. Williams MM, Meisel MM, Williams J, Morris JC. An interdisciplinary outreach model of African 
American recruitment for Alzheimer’s disease research. Gerontologist 2011;51 Suppl 1:S134–41. 
[PubMed: 21173436] 
45. Gauthier MA, Clarke WP. Gaining and sustaining minority participation in longitudinal research 
projects. Alzheimer Dis Assoc Disord 1999;13 Suppl 1:S29–33. [PubMed: 10369515] 
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 16
Figure 1: 
Full process of creation, testing, and approval of the English survey materials.
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 17
Figure 2: 
Full process of creation, testing, and approval of the Spanish survey materials.
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 18
Figure 3: Online testing through Pilot Expression of Interest website.
REDCap was used as the engine for questionnaire testing. For this project, we developed the 
ability to answer a feedback questionnaire for any question.
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 19
Table 1:
Original question sources for the All of Us surveys at launch
All of Us Survey Module: Number of Questions Included
Question Source Overall Health The Basics Lifestyle
ATS-DLD-78 a
6
Audit-C 3
BHLS 3
BRFSS 8 a
1
BS 4
2020 US Census 8
GENIUSS 4
MVP a
7
NESARC 1
NHANES 4
NHCHC 1
NHIS 2
NM-Assist 2
PATH 8
PLCO 2
PROMIS Measures 10
TUS-CPS a
5
UK Bio 7 1
VA Homeless Screening 1
New questions developed by All of Us 0 2 0
a 24 31 22
Total number of questions for All of Us Survey
Key: Audit-C: Alcohol Use Disorders Identification Test; BHLS - Brief Health Literacy Screen; BHLS: Brief Health Literacy Scale; BRFSS - 
Behavioral Risk Factor Surveillance System; BS: Blood bank screening; GENIUSS: Gender Identity in U.S. Surveillance; HINTS - Health 
Information National Trends Survey; MVP: Million Veteran Program; NESARC - National Epidemiologic Survey on Alcohol and Related 
Conditions; NHANES - National Health and Nutrition Examination Survey; NHCHC: National Health Care for the Homeless Council; NHIS - 
National Health Interview Survey; NM-Assist: National Institute on Drug Abuse-Modified Alcohol, Smoking, and Substance Involvement 
Screening Test; PATH - Population Assessment of Tobacco and Health Study; PLCO - Prostate, Lung, Colorectal, and Ovarian Cancer Screening 
Trial; PROMIS Patient-Reported Outcomes Measurement Information System; TUS-CPS - Tobacco Use Supplement - Current Population Survey; 
UK Bio - UK Biobank; VICS - Vanderbilt Inpatient Cohort Study
a
Questions were sourced from more than one original instrument. The final question was a composite of question(s) from multiple instruments 
modified to best fit All of Us.
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 20
Table 2:
Survey topics currently undergoing development and/or publicly released in All of Us
Released to Participants To be Released after 
at Enrollment Enrollment In Development
Future survey topics not yet 
in development
1. Basics 4. Family Health History 7. Diet 14. Oral Health
2. Lifestyle 5. Healthcare Access and 8. Disability 15. Pain
3. Overall Health Utilization 9. Environmental Exposure and Occupational 16. Reproductive Health
6. Personal Medical History Health 17. Sleep
10. Mental Health and Substance Use
11. Physical Activity
12. Social Determinants of Health
13. Medications
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 21
Table 3:
Demographic characteristics of testing cohort for launch surveys before and after content revisions. The 
percentage for each category is in parentheses.
Preliminary testing before content Final testing after content revisions 
revisions (Summer 2016) (Summer 2017)
Cognitive Interview Online Testing Cognitive Interview Online Testing
Total n=74 (%) n=337 (%) n=95 (%) n=236 (%)
Gender
Male 20 (27) 150 (45) 26 (27) 102 (43)
Female 54 (73) 185 (55) 63 (66) 130 (55)
Missing 0 (0) 2 (0) 6 (6) 4 (2)
Race/Ethnicity
White 38 (51) 312 (93) 32 (34) 128 (54)
African American 19 (26) 4 (1) 12 (13) 67 (28)
Asian or Pacific Islander 4 (5) 3 (8) 3 (3) 18 (8)
Hispanic/Latino 5 (7) 8 (2) 53 (56) 21 (9)
a
Other 5 (7) 10 (3) 4 (4) 34 (14)
Prefer not to answer 1 (1)  0 (0) 0 (0) 3 (1)
Age
18–29 7 (9) 46 (14) 24 (25) 45 (19)
30–49 32 (43) 90 (27) 24 (25) 82 (35)
50–64 22 (30) 113 (34) 21 (22) 64 (27)
65+ 13 (18) 86 (26) 21 (22) 41 (17)
Missing 0 (0) 2 (0) 5 (5) 4 (2)
Education
Bachelor’s degree or higher 54 (75) 210 (62) 33 (35) 112 (47)
Some college, but not a Bachelor’s degree 18 (25) 117 (35) 26 (27) 78 (33)
High school/GED or less 2 (0) 10 (3) 33 (35) 40 (17)
Missing 0 (0) 0 (0) 3 (3) 6 (3)
Residential Size
City(Large urban area-more than 50,000 
people) 58 (81) 187 (55) 49 (52) 154 (65)
Town(Small urban area-between 2,500–
50,000 people) 10 (14) 125 (37) 36 (38) 63 (27)
Rural Community(less than 2,500 people) 4 (6) 42 (12) 5 (5) 16 (7)
Missing 2 (0) 17 (5) 5 (5) 3 (1)
Other Demographic Characteristics
Sexual and/or Gender Minorities 5 (7) 23 (7) 14 (15) 40 (17)
Individuals with Low Health Literacy 38 (51) 148 (44) 27 (28) 12 (5)
Individuals with Physical Disabilities 13 (18) 32 (9) 8 (8) 16 (7)
Individuals with 3 or More Chronic 
Conditions 13 (18) 99 (29) 11 (12) 45 (19)
Individuals with Low Tech Literacy 6 (8) 33 (10) 19 (20) 7 (3)
Individuals with no internet access 1 (1) 2 (1) 0 (0) 2 (1)
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 22
Preliminary testing before content Final testing after content revisions 
revisions (Summer 2016) (Summer 2017)
Individuals with Children under 18 24 (33) 61 (18) 28 (29) 51 (22)
Individuals whose preferred language is 
Spanish 0 (0) 0 (0) 47 (49) 0 (0)
a
Other race/ethnicity counts include those individuals who self-identify as more than one race/ethnicity.
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 23
Table 4:
Completion time data for surveys included at All of Us Research Program launch: prior to content revisions 
and afterwards. Participants may not be required to answer certain questions if they are not applicable (e.g., a 
non-smoker will skip the questions asking about smoking history).
Number of Preliminary data before content revisions Final data after content revisions (Summer 
questions (Summer 2016) 2017)
(min – max)
The All of Us Median (min:sec) Range (min:sec) Median (min:sec) Range (min:sec)
Research Program 
Survey
The Basics 26 – 29 10:29 2:44 – 27:51 6:30 2:00–17:51
Lifestyle 22 – 57 2:47 0:33 – 8:00 1:23 0:17–7:05
Overall Health 14 – 24 2:48 1:20 – 7:36 2:00 0:14–6.06
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript
Cronin et al. Page 24
Table 5:
Question modification summaries of Round 1 and Round 2
Question modification summary from round 1 (Summer 2016)
Nature of change as compared with original survey instrument
All of Us Survey Domain Total questions (n) No change or N/A n (%) Minor change n (%) Major change n (%)
The Basics 36 14 (39) 20 (56) 2 (5)
Overall Health 14 14 (100) 0 0
Lifestyle 44 21 (48) 22 (50) 1 (2)
Question modification summary from round 2 (Summer 2017)
Nature of change as compared with original survey instrument
All of Us Survey Domain Total questions (n) No change or N/A Minor change n (%) Major change n (%)
The Basics 31 0 5 (16) 0
Overall Health 24 0 2 (8) 0
Lifestyle 22 0 1 (2) 0
Epidemiology. Author manuscript; available in PMC 2020 July 01.
Author Manuscript Author Manuscript Author Manuscript Author Manuscript