Online survey research generates critical statistics that inform employee benefits policy and market decisions. In fact, the online survey software market is rapidly growing. As such, benefits and financial services providers are challenged to stay current in their review of survey research processes. Doing so ensures an accurate representation of taste and preferences across diverse markets. This featured report from the EBRI Diversity, Equity, and Inclusion (DEI) Council explores online survey considerations for inclusion and considerations, which are both general for all survey methodologies and specific to the field of employee benefits.
Online Survey Considerations I r nt eser ou prrcet e gr ation oups , c ont can p rolr liovi ng d fe orval indus uabtlr e i y, npu emtpl ioy nto er r es siz ee, arch de incom sie, gn and o as wtehe ll. r f In 20 actor 22 s w , EB ill R pr I o and Gree vide valuab nwa le ld R cont es ext ear f ch or col the liabo nter rat pred etat w ion ith of the Hu resulm ts. a n R ights Campaign Foundation and DEI Council members representing the LGBTQ+ Additional Resources for Inclusion Diversity, Equity, and Inclusion Council community to enhance the Retirement Confidence Survey and Workplace Wellbeing Survey by identifying shared experiences and unmet needs of the community. Survey questions were designed to provide actionable steps to Sampling Considerations • Sha, Mandy, and Emily Geisen. “Every voice matters: How to get inclusive research design right.” April 11, 2024 benefits stakeholders serving the LGBTQ+ community. The data were later leveraged as decision support for Qualtrics. Available at https://www.qualtrics.com/ebooks-guides/every-voice-matters/. employers to enhance benefit design and for providers to develop new educational materials and reposition A well-specified research question will help define the sample population. This section briefly discusses three • “28 Questions to Help Buyers of Online Samples.” Qualtrics, April 2019. existing products. online sampling considerations: sample source, the digital experience, and achieving representation. “Wi • t“ h t The A he U.S. p meriopul can Tre ation nds Pa projnel ected t .” Po bec ew Rom esee i arch Cen ncreasi tng er.l y d Avai ive labl rse, e a empl t httoye ps:/r/s a ww re w f.p aci ewres ng new c earch.or hallg/ eng our es - in discerning which workplace benefits are best suited to their employees. To better understand the methods/u-s-surveys/the-american-trends-panel/. Language — Vernacular and English-language proficiency are important considerations when fielding benefits Sample Source — In an instance where the research is focused on benefits decision-making behaviors among physical, mental, and financial wellbeing needs of a diverse work force, it will be important to conduct • “WCAG 2 Overview.” Web Accessibility Initiative, March 2024. Available at surveys across diverse groups. Many financial and insurance terms are not universally understood or accepted, full-time employees that identify with Group A, there are several sample sources for the benefits community to benefits research through the lens of a range of demographics. https://www.w3.org/WAI/standards-guidelines/wcag/. and some jargon may even trigger an emotional response among individuals facing hardship. For example, draw upon. Employers, benefits providers, and online panels can all provide samples for a decision-making study, • U.S. General Services Administration. “Integrating Accessibility into Agency Diversity, Equity, Inclusion asking, “What is your net worth?” at the beginning of a survey that oversamples a lower-income population can though the level of generalizability is related to the sample size and makeup. For example, employers can This document of best practices was created by the EBRI DEI Council as a resource to assist benefits and Accessibility (DEIA) Implementation Plans.” Section508.gov, August 2023. Available at both confuse and demoralize some respondents, compromising completion and suppressing a sense of financial leverage employee engagement surveys to study decision making. While behavioral data of a single employer’s providers and employers with the task of developing more inclusive surveys. Yet, while we acknowledge the https://www.section508.gov/manage/deia-guidance/. wellbeing in later survey questions. Limiting financial jargon and providing simple definitions can improve the population may inform proprietary strategic benefits decision making, results may not be generalizable to the importance of gaining a clear understanding of the needs and challenges of workers, we also know that • Matsumoto, Abigail. “How to create more inclusive surveys.” SurveyMonkey. Available at respondent’s understanding of what’s being asked, ultimately improving the study’s reliability. More obviously, broader population of Group A. Another sample source is benefits providers, who may be able to provide a more some of the recommendations suggested here may be cost prohibitive. If these inclusive survey practices are https://www.surveymonkey.com/curiosity/how-to-create-more-inclusive-surveys/. English language proficiency may impact response rates and results. An understanding of the proficiency of the representative view of Group A behaviors given their client breadth. At the same time, benefits providers may not out of reach, we suggest looking for ways to supplement the data with research conducted by independent, • “Conscious Style Guide.” June 17, 2023. Available at https://consciousstyleguide.com/general/. 5 respondents and offering the survey in a different language may improve data quality. have permission for this type of employee communication across all clients. Both employer and benefits provider objective, fact-based organizations that have done the work of parsing out findings by segments such as • “Guidelines for Inclusive Language.” Linguistic Society of America, November 2016. Available at sample sourcing also assume that respondents in Group A are readily available or are comfortable sharing that race/ethnicity, gender, age, and/or sexual orientation. By doing so, providers may be able to identify https://www.lsadc.org/rc_files/10/Inclusive_Lg_Guidelines.pdf. Placement — As alluded to above, question location can be a strength or weakness of the survey instrument. identity. If employees are reluctant to identify with Group A to their employer or benefits provider, an online potential gaps or ‘blind spots’ in their data and address them accordingly.” 2 Placing what may be sensitive questions toward the end could help improve the data quality and mindset of the provider may be an alternative sample source due to a heightened sense of anonymity among panel participants. The Employee Benefit Research Institute is a private, nonpartisan, and nonprofit research institute based in survey respondents. Questions that might trigger financial trauma or feelings of financial despondency should be – Rodney Bolden, Morgan Stanley, EBRI DEI Council Chair Washington, D.C., that focuses on health, savings, retirement, and economic security issues. EBRI does not lobby reviewed for proper placement. Along these lines, while researchers often desire all questions to be answered and Digital Experience — Online survey methods generally assume that respondents have equal access to digital and does not take policy positions. The work of EBRI is made possible by funding from its members and force mandatory responses, voluntary responses are an option when the question is irrelevant to the target technologies. However, in policy or market-oriented studies asking questions like, “How can access to benefits Online survey research generates critical statistics that inform employee benefits policy and market decisions. sponsors, which include a broad range of public and private organizations. For more information, visit population. For example, a question about the rationale for not delaying Social Security may not resonate among 1 among Group A be improved?” when Group A shares characteristics with traditionally underserved or Yet, the online survey software market is rapidly growing, portending that these information collectors will www.ebri.org. certain groups, suggesting that the question should be eligible to be skipped by the respondent or include a “does marginalized communities that have unequal wealth, they may also have unequal digital resources. The digital experience evolution in business practices over time. As such, benefits and financial services providers are not apply” answer choice. Researchers can also allow for open-ended (fill-in-the-blank) responses alongside divide refers to unequal access to modern information and communications technology based on demographics challenged to stay current in their review of survey research processes. Doing so ensures an accurate 3 multiple-choice questions. and region. For instance, internet access can be less reliable among lower-income communities and in rural or representation of taste and preferences across diverse markets, such as persona refinement that improves ### remote geographies. The type of device also plays a role, as online surveys have different experiences on cell marketing effectiveness. Accessibility — Researchers should be mindful of survey design when targeting respondents with reading or phones, tablets, and personal computers. An online survey respondent of Group A could be using a computer at a 1 Market Growth Reports. Global Online Survey Software Industry Research Report 2023, Competitive Landscape, Market vision difficulties, as not all online survey questions are accessible. Online survey platforms such as public library with time limits or a personal cell phone with data limits. The digital divide may result in EBRI has a long-standing history of partnering with other top research organizations, including Greenwald Size, Regional Status and Prospect. SurveyMonkey and Qualtrics offer accessibility checklists and rating tools relative to Section 508 Compliance unfinished surveys and partial respondents, meaning analysis of only fully completed surveys could introduce a 2 R On esea linre ch a surv nd Q eys ar ua e lt orfiten cs, ass to o ficiate eld and d with di s pse anm eli-n bat ased e ne sam w s pur les, vey typ rically esear ch main focu tain si ed ng by on prtorfad ess itio ion nal alp ly und anel ag er gse reg rved o ators. r A and Web Content Accessibility Guidelines (WCAG) 2.0. Some matrix, rank order, and constant sum question bias against respondents impacted by the digital divide. survey panel is a group of consumers who have opted into market research activities and may be compensated for their time underrepresented communities such as Black and Hispanic Americans, LGBTQ+ community members, designs may not be WCAG2 compliant. In other words, respondents with cognitive or learning disabilities, those through cash or other rewards. caregivers, women, disability-affected Americans, and current and former military personnel. In a recent exercise, with low vision, or those with disabilities who are using mobile devices may have difficulty completing the A chieving Representation — The digital divide is one example of how the online survey experience can impact EBRI’s Diversity, Equity, and Inclusion (DEI) Council discussed online survey considerations for inclusion. 3 survey. Several survey programs include an automated accessibility review to ensure compliance. The Federal Communications Commission’s (FCC’s) Affordable Connectivity Program (ACP), launched in December results. As with any survey, some important considerations for online sampling are the natural incidence of the These considerations, which are both general for all survey methodologies and specific to the field of employee 2021, has helped to address the digital divide through discounts to eligible households, seeking to ensure that households can group being studied, and whether the representation of the group will be achieved through an oversample set by benefits, are summarized below. 4 afford the broadband they need for work, school, health care, and more. Without additional Congressional funding, the ACP quot Anal as, wei ysis an ght di ng, o Inter r a pr co eta mtion binat ion of the two. While decreasing true randomness, the sample size of Group A is set to wind down in April 2024. Researchers may find evidence of the digital divide when reviewing unweighted survey can also be increased by distributing the survey through affinity groups or snowball sampling. When quotas An Inclusive Process demographics. While the pandemic relief programs eased the digital divide for several years and lessened its impact on data become prohibitive due to cost or time constraints, it may be necessary to lower the level of confidence and/or An inclusive research design seeks input throughout the research process. As numerous prior EBRI research has collection, termination of the program may preclude some targeted respondents from online survey participation. increase the margin of error. shown, comparing the financial characteristics and benefits preferences among groups can reveal discernible Transparency and engagement with representatives of the focused population throughout the research process can 4 differences. In consultation with trusted resources, collaborative interpretation of analyses can address how the A random sample of a probability-based panel should yield demographics comparable to the U.S. population, whereas a help to minimize the risk of bias in survey design, deployment, analysis, and interpretation. However, the level to new data advance dialogue; whether certain differences remain statistically significant when controlling for non Qu-e rstion epresen ntativ aire e D pae nve el lo wop um ld e req ntu ire quotas built into the screening logic and post-fielding reweighting to be more which researchers achieve inclusive research design can be limited by resource constraints like time, budget, and income, education, and other variables; and areas of further study, such as products or policies that could help to representative of American households. It is likely that either method would require oversampling and reweighting to ensure staffing. representation among diverse groups of interest. reduce pain points and wealth gaps. One manifestation of collaborative and inclusive interpretation is the survey Including a group’s perspective in questionnaire development can improve the quality of insights and readout sessions hosted by EBRI prior to publication. Such preliminary reviews of findings are opportunities to actionability of findings through thoughtful question design, language, question placement, and accessibility An inclusive research design requires intentionality to incorporate diverse perspectives. This intentionality 5 See Thephasit, Samita, “The Importance of English Proficiency in Financial Wellbeing,” EBRI Fast Facts, no. 495 verify that interpretation accurately represents the group and to ensure that observations are communicated with considerations. naturally starts with developing a research question stemming from a market or policy problem of focus. Across (Employee Benefit Research Institute, March 2024). empathy and objectivity. all policy research, including employment benefits, qualitative and quantitative questions seek causal, relational, Actionable Question Design — Involving groups directly and indirectly in questionnaire development can and descriptive understandings of behavior. An inclusive approach to developing the research question “How do In conclusion, there are many opportunities for inclusion in a research design that leverages online surveys, improve data quality by asking relevant and actionable questions in common terms. For example, when asking, access and take-up of benefits relate to the differences in average financial situations between Group A and though the extent to which this is true is often dependent on timeline and budget. Intentional efforts to be “How do tastes and preferences compare between Group A and Group B, and how does that impact their benefit Group B?” would suggest that representatives of both Group A and Group B are consulted with, in addition to inclusive, even when operating within resource constraints, should result in a more accurate representation of decision-making?” researchers may leverage first-degree connections who identify with the group. Second-degree employers and provider stakeholders who enable access to the benefits. Word choice within the research question diverse populations. connections such as advocates and experts in the field across non-profits, academia, clients, and employee should ensure objectivity and provide clarity on scope and methodology. As described later under Analysis and © 2024, Employee Benefit Research Institute, 901 D St. SW, Suite 802, Washington, DC 20024, 202/659-0670 www.ebri.org 3 4 2