By Dr. Edward Rincón
The Culture of Research
If you are a tax-paying Texas resident, should your opinion matter in decisions related to publicly-funded programs or services in Texas?
Of course, you may say, the opinions of all Texas residents are important. But one Texas state agency thinks that it is acceptable to exclude Spanish-speaking Hispanics from state-funded public opinion polls that are used to decide how tax dollars are spent.
I would like to share the details of an actual case study that vividly illustrates how one Texas agency is being allowed to silence the voice of Texas Hispanics in its public opinion polls.
The State of Texas has plans to spend billions of dollars to improve their transportation system, including the possibility of high speed rail. To ensure that the improved system meets the needs and expectations of Texas residents, the Texas Transportation Institute at Texas A & M has the responsibility for conducting important surveys of Texans that reside in specific geographic areas, or corridors, that are likely to be impacted by these improvements.
A recently released report by the Texas Transportation Institute for the first of these two surveys, conducted during the Fall of 2012, provides concrete evidence that the voice of Texas Hispanics was muted by the survey planners.
Indeed, Hispanics represented only 20 percent of the survey respondents, despite their current representation of 38 percent in Texas (American Community Survey, 2011). Even worse, only 19 percent of the few Hispanics included in the study were interviewed in Spanish – which compares poorly with other state surveys that have shown that 50 to 67 percent of Hispanics prefer a Spanish-language interview.
How could this occur, especially when the study design was reviewed by a “panel of experts” at the Institute? A careful review of the study methodology reveals several missteps in the planning and execution of this survey:
- The 16,000 households selected as respondents received only an English-language version of the survey. The cover letter that was included with the English-language survey was provided only in English, and did not offer respondents any support to complete the survey in another language.
- A question that asked respondents to identify their race-ethnic background provided only one ethnic identifier for Spanish-speaking respondents — “Hispanic” – which could partly explain the under-count of these respondents because other labels are often preferred over the “Hispanic” option.
- A call center was supposedly set up to receive incoming calls from survey respondents that had questions or needed Spanish-language support. But this call center probably received few calls from Spanish-speaking respondents since the cover letter did not provide the needed contact information. Moreover, the report did not include a copy of the Spanish-language telephone survey that was supposedly used by the survey vendor’s call center to capture incoming calls by the survey respondents.
- The study design required that automated advanced calls (or “robo calls”) be made to the selected households prior to the survey mailing. Automated calls are a recognized nuisance from telemarketers and political campaigns that often discourage response rates to legitimate public opinion polls.
- The report indicated that the survey “participation rate” was 34.6 percent – a rate that appears subjectively created and not recognizable by the American Association of Public Opinion Research (2011). Instead, the overall survey response rate was more likely to be a much lower 9.7 percent (1,559 completions /16,000 invited participants) – not surprising given the recognized shortcomings in the methodology.
- While the vendor acknowledged that Hispanic respondents were significantly under-represented and non-Hispanic whites were over-represented, no explanation was provided about the potential causes or consequences of this imbalance.
- The fact that the survey planners ignored a previous warning about the potential flaws in the survey methodology suggests that the poor survey outcomes did not result from just simple carelessness.
To make matters worse, the same survey vendor was awarded a second contract to conduct another public opinion poll of Texas residents using the same flawed methodology. Why are state officials allowing such flawed practices to take place, especially at a time when the state’s population is being heavily impacted by the growing Hispanic population?
As one of the vendors that competed for both survey contracts, Rincón & Associates LLC monitored both competitions with some concern. In both competitions, state procurement staff decided to award the contract to the lowest bidder, which may not have been the brightest decision given the poor study outcomes.
As both studies required a mixed-mode survey methodology that few U.S. companies were capable of executing, more weight should have been given to proven experience using this specialized methodology with diverse communities in Texas. Procurement staff did not have to settle for the lowest bidder as other Texas vendors were ready, willing and able to conduct both studies.
Fortunately, an investigation was initiated on March 3, 2013 to find out why a state agency like the Texas Transportation Institute is allowed to deliberately design a study that minimizes the participation of Hispanic residents, especially Spanish speakers. The outcome of this investigation is important because it could either (a) allow other state agencies to exclude Spanish-speaking Hispanics from state-funded studies, or (b) raise the standards of research for all state agencies to ensure that all state-funded studies provide adequate Spanish-language support.
The practical significance of this issue cannot be over-stated. Spanish-speaking residents are often the most likely to be overlooked in the delivery of public services, the most likely to receive the lowest quality services, and show distinctive attitudes or values that differ significantly from English-speakers. Thus, excluding Spanish-speakers from opinion polls can lead to more positive satisfaction ratings than is actually the case, and result in erroneous public policy decisions.
It is time to require a higher standard for public opinion polling in communities that are linguistically and culturally diverse. Although professional research organizations have always defined quality and ethical standards for the research industry, it is apparent from the case just discussed that public agencies may not feel the need to follow these guidelines. Following are a few ideas suggested from our experiences with public agencies like the Texas Transportation Institute:
- Research firms that compete for opinion polls in the public sector should be required to produce evidence that they have the staff, facilities and past experience to conduct polls in linguistically and culturally diverse communities. If a research firm does not produce a representative sample of such communities in a contracted study, they should not be rewarded with another contract that utilizes the same flawed methodology.
- The committee members convened by public agencies to evaluate research proposals may not have the expertise to judge these proposals in terms of their adequacy for diverse communities. The inclusion of experts with experience in conducting polls in diverse communities may have prevented the missteps in the Texas A & M studies.
- In the haste to award a contract to the lowest bidder, proposal evaluators do not regularly check the references provided by the different bidders, but oddly enough still find a way to rate the relevant experiences of the bidders without this information. Prior to contract award, an audit should be conducted to ensure that such references were verified for all of the vendors that submitted a bid in such competitions.
It is unclear that the State of Texas got the “best value” by selecting the lowest bidder from outside of Texas. Indeed, what is the economic benefit to Texans when a contract is awarded to a non-Texas vendor whose payrolls, taxes and local spending for goods and services will only benefit another state?
Legislators and advocacy organizations, especially those that represent the needs of Texas Latinos, should show their concern about public opinion polling practices that minimize or eliminate the voice of the constituents that they represent.
Can we afford to remain silent on this issue?
The more conservative members of the Texas community may believe that all public work should be conducted only in English, and that no special accommodations should be made to non-English speakers. Unless we are willing to also exempt non-English speakers from the payment of taxes as well, then I believe that they should be given the option of voicing their opinions on topics that impact their quality of life.
Although many Hispanics and Asian residents have proficiency in English and their native language, about 50 to 70 percent of these residents still prefer to express their opinions in their native language. By providing the appropriate linguistic options, public opinion researchers are more likely to establish rapport, increase response rates, and obtain more valid responses to their questions from ethnic respondents – all desirable outcomes for high quality research.
Texas public agencies, especially the Texas Transportation Institute, must be required to raise their standards when conducting opinion polls of Texas residents, and legislators must take a more assertive role to ensure this outcome.
We cannot afford to bury our collective heads in the sand on this issue.
Dr. Edward T. Rincón, Ph.D., is president of Rincón & Associates LLC and is a research psychologist. He has taught courses on Hispanic marketing and survey research methods, and often writes about the sources of bias that undermine the quality of research with culturally diverse populations.
