Superior tools and resources for opinion research

OpTech/Validate automated validation

Automated validation – an opinion research breakthrough

BIRMINGHAM, ALABAMA – Opinion researchers can breathe a sigh of relief. Until now, researchers could never be sure interviewers were conducting surveys properly. If one caller skips questions or types wrong answers, it can invalidate an entire survey; yet callers (often low wage, temporary employees) don’t always take their jobs seriously. But now Opinion-Tech, a software and services vendor to the research industry, has changed things – by introducing 100% automated validation for survey research. In a field with notorious quality control problems, despite random monitoring, statistical analysis, and training, it is now possible to KNOW that survey results are correct.

Opinion-Tech’s new service validates EVERY completed interview, immediately and automatically. This is a big change from the traditional alternative, where human auditors listened in on random calls. Opinion-Tech’s independent validation service works in real-time, collecting and analyzing extensive timings and statistics during each survey, and then placing an automated verification call to every respondent. This all occurs within seconds of the completed interview.

A familiar problem

Survey houses have always struggled to attract and retain good callers. Demand for opinion research varies over time, following political cycles and seasonal business demands. Research firms often face difficult training and personnel challenges, especially on a big job with new staff. Imagine an unscrupulous new caller, trying to finish calls quickly by skipping questions and typing one- word responses, and then daydreaming between calls. One such slacker can ruin the results for a whole room of industrious, trained professionals. Good shops do catch cheaters, eventually; but doing so might take a day or a week, depending on the size of the room, the ratio of callers to supervisors, etc. In the meantime, cheating creates an unacceptable margin of error, and a lack of client confidence.

The solution is to monitor and validate every survey, using automated tools, and to provide job supervisors with real-time results. When rule breakers are caught instantly, and fairly, there is no incentive to cheat.

Production experience

Early experience with Opinion-Tech’s validation service yielded dramatic results. Client survey confidence and reliability improved greatly – since all callers know that they are being monitored 100% of the time. Training issues and caller errors all but disappeared, and deliberate infractions could be confidently ruled out. A level playing field means that monitoring is fair, impartial, and automated, eliminating any temptation for caller cheating. OpTech call centers can implement 100% validation...and can prove it!

How validation works

Opinion-Tech validation uses two key mechanisms: timing analysis and respondent verification.

• Timing analysis is used to flag exceptions, detected by a) timing each of the responses entered by each caller, and then b) flagging abnormally short or long delays, based on the results of other callers on those same questions. (A too-short delay can indicate a partly or entirely skipped question; a too-long delay can indicate deviation from the question text. A key problem tell-tale is a mix of too short and too-long responses, which will happen if a caller enters random responses, with no relationship to the actual call.) Such timing aberrations are flagged for supervisor intervention within ten seconds of the occurrence.
• Respondent verification is built into each call. Surveys end with a statement like the following: "When this survey is completed, our quality assurance department will call you back within ten seconds. Please confirm that you completed the survey by pressing ‘1.’ Unless you validate the survey, it will not be included in the study." This gives both caller and respondent a vested interest in successful validation; otherwise, the time was wasted. Within seconds of survey completion, an automated call is placed to the respondent – verifying that the survey was done, and also giving the respondent an opportunity to report quality issues.

These two mechanisms – question timings and verification – are automatically applied to every survey. The factors are then used to assign a score to each completed questionnaire, immediately posted on the survey manager’s web site, along with other job control data. This information is all made available to management in real time – just as it happens.

Any low scores are ‘red-flagged’ for immediate attention. A supervisor can then take appropriate action – such as calling back the respondent, or questioning the caller. A training issue that might have caused problems throughout a shift can be dealt with instantly; a ‘bad apple’ can be identified and dismissed. And this all happens within seconds or minutes of the problem – not overnight.

Reduced monitoring overhead

Another huge advantage of Opinion-Tech’s new solution is a reduction in the need for traditional monitoring personnel. The new automated system transforms routine monitoring into a part-time function of a single person. The computer system picks up the slack, and is relentless. There is no longer any reason for a supervisor to " watch over a caller’s shoulder" to be sure that work is being done. Instead, effort that formerly went into routine monitoring can instead go toward training and true quality assurance – being sure that callers are using correct pronunciation, recording answers accurately, etc.

Part of Opinion-Tech Network

Automated validation is built into the Opinion-Tech Network, a technology infrastructure allowing member organizations to share resources and workload. Opinion- Tech Network utilizes true distributed interviewing to place a huge ‘virtual workforce’ under the control of a project manager. Callers at many geographic locations can be utilized simultaneously, sharing a single sample, a centralized quota system, and real-time management tools. Automated validation expands the tools available to the project manager, and subjects outside firms to objective quality measures.

Respondent- and job-specific data

Opinion-Tech’s validation system utilizes advanced dynamic text to speech technology, rather than a prerecorded voice. This means that validation calls can incorporate job- and respondent-specific details.

In the next few months, Opinion-Tech will release questionnaire/validation text-substitution capabilities in a new release of OpTech/Interview, the call center interviewing system. This enhancement will allow seamless integration between questionnaire content and validation text. A respondent might thus be asked "In your survey, you indicated that you are a female between the ages of 25-30 who voted in the last election, and you supplied the confirmation information that you were born in Springfield. Please press ‘1’ to confirm these responses." For statistical reasons, most surveys are of course designed with strict quota requirements. Calling houses, in the last hours or minutes of a job, have been known ‘accidentally’ to record a male as a female, or a Democrat as a Republican, in order to fulfill a quota requirement. ("Oh well, at least we got the 600 respondents.") Researchers have nightmares about this problem; but such quota errors can be eliminated with automated verification.

Low cost

Validation services are available to Opinion-Tech Network members in conjunction with distributed interviewing and related services.

Current plans are for validation to be performed for 5¢/completion plus 1¢/minute of interview, with a per- survey cap of 25¢. Thus, 100% validation of a 600 respondent survey would have a cost of just $30 150, depending on the length of the interviews. (Final pricing is still pending and is subject to change.)

Note that validation costs are distinct from Opinion- Tech’s basic surcharge of 1¢/minute for distributed interviewing (i.e. interviewing performed by callers at a ‘satellite’ location under the control of a different ‘host’ organization). Validation is a new service from Opinion-Tech, and is not included in any existing site licenses.

For more information

Today’s pilot validation system can be seen on Opinion- Tech’s demonstration system, accessible at http://demo.opinion-tech.net. For information about Opinion-Tech’s other products and services, visit our website at http://www.opinion-tech.com, or call us at 1-800-513-3511 or email us.