A new study by the Butler Business Accelerator found that ChaCha answered questions better than 10 other mobile search engines and question-and-answer platforms, including Ask.com, Answers.com, Google, Quora, Siri, and Yahoo Answers.
The Butler University Q&A Intelligence Index measured the likelihood that a user could expect to receive a correct answer in a timely manner to any random query using natural language. ChaCha’s mobile application for the Apple iPhone outperformed all the services tested, garnering the highest score on the Butler University Q&A Intelligence Index by responding to 99 percent of the study’s 180 randomly selected questions with the highest degree of accuracy.
The rankings are:
*Mean accuracy of responses originally graded on a five-point scale.
The questions were asked via mobile services and randomized to cover both popular and less common questions. The coverage score measured whether an answer was returned within a three-minute window and assessed the first non-sponsored search result in Q&A platforms. The accuracy score measured how correct answers were in relation to an objective third party’s answer key that accounted for logic and bias.
Questions asked were in the areas of advice (“What if a girl doesn’t want to talk to you?”), objective (“What are the 10 most common names?”), and subjective (“Who would win in a fight, The Hulk or Superman?”).
“ChaCha delivered the highest quality responses consistently across the largest group of categories and question types,” said Trent Ritzenthaler, operating director of the Butler Business Accelerator.
Other findings of the study:
-Ask.com performed better than all others in the single category of questions labeled objective/temporal, such as “When does summer end?”
-Quora was proficient at answering difficult questions that require expert and extensive explanations, but it was generally unable to deliver answers within three minutes for most information searches on mobile devices. Quora answered only 24 percent of the questions at all, and often the match found did not include a viable answer.
-Siri accurately answered only 37.5 percent of the questions posed, but Siri’s biggest strengths are considered to be in local discovery and operating system commands, which were not highly represented in Butler’s study of more mainstream questions.
-Google’s response rate was 100 percent, but the first non-sponsored result on the search results page—which often times was not fully visible as an organic search result on the presented page on a mobile device—only presented an accurate answer about 50 percent of the time.
“Through our extensive study, ChaCha showed the highest level of performance,” Ritzenthaler said. “As the market matures, we anticipate that ChaCha and other mobile search companies will lift the standard of natural language processing and semantic understanding to enhance the user experience with mobile Q&A.”
For further information on the research methodology and analysis of test data, visit http://www.butler.edu/q-a-study/.
Media contact:
Marc Allan
mallan@butler.edu
(317) 940-9822