Loup Ventures has published the results of a new survey that looks at the intelligence of the smart assistants currently on the market. The study asked Apple’s Siri, Google Assistant, Amazon Alexa, and Microsoft’s Cortana the same 800 questions. The results were somewhat surprising. 

The survey found that Google Assistant was able to answer 86 percent of the questions correctly, compared to Siri’s 79 percent. Alexa (61 percent) and Cortana (52 percent) were both far behind. This survey represents the first time Loup Ventures tested the intelligence of Alexa.

The questions in this year’s survey were graded on two metrics:

  • Did it understand what was being asked?
  • Did it deliver a correct response?

The question set was broken into five categories, including:

  • Local – Where is the nearest coffee shop?
  • Commerce – Can you order me more paper towels?
  • Navigation – How do I get to uptown on the bus?
  • Information – Who do the Twins play tonight?
  • Command – Remind me to call Steve at 2pm today.

As Loup Ventures explains, nearly every wrong answer involved a proper noun, such as the name of a local town or restaurant. It notes, “Both the voice recognition and natural language processing of digital assistants across the board has improved to the point where, within reason, they will understand everything you say to them.”

Drilling down, Google Assistant scored the highest in every category except Command, which was led by Siri. Of this, Loup Ventures contends, “We found Siri to be slightly more helpful and versatile (responding to more flexible language) in controlling your phone, smart home, music, etc.”

In the past 15 months, Google Assistant has improved by 11 percentage points, while Siri’s number rose by 13 points. Regardless of the variance between each voice assistant, the survey’s creators are confident each will continue to improve quickly.

They conclude “We’re impressed with the speed at which the technology is advancing. Many of the issues we had just last year have been erased by improvements to natural language processing and inter-device connectivity.”

I’m fairly confident to predict that surveys like this will no longer be necessary within two to three years. By that point, each of the voice assistants will score perfect marks or no longer be around. With that being said, given the popularity and longevity of Amazon’s Echo devices, I’m surprised Alexa scored so low on this survey.

What do you think? Leave your comments below.

Chart: Loup Ventures