Applebot, a web crawler for Siri and Spotlight, unmasked in a support document

siri ios 7

An official support document updated last evening has confirmed that Apple is using a web crawler service called Applebot to index the web for content used by its Siri personal digital assistant on iOS devices and built-in Spotlight search on both iOS and OS X.

An Apple-operated engine, Applebot is described as “the web crawler for Apple” which gathers useful content for products including Siri and Spotlight Suggestions. The document doesn’t state how long AppleBot has been operational in production.

The crawler is hitting web content from the 17.0.0.0/8 block, which has been assigned to just a handful of companies that control an entire A block, among them Apple.

Similarly to Google’s Googlebot, Applebot respects customary robots.txt rules and robots meta tags and identifies itself as “Applebot” in user-agent strings.

“If robots instructions don’t mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions,” reads the document.

While we shouldn’t get ahead of ourselves and speculate that the very existence of Applebot points to an Apple-branded search engine in the works, it’s interesting that Apple is using a web crawler of its own, even more so knowing that Siri mostly pulls content from Bing and Wolfram Alpha for its web searches.

Pictured below: a recently-revamped Siri section on Apple’s website.

new siri site

Assuming a full-on Apple search engine exists in Apple’s labs, Google should be concerned as it gets a lion’s share of mobile ad revenue from searches originated from Macs, iPhones, iPads and iPods.

A post today on Google’s official Inside AdWords blog has confirmed an important milestone as “more Google searches take place on mobile devices than on computers in ten countries including the US and Japan.”

Yahoo CEO Marissa Mayer said recently that the Safari platform is “basically one of the premier search deals in the world if not the premier search deal in the world.”

Google Search is currently the default search engine in Safari on both OS X and iOS devices, paying an estimated $1 billion annually for the privilege. But that search deal is set to expire this year, prompting both Microsoft and Yahoo to fight tooth and nail to supplant Google as the default search engine on the Apple platform.

Is Apple working on something bigger, do you think?

Source: Apple