ARKit, Apple's new framework in iOS 11 for building augmented reality experiences for iPhone and iPad, has been used to recreate the music video for the 1980s hit song “Take on Me” by Norwegian band A-ha in real-time, highlighting ARKit's potential for film-making.
If the awesome ARKit-powered measuring tape hasn't knocked your socks off, how about PCalc, the best scientific calculator available on App Store, in augmented reality?
There's certainly no shortage of impressive demos of awesome augmented reality apps made possible by Apple's new ARKit framework in iOS 11, but this new proof-of-concept app from independent game developer Duncan Walker certainly takes things up a notch.
Windows giant Microsoft will outfit the next version of its mixed reality HoloLens goggles with an extra AI processor that will let the device recognize speech and images in real time, without having to send the data back to the cloud for analysis.
Apple's new ARKit framework for building augmented reality experiences for compatible iPhone and iPad devices is off to a great start and already Facebook is taking notice, according to a new report Thursday from Bloomberg's Mark Gurman.
The author claims that Facebook has been hard at work developing an inexpensive headset, code-named “Pacific”, that is expected to bring virtual reality experiences to the masses without requiring a beefy computer or a compatible smartphone.
“The idea is that someone will be able to pull the headset out of their bag and watch movies on a flight just the way you can now with a phone or tablet,” reads the article.
It should be priced aggressively at $200 and release at some point next year, representing “an entirely new category”. According to people familiar with the plans, the device will provide a similar interface to Samsung’s VR Gear that users could control by a wireless remote.
The headset should be powered by Qualcomm's Snapdragon mobile chip that would make it superior to Samsung's Gear VR headset in terms of gaming in virtual reality. Unlike the current Oculus Rift hardware, the upcoming gizmo won't include positional tracking technology.
An excerpt from the article:
This means that the device won’t be able to tell where its user is spatially, which is useful for tasks like virtual rock climbing. A future version of the product will have that technology, according to a person familiar with the plans.
According to sources, the headset will let users play immersive games, watch video, use social networking apps and so forth. It resembles a more compact version of the current Oculus Rift and will be lighter than Samsung’s Gear VR headset.
Handset maker Xiaomi and its manufacturers are said to build 2018's Oculus-branded device.
And later this year, Facebook allegedly plans to announce a more affordable wireless headset that it is betting will popularize virtual reality “the way Apple did the smartphone”.
Oculus spokesman Alan Cooper said via email:
We don’t have a product to unveil at this time, however we can confirm we’re making several significant technology investments in the standalone VR category.
Facebook's said it’s also working on yet another device, code-named “Santa Cruz” and best described as a wireless Oculus Rift “with the full power of the original device sans PC.“
Facebook acquired Kickstarter-funded Oculus startup in 2014 for about $2 billion.
IDC estimated that Samsung leads the pack in terms of VR device shipments with 22 percent of the global market for VR devices, followed by Sony, HTC and Facebook's Oculus Rift with about five percent of the market, or less than 100,000 units sold.
At last month's Mobile World Congress Shanghai, Samsung showed a secret standalone virtual headset prototype to partners. As spotted on VR Focus, the product uses technologies allowing it to track eye and hand movement as well as determine various facial expressions.
Dubbed Exynos VR III, the head-mounted accessory is apparently a successor to another Samsung headset prototype, called Exynos VR II, that was never officially released. Samsung already offers a virtual reality headset in the form of the Gear VR device which requires the user to dock and undock their smartphone every time they use it.
Thanks to Visual Camp, a VR company that developed eye-tracking technology for the secret VR headset, we know it's powered by a Samsung-designed 10nm hexa-core chip.
The chip includes a pair of Samsung M2 CPU cores clocked at 2.5 GHz, four ARM Cortex-A53 CPU cores clocked at 1.7 GHz and ARM Mali G71 MP20 graphics capable of driving two built-in 2,560-by-1,440 pixel displays at 90Hz or a single 4K external screen at 75Hz.
As mentioned, unlike Samsung's current Gear VR headset that requires a smartphone to process data and render visuals, this all-in-one head-mounted display prototype packs in all the technology needed to render virtual worlds and apps standalone.
Visual Camp's press release announcing the Samsung deal says its eye-tracking tech lets VR headsets conserve power by rendering parts of a scene the user is currently looking at very high resolution while showing anything in peripheral vision in reduced resolution.
This technique is known as “foveated rendering”.
“Several other technologies will be applied to the Exynos 3, in addition to the company's eye-tracking technology, including hand tracking, voice recognition, and facial expression recognition,” reads the press release.
A measurement of the CPU power consumption of Samsung Electronics' Exynos 8890 chip resulted in the relatively low average figure of less than three percent, said Visual Camp.
Companies like Apple, Google and Facebook are researching eye-tracking technology, too.
Apple is rumored to be working on a digital glasses or a virtual headset product that may use optics by German specialists Carl Zeiss, thought to be released in 2018 or 2019. The Cupertino giant recently acquired SensoMotoric Instruments for an undisclosed sum.
SensoMotoric Instruments is a German company that specializes in eye tracking. Their technology also uses foveated rendering, understands facial expressions and recognizes participant gestures and external events.
Apple may have quietly acquired SensoMotoric Instruments, a German company which can track people's eye movements. MacRumors was first to report yesterday that SensoMotoric has been acquired for an undisclosed sum by Apple's shell company, called Vineyard Capital.
The company holds multiple patents relating to eye tracking and virtual reality.
“Apple buys smaller technology companies from time to time, and we generally do not discuss our purpose or plans,” an Apple spokesperson said in a boilerplate statement issued to Axios.
Gene Levoff, Apple's Vice President of Corporate Law representing Delaware's Vineyard Capital Corporation, granted power of attorney to a German law firm to represent the shell company, which in turn acquired SensoMotoric Instruments on June 16. Levoff even notarized the document in Cupertino, California, where Apple is headquartered.
Tellingly, SensoMotoric recently removed over a dozen pages from its official website. It no longer has a jobs portal, news blog, schedule of events and workshops, contact information, list of distributors and resellers or mailing list signup form.
Their managing director Eberhard Schmidt was replaced by Dr. Ali Sahin, one of the German attorneys representing Vineyard Capital Corporation. Christian Villwock, who was the company's Director of OEM Solutions Business, was removed from the website, too.
Here's an example of SensoMotoric's eye-tracking technology in Samsung Gear.
https://www.youtube.com/watch?v=mDvgP2tnMHQ
Another video embedded further below shows off SensoMotoric's eye-tracking glasses with Natural Gaze Head Gear used by young athletes playing tennis to accurately capture their natural gaze, which helps them evaluate and improve their visual performance.
https://www.youtube.com/watch?v=VEZP_corY3Q
Proprietary eyeglass hardware that the video's athletes are wearing can capture a person's natural gaze behavior at a rate of 120 scans per second.
On the hardware side, Apple could use SensoMotoric technology in its rumored augmented reality glasses product. Eye-tracking technology can significantly reduce motion sickness for users of virtual reality headsets such as Facebook's Oculus Rift.
One specific aspect of SensoMotoric's technology, called foveated rendering, allows a virtual reality headset to save power by only showing you in high resolution what you're actually looking at, with anything in your peripheral vision being rendered in less high-resolution.
This reduces the amount of processing power needed to render a virtual world.
https://www.youtube.com/watch?v=-w7r0IGRlTY
On the software side, Apple could improve iPhone 8's rumored 3D facial recognition security feature through eye tracking and even allow apps and games to track the user's eye movement so that they could, for instance, aim in a game with their gaze.
Founded in 1991, SensoMotoric Instruments is headquartered in Teltow, Germany, with a satellite office in Boston, Massachusetts. The company employs about 60 engineers.
“I do think that a significant portion of the population of developed countries, and eventually all countries, will have AR experiences every day, almost like eating three meals a day,” Apple CEO Tim Cook said last year.
Apple Maps on iOS 11 beta 2 features a great new virtual reality (VR) mode that takes advantage of Apple's new ARKit framework to let you move around in 3D by walking.
This unapologetically cool feature seems to be tied to Flyover, which replaces satellite imagery with three-dimensional buildings, landmarks and other points of interest.
The new VR mode on iOS 11 Maps was highlighted yesterday by Twitter user @StijnDV, but it appears to have been originally discovered by Tweetbot developer Paul Haddad on Wednesday.
To try it out yourself, open Maps on iOS 11 beta 2, switch to 3D mode by tapping “3D”, then use the search field at the bottom to find a place that has Flyover.
On the place card, tap the Flyover button and move the device around to rotate the view. Better still, why don't you actually move forward, backward or side to side to explore the map in VR?
Mind blown.
So, how do we know this nifty feature actually uses ARKit? Because it displays a message when you cover the camera, just like any ARKit-powered app does, saying you should aim the device at a different surface because “more contrast is required”.
As a quick backgrounder, ARKit analyzes live camera feed in real-time, using computer vision to find horizontal planes in your real world, such as tables and floors. I was able to successfully test the feature on my iPhone 6s running a second beta of iOS 11. Because I don't currently own an iPad, I couldn't test VR mode in Maps on the Apple tablet.
WOW There is an VR mode in Apple maps on iOS 11! It seems to use ARKit for positioning! pic.twitter.com/IdXiGoed26
— Stijn (@StijnDV) June 24, 2017
At any rate, this appears to be the default mode for Flyover now, not a special setting. But don't you worry, there's the option to switch back to the old Flyover mode where you rotate and zoom your Flyover view using touch interactions.
This is honestly one of the coolest features in iOS 11! pic.twitter.com/Zjr6RRkKHk
— Stijn (@StijnDV) June 24, 2017
This is a wicked cool feature and I cannot help but wonder how it might look like when experienced through Apple's rumored digital glasses that, as per Robert Scoble, should use optics by German lens specialist and optical instruments maker Carl Zeiss.
You can actually move around by walking! This is crazy cool! pic.twitter.com/ttR6RaAo7D
— Stijn (@StijnDV) June 24, 2017
Some people couldn't get Maps' new VR mode to work, but I suspect it may have something to do with their hardware. Maps' VR mode uses ARKit, which tracks your actual position in the real world with the camera but requires newer hardware.
Holy Flyover Magic Window batman. pic.twitter.com/Fb8nPeLT5J
— Paul Haddad (@tapbot_paul) June 22, 2017
According to Apple, ARKit runs on the Apple A9 and A10 processors. “These processors deliver breakthrough performance that enables fast scene understanding and lets you build detailed and compelling virtual content on top of real-world scenes,” says the company.
In other words, anything older than iPhone 6s, iPhone 6s Plus, iPhone 7, iPhone 7 Plus, the 9.7-inch iPad (early-2017 model) or iPad Pro won't be able to run iOS 11 Maps' VR mode.
So, is this cool or what?
We'd obviously love to hear your thoughts and predictions regarding iOS 11 Maps' new VR mode and what it might signify in terms of possible new VR hardware from Apple.
Do us a favor and chime in with your thoughts in the comments section.
Apple's next-generation handset will feature a stainless steel frame, reports Japanese blog Mac Otakara. Citing sources familiar with the matter, the site claims the new iPhone will use an improved version of the Apple Watch's stainless steel casing, which will sit sandwiched between two pieces of glass.
Bloomberg's Mark Gurman in a new report Monday offered a deeper insight into Apple's next big thing—the company's secretive efforts related to augmented reality (AR) technologies which reportedly include an iPhone-connected digital spectacles that the news organization previously said would launch in 2018.
Citing people with knowledge of Apple's plans, the report states that the company's built a team combining the strengths of its hardware and software veterans with the expertise of talented outsiders. The group is allegedly being run by former Dolby Laboratories executive Mike Rockwell and includes engineers who worked on Facebook's Oculus and Microsoft's HoloLens virtual reality headsets “as well as digital-effects wizards from Hollywood.”
When former Android executive Hugo Barra announced three days ago he was leaving Chinese handset maker Xiaomi, he provided standard boilerplate explanation: he wanted to spend more time with his family because living and working abroad has taken a toll on his health and life.
Then again, that's exactly the kind of stuff that you'd expect high-ranking executives to say when jumping ship or seeking greener pastures with another team.
As it turns out, Barra is leaving Xiaomi not because he's burned out but to join Facebook where he will be charged with managing the Oculus team and the company's other virtual reality projects, CEO Mark Zuckerberg announced yesterday.
For years, techy gifts have been on an unstoppable rise and these holidays more than ever, new VR headsets of all kinds were added to the mix. If you are one of the lucky receivers of a Cardboard or any other generic pair of VR goggles compatible with your iPhone, your first address to go for VR-ready apps should of course be the App Store.
To help you get on top the bulk of apps available and break the tedious ‘download, try and delete’ chain, we have been weeding through the vast app-scape and given VR-ready apps a hard look. If you want to put your new toys to good use, here is a list of some of the best VR games and apps we believe you will want to embark on first.