Apple is rumored to be working on an advanced augmented reality headset. Its latest patent filing offers a glimpse into how accurate and reliable gaze tracking could permit users to interact with a computer-generated environment more easily than with hands.

Published Tuesday, the United States Patent and Trademark Office granted Apple a patent for an “Electronic device with gaze tracking system” aims to solve the drift issue using probability and scene detection.

What’s the drift issue, you ask.

AppleInsider has more on that:

Eye tracking could provide an alternate way for users to interact with a virtual reality environment, by detecting what the user is specifically focusing on within the headset’s displays. This could provide a more natural and accurate interaction with a menu or other software system than using hands, as there is less awkwardness than grasping and selecting a virtual object in real-world space.

However, eye tracking is limited in its accuracy as the user’s headset is constantly in motion. While fixed to the user’s head, the headset will still move around, with the distance between the user’s eyes and the eye tracking system varying all the time, minimizing how effective it could be.

Apple’s patent proposes the use of a software system with an established eye tracking system to enhance the ability to select items. The system would take advantage of a “saliency map,” which is described as a probability density map for content being shown on the display, to identify interactive elements within the user’s field of view, like selectable buttons and text.

“In normal use, if a user wants to select something, the gaze detection can determine where the user is looking, and using the saliency map, guess as to what they are specifically looking at, with a separate controller used to confirm the selection,” AppleInsider explains.

From the filing:

User input such as mouse clicks, voice commands, and other commands may be used by the control circuitry in identifying when a user is gazing on particular items within the displayed content. Information on a user’s actual on-screen point of gaze that is inferred using the saliency map information and user input can be compared to measured eye position information from the gaze tracking system to calibrate the gaze tracking system during normal operation of the electronic device.

Companies like Apple often register numerous patents on a weekly basis, but most of the filings are made as a defensive move. In other words, there are absolutely no guarantees that any of the reported patents will make it into real products. With that in mind, these filings do offer an interesting look into Apple’s research and development efforts.

On a related noted, Apple back in 2017 acquired German company SensoMotoric Instruments which specializes in eye-tracking technology.

Image: an Apple augmenter reality headset concept by 3D artist Antonio De Rosa