Apple researching iSight cameras with 3D imaging and facial gestures recognition

By , Mar 29, 2012

According to a patent filing that surfaced today in the US Patent and Trademark Office’s (USPTO) database, Apple is researching a much-improved camera for mobile devices that could recreate 3D models of scanned objects as well as capture gestures and facial expressions. It works with both stills and video and employs one or more dedicated cameras to capture 3D objects.

The system is based on new depth-detection sensors, such as LIDAR, RADAR and laser, that create stereo disparity maps in 3D imagery. With the ability to both capture and recreate 3D images, this killer imaging system could elevate the already powerful imaging capabilities of the five-megapixel iSight camera on the new iPad and its eight-megapixel counterpart found on the iPhone 4S.

Can we have this on the iPhone 5, please?

Patently Apple provides a nice overview of facial gestures:

The three-dimensional imaging apparatus may be used for recognizing facial gestures. Facial gestures may include, but are not limited to, smiling, grimacing, frowning, winking, and so on and so forth. In one embodiment, this may be accomplished by detecting the orientation of various facial muscles using surface geometry data, such as the mouth, eyes, nose, forehead, cheeks, and so on, and correlating the detected orientations with various gestures.

A previous patent filing from December 2011 similarly outlines facial recognition features on portable devices.

As for 3D imaging, one of the proposed depth-detection techniques includes a Light Detection and Ranging (LIDAR) sensor that emits laser pulses. The LIDAR sensor then picks the pulses reflected from objects to calculate the distance by measuring the time delay between transmission of a laser pulse and the detection of the reflected signal.

It’d be capable of capturing oneself’s representation in a three-dimensional space via a Photo Booth-like interface, depicted at the bottom. Once captured, a 3D image of your face could be used for your three-dimensional avatar, Apple says.

More complex scenarios are discussed as well, including this:

In another related embodiment, multiple photographs or video may be taken while the image sensing device is moved relative to the object, and used to construct a three-dimensional model of the objects within the captured image(s). For example, a user may take video of a home while walking through the home and the image sensing device could use the calculated depth and surface detail information to create a three-dimensional model of the home. The depth and surface detail information of multiple photographs or video stills may then be matched to construct a seamless composite three-dimensional model that combines the surface detail and depth from each of the photos or video.

Apple credits engineers Brett Bilbrey, David Simon, Rich DeVaul, Mushtag Sarwar, Michael Culbert and David Gere with the invention. To get more information on this patent filing, type in an ID number 20120075432 into the USPTO search engine.


Patent drawing courtesy of AppleInsider.

I like how Apple isn’t standing still and continues pursuing advanced imaging techniques. We’ve come a long way since the original iPhone and its sub-par camera.

Now we have iOS gadgets that capture 1080p video at thirty frames per second and high-quality stills. Apple isn’t just using off-the-shelf CMOS sensors, they’re improving them with own solutions such as customized optics with a fifth lens and real-time processing to detect faces and stabilize video.

Sure, other phone cameras have those features as well – I’m just observing that the iPhone no longer has a crappy camera. It’s also interesting that Apple appears to be really interested in 3D imaging, as this patent filing proves.

I’m wondering whether those camera- and imaging-related patents could mean an Apple-branded digital camera down the road? After all, Apple did re-brand itself as a consumer electronics powerhouse.

Surely widening their product portfolio and branching out into digital cameras at some point won’t hurt.

Share your thoughts in the comment section below.

  • Share:
  • Follow:
  • http://twitter.com/e420kush e420Kush

    Apple needs to research how to install an iSight camera on the front of the iPad….not the rear.

    • Anonymous

      I’ve always wondered the same thing….I don’t FaceTime much but if they had a better camera in the front I prob would

  • http://twitter.com/omakad Omar Kadrovic

    This could be used for controlling the device with facial gestures rather than with touch. I too hope they continue working on this.

  • http://twitter.com/Dongiuj Giulio

    3D camera. Sharp’s android phone already has this, since way back last year some time. But I guess it will be different in some way. I mean apple said they changed how we use mobile phones by adding video calls. Which wasn’t true because they were 5 years behind Japanese mobile phones on that one and with apple it’s only over wi-fi and not 3G ( I know, 3G is expensive to make video calls ) also you’re restricted to only calling somebody with another apple product.

  • http://www.facebook.com/luisreyespr Luis

    This Could be like this camera on bestbuy that took pictures whenever u smiled.

  • http://adam.hbang.ws/ Adam D

    Anyone noticed the half-size bezel in the drawing?

    Ad@m