Finger painting and 3D sculpting in augmented reality with ARKit

A lot of iDownloadBlog readers really liked yesterday’s food-ordering demo which renders virtual foods on a plate so we’re featuring yet another novel use case for Apple’s new ARKit framework: finger painting and 3D object sculpting through augmented reality.

Developer Fabin Rasheed has put together an app called Maker Studio which lets you sculpt and paint 3D objects with your finger or Apple Pencil on an iPad Pro, all in augmented reality.

If you’d like, you can sign up for the beta and updates through the official website.

“This is my recent work exploring sculpting and painting using ARKit,” he wrote. “It was a beauty to use Apple Pencil for sculpting and painting and ARKit gave perspective to it.”

I’m not sure how practical 3D modeling in augmented reality is, but the demo sure looks fun.

A 3D artist could use a full ARKit modeling and painting platform like that to visualize just about anything. Architects could easily model extensions for pre-existing buildings with it, designers could dress virtual models in their latest outfits and so forth.

Another demo comes via developer Osama Abdel-Karim, who built his app using a combination of iOS 11’s ARKit and Vision frameworks to create the finger painting effect.

According to Osama:

One of the cool libraries that Apple introduced in iOS 11 is Vision Framework. It provides some computer vision techniques in a pretty handy and efficient way. In particular, we are going to use the object tracking technique.

Object tracking works as follows: First, we provide it with an image and coordinates of a square within the image boundaries for the object we want to track. After that we call some function to initialize tracking. Finally, we feed in a new image in which the position of that object changed and the analysis result of the previous operation.

Given that, it will return for us the object’s new location.

Apple will most likely feature ARKit-driven apps in a special section on App Store.

The company has said that ARKit apps will require an iPhone or iPad with the A9, A10 or later processor, meaning owners of iPhone 6, iPad Air 2 and older devices will be left out in the cold.