In our previous article about Augmented Reality with ARKit, we learned the basics of the new augmented reality platform from Apple. In this post, we are going to continue our exploration of its possibilities by learning how detecting planes works, and how to visualize them in our applications. Let’s start!

As always, you can get the full code from my Github repository.

Why Detecting Planes is So Important

The whole concept behind Augmented Reality is blending together the reality around us with virtual objects that exist only within our App. In order to be able to do that successfully, we need to be aware of the geometry of our surroundings.

In other words, we need to be able to identify the ceilings, walls, tables and other physical objects.

Then, we can start adding objects to the scene so that they will look real. Of course, size and geometry are just one part of the equation. There are many other important factors, such as the lighting of the scene, that need to be taken into account to place a realistic virtual object.

However, in this article, we are going to focus on detecting planes in our scene and being able to visualize them.

When you have finished reading this, you will be able to build an app like this one that automatically detects and updates the planes of the scene.

Augmented Reality with ARKit in iOS. Detecting planes and placing objects in augmented reality.

How Plane Detection in ARKit Works

Detecting planes from a scene in ARKit is possible thanks to feature points detection and extraction.

During the augmented reality session, ARKit continuously reads video frames from the device’s camera. Then, it tries to extract feature points to identify objects in the scene such as ceilings or furniture.

These feature points can be anything that helps identify objects: corners, structure lines, fabric characteristics, gradients, changes in color or form, edges of objects, etc…

Currently, ARKit can identify horizontal planes. This is done by means of a configuration of the ARKit scene (more about this later). Perhaps in the future, we will be able to identify vertical planes or even objects. However, for now, that’s all we get.

ARKit does an amazing job detecting planes. Nevertheless, some work is required in order to recognize and update these planes properly.

Getting Into the Scene

All the ARKit magic in your application happens in the Scene. As we saw in the previous post, depending on the engine you are using for displaying the virtual objects (SceneKit, SpriteKit, Metal…), you will use a different type of view.

In our case, as we are using SceneKit, we will use an ARSCNView. This view gets initialized with a configuration and a SceneKit scene instance.

Let’s see how to add plane detection to our scene, starting with the scene configuration.

ARKit VS SceneKit Coordinate System

It is important to note when using SceneKit to display ARKit virtual objects, that both systems use different coordinate systems. Planes in SceneKit are vertical when compared to ARKit. Thus, when translating coordinates from ARKit into SceneKit, we will use the Z-axis coordinate as the Y-coordinate of a SceneKit plane.

We’ll see a concrete example later when defining our planes in response to ARKit events.

Configuring the Scene for World Tracking

First, we will configure our scene to allow world tracking. We can do this at the viewWillAppear method.

override func viewWillAppear(_ animated: Bool) {
  super.viewWillAppear(animated)
 
  // Create a session configuration
  let configuration = ARWorldTrackingConfiguration()
  configuration.planeDetection = .horizontal
 
  // Run the view's session
  sceneView.session.run(configuration)
}

Of course, our view controller will act as the delegate of the scene view, as we can see in viewDidLoad. Also, we will assign the SceneKit scene as the scene of our scene view:

@IBOutlet var sceneView: ARSCNView!
 
override func viewDidLoad() {
  super.viewDidLoad()
 
  // Set the view's delegate
  sceneView.delegate = self
  // Create a new scene
  let scene = SCNScene()
 
  // Set the scene to the view
  sceneView.scene = scene
}

We need to track three different situations in response to delegate calls regarding plane detection. First, the delegate will be called when new nodes are detected and added to the scene via the method renderer(SCNSceneRenderer, didAdd: SCNNode, for: ARAnchor).

Then, every time a node is updated due to the movement of the camera, objects of the world, or any other situation, the delegate will receive a call to renderer(SCNSceneRenderer, didUpdate: SCNNode, for: ARAnchor).

Finally, if a node is deleted from the scene, the delegate will receive a call to renderer(SCNSceneRenderer, didRemove: SCNNode, for: ARAnchor).

Here, ARAnchor represents an “anchor” element indicating the point and dimensions where the node has been detected or modified.

The Virtual Plane Object

We will use a class called virtualPlane to represent the planes we add, update or remove from the scene in response to the delegate calls.

Our VirtualPlane object will store the anchor and a plane SceneKit node containing the visual representation of the plane.

class VirtualPlane: SCNNode {
   var anchor: ARPlaneAnchor!
   var planeGeometry: SCNPlane!
}

Initializing a VirtualPlane from an Anchor

When the delegate gets a call to renderer(SCNSceneRenderer, didAdd: SCNNode, for: ARAnchor), we will create a new VirtualPlan with the following init method.

init(anchor: ARPlaneAnchor) {
  super.init()
 
  // (1) initialize anchor and geometry, set color for plane
  self.anchor = anchor
  self.planeGeometry = SCNPlane(width: CGFloat(anchor.extent.x), height: CGFloat(anchor.extent.z))
  let material = initializePlaneMaterial()
  self.planeGeometry!.materials = [material]
  
  // (2) create the SceneKit plane node. As planes in SceneKit are vertical, we need to initialize the y coordinate to 0, 
  // use the z coordinate, and rotate it 90º.
  let planeNode = SCNNode(geometry: self.planeGeometry)
  planeNode.position = SCNVector3(anchor.center.x, 0, anchor.center.z)
  planeNode.transform = SCNMatrix4MakeRotation(-Float.pi / 2.0, 1.0, 0.0, 0.0)
  
  // (3) update the material representation for this plane
  updatePlaneMaterialDimensions()
  
  // (4) add this node to our hierarchy.
  self.addChildNode(planeNode)
}

Let’s analyze this code step by step. The important concept you must understand is the difference between the physical dimensions (geometry) of the plane, and its representation with a concrete material and visual area in a SCNNode.

Step By Step

First, we will define the geometry for our virtual plane. The extent variable of the anchor gives us the area of the plane. As we mentioned earlier, planes in SceneKit are vertical when compared to ARKit, so we will use the Z coordinate of the extent for the Y-axis of the SCNPlane.

To give the plane a physical appearance and make it visible, we need to specify a material for our plane node. We will do this with the method initializePlaneMaterial. This simple method just creates a semi-transparent gray surface.

func initializePlaneMaterial() -> SCNMaterial {
   let material = SCNMaterial()
   material.diffuse.contents = UIColor.white.withAlphaComponent(0.50)
   return material
}

Then, we can define a SceneKit node to visually show this plane geometry we just defined. We will use a SCNNode, and make sure to adjust the coordinate system accordingly to account for the differences between ARKit and SceneKit. Thankfully, we can do this easily with a 90º rotation using SCNMatrix4MakeRotation.

Next, we need to make sure we adjust the material of the plane to have the right dimensions. If we do this on a separate method, updatePlaneMaterialDimensions, we will be able to reuse it when updating the planes later.

func updatePlaneMaterialDimensions() {
   // get material or recreate
   let material = self.planeGeometry.materials.first!
 
   // scale material to width and height of the updated plane
   let width = Float(self.planeGeometry.width)
   let height = Float(self.planeGeometry.height)
   material.diffuse.contentsTransform = SCNMatrix4MakeScale(width, height, 1.0)
}

Similarly to the plane geometry, we must address the rotation for the different coordinate systems between ARKit and SceneKit.

Finally, we add this SceneKit node to our VirtualPlane. At that moment, our plane will be ready to display in the ARKit session.

From Plane Detection to Adding our Virtual Plane

Augmented Reality with ARKit in iOS. Detecting planes.So, what do we do when renderer(SCNSceneRenderer, didAdd: SCNNode, for: ARAnchor) gets called?

We will keep a list of virtual planes.

Each time the didAdd method is called, we will add one plane to our list. Every ARAnchor has a UUID identifier.

Then, we can use this identifier to tell if a plane is new and should be added to the scene, or it’s one of our previously detected planes and we should update it.

We can use a dictionary [UUID: VirtualPlane] for that in our UIViewController.

Thus, this will be the implementation of the delegate call.

func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
   if let arPlaneAnchor = anchor as? ARPlaneAnchor {
      let plane = VirtualPlane(anchor: arPlaneAnchor)
      self.planes[arPlaneAnchor.identifier] = plane
      node.addChildNode(plane)
   }
}

Updating our Planes

As you move around the scene, go forward or change the position of the camera, ARKit will update the planes. We should respond to the delegate call to update the visual representation of our planes accordingly.

First, we will add an update method to our VirtualPlane class making use of some of the methods we have already defined.

func updateWithNewAnchor(_ anchor: ARPlaneAnchor) {
   // first, we update the extent of the plane, because it might have changed
   self.planeGeometry.width = CGFloat(anchor.extent.x)
   self.planeGeometry.height = CGFloat(anchor.extent.z)
 
   // now we should update the position (remember the transform applied)
   self.position = SCNVector3(anchor.center.x, 0, anchor.center.z)
 
   // update the material representation for this plane
   updatePlaneMaterialDimensions()
}

Next, we just implement the delegate method in our view controller.

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
   if let arPlaneAnchor = anchor as? ARPlaneAnchor, let plane = planes[arPlaneAnchor.identifier] {
      plane.updateWithNewAnchor(arPlaneAnchor)
   }
}

And that’s all we need to keep our planes updated as we move through the scene.

Similarly, we just need to delete the planes when ARKit tells us to.

func renderer(_ renderer: SCNSceneRenderer, didRemove node: SCNNode, for anchor: ARAnchor) {
   if let arPlaneAnchor = anchor as? ARPlaneAnchor, let index = planes.index(forKey: arPlaneAnchor.identifier) {
      planes.remove(at: index)
   }
}

Finally, run the application, move around the scene, and you should see the semi-transparent planes appearing as ARKit detects them.

About Lighting, Textures, and Surfaces when Detecting Planes

Even though ARKit does an excellent job recognizing planes and surfaces, its efficiency greatly depends on several factors:

Lighting

Obviously, lighting has an important influence on the ability of ARKit of detecting planes. Computer vision algorithms, in general, behave poorly under bad lighting conditions.

You can check that directly in the application. Try the same path with different lighting environments (all lights switched on, only ambient light, etc), and you will see how it affects plane detection.

Textures and Surfaces

ARKit works best when it has some good, rich textures and surfaces with enough points to extract features.

This implies that a white, flat wall without texture or differentiating points will not give you good planes. On the contrary, corners, rich textures and well-defined surfaces will help ARKit do a good job when detecting planes from the scene.

Debugging Feature Points

Augmented Reality with ARKit in iOS. Detecting planes.In order to get an idea of how well features are being recognized on different scenarios, we can activate two debugging options in our sceneView object: “Show feature points” and “Show world origin”.

The first one will ask ARKit to render the “feature points” as they are being identified. They will appear as yellow dots. The second one will show a triple X-Y-Z axis at the world origin.

Experiment a little with this debugging options enabled and check how feature points appear on differents scenarios. Check corners, different fabrics, ceilings, vertical and horizontal surfaces, etc.

Enabling them is easy with just a line of code in viewDidLoad.

// debug scene to see feature points and world's origin
self.sceneView.debugOptions = [ARSCNDebugOptions.showFeaturePoints, ARSCNDebugOptions.showWorldOrigin]

 

About Initialization Times, Going into Background and Detecting Planes

It’s important to take into account that ARKit takes some time to initialize. Obviously, during that time, no plane detection will be available, and your UI should clearly indicate that to the user.

Similarly, when the app goes into the background, ARKit stops tracking the world, and plane detection functionality is disabled. As a result, when the app comes into the foreground again, there will be a delay also before ARKit starts recognizing planes and objects around.

Then, if during that time the user moves its position or the camera changes significantly, all previously detected planes will not be accurate. Thus, you might want to remove or try to adjust all the planes when this situation occurs.

Conclusion

In this second article of “Augmented Reality with ARKit”, we learned about plane detection and how to visualize the planes we extract from the scene in SceneKit.

This knowledge will be the foundation for putting virtual objects in the scene. We will address that in a future article.

I hope you will have some fun moving around, detecting planes and playing with ARKit. Did you enjoy the post? Did you find it useful? Please don’t hesitate to leave a comment.