Development

Part 2 // Manhole Monsters: An Introduction to Augmented Reality in iOS

Manhole Monster Part 2 Hero

BUILT & AUTHORED BY:

David Sklenar & Lemon Garrett

CONCEPT BY:

Mike Judge

Part 2: Anchoring the Model

Welcome back! In Part 1 of this series, we introduced our challenge: placing animated 3D objects on custom objects in the real world while using Core ML to find custom objects. Now, in Part 2, it’s time to place them.

RealityKit - Reality Composer

Reality Composer is a tool that offers an interface in which to create, manipulate and customize AR experiences. Out of the box, Reality Composer comes with an impressive collection of customizable assets. It’s also possible to import your own, provided they’re of file type `usdz` (conversion instructions here), and use them to curate your AR experience. Start off by selecting how you might like to anchor your scene, then you can position your scene however you might like. The option of adding sound effects or animation according to user interaction is also possible simply by using this interface. It’s easy to test how the scene looks on a device, provided it supports ARKit since Reality Composer allows you to sync your session on your Mac with your iOS device.

Anchor
RealityKit provides a host of automatic anchoring strategies. These include: horizontal and vertical planes, images (in 2D planes), faces, and scanned objects.
P11
Here, we tried building out an elaborate spread with some of the objects that come with Reality Composer. (Note: You might have to download some assets to view them in Reality Composer.)
P12
We chose one of the models that ships with Reality Composer to place in our app: a toy drummer.
P13
Adding behaviors is basically a form of visual programming. Click the + button to add triggers and actions. Here you can see that tapping the model scales it by 2X over a second with an ease-in and ease-out animation.

One really cool feature is the ability to add behaviors to objects in your scene, making them interactive, without writing any code. You can trigger behaviors off a host of common situations, including tapping an object, getting within a certain distance, colliding the object with another object, or custom triggers the developer defines. Some of the behaviors that these triggers can cause are moving, scaling, or animating the position of an object, triggering a predefined USDZ animation, or changing the scene entirely. These built-in functions can get you pretty far.

RealityKit - Placing a 3D Model

P14
Once we knew where the book was in the camera, we utilized RealityKit to place the model in 3-dimensional space. To do so, we raycasted the 2-dimensional center of the bounding box returned from our custom machine learning model into a 3D anchor. Apple defines raycasting as, “the preferred method for finding positions on surfaces in the real-world environment.”
extension ARView {
  @discardableResult
  func raycastAndPlaceObject(realityKitModel: Experience.Nutcracker, point: CGPoint) -> Bool {
    let scaledPoint = CGPoint(x: point.x * bounds.width, y: point.y * bounds.height)

    guard let raycastQuery = makeRaycastQuery(from: scaledPoint, allowing: .estimatedPlane, alignment: .any) else {
      print("failed to make raycast query")
      return false
    }

    guard let result = session.raycast(raycastQuery).first else {
      print("no raycast found")
      return false
    }

    let transformation = Transform(matrix: result.worldTransform)

    let raycastAnchor = AnchorEntity(raycastResult: result)

    realityKitModel.generateCollisionShapes(recursive: true)
    realityKitModel.transform = transformation
    raycastAnchor.addChild(realityKitModel)

    self.scene.addAnchor(raycastAnchor)

    return true
  }
}

Note that this operation doesn’t always succeed, because, behind the scenes, RealityKit may not have figured out surfaces, planes, and objects in the real world. We suspect that if you were using the new iPhone or iPad, the detailed depth information provided by the LiDAR Scanner would really help out.

And there you have it! We found a new object in the camera feed, used RealityKit to find its 3D location, and anchored our Reality Composer model to it.

Ezgif com gif maker
Our model isn’t robust enough to handle complex patterns surrounding the book very well, so when I picked it up and got more detail registering it worked.
P17vid 2
On a plain background, it worked right away!
P18vid
Example of the 3D model disappearing after getting 5 meters away from the anchor.

One thing we noticed is that objects lose their relevance once you get far enough away from them, especially if you’re on a walk and the placed object might be behind a building or out of sight. Reality Composer surprisingly doesn’t provide a trigger to remove an object when it gets a certain distance away, so we had to hook up that logic ourselves. Check out the demo above to see how well it works.

func session(_ session: ARSession, didUpdate frame: ARFrame) {
 // Remove the model from the scene if it has been placed and is over 5 meters away.
  if
    let scene = view?.scene,
    let anchor = scene.anchors.first
    {
      let cameraPosition = frame.camera.transform.columns.3
      let anchorPosition = anchor.transform.matrix.columns.3

      let cameraToAnchor = cameraPosition - anchorPosition
      let distance = length(cameraToAnchor)

      if distance > 5 {
        print("Removing model.")
        parent.scanningState = .scanning
        return
      }
   }
}

Conclusion

Next week, we'll share our final installment in the Manhole Monster series: exploring building out a basic user interface using Swift UI and exploring the challenges around accessibility in augmented reality.

David Sklenar Headshot

David Sklenar (he/him)

Senior Developer

David is a software developer with over 10 years of experience. He’s spent most of that time developing iOS apps (most recently at Nike) and has used technology to solve problems in poetry publishing, medieval architectural history, cardiac health, and digital storytelling. More recently, he’s become interested in machine learning and its applications and implementation. He most enjoys the collaborative and social nature of software development and the endless opportunities for learning.

Lemon author

Lemon Garrett (they/them/she/her)

Developer, Substantial

A bit of a renaissance person, Lemon spent eight years as a traditional artist before eventually transitioning into a career as a software engineer, and found their way to Substantial in October 2019. As a full-stack developer, Lemon is a steadfast accessibility advocate and a zealous promoter of diversifying the tech industry in pursuit of creating and maintaining a more inclusive, supportive environment. Passionate about leveraging their knowledge in order to help make web and mobile apps universally accessible, Lemon is constantly seeking new, straightforward ways to integrate accessibility best practices into their teams’ workflows and approach to development.

Let's build a better future, together.