ANIMA - Onlybots + App Clip

Overview

Onlybots is an immersive augmented reality experience that lets users discover, interact with, and control virtual characters in their own environment. Built with Swift and RealityKit, the project included a full iOS app and an App Clip, allowing users to instantly launch an AR experience by scanning a QR code without needing to install the full app.

Users could browse a collection of virtual characters, switch between AR and non-AR viewing modes, and move characters around their physical space using intuitive touch gestures.

My Role

Lead iOS Engineer

I led the iOS engineering effort, developing much of the core RealityKit implementation and designing the character interaction systems. My work focused on building custom controls, integrating advanced rendering techniques, and extending the capabilities of RealityKit to support the experience.

Key Contributions

Networking

I used HTTP networking with a REST API through the Alamofire Swift package to download characters, which were represented as voxel data. I used that data to procedurally generate the geometry in RealityKit, rather than loading USD files the way the ANIMA NFT viewer did. The app also kept track of which characters a user had unlocked and associated that with their account.

AR Placement

I built a custom placement system: Once a bot was invoked in AR (such as via an app clip link or a user tapping to open a certain bot), the app would show a glowing white ring where the user could place the bot. If pointing at another bot, the ring would disappear. If pointing at an invalid surface, there would be a glowing red ring instead.

I built a cloud of swirling white voxels / cubes. Once the user tapped to place the bot, the voxels would converge into the bot's shape and fade from white to the bot's own colors.

Custom Character Controls & Interactions

Designed and implemented a custom character controller and gesture recognition system, replacing RealityKit's built-in interaction behaviors with more flexible controls.

This allowed users to directly manipulate virtual characters, reposition them within their environment, and interact with them through intuitive touch gestures.

Custom Rendering & Glowing Eye Effects

I wrote a shader for the RealityKit side of the eyes which modulated their color depending on the amplitude of the audio. The more loudly the bot spoke, the more white their eyes became. If they were not talking, the eyes would have that particular bot's original eye color.

For the glowing eyes effect, I used a post process, but this post process needed a mask of where the eyes were in the frame. I integrated a parallel rendering pipeline to overcome limitations in RealityKit's rendering capabilities. Because RealityKit did not expose a stencil buffer, I integrated Satin, a Metal-based rendering framework, to render character eyes in a separate pass. This enabled a custom post-processing effect that gave the characters glowing eyes.

Physics Engine Integration

Integrated Bullet Physics into the Swift application through a customized fork of PhyKit, a Swift bridge to the C++ physics engine.

This extended the application's physics capabilities beyond RealityKit's built-in functionality and provided greater control over the underlying simulation.

App Clips & Seamless AR Experiences

Helped deliver an experience across both a full iOS app and a lightweight App Clip.

Users could scan a QR code to launch the App Clip immediately, or open the full application if it was already installed, creating a low-friction entry point into the AR experience.

AR & Non-AR Character Viewing

Implemented character experiences that supported both augmented reality and non-AR modes, giving users the flexibility to view and interact with virtual characters with or without their camera.

The full app also featured a browsable collection of characters. The nonAR view had a parallax effect that used data from the device's motion sensors.

Technologies

Swift · RealityKit · ARKit · App Clips · Satin · Metal · Bullet Physics · PhyKit · C++ · Alamofire · REST API