RealityKit Body Tracking

BodyTracking

I created BodyTracking to simplify human tracking in ARKit and RealityKit applications. The open-source Swift package provides reusable, RealityKit-friendly APIs for building interactive augmented reality experiences without repeatedly implementing low-level tracking infrastructure.

Features

• 2D and 3D body tracking

• 2D and 3D hand tracking

• Face and eye tracking

• Face geometry morphing

• Person occlusion

• Joint-based attachment of UI and 3D content

• Joint position and angle calculations

The package uses ARKit, RealityKit, and Vision, with a modular architecture built around RealityKit’s entity-component system. It is distributed through Swift Package Manager and separates body, hand, and face tracking into focused libraries that developers can integrate as needed.

View the source code on GitHub

I also made a course that has sample projects showing how to use the swift package.

Face Tracking

Before RealityKit supported blendshapes, I found a way to perform geometry morphing interpolating between target geometries. I forked a repo and used this to create a memoji style effect with a turtle head.

Body Tracked Entity

I figured out how to rig my own BodyTrackedEntity in Blender for use in RealityKit. You can see the example with the green biped robot responding to my movements in realtime.