Hand Tracking

How it works

Hand Tracking estimates 3D joint transforms from Apple’s Vision 2D hand pose and an ARKit depth buffer. It stays on native frameworks, so it does not depend on a third-party hand model. The sample project draws a box on every joint so you can see the skeleton in the camera view.

Features

• attach(entity:to:) parents an entity to one joint. One entity is tracked per joint. Extra views should be children of that entity.

• HandAnchor.requestRate controls how often the Vision request runs. Every frame is the default. Half or quarter rate trades smoothness for speed.

• With autoEnableInitially and autoToggleContinually, the hand stays disabled until Vision sees it, and can disable again when the hand leaves the frame.

Notes

A HandAnchor owns a HandTracker2D and any HandTracker3D entities attached to it. Each frame, HandTracking3DSystem reads the 2D joint positions, samples depth, and writes joint transforms.

• Depth defaults to smoothed scene depth on LiDAR devices. Scene depth and person segmentation with depth are the other options. Devices without smoothed scene depth fall back to person segmentation.

• All joint depths are read from the depth map in one lock, using normalized AVFoundation coordinates.

• Each screen point is unprojected with ARView.ray(through:). The ray origin plus the ray direction times depth, in meters, is the world position.

• The wrist sets the anchor position.

• Other joints are stored in the hand’s model space. A joint aims at its child. Fingertips aim outward from the previous joint, because fingertip depth is copied from that joint.

• Depth is smoothed with a lerp. If a sample jumps more than the threshold limit from the middle knuckle, it keeps the previous depth, or the middle knuckle depth when the hand is moving quickly toward or away from the camera.

Limits

• The update loop tracks one hand. Multiple hands are not supported yet.

• Depth on non-LiDAR devices is a known gap. The fallback buffer is less reliable than smoothed scene depth.

• 2D screen positions are stable. Depth is not, so the 3D pose is an estimate. For a more robust skeleton, train your own model and feed it through ARKit. Apple’s recognizing and labeling arbitrary objects sample is the reference for that path.

More body tracking work can be seen on this post: Body Tracking.