3D Collision Patch & HandTracking with Spark AR

Dn Cherry
Jul 1, 2022
2 min read

Jump Ahead: 3D Collision Patch | Hand-Tracking

3D bounding box collision patch for augmented reality using spark AR

The Singing Bowl filter explores Spark AR’s new target tracking feature for Instagram. Many hand tracking filter tutorials cover a two-dimensional approach, and my goal here was to analyze 3D usage. In this post, you can download the patch created to detect collision between two 3D objects and discover the third way of connecting objects to the hand tracker.

With the filter, users can experience the sensation of singing bowls. The experience was one of the award recipients for Lenlist’s 70th Weekly Selection. I highly recommend checking out their page and joining the community.

In the filter, users can:

Change the color of the bowl to a set of options
Hear the sounds from the singing bowl by rotating the stick around the bowl
Experience a kaleidoscope effect every third time the bowl is touched

Key features:

Collision detection between the bowl and stick trigger the sound effect
Sound distortion based on the rotation speed around the bowl
Textures for the singing bowl set were generated in photoshop
SDFs and fun math are used to create the kaleidoscope effect

Try the filter here: https://www.instagram.com/ar/405824541235665/

Patch graph for augmented reality filter in spark AR — Patch Graph

3D Bounding Box Collision Patch

Collision detection between the drum stick and bowl was needed for the sound to know when to play. Following tips from the Spark AR documentation and math from this article on 3D Collision Detection, I created the patch below. Connecting two objects enables collision detection with a boolean signal as the output.

3D bounding box collision patch with math for augmented reality software spark ar — Collision Patch Breakdown

Download the patch below.

Hand-Tracking

Spark AR covers two ways of incorporating the hand tracking feature:

In the Scene panel, by placing the object as a child
Through patches utilizing the hand bounding box

The third way of connecting objects to the hand tracker is a combination of the two methods above. This method is needed to utilize the 3D bounding box patch. The hand bounding box method is only in 2D space and won’t consider the three-dimensions of objects in your hand.

Add a hand tracker to the Scene panel
Drag & drop the hand tracker to the patch editor or click create under Interactions from the right-hand panel
Connect the desired 3D object's position to handtracker0

From my experience, this allows for better debugging, control, and interactivity. Below is the setup for the drum stick tracking with the hand. Adjustments to the position of your object can be made by unpacking the vector set and adding or subtracting the desired coordinate by the given value.

hand tracking for spark ar — Hand Tracking Setup

The ability to pull specific coordinates took the collision detection to another level. Based on the rotation around the bowl (rate of change between x & z), while the collision was detected, the distortion of the music would alter.