Tutorial: Iris Tracking

Adding iris tracking in Spark AR Studio allows you to create effects that change the appearance of someone's eyes, by tracking 3D objects to the irises.

Download this example project to try it out. In the unfinished project, you'll see:

• A face tracker and face mesh. We've applied an occluder material to the face mesh, which will block out whatever's behind it. This means the irises will be covered in a natural way - so they're not visible when someone blinks.
• 3D objects with textures applied to them.

You can adapt the example project to make your own iris tracking AR effects.

You can either use the Patch Editor or scripting to create an effect with iris tracking. We've included instructions and a script for doing both in this guide.

If you haven't used the Patch Editor before, it's worth learning the basics.

Using the Patch Editor to add iris tracking

Drag the face tracker from the Scene panel into the Patch Editor. You'll see a group of 3 patches: Face Finder, Face Select and faceTracker0.

The Face Finder patch is usually used to detect and follow the entire face. You'll need to replace this patch with an Eyeball patch, which can detect and follow someone's eyes. To do this:

1. Add an Eyeball patch.
2. Connect the Face output port of the Face Select patch to the Face input on faceTracker0.
3. Connect the Face output port of faceTracker0 to the Face input port of the Eyeball patch.

Here's how your graph will look:

The Eyeball patch has 6 outputs:

• Left and Right Iris Position: the 3D position of the the pupil on the surface of the eye, relative to the face tracker.
• Left and Right Eyeball Center Position: the position of the center of the eyeball, relative to the face tracker.
• Left and Right Eyeball Rotation: the 3D rotation of the eyeball. For example, if the person is looking down, this rotation will be rotated with a positive angle around the X axis.

For this effect you'll use the Left and Right Iris Position and the Left and Right Iris Rotation output ports.

You'll need to connect these ports to patches that represent the Position and Rotation properties of the left and right iris 3D objects:

To do this:

1. Select the left_iris and right_iris objects in the Scene panel.
2. In the Inspector, click the arrows next to both their Position and Rotation values.

This should create 4 yellow patches, representing each value:

Connect:

1. The Iris Position ports in the Eyeball patch to the 3D Position ports of their corresponding object patches.
2. The Iris Rotation ports in the Eyeball patch to the 3D Rotation ports of their corresponding object patches.

Here's how this part of your graph will look:

And your finished graph will look like this:

The objects should cover and track the eyes in your finished effect:

Using scripting

In the Assets view, click +, then Create New Script.

You'll see a new script in the Assets panel.

Double-click the script to open it in a text editor, then paste this script:

    var FaceTracking = require("FaceTracking");
    var IrisTracking = require("IrisTracking");
    var Scene = require("Scene");
        
    var leftIris = Scene.root.find("left iris");
    var rightIris = Scene.root.find("right iris");
        
    var leftEyeballInfo = IrisTracking.leftEyeball(FaceTracking.face(0));
    var rightEyeballInfo = IrisTracking.rightEyeball(FaceTracking.face(0));
        
    leftIris.transform.position = leftEyeballInfo.iris;
    rightIris.transform.position = rightEyeballInfo.iris;
        
    leftIris.transform.rotation = leftEyeballInfo.rotation;
    rightIris.transform.rotation = rightEyeballInfo.rotation;
      

The Eyeball object stored in leftEyeballInfo and rightEyeballInfo has three properties:

• iris, which gives the pupil's central position relative to the face tracker.
• center, which gives the position of the center of the whole eyeball relative to the face tracker.
• rotation, which gives the rotation of the eyeball.

Learn more

• Position 3D animated objects on different parts of the face in this tutorial.
• Experiment with different face landmark patches.