Iris Tracking

Adding iris tracking in Spark AR studio exposes details about the position and rotation of a user’s eye. This allows you to create different effects, including changing the appearance of someone's eyes, by tracking 3D objects to the eye.

In this tutorial, you’ll learn how to add iris tracking to an AR effect.

Opening the Unfinished Project

Download the sample content follow along with this tutorial.

In the unfinished project, you'll see:

  • A face tracker and face mesh. We've applied an occluder material to the face mesh, which will block out whatever's behind it. This means the irises will be covered in a natural way - so they're not visible when someone blinks.
  • Eyeball 3D Objects
  • A texture which we’ll apply to the eyeball 3D objects.
  • A Patch Asset labelled Eyeball Shader, imported from the AR Library.

You can adapt the example project to make your own iris tracking AR effects.

The eyeball 3D objects will be visible in the Viewport and Simulator. They’re positioned behind each other in the middle of the face because they’re not tracked to the eyes yet.

The eyeball object can be seen on top of the user's nose.

This effect uses the Patch Editor. If you haven't used the Patch Editor before, it's worth learning the basics Follow our tutorial or find more information in our documentation.

Tracking the 3D objects to the eyes

Open the Patch Editor and drag faceTracker0 from the Scene panel into the Patch Editor. You'll see a group of 3 patches: Face Finder, Face Select and faceTracker0.

The Face Finder patch is usually used to detect and follow the entire face. In this effect we need to capture the positioning and orientation of the eyes. To do this, you’ll need to attach an Eyeball patch:

  1. From the Patch Editor menu, select Face Landmarks.
  2. Select the Eyeball
  3. Click Add Patch.
  4. Connect the Face output port of faceTracker0 to the Face input port of the Eyeball patch.

Here's how your graph will look:

A patch graph containing four patches.

The Eyeball patch has 6 outputs. For this effect you'll use the following ports:

  • Left and Right Eyeball Center Position: the position of the center point of the eyeball (i.e. the center of the sphere), relative to the face tracker.
  • Left and Right Eyeball Rotation: the 3D rotation of the eyeball (e.g. if the person is looking down, then this rotation will be rotated with a positive angle around the X axis).

You'll need to connect these ports to patches that represent the Position and Rotation properties of the left and right eyeball 3D objects:

To do this:

  1. Select the eyeball_left and eyeball_right objects in the Scene panel.
  2. In the Inspector, click the arrows next to both their Position and Rotation properties.

This will create 4 yellow patches, to represent these values:

A patch graph containing six patches.

The Eyeball Position ports in the Eyeball patch to the 3D Position ports of their corresponding object patches.

The Eyeball Rotation ports in the Eyeball patch to the 3D Rotation ports of their corresponding object patches.

Your finished graph will look like this:

A patch graph containing eight patches.

The 3D eyeball objects should cover and track the eyes:

The 3D eyeballs are now tracked to the right place.

Applying textures to change the appearance of the eyes

To apply the the texture included in this tutorial:

  • Select eyeball_mat in the Assets panel.
  • Select the dropdown to the right of Texture in the Inspector.
  • Select the eyeball texture.

Your effect will look like this:

The user's eyes are an unnatural blue and white.

You can apply any texture you wish to the 3D eyeball objects. For example, you can experiment by importing copyright-free images and using them as textures or creating your own.

Using the Eyeball Shader

In the Assets panel there’s a patch asset called Eyeball Shader. This is made of several visual shader patch graphs. The patch asset can be used to apply a more complex texture to the material covering the eyeball 3D objects.

You can easily customize this texture by editing the values in the Eyeball Shader to change how the eyes look.

Removing the Eyeball texture

Start off by removing the eyeball texture already applied to the material.

  1. In the Assets panel, select eyeball_mat.
  2. In the Inspector select the dropdown next to Texture and select Remove.
The texture is removed from the eyeball material

Connecting the patch asset to the material

In the Patch Editor:

  1. Drag the Eyeball Shader into the Patch Editor.
  2. Select eyeball_mat and under Diffuse in the Inspector, click the arrow next to Texture to create a patch representing the diffuse texture of the eyeball material.
  3. Connect the Texture output port to the input port of the Diffuse Texture patch.

Your graph will look like this:

The Eye Shader patch is connected to the eyeball material.

Customizing the material

You can easily customize your effect by editing the values in the Eyeball Shader to change attributes such as Iris Color, Pupil Color and Opacity. For example, to change the Iris Color:

  1. In the Eyeball Shader click the rectangles next to the Iris Color A, B and C values.
  2. Choose a color from the color wheel for each value and click OK when you’re happy with how your effect looks.
Choosing colors for the eyeball texture.

If you choose to keep the original values in the Eyeball Shader your final effect will look like this!

The user is seen in the Simulator with pink/purple eyes.