Creating a Face Tracking Effect

Hello! In this guide, you'll learn how to make an effect that responds to someone's face. You'll use a face tracker, face mesh, material and texture.

Download the sample content to follow along with this tutorial. It includes the custom texture you'll need to make this effect. Everything else can be created in Spark AR Studio.

To help you make your own textures that map to the face, we've created a template that can be used in Adobe® Photoshop®. Learn how to use it to make your own textures in this article.

AR effects made with Spark AR Studio can be used in the Facebook and Instagram cameras. To create a frame for a Facebook profile picture, use Frame Studio.

Setting up

To get started, open the unfinished effect in the sample content folder.

The space in the middle of the screen is called the Viewport. This is where you can see your effect as you're building it.

The Simulator will show how your effect will look on a device.

The Simulator is highlighted.

We've already imported the texture you'll need for this effect. It's called Moustache and listed in the Assets panel:

The moustache texture can be seen in the Assets panel.

You'll combine this texture with a material to create the effect of a moustache drawn across a person's face.

Spark AR Studio supports PNG and JPG texture files. Files can be a maximum of 1014x1024 in size. Learn more about using textures.

The face tracker

The first thing you need to do is add a face tracker to your project. The face tracker detects the position and orientation of the face of the person using your effect. It's the foundation of any effect that responds to someone's face.

To add any object to you scene, just click the Add Object button. You'll see a list of all the objects available in Spark AR Studio. For this tutorial:

  1. Select the Face Tracker from the list.
  2. Click Insert .
An object is being added to the Scene panel by clicking Add Object.

When the face tracker is selected in the Scene panel, you'll see axes moving with the face in the Viewport.

Adding a face mesh

You've already added a face tracker, so now you can insert a face mesh. The face mesh is a 3D model that responds to facial expressions tracked by the face tracker. You'll apply materials to the face mesh to change how it looks - creating the mask effect.

You'll add the face mesh as a 'child' of the face tracker. When an object is the child of another object it will take on any changes made to the object it's a child of - the parent object. It can also respond to the parent object. The face mesh will respond to the movement detected by the face tracker.

To add the face mesh as a child of the face tracker:

  1. Right-click on the face tracker in the Scene panel.
  2. Select Add.
  3. Choose the Face Mesh from the list.

You’ll see the face mesh object in the Viewport and Simulator:

The face mesh object (a checkerboard mask) can be seen tracking the face in the Simulator and Viewport.

You can tell if an object is a child of another by clicking the arrow to the left of the parent object in the Scene panel. This will show or hide any of its children.

Creating a material

To create a material for the face mesh, make sure it's selected in the Scene panel.

On the right of the interface is the Inspector. This is where you can make changes to the object or asset you've selected in the Scene panel - including creating materials.

In the Inspector, click + next to Material to create a material for the face mesh.

You’ll see the material covering the face mesh in the Viewport and Simulator:

A material has been applied to the facemesh and it has lost its checkerboard pattern.

The material will be listed in the Assets panel as material0:

The material and texture can both be seen in the Assets panel.

Materials can also be created in the Assets panel. To do this, click Add Asset and select Material. To add the material to an object in the scene, go to the Inspector and click the dropdown next to Materials.

You can rename objects and assets in Spark AR Studio. It's a good idea to do this, to help stay organized as you build your project. All you need to do is right-click on the object and select rename. For this tutorial:

  1. Right-click on material0.
  2. Select Rename.
  3. rename the material mask_material.

Setting the Shader Type

Go back to the Inspector to edit the material.

First let's take a look at this option at the top - Shader Type:

The shader type for material 0 being selected in the Inspector.

You can use this option to choose from several different built-in shaders in Spark AR Studio to define the appearance of materials in your effects.

The mask that we're going to make will show the user's face behind it, so we're going to choose the Face Paint shader. It's the best option for mask effects where a texture is painted on top of the face because it preserves the luminance of the face, but removes the color. This means your own texture can be multiplied over it.

To change the shader:

  1. Click the dropdown menu next to Shader Type.
  2. Select Face Paint.

Now you can see the face through the material:

The Face Paint shader type has been selected, allowing us to see the user's face through the shader mask in both the Viewport and Inspector.

Applying the texture

Next let's apply the texture. Next to Texture:

  1. Select Choose File....
  2. Select moustache.
The mask is shown changing as our moustache texture is applied to the material.

Alright, we've finished the mask! Now let's take a look at how to add the option to use an effect with a friend - by adding a second face tracker.

Adding a second face

To try this out, you can duplicate the face tracker you've just made. This will copy its children, too. Simply:

  1. Select the face tracker in the Scene panel.
  2. Hold down control and c on your keyboard, then control and v.

Another face tracker and face mesh will be listed in the Scene panel.

You'll need to make one edit in the Inspector. Make sure the second face tracker, faceTracker1, is selected in the Scene panel. Over in the Inspector you'll see a property called Tracked face. Use this to assign a particular face to a particular face tracker.

It's set to Face 1 by default. Change it to Face 2:

The properties of face tracker 1 are being changed in the Inspector.

This will make it responds to the second face that appears in the effect. And there we go, your effect can be used by two people. You could add a different texture to the material applied to the second face mesh, to add more variety to your effect.

Congratulations! You've finished an effect with face tracking! Keep on learning and complete the next tutorials.

Was this tutorial helpful?