Tracking a Texture to the Hands

As the user raises each hand, the camera zooms into their face. When both hands are raised, a transparent square appears between their hands.

Understanding the unfinished project

The unfinished project is a slightly-modified version of the hand movement template available in the Spark AR Welcome screen. We’re not using any textures or animations to create our effect, so we removed:

The first two rectangles.
The hand_mat material.
The replaceMe assets.

In the Layers panel, we’ve also moved the betweenhands layer to the top of the list above layer0. This means the colorful gradient shape will render in front of the user. Finally, in the Assets panel, we added the Quad Gradient Patch Asset from AR Library. We’ll use this later in this tutorial.

Learn how the unfinished project works or skip this section and go straight to Getting started, below.

How the unfinished project works

Open the unfinished project and in the Patch Editor, you’ll see rect_hand0 and rect_hand1 comment boxes. The hand tracking patches in the green rect_hand0 box track the first hand and the patches in the blue rect_hand1 box track the second hand found in the camera.

How the hand tracking patches work

Each Hand Select patch tracks a hand in the scene. The Hand Bounding Box patch then plots an invisible box around the four corners of each tracked hand. The Bounding Box 2D Unpack patch outputs the X and Y coordinates of the four corners of this box, as well as the Center, Width and Height of each hand.

When the hand is tracked, the center position of this hand is output to either the hand0 Center or hand1 Center Sender patch, in blue.

The bounding box 2D unpack patch is seen connected to a blue sender patch.

This positional data is stored in either the hand0 Center or hand1 Center blue receiver patches, found in the purple comment box, and then sent to the rest of the patches in this box.

The two sender patches are connected to ten other patches.

The logic in this box uses math patches (Distance, Multiply, Add and Divide) to calculate and control the position and scale of the rectangle that appears between the tracked hands. The Hand calculations and Angle Calculation patch assets contain a series of math patches that correctly calculate the angles the rectangle should rotate at based on the movements of the user’s hands. These are useful patches to reuse when making your own hand tracking projects

Making the rectangle object visible when both hands are tracked

The Equals Exactly patch checks whether the hand count number is equal to two and makes the rectangle visible when both the first and second hands are tracked.

Now you understand the unfinished project, let's get started!

Getting started

Setting up

To see what an effect with hand tracking looks like, you'll need to switch on your computer's camera. This will let you see your own hands in the Simulator when it comes to testing out the hand tracking effect.

Creating the colorful gradient texture

We’ll start by creating a new diffuse texture for the rect_betweenhands object that's already added to the unfinished project. First, create a Diffuse Texture property patch for the betweenhands_mat:

In the Assets panel, select betweenhands_mat.
Click the arrow to the left of Texture in the Inspector.
Click Add patch.
Drag the patch to a clear space in the Patch Editor, where it’s easier to work with.

Next, we'll use a patch asset to create the colorful texture. For this tutorial we're using the Quad Gradient patch asset from AR library. It’s listed in the Assets panel. To add it to your graph:

Drag the Quad Gradient from the Assets panel to the Patch Editor.
Connect the patch to the Diffuse Texture property patch.
Finally, in the Assets panel, select betweenhands_mat and In the Inspector, change the Blend Mode to Add and the Opacity to 50%.

Your graph will look like this:

The Quad Gradient and material texture patches.

Hold up both hands and you’ll see the new colorful material in the Simulator!

The gradient has been applied to the rectangle.

Creating the zoom effect

Next you’ll use the Patch Editor to make the camera texture bigger and smaller, creating the zoom effect we need.

First, add a rectangle:

In the Scene panel, click Add Object.
Select Rectangle.
Rename it to cam_tex.

Next, ensure the rectangle is set in the correct layer and that it fills the entire screen. You can do both in in the Inspector, with cam_tex selected:

To the right of Layer, make sure that layer0, is selected. This is because in the Layers panel, we arranged this layer to be rendered last, so it doesnt cover the colorful gradient.
To the right of Width and Height, click the value.
From the dropdowns that appear, select Fill Width, and Fill Height.

This will temporarily cover the camera feed input with a checked texture. To create a material for the rectangle:

With your new rectangle selected, go to the Inspector and to the right of Materials, click + , then Create New Material.
Rename the material, cam_tex_mat.
In the Asset panel, select cam_tex_mat.
In the Inspector, change the material’s Shader Type to Flat.

Staying inside the Inspector, click the arrow to the left of Texture. This will create a patch representing the material’s texture.

Then create the camera texture:

In the Scene panel, select the Camera.
In the Inspector, to the right of Texture Extraction, click +

It'll be listed as cameraTexture0 in the Assets panel. Drag cameraTexture0 from the Assets panel into the Patch Editor.

Finally, in the Patch Editor, connect the RGBA port to the yellow patch representing cam_tex_mat. Your graph will look like this:

The camera texture patch is connected to a yellow patch representing the camera texture material's diffuse texture.

You’ll see the live video feed back in the Simulator. We’ll add the zoom logic next.

Scaling the camera texture

To create the zoom effect, we need to scale the camera texture up and down dynamically as the hands are tracked. We’ll start with creating the patch graph to scale the texture. First add a Texture Transform and 2D transform pack pacth, then connect:

The CameraTexture0 RGBA output to the Texture input in the Texture Transform patch.
The output of the Texture Transform patch to the input of the camTextMat Diffuse Texture. This replaces the connection between cameraTexture0 and camTextMat.
The 2D Transform Pack patch to the Transform input of the Texture Transform patch.

You won’t see any zoom effect in the camera because we haven’t scaled the camera texture up or down yet. To do this, we would need to edit the x and yScale values in the 2D Transform Pack patch. In the example below, we’re manually increasing the scale on the x and y axis by 0.3. In the next section we’ll show you how to do this dynamically, as each hand is tracked.

As the values are changed, the effect zooms into the user.

Dynamically scaling the camera texture

To scale the camera texture dynamically, we need to trigger an animation patch when a hand is found in the camera, connected to a transition patch, which will dynamically change the scale of the camera texture. We’re going to do this for one hand at a time, so start by locating the first Hand Bounding Box patch — the one in the green comment box.

Next add a pulse patch, an animation patch and a transition patch.

Then connect:

The Is Tracked output of the Hand Bounding Box to the Pulse patch.
The Turned On output of the Pulse patch to the Play input in the Animation patch.
The Turned Off output of the Pulse patch to the Reset input in the Animation patch. We’ve left the animation duration at the default 1, but you can edit this to suit.
The Progress output in the Animation patch to the Progress input in the Transition patch. At the bottom of the Transition patch, change the data type to Vector 2, as we’re working with 2D coordinates (the x and y values of the camera texture).
Edit the values in the Transition patch by changing the Start x and y values to 1 and the End x and y values to 1.3. This way, we are telling Spark AR to dynamically scale the camera texture from by 0.3 when a hand is tracked in the scene.

The Transition patch to the Scale input in the 2D Transform Pack patch.

Your graph will look like this:

Your effect looks like this:

As one hand is raised, the camera zooms in.

You’ll notice that the zoom transition seems quite abrupt. We’ll add an exponential smoothing function between the transition and scaling to smooth it and make it more natural. First add:

An Unpack patch. Change the data type to Vector 2.
A Pack patch. Change the data type to Vector 2.
Two Exponential Smoothing patches.

Then connect:

The Transition patch to the Unpack Patch. This splits the x and y signals into two separate number values, required for the Exponential Smoothing patch.
Connect the X of the Unpack patch to one Exponential Smoothing patch and the Y output to the other. This adds smoothing to both signals.
Connect the Outputs of the Exponential Smoothing patches to the inputs of the Pack patch.
Connect the Vector output of the Pack patch to the Scale input of the 2D Transform Pack. This packs the X and Y data streams back into a vector 2 data type.

Increasing the zoom effect with the second hand

At the moment the zoom effect only works for one hand. To create an increased zoom effect when the second hand is tracked, we need to set up some logic to check if the hand count in the hand finder patch is equal 2 and, if so, scale the camera texture to 1.6. First add an Equals Exactly patch and an If then Else patch. Then connect:

The count output in the Hand Finder patch to the First Number input in the Equals Exactly patch.
The Equals Exactly patch to the Condition input of the If then Else patch. Make sure to change the If then Else patch to Vector 2 data type.
The If Then Else patch to the End input in the Transition patch.

Edit the values

In the Equals Exactly patch, edit the Second Number value to 2. This means a boolean true signal will be sent to the If Then Else patch when the hand count in the Hand Finder patch is 2.

Finally in the If Then Else patch edit the Else value to 1.3 and the Then value to 1.6 on both axis. This means that if the condition is true, and there are two hands found, the x and y values of the camera texture will increase to 1.6. If only one hand is found the value increases to 1.3.

Your graph will look like this:

Here’s how your final effect will look!