In this tutorial, you’ll learn how to create an effect that zooms in each time a hand is found in the camera. When two hands are found, a colorful gradient shape appears between them, and resizes with the hands.
To follow along, download the example project and open the unfinished project.
To see what an effect with hand tracking looks like, you'll need to switch on your computer's camera. This will let you see your own hands in the Simulator to test the effect.
The unfinished project is a slightly-modified version of the hand movement template available in the Spark AR Welcome screen. We’re not using any textures or animations to create our effect, so we removed:
In the Layers panel, we’ve also moved the betweenhands layer to the top of the list above layer0. This means the colorful gradient shape will render in front of the user. Finally, in the Assets panel, we added the Quad Gradient Patch Asset from AR Library. We’ll use this later in this tutorial.
Learn how the unfinished project works or skip this section and go straight to Getting started, below.
Open the unfinished project and in the Patch Editor, you’ll see rect_hand0 and rect_hand1 comment boxes. The hand tracking patches in the green rect_hand0 box track the first hand and the patches in the blue rect_hand1 box track the second hand found in the camera.
How the hand tracking patches work
Each Hand Select patch tracks a hand in the scene. The Hand Bounding Box patch then plots an invisible box around the four corners of each tracked hand. The Bounding Box 2D Unpack patch outputs the X and Y coordinates of the four corners of this box, as well as the Center, Width and Height of each hand.
When the hand is tracked, the center position of this hand is output to either the hand0 Center or hand1 Center Sender patch, in blue.
This positional data is stored in either the hand0 Center or hand1 Center blue receiver patches, found in the purple comment box, and then sent to the rest of the patches in this box.
The logic in this box uses math patches (Distance, Multiply, Add and Divide) to calculate and control the position and scale of the rectangle that appears between the tracked hands. The Hand calculations and Angle Calculation patch assets contain a series of math patches that correctly calculate the angles the rectangle should rotate at based on the movements of the user’s hands. These are useful patches to reuse when making your own hand tracking projects
Making the rectangle object visible when both hands are tracked
The Equals Exactly patch checks whether the hand count number is equal to two and makes the rectangle visible when both the first and second hands are tracked.
Now you understand the unfinished project, let's get started!
Setting up
To see what an effect with hand tracking looks like, you'll need to switch on your computer's camera. This will let you see your own hands in the Simulator when it comes to testing out the hand tracking effect.
We’ll start by creating a new diffuse texture for the rect_betweenhands object that's already added to the unfinished project. First, create a Diffuse Texture property patch for the betweenhands_mat:
Next, we'll use a patch asset to create the colorful texture. For this tutorial we're using the Quad Gradient patch asset from AR library. It’s listed in the Assets panel. To add it to your graph:
Your graph will look like this:
Hold up both hands and you’ll see the new colorful material in the Simulator!
Next you’ll use the Patch Editor to make the camera texture bigger and smaller, creating the zoom effect we need.
First, add a rectangle:
Next, ensure the rectangle is set in the correct layer and that it fills the entire screen. You can do both in in the Inspector, with cam_tex selected:
This will temporarily cover the camera feed input with a checked texture. To create a material for the rectangle:
Staying inside the Inspector, click the arrow to the left of Texture. This will create a patch representing the material’s texture.
Then create the camera texture:
It'll be listed as cameraTexture0 in the Assets panel. Drag cameraTexture0 from the Assets panel into the Patch Editor.
Finally, in the Patch Editor, connect the RGBA port to the yellow patch representing cam_tex_mat. Your graph will look like this:
You’ll see the live video feed back in the Simulator. We’ll add the zoom logic next.
Scaling the camera texture
To create the zoom effect, we need to scale the camera texture up and down dynamically as the hands are tracked. We’ll start with creating the patch graph to scale the texture. First add a Texture Transform and 2D transform pack pacth, then connect:
You won’t see any zoom effect in the camera because we haven’t scaled the camera texture up or down yet. To do this, we would need to edit the x and yScale values in the 2D Transform Pack patch. In the example below, we’re manually increasing the scale on the x and y axis by 0.3. In the next section we’ll show you how to do this dynamically, as each hand is tracked.
Dynamically scaling the camera texture
To scale the camera texture dynamically, we need to trigger an animation patch when a hand is found in the camera, connected to a transition patch, which will dynamically change the scale of the camera texture. We’re going to do this for one hand at a time, so start by locating the first Hand Bounding Box patch — the one in the green comment box.
Next add a pulse patch, an animation patch and a transition patch.
Then connect:
The Transition patch to the Scale input in the 2D Transform Pack patch.
Your graph will look like this:
Your effect looks like this:
You’ll notice that the zoom transition seems quite abrupt. We’ll add an exponential smoothing function between the transition and scaling to smooth it and make it more natural. First add:
Then connect:
Increasing the zoom effect with the second hand
At the moment the zoom effect only works for one hand. To create an increased zoom effect when the second hand is tracked, we need to set up some logic to check if the hand count in the hand finder patch is equal 2 and, if so, scale the camera texture to 1.6. First add an Equals Exactly patch and an If then Else patch. Then connect:
Edit the values
In the Equals Exactly patch, edit the Second Number value to 2. This means a boolean true signal will be sent to the If Then Else patch when the hand count in the Hand Finder patch is 2.
Finally in the If Then Else patch edit the Else value to 1.3 and the Then value to 1.6 on both axis. This means that if the condition is true, and there are two hands found, the x and y values of the camera texture will increase to 1.6. If only one hand is found the value increases to 1.3.
Your graph will look like this:
Here’s how your final effect will look!