In this effect we've used the Patch Editor to animate a 3D object with a skeleton and joints. The object looks like a teddy.
We've connected the object to the face tracker, so the teddy moves its body and face in sync with the movement of the person using the effect - like a virtual puppet.
We've also added an extra layer of interactivity, so the clothes the teddy is wearing change color and style when the screen is tapped:
Download the sample content to follow along with this tutorial. You can adapt any of the assets in this effect in your own projects.
If you want to build this effect yourself, open the unfinished effect in the sample content folder. We've already imported the assets you need, to help you get started quickly.
We've used patches to add a simple gradient to the background of the scene. You can change the color of the background by adjusting the Previous Step and New Color values in the Gradient Step patch.
We've also instantiated the 3D object - teddy - in the scene, as a child of a null object. We've adjusted the Transformations in the null object, to position the teddy correctly.
The teddy contains a skeleton with several joints. You can see the skeleton inside the object in the Viewport, it's looks like grey lines.
Finally, we've applied a texture and material to the teddy. The project looks like this:
The 3D Object
Before you start building the effect it's worth taking a look at the 3D Object. It's listed as teddy in the Scene panel:
The skeleton it contains is made up of:
Follow these steps to add procedural animation to the teddy using the Patch Editor.
Moving up and down
You'll use the Position properties in two of the joints - jack_base and spring_mid - to make the teddy move up and down:
A yellow patch will be created for the Position property of both joints. We chose the position property, because we'll use a Transition patch to set the values that the position of the joints move between, and the Loop Animation patch to animate this movement.
Right-click in the Patch Editor to open the menu. Select 2 Transition patches and a Loop Animation patch. Then connect the patches, so your graph looks like this:
You'll see the teddy is moving slightly. For the object to move correctly, you'll need to adjust the properties in the Transition and Loop Animation patches.
In the Loop Animation Patch:
In the Transition patch connected to jack_base:
It'll now move up and down from the base. You could experiment with adjusting these values to make the movement more or less dramatic.
In the other Transition patch:
Here's how the movement will look:
Moving side to side
You'll use the Rotation properties in sping_mid and jack_neck to make the puppet move from side to side.
Again, create 2 Transition patches and a Loop Animation patch. You'll should see these 5 patches:
You're going to connect a patch representing the face tracker to this part of the graph. This will make the movement of the teddy mirror the movement of the person using the effect, like a puppet. To create this patch:
Three patches will be added:
For this part of the graph you'll need to add 4 more patches from the menu in the Patch Edition: A Delay patch, a Multiply patch and and 2 Add patches.
You'll use the Delay patch to add a slight delay to the movement of one of the joints - to create a more natural effect. The Add patches will add the data from the face tracker to the movement of the teddy, while the Multiply patch will control how dramatic the movement of the teddy is.
Start by connecting:
Here's how the graph will look:
Now edit the values in the patches:
The teddy's head will be moving from side to side:
Next connect the remaining Transition and Add patches to the Loop Animation patch and the patch representing spring_mid joint.
In the Transition patch, set:
Finally, connect the face tracker. Connect the 3D Rotation output in the face tracker patch to the first input in the Add patch connected to jack_neck.
Then connect the 3D Rotation output in the face tracker patch to the first input in the Multiply patch. Like with the Add patches, make sure the Multiply patch is set to Vector 3. To finish this part of the graph:
Here's how the new section of the graph will look:
The teddy's head will now match the movement of the person using the effect. You test this out on yourself my changing the video in the Simulator to your own camera.
Raising the teddy's arms
This section of the graph will animate the movement of the teddy's arms.
It uses the Rotation property in the joints called wrist_r, wrist_l, arm_l and arm_r. These joints are children of the null objects listed under jack_base in the Scene panel.
So, like we did before, start by creating patches to represent the Rotation property of wrist_r, wrist_l, arm_l and arm_r.
From the menu in the Patch Editor, select:
Connect them, so your graph looks like this:
Again, make a few changes to the Transition patches:
In the Loop Animation patch:
In the next part of this tutorial, we'll connect this part of the graph to a patch that will detect an open mouth. As a result, this animation will become more dramatic when the user of the effect opens their mouth. That's why we need the Multiply patch - to multiply the signal from the Mouth Open patch, creating a more dramatic effect.
For now, in the Multiply patch change the lower input value to 30.
Now let's make the movement of the teddy's face match the movement of the user.
Responding to an open mouth
This part of the graph will make the teddy's mouth open, at the same time as the mouth of the person using the effect. It uses the Rotation property in the joint called mouth_lower, which is a child of jack_neck in the Scene panel:
From the menu in the Patch Editor, select:
Connect the patches, so the section of the graph looks like this:
The final step is to add values to the Multiply patch - to set how much the mouth should open. Set X on the bottom row of the Multiply patch to 20.
The teddy's mouth should open at the same time as the user:
Now we'll connect the Mouth Open patch to the part of the graph that animates the teddy's arms. Connect the Mouth Openness port to the top input in the Multiply patch created earlier. We've selected the patches you'll need to connect in the image below:
Making the teddy blink
Start by selecting an Eyelid patch from the menu in the Patch Editor. This patch captures data on the position of someone's eyelids - so it always needs to be connected to the Face port in the face tracker patch.
Next:
Connect the patches, so the section of the graph looks like this:
You'll notice we've only used the Left Openness port in the Eyelid patch. We've then used Subtract and Pack to make this signal drive the scale of both eyes. This is because if we used both the left and right eye openness ports, the eyes would likely appear to blink at very slightly different times, in sync with the person using the effect. We felt that for this effect making both eyes move at exactly the same time would look better.
You'll need to make some changes to the values in the patches:
The teddy's eyes should blink at the same time as yours.
Moving the eyebrows
This graph will make the teddy's eyebrows move at the same time as the person using the effect.
Select an Eyebrow patch from the menu in the Patch Editor. This patch captures data on the position of someone's eyebrows.
Next:
Connect the patches so the section of the graph looks like this:
For each Add patch, set the Y value to 0.07 and the Z value to 0.12. This sets the co-ordinates that each eyebrow moves between. You could experiment with changing these values for a more or less dramatic effect.
The teddy's eyebrows should move at the same time as yours. Now we can move on to the teddy's outfit!
This effect uses a patch group to change the number on the teddy's vest and hat, and the color of the outfit when the user taps the screen. You can test this out in the finished effect, by selecting Simulate Touch in the Simulator:
The different numbers that appear on the teddy's vest are textures, imported as a sequence. They're listed in the Assets panel under Animations, as inventory_tex.-animation.
The patch group is listed in the assets panel under Patches, it's called Teddy_outfit.
Drag the patch group into the Patch Editor to create a patch:
You can click the button in the right corner of the patch to expand the group.
If you want, you can change the colors in the Option Picker to choose different colors for the effect:
You'll notice the colors in the option picker are darker than the colors you can see in the finished effect. This is because the texture we'll apply to the teddy will be combined with these colors. The texture has a light area covering the teddy's shirt and hat, which makes the outfit lighter.
Click Back to Main in the top left of the Patch Editor to return to the rest of your graph:
The patch group has 2 input ports:
It has 2 outputs:
From the Assets panel, drag teddy_tex and inventory_tex into the Patch Editor. Two orange patches will be created, representing the texture and the animation sequence:
To create a patch representing the material we want to apply the texture to, select teddy_outfit_mat in the Assets panel. In the Inspector:
Next we need a patch representing the Current Frame property of the animation sequence. Select inventory_tex.-animation in the Assets panel. In the Inspector:
Connect the patches, so your graph looks like this:
If you tap the teddy in the Simulator, you'll see the outfit and color changes.
To complete the effect, add a little extra fun and interest with an audio clip. You’ll set it to play on a loop in the background of the effect:
We’ve already added the clip you’ll need, it’s listed in the Assets panel under Audio as Skeleton_Animation_Groove.m4:
Use an object called a speaker and an asset called the audio playback controller to render it in the scene. First add the speaker:
Next add the audio playback controller:
To render the audio clip in the scene, connect the audio file, audio playback controller and speaker:
The audio playback controller is where you’ll set how the audio file plays in the effect. To make it loop continuously:
The audio clip will now be playing! You might need to click the Refresh button in the toolbar to hear it.
You’ve now completed the effect!