- Meta Spark Studio
- Meta Spark Hub
- Features
- Learn
- Programs
- Blog
- Community
- Create AR effects for Instagram and Meta.Get Started
2D Body Tracking
In Meta Spark Studio, use 2D body tracking to create effects that move with one or multiple user's bodies. There are two ways to implement this depending on which key points of the body you want to track. You can use either:
- The Body Landmark patches. In this implementation, the camera tracks specific parts of the user’s body (upto 20 joints in total). You could use this to create line trail effects where particles emit from body key points. Take a look at how we've used body landmark patches to create a tattoo effect in this tutorial.
- The Body Bounding Box patch. In this implementation, the camera tracks the top and bottom corners of the body creating an invisible rectangle. The bounding box can be used to create body swapping effects and more.
Both methods also use the Body Finder and Body Select patches. All these patches are found in the patch menu. If you’re new to working with patches, take a look at the patch editor overview.
Set up body tracking
When creating a body tracking effect, it’s a good idea to change the Simulator video to one that shows an entire person. You can also select your computer’s built-in camera to see the effect on yourself.
Add and connect the body tracking patches
The first two patches you need to create a body tracking effect are available in the Patch Menu. They are:
- Body Finder — This patch represents the body/bodies detected by the camera.
- Body Select — Adjust the value in this patch to select a specific body detected by the Body Finder patch
Once you’ve connected these patches as in the example above, you need to connect the Body output of Body Select to the input of either a Body Bounding Box patch or one of the six Body Landmarks patches (Head 2D, Neck 2D, Right Arm 2D, Left Arm 2D, Right Leg 2D, Left Leg 2D).
The example graph below shows all seven patches connected to the output of the Body Select patch. This means all possible body key points are being tracked.
Take a look at the examples below to learn how to create effects that respond to these tracked points.
Instructions
If your effect has body tracking patches in it, the automatic instructions capability is added to your project by default and an instruction saying Stand in View (front facing camera) or Find a Friend (back facing camera) will appear until a body is detected by the camera. You can turn off these automatic instructions and add custom instructions to your effect if you prefer.
Example 1 — Attach 2D textures/text to the position of the user's joints
In this example, we’ve created the effect of red tracking points following specific parts of the body. We did this by attaching 2D rectangles to the 20 key points of the body.
You can customize this effect by adding your own materials and 2D textures to the rectangles. You could also make 2D text follow the body instead.
Add the canvas and rectangles
To build this effect we first added a canvas. We then added 20 resized rectangles, each with a red material applied. We grouped the rectangles under the canvas and renamed them to reflect where they’ll be positioned on the body.
Add the body tracking patches
Next, we added the Body Finder, Body Select and the six Body Landmarks patches. We connected:
- The Bodies output of the Body Finder patch to the Bodies input of the Body Select patch.
- The Body output of the Body Select patch to the input of the six Body Landmarks patches.
Finally, we connected each Body Landmark patch output to the yellow patches representing the corresponding positions of the rectangles. Here’s how the final patch graph looks:
Next steps
You might want to create an effect where 3D objects, like a particle system or a rigged 3D model follows the user's body. You can do this by attaching 3D planes to the Body Landmark patches instead of 2D rectangles. This implementation is more complex because you need to add additional patches to your graph to convert the 2D data output from the body landmarks patches to 3D data
We’ve set these patches up for you already in the Body Movement Template available in the Meta Spark Studio welcome window. It’s a good idea to customize this template to easily set up complex body tracking effects.
Example 2 — Create a shape around the body that moves and resizes with the person’s movement
In this example we’ve used the body bounding box patch to create a rectangle around the body that moves and resizes as the person moves.
Add the canvas and rectangle
To build this effect we first added a canvas and a rectangle. The rectangle is automatically added to the top left because this is where the default pivot point is located in Meta Spark Studio.
Edit the rectangle
We then applied a flat, cyan-colored material to the rectangle and edited the render options to get the transparent look. We chose the following render options:
- Blend mode: Screen
- Opacity: 70
Add the body tracking patches
Next we added the Body Finder, Body Select and Body Bounding Box patches. We connected:
- The Bodies output of the Body Finder patch to the Bodies input of the Body Select patch.
- The Body output of the Body Select patch to the input of the Body Bounding Box.
Make the rectangle move with the user
To make the rectangle move with the user, the rectangle’s position must be defined by the bounding box coordinates. To do this, we:
- Created a patch representing the 2D position of the rectangle
- Connected the patch to the Top Left 2D Coordinates of the Body Bounding Box patch. We chose the top left coordinates to align with the Meta Spark default pivot point.
Make the rectangle resize with the user
To make the rectangle resize based on the user, the height and width of the rectangle must align with these same properties of the bounding box. To do this, we:
- Created patches representing the Width and Height properties of the rectangle.
- Connected these to the Width and Height outputs of the Body Bounding Box patch.
Here’s how the final graph looks:
Limitations
There are some limitations to consider with body tracking effets. Keep in mind that:
- Certain effects require the user to stand further away so the camera can capture and track your whole body and joints.
- The user’s face needs to be in the device frame.
- People in the scene mustn't overlap significantly. The tracker is able to track multiple bodies in a scene but certain limited cases, such as a person leaving and coming back or two people overlapping significantly can cause the effect rendered on each body to swap or seem as if one body isn’t being tracked.
Tracking is weaker or lost when the person:
- Moves their legs higher than 30 degrees.
- Places one leg behind the other.
- Is turned to the side instead of facing the camera.
- Has joints hidden/occluded behind other objects.
- Tilts their head up at extreme angles.
