Use segmentation in Spark AR Studio to detect and separate parts of the camera input. For example, to separate someone’s hair from the rest of the scene - so you can change the colour without changing anything else in the effect.
There are 3 types of segmentation available:
You can combine all 3 types of segmentation to achieve a more complex effect.
In this article we'll look at the different types of segmentation available, and how to edit how segmentation looks in your effects. We'll also walk through an example of how to implement person segmentation. To learn more about segmentation, try out our background segmentation and hair segmentaiton tutorials.
Keep in mind that not all devices support segmentation. Your effect will only be distributed to newer iOS and Android devices.
There are some limitations to consider with segmentation effects. Segmentation in Spark AR Studio:
Add segmentation to an effect by applying a ‘segmentation texture’ to the Texture property of a material. Typically, you’ll then apply this material to a rectangle that covers the full width and height of the device screen.
There are 3 segmentation textures available in Spark AR Studio - one for each of the 3 segmentation types.
To create a segmentation texture:
The segmentation texture will be listed in the Assets panel. To change the type of segmentation texture:
In the Inspector, you can edit the following properties of a segmentation texture to change how it appears in your effect.
Segmentation creates an outline around the person detected in the scene. Edit Edge Softness to soften this outline.
Adjust the size of the outline around the part of scene the texture is segmenting.
Select whether the texture separates someone’s skin, hair or the whole person.
Add rectangles to render segmentation in the scene
The first step is to add 2 rectangles to your scene. One will be used to render the user in the foreground and the other will create the background.
Start by adding a canvas:
The canvas is always the same size as the device that's using the effect. You'll insert the rectangles as children of the canvas, so they'll always be the right size for the device. To do this:
Repeat these steps, so you have two rectangles in your scene. It's worth renaming the the rectangles to help keep track. To rename an object in Spark AR Studio:
Your Scene panel should look like this:
Next change the size of the rectangles, so they're the same size as the canvas:
Your project should look like this:
Adjust the layers
The background rectangle needs to be added to another layer, so it renders after the foreground rectangle. To do this:
In the Layers tab you can adjust the order of the layers to change the order they render in. The new layer with our background rectangle (layer1) needs to be below the layer with our user rectangle (layer0):
Create materials for the rectangles
You'll apply a segmentation texture to the material on the user rectangle. You can add a color or your own textures to the material on the background rectangle.
To create the materials:
Repeat these steps for the background rectangle. This time, rename the material background_material.
Create a segmentation texture
The segmentation texture separates what the camera can see in the foreground, from the background. To create it:
Extract the camera texture
Extracting the camera texture allows you to use the video captured by the camera as the effect plays as a texture in the effect. You'll add this texture to the user_material, to render the user in the scene. To create it:
Add the textures to the user material
Select the user material in the Assets panel. In the Inspector:
You should see the user in your scene:
Build the background material
Select background_material. You can make any edits you want to this material - for example, import your own texture or change the color. Below, we've edited the color under Diffuse:
When the segmentation texture is selected in the Assets panel, you can make changes how it works in your effect in the Inspector.
Segmentation creates an outline around the person detected in the scene. Use Edge Softness to blur or soften this outline.
Adjust the outline of the person in the scene.
Similar changes to softness and mask size can be made in the Patch Editor using a Segmentation Processor patch group.