Segmentation

A screenshot from Spark AR Studio showing a user who has been separated from their background

Have a question?

Ask the community

In this article we'll look at the different types of segmentation available, and how to edit how segmentation looks in your effects. We'll also walk through an example of how to implement person segmentation. To learn more about segmentation, try out our background segmentation and hair segmentaiton tutorials.

Keep in mind that not all devices support segmentation. Your effect will only be distributed to newer iOS and Android devices.

Limitations

There are some limitations to consider with segmentation effects. Segmentation in Spark AR Studio:

Can works best when detecting people from the chest upwards in the immediate foreground, in settings with balanced lighting.
Only works on newer devices. iOS devices must be iPhone 6S or later. Android devices must be Samsung Galaxy S6, Sony Xperia Z2 or equivalent, or later.

Adding segmentation

Add segmentation to an effect by applying a ‘segmentation texture’ to the Texture property of a material. Typically, you’ll then apply this material to a rectangle that covers the full width and height of the device screen.

There are 3 segmentation textures available in Spark AR Studio - one for each of the 3 segmentation types.

To create a segmentation texture:

In the Scene panel, select the Camera.
In the Inspector, click + to the right of Segmentation, and select either Hair, Person or Skin from the list.

The segmentation texture will be listed in the Assets panel. To change the type of segmentation texture:

Select the segmentation texture in the Assets panel.
In the Inspector, go to Type.
Select another option from the dropdown list.

The Type property being changed in the properties panel of a segmentation texture.

Segmentation texture - properties

In the Inspector, you can edit the following properties of a segmentation texture to change how it appears in your effect.

Edge Softness

Segmentation creates an outline around the person detected in the scene. Edit Edge Softness to soften this outline.

Mask Size

Adjust the size of the outline around the part of scene the texture is segmenting.

Type

Select whether the texture separates someone’s skin, hair or the whole person.

Example - person segmentation

Add rectangles to render segmentation in the scene

The first step is to add 2 rectangles to your scene. One will be used to render the user in the foreground and the other will create the background.

Start by adding a canvas:

Click Add Object.
Select Canvas.

The canvas is always the same size as the device that's using the effect. You'll insert the rectangles as children of the canvas, so they'll always be the right size for the device. To do this:

Right-click on the canvas.
Select Add.
Select Rectangle.

Repeat these steps, so you have two rectangles in your scene. It's worth renaming the the rectangles to help keep track. To rename an object in Spark AR Studio:

Right-click on the object.
Select Rename.
Call the first rectangle in the list user, and the second background.

Your Scene panel should look like this:

A screenshot showing two rectangles under a canvas in the Scene panel.

Next change the size of the rectangles, so they're the same size as the canvas:

Select each rectangle in the Scene panel.
In the Inspector, click each box next to Fill Width and Fill Height.

The size of each rectangle being changed in the Properties panel.

Your project should look like this:

A screenshot showing the two rectangles have filled the device simulator with a checkerboard pattern.

Adjust the layers

The background rectangle needs to be added to another layer, so it renders after the foreground rectangle. To do this:

Select the background rectangle in the Scene panel.
Click the dropdown next to Layer in the Inspector.
Select Create New Layer.

In the Layers tab you can adjust the order of the layers to change the order they render in. The new layer will be set to render last by default, so you won't need to make any changes here.

Create materials for the rectangles

You'll apply a segmentation texture to the material on the user rectangle. You can add a color or your own textures to the material on the background rectangle.

To create the materials:

Select the user rectangle in the Scene panel.
Click + next to Material in the Inspector.
The material will be listed in the Assets Panel - rename the material user_material.

Repeat these steps for the background rectangle. This time, rename the material background_material.

Create a segmentation texture

The segmentation texture separates what the camera can see in the foreground, from the background. To create it:

Select the Camera in the Scene panel.
Click + next to Segmentation in the Inspector.
Click Person.

Extract the camera texture

Extracting the camera texture allows you to use the video captured by the camera as the effect plays as a texture in the effect. You'll add this texture to the user_material, to render the user in the scene. To create it:

Select the Camera in the Scene panel.
Click + next to Texture Extraction in the Inspector.

Add the textures to the user material

Select the user material in the Assets panel. In the Inspector:

Change the Shader Type to Flat.
Under Diffuse, click the dropdown next to Texture and select cameraTexture0.
Check the box next to Alpha.
Under Alpha, click the dropdown next to Texture and select personsegmentationMaskTexture0.

You should see the user in your scene:

The user is now visible in the simulator.

Build the background material

Select background_material. You can make any edits you want to this material - for example, import your own texture or change the color. Below, we've edited the color under Diffuse:

The simulator now shows the user against a blue background.

Editing the segmentation texture

When the segmentation texture is selected in the Assets panel, you can make changes how it works in your effect in the Inspector.