Learn Spark AR Studio

Everything you need to know to create an interactive augmented reality experience.

Clear
Back

Background Segmentation

Use segmentation in Spark AR Studio to create an AR effect that separates someone from their background. You can then transform the environment behind them using textures and objects.

Segmentation can only identify people from the chest upwards, in the immediate foreground.

Segmentation only works on newer devices. iOS devices must be iPhone 6S or later. Android devices must be Samsung Galaxy S6, Sony Xperia Z2 or equivalent, or later.

Learn about:

Creating an effect with background segmentation

Add the rectangles to render segmentation in the scene

You'll need to add 2 rectangles to your scene. One will be used to render the user in the foreground of your scene, and the other will create the background.

First, add a canvas:

  1. Click Add Object.
  2. Select Canvas.

The canvas is always the same size as the device that's using the effect. You'll insert the 2 rectangles as children of the canvas, so they'll always be the right size for the device. To do this:

  1. Right-click on the canvas.
  2. Select Add.
  3. Select Rectangle.

Repeat these steps, so you have two rectangles in your scene. It's worth renaming the the rectangles, to help keep track of your project. To rename an object in Spark AR Studio, just right-click on the object and select Rename. Call the first rectangle in the list user, and the second background.

Your Scene Panel should look like this:



Next, change the size of the rectangles, so they're the same size as the canvas:

  1. Select each rectangle in the Scene Panel.
  2. In the Inspector, click each box next to Size and select Fill Width, and Fill Height.


Your project should look like this:

Adjust the layers

The background rectangle needs to be added to another layer, so it renders after the foreground rectangle. To do this:

  1. Select the background rectangle in the scene tab.
  2. Click the dropdown next to Layer in the inspector panel.
  3. Select Create New Layer.

In the layers tab, you can adjust the layer order to change the order they render in. The new layer will be set to render last by default - so you won't need to make any changes here.

Create materials for the rectangles

You'll apply a segmentation texture to the material on the user rectangle. You can add a color or your own textures to the material on the background rectangle.

To create the materials:

  1. Select the user rectangle in the Scene Panel.
  2. Click + next to Material in the Inspector.
  3. The material will be listed in the Assets Panel - rename the material user_material.

Repeat these steps for the background rectangle. This time, rename the material background_material.

Create a segmentation texture

The segmentation texture separates what the camera can see in the foreground, from the background. To create it:

  1. Select the Camera in the Scene Panel.
  2. Click + next to Segmentation in the Inspector.

Extract the camera texture

Extracting the camera texture allows you to use the video captured by the camera as the effect plays, and use it as a texture in the effect. For segmentation effects, you'll add this texture to the user material, to render the user in the scene. To create it:

  1. Select the Camera in the scene tab.
  2. Click + next to Texture Extraction in the Inspector.

Add the textures to the user material

Select the user material in the assets panel. In the Inspector:

  1. Change the Shader Type to Flat.
  2. Under Diffuse, click the dropdown next to Texture and select cameraTexture0.
  3. Check the box next to Alpha.
  4. Under Alpha, click the dropdown next to Texture and select segmentationMaskTexture0.

You should see the user in your scene:



Build the background material

Select background_material. You can make any edits you want to this material - for example, import your own texture or change the color. Below, we've edited the color under Diffuse:



Editing the segmentation texture

When the segmentation texture is selected in the assets panel, you can make changes how it works in your effect in the Inspector.

Edge Softness

Segmentation creates an outline around the person identified in the scene. Use Edge Softness to blur or soften this outline.

Mask Size

Adjust the outline of the person in the scene.

Optimization

We've found that effects with both segmentation and the face tracker don't perform well. It's best to avoid using these capabilities in the same effect.

If there are no 3D objects in your segmentation effect, you can remove the ambient and directional light that are included in Spark AR Studio projects by default.

Join the Spark AR Creator's community

Find inspiration, see examples, get support, and share your work with a network of creators.

Join Community

Frequently asked questions

Have a specific question? Maybe it's been answered.


Read FAQs
Was this document helpful?