Tutorial: Segmentation and the Camera Texture

In this tutorial you'll learn how to use segmentation and the camera texture to create an effect that separates someone from their background.

You'll use a segmentation mask texture to divide the foreground and background of the scene. You'll then apply the camera texture to a material in the foreground - to stream a video of the person using the effect back into the scene. You can then transform the environment behind them.

Something Went Wrong

We're having trouble playing this video.

Learn More

You don't need any custom assets to create this effect. All you need to do is open Spark AR Studio to get started.

Learn about:

• Creating a segmentation mask texture.
• Creating the camera texture.
• Making the background.
• Adding segmentation.

Creating a segmentation mask texture

Start by creating the segmentation mask texture. This texture separates the foreground - with the user in it - from their background.

To do this:

1. Select the Camera in the Scene panel.
2. In the Inspector, click + next to Segmentation.

You'll now see the texture in the Assets panel - it's called segmentationMaskTexture0.

Creating the camera texture

Next create the camera texture:

1. Make sure the Camera is selected in the Scene panel.
2. Click + next to Texture Extraction.

You'll see a texture called cameraTexture0 in the Assets panel.

Making the background

To render the user in the scene and create the background, you'll use two 2D objects called rectangles.

One rectangle will have a material with the camera texture and segmentation mask texture applied to it, creating the foreground. The other will have a single material applied to it, creating a background.

Rectangles are always children of a canvas. In this effect, this will mean the rectangles rescale with the screen of the device.

Start by making the rectangle that will form the background:

1. Click Add Object at the bottom of the Scene panel.
2. Select a rectangle from the menu.

It's a good idea to rename the rectangle, to help you keep track. To do this just:

1. Right-click on the rectangle.
2. Select rename.
3. Call it background_rectangle.

In the Viewport you can see that the rectangle is quite small, and positioned in the middle of the scene. We want it to fill the whole canvas and therefore the full screen of the device.

To do this, go over to the Inspector panel. Next to Size you'll see two options. You can use the first one to set the width of the rectangle, and the second one to set the height. Click in the first box and select the option for Fill Width.

Do the same for the height option:

1. Click in the second box.
2. Select Fill Height.

Next create a material for the rectangle. At the bottom of the Inspector:

1. Click + next to Materials.
2. Select Create New Material.

The material will be listed in the Assets panel. Right-click on the material and rename it - we suggest background_material.

You can achieve all kinds of effects with segmentation - for example using your own custom textures for the background. For this tutorial, we're going to keep it simple and pick a color.

To do this, select the material. In the Inspector panel, change the Shader Type to Flat.

Then, under Diffuse:

1. Click the box next to Color.
2. Choose a color for the background from the menu - I'm going to pick blue.
3. Click OK.

Adding segmentation

Now let's build the foreground, with the user in it.

Right-click on the canvas and add another rectangle. Like before, rename the rectangle - go with something like user_material. In the Inspector, again select Fill Width and Fill Height next to Size.

Create a material for the rectangle and select it in the Assets panel - it'll be listed as Material0. Like before, rename it. You could choose something like user_material. Over in the Inspector change the Shader Type to Flat

Then, under Diffuse:

1. Click the dropdown next to Texture.
2. Select cameraTexture0 from the list.

The user has appeared in the scene, but we can't see the background. This is because we haven't applied the segmentation mask texture yet.

To apply the segmentation mask texture:

1. Check the box next to Alpha.
2. Next to Texture, click the dropdown and select the segmentation mask texture.

Learn more

• Build a frame with 2D objects.
• Combine segmentation with particles to create a firefly effect.