Learn Spark AR Studio

Everything you need to know to create an interactive augmented reality experience.

Clear
Back

Sample Effect: Visual Shaders

Learn to create an effect with visual shader patches. In this tutorial, you'll take custom texture and use patches to transform it. You'll adding color and change its position, so it looks and moves across the face like a realistic beam of light.

Download the sample content to follow along with this tutorial. If you open the unfinished project in the sample content folder, you'll see we've already imported the texture you'll need - it's listed in the Assets panel as lightLeak_tex, and looks like a beam of light.

In this guide, you'll learn about:

Adding the custom texture, camera texture and segmentation

In the finished effect, a beam of light shines across the user's face. It isn't visible in the background because we've separated the foreground and background using segmentation.

Materials in both the foreground and background also have the camera texture applied to them. In the foreground, we'll combine segmentation and the camera texture with the light leak texture, and a gradient effect too. In the background, we'll just add a gradient.

Adding segmentation

In Spark AR Studio, segmentation effects are created using a texture called a segmentation mask texture. Start by inserting the 2 rectangles that will render the textures in the scene.

  1. Click + in the menu bar, and select a Rectangle. It'll automatically be inserted as a child of the canvas. The canvas ensures rectangles scale up and down with the size of the device screen.
  2. Right-click the canvas, select Insert then Rectangle.

To help you keep track, rename:

  1. The first rectangle in the list background.
  2. The second rectangle in the list user.

Your Scene tab should look like this:



Like before, it's worth renaming the materials. Rename them user_material and background_material, to correspond with the rectangles they're applied to.

Finally, resize the rectangles so they fill the screen of the device:

  1. Select the rectangles - you can select them both at the same time by holding command on your keyboard.
  2. Click Fill Parent in the Inspector panel.

Next, create the segmentation mask texture:

  1. Select the Camera in the Scene tab - it's listed underneath Device.
  2. In the Inspector panel, click + next to Segmentation Mask Texture.

You'll see the texture in the Assets panel, listed as Segmentation Mask Texture.

To render segmentation in the scene, apply this texture to both the materials you've created:

  1. Select the material in the Assets panel.
  2. In the Inspector panel, tick the box next to Alpha.
  3. Click the dropdown next to Texture, and select Segmentation Mask Texture from the list.

Your project should look like this:



The camera texture

The camera texture is a video of what's being captured by the camera when the effect is open on a device. Applying the camera texture will show the user and background in the effect again.

To create the camera texture:

  1. Select the Camera in the Scene tab.
  2. In the Inspector panel, click + next to Texture Extraction.

You'll see the texture listed in the Assets panel, as CameraTexture.

You'll finish creating this material in the Patch Editor, so you can combine multiple textures in the same material. Open the Patch Editor, by selecting View in the menu bar, then Patch Editor.

Next create a patch to represent the Camera Texture. You can do this by dragging it from the Assets panel, into the Patch Editor.



You'll also need to create a patch to represent the Diffuse Texture property of each material. You can do this by:

  1. Selecting the material in the Assets panel.
  2. Clicking the circle next to Texture in the Inspector panel:


Repeat this step - so you've created a patch for both the user_material and the background_material.

Next, connect the RGBA output in Camera Texture to the Diffuse Texture port in background_material.

If you change the video to your own camera, you'll see the background has appeared in the Simulator.

Adding the texture

Now it's time to complete user_material, by combining it the camera texture and custom texture, using an Add patch.

The texture is listed in the assets panel as lightLeak_tex. Drag it into the Patch Editor to create a patch.

Right-click in the Patch Editor, and select an Add patch from the menu.

Connect:

  1. The RGBA port in the CameraTexture and lightLeak_tex patches to the Add patch.
  2. The output in the Add patch to the Diffuse Texture port in user_material.

Your project should look like this:

Adding a color gradient

Now let's add the gradient effect. You'll need to break some of the connections between patches you've already made. It's simple to do this - just click on a port, and press delete on your keyboard.

Disconnect:

  1. The connection between Camera Texture and the Diffuse Texture port in background_material.
  2. The connection between the Add patch and the Diffuse Texture port in user_material:


Now create the patches you'll need for the gradient effect. Right-click in the Patch Editor and select:

  1. A Gradient and Gradient Step patch.
  2. 2 Multiply patches.
  3. An Add patch.

You'll use:

  1. The Multiply patches to multiply the strength of the color gradient.
  2. The Add patch to combine the gradient with the camera texture and custom texture we've already added.

So, connect:

  1. The output of the purple Gradient patch to the input of the Gradient Step patch.
  2. The Output port in the Gradient Step patch to the input port in each Multiply patch.
  3. The output of one Multiply patch to the Diffuse Texture port in the background material patch:


Next, make some adjustments to the Gradient Step patch:

  1. Change the color next to Previous Step to a pale yellow and the color next to New Color to white. You could pick different colors for a different effect.
  2. Set the Start Range to 0.6.

The shape of the gradient will be set to Horizontal by default. For this effect, change the shape of the gradient by clicking on the patch, and selecting Circular from the menu.

Then, connect the output of the Add patch connected to the two orange texture patches, to the bottom input in the Add patch you've just created:



Connect:

  1. The output of the second Add patch to the Diffuse Texture input in user_material.
  2. The output of the remaining Multiply patch to the input in the second Add patch.


After connecting these patches, you'll see the color in the effect is super bright. To adjust this, change the values in the Multiply patches. Change:

  1. The value in the Multiply patch at the top of the graph to 0.7.
  2. The value in the Multiply patch at the bottom of the graph to 0.2.

We experimented to work out the best values for the Multiply patches. You could make the color strong or weaker by adjusting these numbers.

At this point, your project should look like this:



At the moment, the texture looks flat and unrealistic. To add realism, we'll make the light leak texture bend and distort relative to the camera texture, so it appears to fall in three dimensions across the face.

Distorting the Texture

In the finished effect, we've grouped the collection of patches that distort and transform the texture. It's called Texture Distortion Shader:



When it's connected to the rest of the graph, it combines and distorts the camera texture and custom texture, adding realism to the effect. You can take a look at the patches that make up the Texture Distortion Shader by clicking the icon in the right corner of the patch group, to expand it.

This part of graph works by using Vertex Attribute patches to read UV values, which define how a texture is laid across a surface. In this case, the surface is the rectangles our materials are applied to. This graph will take those values - normalized from 0 to 1 horizontally and vertically. It will then offset them using RG and RGB values from the camera texture - using a Multiply patch to amplify the effect.

These UVs, distorted by the camera texture and applied to the light leak texture, will causes the beam of light to distort relative to the camera texture, by looking up pixels in the light leak texture using the Texture Sampler patch. This gives the effect of the light rolling across a 3D face, even though the distortion is happening based on the RGB values, not actual depth or 3D information.

So you can quickly build this part of the effect, we've already added most of the patches you'll need for the Texture Distortion Shader and connected them in a graph. The graph is listed as an asset in the Assets panel, under Patches. It's called Distortion Shader:



Drag and drop the Distortion Shader from the Assets panel into the Patch Editor. Right-click on the patch group, and select Ungroup. The graph will look like this:



We still need to adjust the values in these patches, and connect them to the textures and materials, so you're still going to learn exactly what each patch is doing.

You'll need to create a couple more patches too. Right-click in the Patch Editor and select:

  1. 2 Vertex Attribute patches.
  2. A Pack patch.

Change:

  1. The value in the Vertex Attribute patches to Texture Coordinates, to read the texture coordinates from the rectangles.
  2. The Type in the Pack patch to Vector 2.

Connect:

  1. The output of one Vertex Attribute patch to the UV input in the Texture Sampler at the start of the graph.
  2. The output of the other Vertex Attribute patch to input in the Add patch that's connected to the UV input in the second Texture Sampler.
  3. The output of the Multiply patch at the end of the graph, to the input in the Pack patch:


Next we'll connect these patches to the rest of the graph.

Connecting the camera texture and custom texture

Like before, you'll need to break some connections again to do this. Disconnect the CameraTexture and LightLeak_tex patches from the rest of the graph.

You can delete the Add patch that was connected to lightLeak_tex - we don't need it anymore.

Instead, connect:

  1. The RGB port in lightLeak_tex to the Texture port in the Texture Transform patch.
  2. The RGB port in CameraTexture to both the Texture input in the Texture Sampler patch, and the top input in the final Multiply patch in the graph:

Connecting the materials

Go over to the right end of the graph, and connect the output port in the Pack patch to the remaining input port in the remaining Add patch. The end of the graph should look like this:



Now it's time to adjust the values in the patches.

Adjusting values

Start with the Pack patch at the end of the graph. Change the input value in the second row to 1.

You'll see the color in the scene, but the textures are really dark and the beam of light isn't visible.



So you can see the texture, adjust the value in the Multiply patch at the bottom of the graph. Change it to -0.1. Then, change the value in the Swizzle patch to xy.

Let's adjust the brightness now. Do this by changing the input value in the Multiply patch that sits between the Texture Transform and Add patches to 1.5.

Change the value in the Add patch connected to it to 0.5. The Add patch lifts the black levels of the light leak texture, so when it'is multiplied over the camera texture, it doesn't darken it too much.



Next let's flip the custom texture over adjust the rotation and scale of the texture in the 2D Transform Pack:

  1. Next to Translation, change X to -0.09 and Y to 0.013.
  2. Next to Scale, change X to -1.1.
  3. Change the Rotation to 180.
  4. Next to Pivot change X to -0.5 and Y to 0.5.

And there we have it - you've used visual shaders to combine and distort the camera texture and a custom texture, creating the appearance of a beam of light:

Normalizing values

Now you have the texture in the scene, you're ready to add interaction - so the texture reacts to the user's face, and moves when the screen pans. You'll also add animation, to make the beam of wobble slightly from side to side.

This part of the effect will use the face tracker to detect the position of the face, the Screen Pan patch to detect screen interaction, and the Loop Animation patch to animate the texture.

The values these patches generate are high. If you connected, for example, the Screen Pan patch directly to the rest of the graph, the tiniest touch of the screen would make the texture move very dramatically.

To solve this problem, you'll normalize these values using a Divide patch. This will divide the values outputted by these patches by a high number, so they're much smaller.

You'll also create Add patches, which you'll use later to connect the animation patches and the Screen Pan patch to the rest of the graph.

Right-click in the Patch Editor and select 2 Add patches and a Divide patch.

Change the Type of the Add patches to Type 2D, because these patches will move the texture in 2D space - on the X and Y axes. Click on the patches, and change the Type to 2D.

Connect them, so your graph looks like this:



Then:

  1. Connect the output of the Divide patch to the Translation input in the 2D Transform Pack patch.
  2. Change the value in the Divide patch to 1000.

Adding the face tracker

The face tracker will detect the movement of the user's face. This data will be used to drive the movement of the beam of light.

First you'll need to add a face tracker to your project:

  1. Click Insert in the Menu bar.
  2. Select Face Tracker from the menu.

To create a patch representing the face tracker, drag and drop it into the Patch Editor.

You'll also need to select an Unpack, Pack and Multiply patch from the menu.

You'll use Unpack to split the X,Y and Z data stream representing the 3D Position of the face into individual channels. The Pack patch converts this data in to a Vector 2 data stream - X and Y. This data can then be read by the 2D Transform Pack patch.

You'll edit the Multiply patch to amplify the effect of the movement of the face, causing the light leak texture to move a bit faster in line with the face's movement.

Like with other patches in this effect, change the Type of the Pack patch to Vector 2, and the Multiply patch to Point 2D.

Connect the patches, so your graph looks like this:



Finally, change the bottom values in the Multiply patch to -5 and -7, to amplify the movement of the face.

Adding the animation

Next we'll add extra interest to the effect with an animation that makes the beam of light move slightly.

From the menu in the Patch Editor, select a:

  1. Loop Animation patch.
  2. Transition patch.

Change the Type of the Transition patch to Point 2D.

Connect:

  1. The Progress port in the Loop Animation patch to the Progress port in the Transition patch.
  2. The output in the Transition patch to the lower input in the Add patch.

The section of the graph should look like this:



To set up the animation, adjust the values in the Loop Animation and Transition patches:

In Loop Animation:

  1. Check the box next to Enable.
  2. Change Duration to 3.
  3. Check the box next to Mirrored.

In Transition, adjust the values to set where the position on the screen that the light beam will move between, and how it will move:

  1. Next to Start, set X to -50 and Y to -120.
  2. Next to End, set X to 50 and Y to -150.
  3. Change the Curve to Quadratic In-Out.

The light beam should now move slightly across the face. Just one more step now - adding an interactive element to the effect using the Screen Pan patch.

Adding interactivity

Right-click in the Patch Editor, and select a Screen Pan patch. This patch will detect when a user moves their finger along the screen in one continuous motion.

Connect the 2D Offset port in Screen Pan to he remaining port in the second Add patch:



You've now completed the effect!

Join the Spark AR Creator's community

Find inspiration, see examples, get support, and share your work with a network of creators.

Join Community

Frequently asked questions

Have a specific question? Maybe it's been answered.


Read FAQs