Everything you need to know to create an interactive augmented reality experience.
Learn to create an effect with visual shader patches. In this tutorial, you'll take custom texture and use patches to transform it. You'll adding color and change its position, so it looks and moves across the face like a realistic beam of light:
In this guide, you'll learn about:
Download the sample content to follow along with this tutorial. If you open the unfinished project in the sample content folder, you'll see we've already imported the texture you'll need - it's listed in the Assets Panel as lightLeak_tex, and looks like a beam of light.
In the finished effect, a beam of light shines across the user's face. It isn't visible in the background because we've separated the foreground and background using segmentation.
Materials in both the foreground and background also have the camera texture applied to them. In the foreground, we'll combine segmentation and the camera texture with the light leak texture, and a gradient effect too. In the background, we'll just add a gradient.
In Spark AR Studio, segmentation effects are created using a texture called a segmentation mask texture. Start by inserting the 2 rectangles that will render the textures in the scene.
To help you keep track, rename:
Your Scene Panel should look like this:
Next, create a material for each rectangle:
Like before, it's worth renaming the materials. Rename them user_material and background_material, to correspond with the rectangles they're applied to.
Finally, resize the rectangles so they fill the screen of the device:
Now create the texture you'll use to add segmentation - the segmentation mask texture:
You'll see the texture in the Assets Panel, listed as segmentationMaskTexture0.
To render segmentation in the scene, apply this texture to both the materials you've created:
Your project should look like this:
The camera texture is a video of what's being captured by the camera when the effect is open on a device. Applying the camera texture will show the user and background in the effect again.
To create the camera texture:
You'll see the texture listed in the Assets Panel, as cameraTexture0.
You'll finish creating this material in the Patch Editor, so you can combine multiple textures in the same material. Open the Patch Editor, by selecting View in the menu bar, then Show/Hide Patch Editor.
Next create a patch to represent the Camera Texture. You can do this by dragging it from the Assets Panel, into the Patch Editor.
You'll also need to create a patch to represent the Diffuse Texture property of each material. You can do this by:
You'll see two new patches in the Patch Editor:
Next, connect the RGBA output in Camera Texture to the Diffuse Texture port in background_material.
If you change the video to your own camera, you'll see the background has appeared in the Simulator.
Now it's time to complete user_material, by combining it the camera texture and custom texture, using an Add patch.
The texture is listed in the Assets Panel as lightLeak_tex. Drag it into the Patch Editor to create a patch.
To combine this texture with the camera texture, you'll use an Add patch. Right-click in the Patch Editor, and select an Add patch from the menu. You'll need to to change the data type of the Add patch to Vector 2 - because we're working with 2D textures:
The graph should look like this:
You'll see the texture in the Simulator and Viewport:
Now let's add the gradient effect. You'll need to break some of the connections between patches you've already made. It's simple to do this - just click on a cable connecting one patch to another, and press delete on your keyboard.
Now create the patches you'll need for the gradient effect. Right-click in the Patch Editor and select:
To complete the gradient effect in the background of the scene, you'll need to make some edits to background_material. In the Inspector:
Next, make some adjustments to the Gradient Step patch:
The shape of the gradient will be set to Horizontal by default. For this effect, change the shape of the gradient by clicking on the patch, and selecting Circular from the menu.
Then, connect the output of the Add patch connected to the two orange texture patches, to the bottom input in the Add patch you've just created:
Then, connect the output of the Add patch connected to the two texture patches, to the bottom input in the Add patch you've just created:
The color effect is super bright:
To adjust this, change the values in the Multiply patches. Change:
We experimented to work out the best values for the Multiply patches. You could make the color strong or weaker by adjusting these numbers.
At this point, your project should look like this:
At the moment, the texture looks flat and unrealistic. To add realism, we'll make the light leak texture bend and distort relative to the camera texture, so it appears to fall in three dimensions across the face.
In the finished effect, we've grouped the collection of patches that distort and transform the texture. It's called Texture Distortion Shader:
When it's connected to the rest of the graph, it combines and distorts the camera texture and custom texture, adding realism to the effect. You can take a look at the patches that make up the Texture Distortion Shader by clicking the icon in the right corner of the patch group, to expand it.
This part of graph works by using Vertex Attribute patches to read UV values, which define how a texture is laid across a surface. In this case, the surface is the rectangles our materials are applied to. This graph will take those values - normalized from 0 to 1 horizontally and vertically. It will then offset them using RG and RGB values from the camera texture - using a Multiply patch to amplify the effect.
These UVs, distorted by the camera texture and applied to the light leak texture, will causes the beam of light to distort relative to the camera texture, by looking up pixels in the light leak texture using the Texture Sampler patch. This gives the effect of the light rolling across a 3D face, even though the distortion is happening based on the RGB values, not actual depth or 3D information.
So you can quickly build this part of the effect, we've already added most of the patches you'll need for the Texture Distortion Shader and connected them in a graph. The graph is listed as an asset in the Assets Panel, under Patches. It's called Distortion Shader:
Drag and drop the Distortion Shader from the Assets panel into the Patch Editor. Right-click on the patch group, and select Ungroup. The graph will look like this:
We still need to adjust the values in these patches, and connect them to the textures and materials, so you're still going to learn exactly what each patch is doing.
You'll need to create a couple more patches too. Right-click in the Patch Editor and select:
Connect the output of one Vertex Attribute patch to the UV input in the Texture Sampler at the start of the graph:
Now connect the output of the other Vertex Attribute patch to input in the Add patch that's connected to the UV input in the second Texture Sampler:
Then connect the output of the Multiply patch at the end of the graph, to the input in the Pack patch:
Next we'll connect these patches to the rest of the graph.
Like before, you'll need to break some connections again to do this. Disconnect the CameraTexture and LightLeak_tex patches from the rest of the graph.
You can delete the Add patch that was connected to lightLeak_tex - we don't need it anymore:
Go over to the right end of the graph, and connect the output port in the Pack patch to the remaining input port in the remaining Add patch. The end of the graph should look like this:
Now it's time to adjust the values in the patches.
Start with the Pack patch at the end of the graph. Change the input value in the second row to 1:
You'll see the color in the scene, but the textures are really dark.
So you can see the texture, adjust the value in the Multiply patch at the bottom of the graph. Change it to -0.1. Then, change the value in the Swizzle patch to xy.
Let's adjust the brightness now. Do this by changing the input value in the Multiply patch that sits between the Texture Transform and Add patches to 1.5.
Change the value in the Add patch connected to it to 0.5. The Add patch lifts the black levels of the light leak texture, so when it'is multiplied over the camera texture, it doesn't darken it too much.
Next let's flip the custom texture over adjust the rotation and scale of the texture in the 2D Transform Pack:
And there we have it - you've used visual shaders to combine and distort the camera texture and a custom texture, creating the appearance of a beam of light:
Now you have the texture in the scene, you're ready to add interaction - so the texture reacts to the user's face, and moves when the screen pans. You'll also add animation, to make the beam of wobble slightly from side to side.
This part of the effect will use the face tracker to detect the position of the face, the Screen Pan patch to detect screen interaction, and the Loop Animation patch to animate the texture.
The values these patches generate are high. If you connected, for example, the Screen Pan patch directly to the rest of the graph, the tiniest touch of the screen would make the texture move very dramatically.
To solve this problem, you'll normalize these values using a Divide patch. This will divide the values outputted by these patches by a high number, so they're much smaller.
You'll also create Add patches, which you'll use later to connect the animation patches and the Screen Pan patch to the rest of the graph.
Right-click in the Patch Editor and select 2 Add patches and a Divide patch.
Change the Type of the Add patches to Vector 2, because these patches will move the texture in 2D space - on the X and Y axes. Click on the patches, and change the Type to Vector 2.
Connect them, so your graph looks like this:
The face tracker will detect the movement of the user's face. This data will be used to drive the movement of the beam of light.
First you'll need to add a face tracker to your project:
To create a patch representing the face tracker, drag and drop it into the Patch Editor.
You'll also need to select an Unpack, Pack and Multiply patch from the menu.
You'll use Unpack to split the X,Y and Z data stream representing the 3D Position of the face into individual channels. The Pack patch converts this data in to a Vector 2 data stream - X and Y. This data can then be read by the 2D Transform Pack patch.
You'll edit the Multiply patch to amplify the effect of the movement of the face, causing the light leak texture to move a bit faster in line with the face's movement.
Like with other patches in this effect, change the Type of the Pack patch to Vector 2, and the Multiply patch to Point 2D.
Connect the patches, so your graph looks like this:
Finally, change the bottom values in the Multiply patch to -5 and -7, to amplify the movement of the face.
Next we'll add extra interest to the effect with an animation that makes the beam of light move slightly.
From the menu in the Patch Editor, select a:
Change the Type of the Transition patch to Point 2D.
The section of the graph should look like this:
To set up the animation, adjust the values in the Loop Animation and Transition patches:
In Loop Animation:
In Transition, adjust the values to set where the position on the screen that the light beam will move between, and how it will move:
Here's how the patches should look:
The light beam should now move slightly across the face. Just one more step now - adding an interactive element to the effect using the Screen Pan patch.
Right-click in the Patch Editor, and select a Screen Pan patch. This patch will detect when a user moves their finger along the screen in one continuous motion.
Connect the 2D Offset port in Screen Pan to the remaining port in the second Add patch:
You've now completed the effect!
Find inspiration, see examples, get support, and share your work with a network of creators.Join Community