This AR effect uses visual shader patches to create the appearance of a realistic beam of light falling across the face.
Background segmentation separates the person using the effect from the background of the scene, so the light appears on the face and not the background.
Download the sample content to follow along with this tutorial. If you open the unfinished project in the sample content folder, you'll see we've already imported the texture you'll need - it's listed in the Assets panel as lightLeak_tex.
In the finished effect, a beam of light shines across the user's face. It isn't visible in the background because we've separated the foreground and background using segmentation.
Materials in both the foreground and background also have the camera texture applied to them. In the foreground, we'll combine segmentation and the camera texture with the custom texture and a gradient effect too. In the background, we'll just add a gradient.
In Meta Spark Studio, segmentation effects are created using a texture called a segmentation mask texture. Start by inserting the 2 rectangles that will render the textures in the scene.
To help you keep track, rename:
Your Scene panel should look like this:
Next, create a material for each rectangle:
Like before, it's worth renaming the materials. Rename them user_material and background_material, to correspond with the rectangles they're applied to.
Next resize the rectangles so they fill the screen of the device:
Now create the segmentation mask texture:
You'll see the texture in the Assets panel, listed as segmentationMaskTexture0.
Apply this texture to both the materials you've created:
Your project should look like this:
The camera texture
The camera texture is a video of what's being captured by the camera when the effect is open on a device. Applying the camera texture will show the user and background in the effect again.
To create the camera texture:
You'll see the texture listed in the Assets panel as cameraTexture0.
You'll finish creating this material in the Patch Editor, so you can combine multiple textures in the same material. Open the Patch Editor by selecting View in the menu bar, then Show/Hide Patch Editor.
Next create a patch to represent the Camera Texture. You can do this by dragging it from the Assets panel, into the Patch Editor.
You'll also need to create a patch to represent the Diffuse Texture property of each material. You can do this by:
You'll see two new patches in the Patch Editor:
Next, connect the RGBA output in Camera Texture to the Diffuse Texture port in background_material.
If you change the video to your own camera, you'll see the background has appeared in the Simulator.
Adding the texture
Now it's time to complete user_material, by combining it with the camera texture and custom texture. Use an Add patch to do this.
The texture is listed in the Assets panel as lightLeak_tex. Drag it into the Patch Editor to create a patch.
Right-click in the Patch Editor, and select an Add patch from the menu. You'll need to to change the data type of the Add patch to Vector 4 - because there are 4 values to pass into the user material - R, G, B and A:
The graph should look like this:
You'll see the texture in the Simulator and Viewport:
Now let's add the gradient effect. You'll need to break some of the connections between patches you've already made. It's simple to do this - just click on a cable connecting one patch to another, and press delete on your keyboard.
Now create the patches you'll need for the gradient effect. Right-click in the Patch Editor and select:
To complete the gradient effect in the background of the scene, you'll need to make some edits to background_material. In the Inspector:
Next make some adjustments to the Gradient Step patch:
The shape of the gradient will be set to Horizontal by default. Change the shape of the gradient:
Then connect the output of the Add patch connected to the two orange texture patches, to the bottom input in the Add patch you've just created:
Then, connect the output of the Add patch connected to the two texture patches, to the bottom input in the Add patch you've just created:
The color effect is super bright:
Change the values in the Multiply patches to fix this. Change:
We experimented to work out the best values for the Multiply patches. You could make the color stronger or weaker by adjusting these numbers.
At this point, your project should look like this:
At the moment the texture looks flat and unrealistic. To add realism, we'll make the light leak texture bend and distort relative to the camera texture, so it appears to fall in three dimensions across the face.
In the finished effect we've grouped the collection of patches that distort and transform the texture. It's called Texture Distortion Shader:
When it's connected to the rest of the graph it combines and distorts the camera texture and custom texture, adding realism to the effect. You can take a look at the patches that make up the Texture Distortion Shader by clicking the icon in the right corner of the patch group.
This part of graph works by using Vertex Attribute patches to read UV values, which define how a texture is laid across a surface. In this case, the surface is the rectangles our materials are applied to. This graph will take those values - normalized from 0 to 1 horizontally and vertically. It will then offset them using RG and RGB values from the camera texture - using a Multiply patch to amplify the effect.
These UVs, distorted by the camera texture and applied to the light leak texture, will causes the beam of light to distort relative to the camera texture, by looking up pixels in the light leak texture using the Texture Sampler patch. This gives the effect of the light rolling across a 3D face, even though the distortion is happening based on the RGB values, not actual depth or 3D information.
So you can quickly build this part of the effect, we've already added most of the patches you'll need for the Texture Distortion Shader and connected them in a graph. The graph is listed as an asset in the Assets panel, under Patches. It's called Distortion Shader:
Drag and drop the Distortion Shader from the Assets panel into the Patch Editor. Right-click on the patch group, and select Ungroup. The graph will look like this:
We need to adjust the values in these patches and connect them to the textures and materials, so you're still going to learn exactly what each patch is doing.
You'll need to create a couple more patches too. Right-click in the Patch Editor and select:
Connect the output of one Vertex Attribute patch to the UV input in the Texture Sampler at the start of the graph:
Now connect the output of the other Vertex Attribute patch to input in the Add patch that's connected to the UV input in the second Texture Sampler:
Then connect the output of the Multiply patch at the end of the graph to the input in the Pack patch:
Next we'll connect these patches to the rest of the graph.
Connecting the camera texture and custom texture
Like before, you'll need to break some connections again to do this. Disconnect the CameraTexture and LightLeak_tex patches from the rest of the graph.
You can delete the Add patch that was connected to lightLeak_tex - we don't need it anymore:
Connecting the materials
Go over to the right end of the graph, and connect the output port in the Pack patch to the remaining input port in the remaining Add patch. The end of the graph should look like this:
Now it's time to adjust the values in the patches.
Start with the Pack patch at the end of the graph. Change the input value in the second row to 1:
You'll see the color in the scene, but the textures are really dark.
So you can see the texture, adjust the value in the Multiply patch at the bottom of the graph. Change it to -0.1. Then, change the value in the Swizzle patch to xy.
Let's adjust the brightness now. Do this by changing the input value in the Multiply patch that sits between the Texture Transform and Add patches to 1.5.
Change the value in the Add patch connected to it to 0.5. The Add patch lifts the black levels of the light leak texture, so when it's multiplied over the camera texture, it doesn't darken it too much.
Next let's flip the custom texture over. To do this, adjust the rotation and scale of the texture in the 2D Transform Pack:
And there we have it - you've used visual shaders to combine and distort the camera texture and a custom texture, creating the appearance of a beam of light:
Now you have the texture in the scene, you're ready to add interaction - so the texture reacts to the user's face and moves when the screen pans. You'll also add animation to make the beam of wobble slightly from side to side.
This part of the effect will use the face tracker to detect the position of the face, the Screen Pan patch to detect screen interaction, and the Loop Animation patch to animate the texture.
The values these patches generate are high. If you connected, for example, the Screen Pan patch directly to the rest of the graph, the tiniest touch of the screen would make the texture move very dramatically.
To solve this problem, you'll normalize these values using a Divide patch. This will divide the values outputted by these patches by a high number, so they're much smaller.
You'll also create Add patches, which you'll use later to connect the animation patches and the Screen Pan patch to the rest of the graph.
Right-click in the Patch Editor and select 2 Add patches and a Divide patch.
Change the Type of the Add patches to Vector 2, because these patches will move the texture in 2D space - on the X and Y axes.
Connect them, so your graph looks like this:
The face tracker will detect the movement of the user's face. This data will be used to drive the movement of the beam of light.
First you'll need to add a face tracker to your project:
To create a patch representing the face tracker, drag and drop it into the Patch Editor.
You'll also need to select an Unpack, Pack and Multiply patch from the menu.
You'll use Unpack to split the X,Y and Z data stream representing the 3D Position of the face into individual channels. The Pack patch converts this data in to a Vector 2 data stream - X and Y. This data can then be read by the 2D Transform Pack patch.
You'll edit the Multiply patch to amplify the effect of the movement of the face, causing the light leak texture to move a bit faster in line with the face's movement.
Like with other patches in this effect, change the Type of the Pack and Multiply patches to Vector 2.
Connect the patches, so your graph looks like this:
Finally, change the bottom values in the Multiply patch to -5 and -7, to amplify the movement of the face.
Next we'll add extra interest to the effect with an animation that makes the beam of light move slightly.
From the menu in the Patch Editor, select a:
Change the Type of the Transition patch to Vector 2.
The section of the graph should look like this:
To set up the animation, adjust the values in the Loop Animation and Transition patches:
In Loop Animation:
In Transition, adjust the values to set the position on the screen that the light beam will move between, and how it will move:
Here's how the patches should look:
The light beam should now move slightly across the face.
Right-click in the Patch Editor and select a Screen Pan patch. This patch will detect when a user moves their finger along the screen in one continuous motion.
Connect the 2D Offset port in Screen Pan to the remaining port in the second Add patch:
Finish the project by adding subtle, ethereal music playing on a continuous loop:
You’ll use a free audio file from AR Library. It’s listed in the Assets panel as arl_mus_RelaxingMeditationSpace_lp_01.m4a.
Adding audio patches will alter the sound slightly, adding extra interest.
Adding the audio clip
To render sound in any effect, you’ll need a scene object called a Speaker. To add it:
Because the sound in this effect is supposed to be subtle, reduce the volume of the speaker:
The network of patches you’ll build will send a signal to a patch representing the Speaker, playing the sound in the scene.
To create a patch for the Speaker:
A yellow patch will be added to the Patch Editor:
Now create a patch representing the audio clip. Select arl_mus_RelaxingMeditationSpace_lp_01.m4a in the Assets panel and drag it into the Patch Editor. An orange patch will be added:
To connect the audio file to the speaker, use a Pulse, Single-Clip Controller and Audio Player patch. You can create these using the menu in Patch Editor. Connect:
You’ve now set up the audio file to play on a loop. Here’s how the graph will look:
Processing the sound
Three audio effect patches will change the sound slightly to better fit the atmosphere in the effect:
Create these patches using the menu in the Patch Editor:
Add the patches to your graph, between the Audio Player and speaker patch:
To complete the sound effect, make some adjustments to the audio effect patches:
In the Reverb patch: