With the multipeer capability you can create group effects that can communicate with other instances of the same effect running in a group video call.
In a script this is implemented using the MultipeerModule
API, which provides the ability to create message channels that effects can send JSON formatted messages to.
The effects can also listen out for messages that other instances have sent to a given message channel, enabling data to be passed between them to create coordinated group experiences.
In this tutorial, you'll learn how to implement multipeer functionality using the API, by creating an effect where video call participants can change the other participants’ backgrounds by tapping the screen.
Download the unfinished project to follow along.
In the example project, you’ll find a number of assets already set up, including a canvas group, materials and imported textures.
Additionally, in the Patch Editor you’ll find a patch graph which sends a tap event to a script and another which handles the changing of the user’s background.
The script.js asset already contains some code which is used to track the current background image and to subscribe to the event sent from the Patch Editor.
// Load in the required modules const Patches = require('Patches'); const Diagnostics = require('Diagnostics'); (async function () { // Enable async/await in JS [part 1] // Initialize background tracking variables const totalBackgroundCount = 3; var backgroundIndex = 0; // Get the tap event from the Patch Editor const screenTapPulse = await Patches.outputs.getPulse('screenTapPulse'); // Subscribe to the screen tap event screenTapPulse.subscribe(() => { }); })(); // Enable async/await in JS [part 2]
When a call participant opens a group effect it's automatically applied to all other participants on the call, though each participant will be running their own instance.
To communicate between the various instances of the effect, we’ll implement logic that allows them to both send messages to, and receive them from, the other instances.
This way, each instance is notified of any state changes and can be updated accordingly.
For the example project we'll be following the logic below, which describes what will happen every time a screen tap occurs on any of the running instances.
For each screen tap interaction, the participant that taps on the screen is classed as the message sender, while the other participants are classed as the message receivers.
Message sender
Messages are broadcast to all peers except the instance that the message was broadcast from - an effect can’t broadcast a message to itself. Step 2 updates the sender's local state and is an important part of ensuring a unified experience across the various instances.
Message receiver
Now that we have an idea about the logic required, we can begin to implement the functionality in a script.
Open the script.js file in a text editor.
On line 4, import the MultipeerModule into the script with the following line of code so that you can make use of the multipeer API.
const Multipeer = require('Multipeer');
We need to create a message channel where messages can be sent to and retrieved from.
The multipeer API supports the use of multiple message channels but we'll only be creating one in this example.
On line 14, add the following lines of code:
// Create a message channel const backgroundIndexChannel = Multipeer.getMessageChannel('backgroundIndexTopic');
The getMessageChannel()
method takes a single argument, the name of the message channel to create. With the line above we’ve created the BackgroundIndexTopic message channel.
We’ll need the channel name to send and retrieve messages later on in our code.
If the method is called with a name that already exists, the existing channel is retrieved and no new channel is created.
Before we send a message to the message channel, we need to update the variable keeping track of the active background image and send its value to the Patch Editor.
We’ll do this within the tap event callback function that was already set up in the script.
Starting on line 20 copy and paste the code snippet below.
// Increment the background index to show the next background graphic backgroundIndex++; // If the index value is equal to the total number of background images, // reset the value so that the displayed graphic loops. if (backgroundIndex >= totalBackgroundCount) { backgroundIndex = 0; } // Send the new background index value to the Patch Editor, so that it can be // used to update the background graphic displayed Patches.inputs.setScalar('msg_background', backgroundIndex);
To send a message to a message channel you can call the sendMessage()
method exposed by MessageChannel
objects.
Paste the code snippet below immediately after the previous snippet, on line 33. Make sure to add it before the callback function’s closing curly bracket (the ‘}’).
// Broadcast the new background index value to other participants on the // previously created backgroundIndexChannel message channel backgroundIndexChannel.sendMessage({'background': backgroundIndex }, true).catch(err => { // If there was an error sending the message, log it to the console Diagnostics.log(err); });
The method’s first argument is a JSON formatted object containing a name and a value to send to the channel. In our case, we’re sending the backgroundIndex
variable to the channel backgroundIndexChannel
under the name background
.
The second argument, true
, refers to whether the message should be sent using the Real-Time Channel messaging backend, which has a high bandwidth for message streams but doesn’t guarantee a synchronized message order across participants.
If false
had been passed, the Sync Channel backend would have been used instead, which synchronizes message order across participants but has a message rate limit of 75 messages every 30 seconds.
The catch()
method and subsequent console log is there to notify us if anything goes wrong when sending the message.
Now that we’ve implemented the message sending logic, we also need to make sure that we’re able to listen out for messages broadcast to the message channel from other participants.
To do so, we can subscribe to the EventSource
returned by the onMessage
property exposed by MessageChannel
objects.
The event’s callback function contains a single argument, the JSON object broadcast to the message channel.
Within the callback function we can access the data by the name it was assigned in the sendMessage
method call. In our example, it’s background
.
In the callback function we’ll also need to send the new background value to the Patch Editor.
Copy and paste the following code starting on line 44:
// Listen out for messages sent to the syncBGChannel backgroundIndexChannel.onMessage.subscribe((msg) => { // Retrieve the 'background' attribute from the JSON object received backgroundIndex = msg.background; // Send the value to the Patch Editor Patches.inputs.setScalar('msg_background', msg.background); });
With that, our script is complete.
// Load in the required modules const Patches = require('Patches'); const Diagnostics = require('Diagnostics'); const Multipeer = require('Multipeer'); (async function () { // Enable async/await in JS [part 1] // Initialize background tracking variables const totalBackgroundCount = 3; var backgroundIndex = 0; // Get the tap event from the Patch Editor const screenTapPulse = await Patches.outputs.getPulse('screenTapPulse'); // Create a message channel const backgroundIndexChannel = Multipeer.getMessageChannel('BackgroundIndexTopic'); // Subscribe to the screen tap event screenTapPulse.subscribe(() => { // Increment the background index to show the next background image backgroundIndex++; // If the index value is equal to the total number of background images, // reset the value so that the displayed image loops. if (backgroundIndex >= totalBackgroundCount) { backgroundIndex = 0; } // Send the new background index value to the Patch Editor, so that it can be // used to update the background image displayed. This only updates the // background for the participant that tapped on the screen Patches.inputs.setScalar('msg_background', backgroundIndex); // Broadcast the new background index value to other participants on the // previously created backgroundIndexChannel message channel backgroundIndexChannel.sendMessage({'background': backgroundIndex }, true).catch(err => { // If there was an error sending the message, log it to the console Diagnostics.log(err); }); }); // Listen out for messages sent to the backgroundIndexChannel backgroundIndexChannel.onMessage.subscribe((msg) => { // Retrieve the 'background' attribute from the JSON object received backgroundIndex = msg.background; // Send the value to the Patch Editor Patches.inputs.setScalar('msg_background', msg.background); }); })(); // Enable async/await in JS [part 2]
The changing of each participants' backgrounds is handled by the Patch Editor, using the backgroundIndex
variable sent to it from the script:
When building group effects, it's important to test regularly to ensure everything works as intended. The Meta Spark Extension for Visual Studio Code provides a built-in tool that allows you to debug effects that use the multipeer capability.
You can also click the Login with Facebook button to log in with your Facebook account.
You can use the MultipeerModule API or patches in a block as you would in the main part of your project. The only difference is the scoping:
Blocks.instantiate('mp_block', {name: "block1"}, {multipeerId: 'block1'})