Method one: Adding a hand tracker object to the Scene panel

This method allows you to easily track 3D objects to a user’s hand. To get started, we’ll be adding a Hand Tracker to the Scene panel:

To add a Hand Tracker:

1. In the Scene panel, click +.
2. Click Hand Tracker.
3. Click Insert.

To make a 3D object appear on the hand, make it a child of the hand tracker in the Scene panel. To do this, either:

• Right click the hand tracker in the Scene panel and select 3D Object.
• Drag an object from either the Assets panel or your desktop, straight onto the hand tracker in the Scene panel.

In the example below, you’ll see that a 3D object titled Sphere is being tracked to the hand:

To learn more, follow along with our hand tracking effect tutorial.

Hand Tracker object properties

When you've selected the hand tracker in the Scene panel, you'll be able to edit its properties in the Inspector on the right of your screen.

Instructions

If your effect has a hand tracker or hand tracking patches in it, the automatic instructions capability is added to your project by default. This mean an instruction saying Hold up a hand will appear until a hand is detected by the camera. You can add custom instructions to your effect if you prefer.

Method two: Using hand bounding box patches

Hand bounding boxes can be used to create effects that respond to both the position and size of a user’s hand. They plot an invisible rectangle around a selected hand, making the 2D coordinates of all four corners and the center of the rectangle — as well as its height and width — available for you to use in your effect.

Follow our tutorial to build an effect using this method.

To create effects using a hand bounding box, we’ll need to add a series of patches to our project:

1. Add the Hand Finder patch. Double click on the Patch Editor and find it under Utility.
2. Add a Hand Select patch, also found under Utility. If you’d like to create bounding boxes for two hands, you’ll need to add two Hand Select patches.
3. Connect the Hands output of the Hand Finder patch to the Hands input(s) of your Hand Select patches.

If you are tracking two hands, your patches will look like this:

Each hand detected by the Hand Finder is given a unique index number. This number (0 or 1) is assigned based on the order in which hands appear in the scene.

You’ll notice that by default, both your Hand Select patches will have an index of 0. To track both hands, change the index value on one of your Hand Select patches to 1, as in the example below:

Next, we’ll be adding the Hand Bounding Box patch to plot the bounding box:

1. Add one Hand Bound Box patch for each of your Hand Select patches. Double click on the Patch Editor and find it under Utility.
2. Connect the Hand output of each Hand Select patch to the Hand input of a Hand Bounding Box patch.

If you are tracking two hands, your patches will now look like this:

Finally, we’ll be adding the Bounding Box 2D Unpack patch. This patch makes the coordinates and values generated by the Hand Bounding Box into outputs. We can then use those outputs to create our hand tracking effect.

To add the Bounding Box 2D Unpack patch:

1. Add one Bounding Box 2D Unpack patch for each of your Hand Bounding Box patches. Double click the Patch Editor and find it under Utility.
2. Connect the Hand Bounding Box output of each Hand Bounding Box patch to the Bounding Box input of each Bounding Box 2D Unpack patch.

If you are tracking two hands, your patches will look like this:

You’ll see that the Bounding Box 2D Unpack patches have outputs for each value calculated by our Hand Bounding Box patches.

In the example below, two rectangles have been added to the scene. Their width, height and position are determined by the outputs of our two Bounding Box 2D Unpack patches.

Scripting hand tracking effects

You can use the HandTracking API to create hand tracking effects via script.

The example below shows the implementation of the Hand Movement template. Download the project file here.

// Load in the required modules
const Scene = require('Scene');
const Reactive = require('Reactive');
const CameraInfo = require('CameraInfo');
const HandTracking = require('HandTracking');

// Get the screen size dimensions
const previewSizeWidth = CameraInfo.previewSize.
  			   width.div(CameraInfo.previewScreenScale);
  
const previewSizeHeight = CameraInfo.previewSize.
  			    height.div(CameraInfo.previewScreenScale);


// Create a reference to the detected hands
const hand0 = HandTracking.hand(0);
const hand1 = HandTracking.hand(1);
  
// Create a reference to the number of tracked hands
const handCount = HandTracking.count;
  
const TEXTURE_SCALE = 1.2;

(async function() { // Enable async/await in JS [part 1]

  // Locate the rectangles in the scene  
  const [
    rect_hand0,
    rect_hand1,
    rect_betweenhands
  ] = await Promise.all([
    Scene.root.findFirst('rect_hand0'),
    Scene.root.findFirst('rect_hand1'),
    Scene.root.findFirst('rect_betweenhands')
  ]);

  // Create references to the hand bounding boxes
  const bb0 = hand0.boundingBox;
  const bb1 = hand1.boundingBox;

  // Find the top left of the bounding boxes relative to the screen size
  const scaledX0 = bb0.x.mul(previewSizeWidth);
  const scaledY0 = bb0.y.mul(previewSizeHeight);

  const scaledX1 = bb1.x.mul(previewSizeWidth);
  const scaledY1 = bb1.y.mul(previewSizeHeight);

  // Bind the position of the rectangles to the bounding box positions
  rect_hand0.transform.x = scaledX0;
  rect_hand0.transform.y = scaledY0;

  rect_hand1.transform.x = scaledX1;
  rect_hand1.transform.y = scaledY1;

  // Bind the rectangle dimensions to the bounding box dimensions
  // relative to the screen size
  rect_hand0.width = bb0.width.mul(previewSizeWidth);
  rect_hand0.height = bb0.height.mul(previewSizeHeight);

  rect_hand1.width = bb1.width.mul(previewSizeWidth);
  rect_hand1.height = bb1.height.mul(previewSizeHeight);

  // Show each rectangle only when its respective hand is tracked
  // To test this, preview on a device
  rect_hand0.hidden = hand0.isTracked.not();
  rect_hand1.hidden = hand1.isTracked.not();

  // Find the midpoint between the hands
  const scaleRect = rect_betweenhands.width.div(2.0);
  
  const midPosX = bb0.center.x.add(bb1.center.x).div(2.0).
  		    mul(previewSizeWidth).sub(scaleRect);
  
  const midPosY = bb0.center.y.add(bb1.center.y).div(2.0).
  		    mul(previewSizeHeight).sub(scaleRect);

  // Find the distance between the hands, and allow for a 
  // scale factor depending on the texture
  const scaled0 = Reactive.pack2(scaledX0, scaledY0);
  const scaled1 = Reactive.pack2(scaledX1, scaledY1);
  const handDistance = scaled0.distance(scaled1);
  const scaledHandDistance = handDistance.mul(TEXTURE_SCALE);

  // Set position and scale of the rectangle between the hands
  rect_betweenhands.transform.x = midPosX;
  rect_betweenhands.transform.y = midPosY;
  rect_betweenhands.transform.scaleX = rect_betweenhands.transform.
  				         scaleY = scaledHandDistance;

  // Show the rectangle between hands only when both hands are tracked
  // To test this, preview on a device
  rect_betweenhands.hidden = handCount.lt(2);

  // Determine left and right hands
  const leftHandCenterX = Reactive.min(bb0.center.x, bb1.center.x);
  const rightHandCenterX = Reactive.max(bb0.center.x, bb1.center.x);
  
  const leftHandCenterY = Reactive.lt(bb0.center.x, bb1.center.x).
  			    ifThenElse(bb0.center.y, bb1.center.y);
  
  const rightHandCenterY = Reactive.gt(bb0.center.x, bb1.center.x).
  			     ifThenElse(bb0.center.y, bb1.center.y);

  // Calculate and set angle between hands
  const angleBetweenHands = Reactive.atan2(leftHandCenterX.
  			      sub(rightHandCenterX), leftHandCenterY.
  			      sub(rightHandCenterY));
  
  rect_betweenhands.transform.rotationZ = angleBetweenHands.add(Math.PI/2);

  // Mirror left hand image
  rect_hand0.transform.scaleX = bb0.center.x.eq(leftHandCenterX).
  				  ifThenElse(1, -1);
  
  rect_hand1.transform.scaleX = bb1.center.x.eq(leftHandCenterX).
  				  ifThenElse(1, -1);

  
})(); // Enable async/await in JS [part 2]