AR Try-On: Three.js + MediaPipe in React Native

Barva Skin Therapie needed an AR lipstick try-on so customers could see shades on their face before buying. ARKit and ARCore felt heavyweight — separate native modules, ~25MB each, platform-specific code to maintain. The solution: a WebView-based approach using Three.js and Google's MediaPipe Face Landmarker running in WASM. One implementation, both platforms, ~3MB payload.

Architecture overview

The React Native layer uses VisionCamera to stream frames at 30fps. Each frame is converted to a base64 JPEG and sent to the WebView via postMessage. Inside the WebView, MediaPipe's Face Landmarker processes the frame and returns 468 3D landmark coordinates. Three.js maps specific lip landmark indices to a ShapeGeometry and renders a semi-transparent MeshBasicMaterial in the selected lipstick colour. The composited canvas is drawn back over the camera feed.

Setting up MediaPipe in WASM

MediaPipe's FaceLandmarker runs in a Web Worker so it doesn't block the render thread. Initialisation loads the face_landmarker.task model (~6MB) from a CDN. The first inference call has ~400ms overhead for WASM JIT compilation; subsequent calls run at 25–30fps on a mid-range device.

const { FaceLandmarker, FilesetResolver } = await import(
  'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision'
);

const filesetResolver = await FilesetResolver.forVisionTasks(
  'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision/wasm'
);

faceLandmarker = await FaceLandmarker.createFromOptions(
  filesetResolver,
  {
    baseOptions: { modelAssetPath: FACE_LANDMARKER_URL },
    runningMode: 'IMAGE',
    numFaces: 1,
  }
);

Mapping lip landmarks to Three.js geometry

The MediaPipe face mesh has 468 landmarks. I extracted the 40 indices that trace the outer and inner lip contours and mapped them to Three.js Vector3 coordinates, scaling by canvas width and height. The z-depth is multiplied by a constant to give the lip geometry a slight 3D curvature that follows the face.

const LIP_OUTER = [61,185,40,39,37,0,267,269,270,409,291,375,321,405,314,17,84,181,91,146];
const LIP_INNER = [78,191,80,81,82,13,312,311,310,415,308,324,318,402,317,14,87,178,88,95];

function updateLipMesh(landmarks, W, H) {
  const points = [...LIP_OUTER, ...LIP_INNER].map(i =>
    new THREE.Vector3(
      landmarks[i].x * W - W / 2,
      -(landmarks[i].y * H - H / 2),
      landmarks[i].z * 200
    )
  );
  lipMesh.geometry.setFromPoints(points);
  lipMesh.geometry.computeBoundingSphere();
}

Colour switching and opacity

Each lipstick SKU has a hex colour stored in Shopify metafields. When a user taps a colour swatch, React Native sends a postMessage with the hex value to the WebView, which updates the MeshBasicMaterial colour and opacity (0.55 for a natural finish, 0.75 for bold shades). The switch is instant — no re-render, just a material property update.