/Virtual Jewelry Try-On
AR / E-commerceWebActive Developmentrole: Solo Full-Stack Frontend Developer

Virtual Jewelry Try-On

Real-Time AR Jewelry Try-On in the Browser

Solo Build
Users
3+
Screens
React 19Vite 7TypeScriptTailwind CSS 3.4React Router 7MediaPipe FaceMeshHTML5 Canvas 2DWebRTC getUserMediaShopify Storefront GraphQL APIlucide-reactESLint 9

Browser-based AR try-on at 30+ FPS using MediaPipe FaceMesh (468 landmarks) — anatomy-aware anchors, custom landmark smoother, head-rotation compensation, and Shopify Storefront GraphQL integration with deep-linked variants.

About the Project

Virtual Jewelry Try-On is a browser-based AR experience that lets shoppers see how necklaces and earrings look on them in real time — no native app, no paid third-party SDK. It uses MediaPipe FaceMesh to extract 468 3D facial landmarks from a 640×480 webcam feed every frame, then composites jewelry images onto a transparent HTML5 canvas layered above the video using a requestAnimationFrame loop. The catalog is pulled directly from a merchant's Shopify storefront via the Storefront GraphQL API, so inventory and variants are managed in one place and instantly available in the AR experience.

Anatomy-aware placement anchors earrings to ear tragion + earlobe landmarks (indices 234/58 and 454/288) and necklaces to chin landmarks (132, 152, 361), dynamically scaling to the user's neck width. A custom LandmarkSmoother class applies an exponential moving average (smoothing factor 0.8) across frames to eliminate jitter. Head-rotation compensation computes yaw / pitch / roll from nose-tip and outer-eye landmarks so jewelry tracks naturally when the user turns their head. A FacePositionGuide scores how well-centered and properly-sized the face is (0–100) and surfaces feedback like 'Move closer' or 'Move face left' so the try-on always looks ideal.

The SPA is component-driven with two custom hooks (useCamera for MediaStream lifecycle, useFaceTracking for FaceMesh wiring), a domain utilities layer (LANDMARK_INDICES, LandmarkSmoother, FacePositionGuide, getEarPositions, getNecklacePosition, getDynamicNecklacePosition, getHeadRotation, adjustForHeadRotation), and a Shopify data-access layer (typed GraphQL fetch wrapper + product-by-handle, get-all-images, products list). ?handle and ?id query parameters let a Shopify product page deep-link straight into the try-on with the correct variant pre-selected. Cross-origin Image loading (crossOrigin='anonymous') keeps the canvas export-safe when drawing Shopify CDN assets. All face detection and rendering happens entirely on-device — no video frames or landmarks are uploaded.

Key Highlights

▸Real-time face tracking with MediaPipe FaceMesh — 468 3D landmarks, refined lip & eye detail
▸30+ FPS canvas overlay loop on requestAnimationFrame for smooth necklace + earring rendering
▸Custom LandmarkSmoother (exponential moving average, factor 0.8) eliminates per-frame jitter
▸Head-rotation compensation: yaw/pitch/roll from nose + outer-eye landmarks keeps jewelry attached
▸Anatomy-aware anchors — earrings on ear tragion/earlobe (234/58, 454/288), necklaces on chin (132/152/361)
▸Scale-invariant sizing: necklace width from inter-chin distance, earring size from face width
▸Shopify Storefront GraphQL integration — products, variants, and image carousel pulled live
▸Deep-linking via ?handle & ?id so Shopify product pages launch the try-on with the right variant
▸FacePositionGuide scoring (0–100) with coaching cues like 'Move closer' / 'Move face left'
▸100% on-device processing — no video frames or landmarks ever leave the browser

Technical Breakdown

Feature areas & implementation details

Camera & Face Tracking
▸WebRTC getUserMedia front camera at 640×480 with clean MediaStream start/stop lifecycle
▸MediaPipe FaceMesh extracting 468 3D landmarks per frame with refined lips/eyes
▸useCamera + useFaceTracking custom hooks with proper cleanup to avoid GPU/WASM leaks
▸Status pill: Camera Off / Detecting / Face Detected so users always understand state
▸Permission denial errors surfaced clearly to the user
AR Overlay & Anatomy
▸Transparent HTML5 canvas layered above the video, redrawn on every requestAnimationFrame
▸LandmarkSmoother — exponential moving average (factor 0.8) over 468 landmarks per frame
▸Earrings anchored to ear tragion / earlobe landmarks (indices 234/58 and 454/288)
▸Necklaces anchored to chin landmarks (132, 152, 361) with dynamic neck-width scaling
▸Head-rotation compensation (yaw/pitch/roll) so jewelry tracks turning heads
▸FacePositionGuide — 0–100 score plus 'Move closer / Move face left' coaching feedback
Shopify Integration
▸Storefront GraphQL fetch wrapper with typed Product interfaces
▸Per-handle product loading + 'browse all images' fallback for the catalog carousel
▸Variant-aware deep-linking: ?handle and ?id query params pre-select the right variant
▸Cross-origin image loading (crossOrigin='anonymous') keeps canvas export-safe
▸Manual fine-tuning controls — per-item scale and X/Y offset for unique anatomy
Architecture & Data Flow
▸Component-driven SPA: App, Camera, JewelryCanvas, JewelryControls, ProductView
▸Hooks layer: useCamera (MediaStream), useFaceTracking (FaceMesh)
▸Domain utils: LANDMARK_INDICES, LandmarkSmoother, FacePositionGuide, getEarPositions, getNecklacePosition, getDynamicNecklacePosition, getHeadRotation, adjustForHeadRotation
▸Data access: shopify.ts, shopify-get-product, shopify-get-product-by-handle, shopify-get-all-images, shopify-products
▸Flow: webcam → FaceMesh → smoother → JewelryCanvas draw loop → 2D overlay above video
Performance & UX
▸Single 2D canvas pass per animation frame; video element kept separate to avoid recompositing
▸MediaPipe model lazy-loaded from CDN to keep the initial bundle small
▸9:16 portrait camera frame (max-width 420px) — intentional on phones, clean on desktop
▸Progressive-disclosure carousel: compact thumbnail expands into a horizontal picker
▸Tested on Chrome, Safari (iOS + macOS), Edge, Firefox; requires WebRTC + Canvas + WASM + ES2022
▸100% on-device privacy — no frames or landmarks uploaded

Other Projects

View All →
Arkmantra
Ayurvedic Cosmetics E-Commerce Storefront & Admin

A highly hardened, full-featured D2C storefront and admin control panel designed...

Soul33 – The Beacon
Spirituality, Meditation & Mindfulness Platform

A cross-platform spiritual wellness app live on iOS & Android with Free, Seeker ...

Yoke Yoga
Social Wellness, Live Yoga & Learning Platform

Production yoga social platform — TikTok-style Reels, ZEGOCLOUD live streaming +...