SCRUB home

Control the screen without breaking scrub.

SCRUB is a hand tracker and gesture interface, in development, built for surgical gloves. It lets a scrubbed-in clinician move a cursor, click and scroll with mid-air gestures. Trackers built for bare skin return nothing on gloves; SCRUB recovers the hand.

  • NO HAND off-the-shelf, on gloves
  • TRACKED SCRUB, on the same gloves
  • COMMITTED only deliberate gestures act
Why it matters

Why it matters

In a sterile field, every mouse is out of reach.

Once scrubbed in, a surgeon cannot touch a keyboard, mouse or touchscreen without breaking sterility. To scroll through imaging mid-procedure, they step away and re-scrub, or direct someone else to click for them. A review of 55 touchless interaction systems for surgery found that most were built to control image viewers, yet fewer than one in six had been tested clinically (Mewes et al. 2017).

Camera-based hand tracking looks like the answer, until the hands are gloved. In a clinical study, a widely used tracker detected hands 0% of the time on blue, purple and tan medical gloves (Kiefer et al. 2024). The gloves themselves are the problem.

SCRUB is a hand tracker and gesture interface designed for gloved hands, so a clinician can drive the screen without touching anything.

See it

One frame from a real operation.

A real cardiac procedure from the public Surgical Hands dataset: white surgical gloves under operating-room lighting. Green marks the annotated joints. Blue is SCRUB's estimate. An off-the-shelf tracker, run on the identical frame, returned no hand at all.

Gloved surgeons' hands in an open-chest operation, with SCRUB's estimated hand skeleton in blue closely following the annotated joints in green
TRACKED OFF-THE-SHELF: NO HAND
White gloves · surgical lighting · recorded evaluation output Annotated joints SCRUB Off-the-shelf (none returned)

The problem underneath

Gloves erase the texture hand trackers depend on.

Skin carries fine detail: creases, nail beds, knuckles. Hand trackers learn to lock onto it. A surgical glove replaces all of that with a smooth, glossy, single-colour surface. That does not make the problem slightly harder; it makes it a different problem, and trackers trained on bare hands fall into a regime where they see nothing.

What the camera captures

The image's frequency content. Bare skin produces a crisp, structured pattern; a glove smears the energy diffusely.

Frequency spectrum of a bare hand: a sharp, structured cross
Bare
Frequency spectrum of a gloved hand: diffuse, star-like spread
Gloved

Where the pose model looks

The model's confidence map. On bare skin it traces each finger; on a glove it collapses into a blur.

Confidence map on a bare hand: sharp, finger-shaped ridges
Bare
Confidence map on a gloved hand: a broad, blurred blob
Gloved

A representative bare and gloved hand, not the same person. Brighter means more energy (left) or more confidence (right).

Hands-free

Six gestures. No accidental clicks.

SCRUB maps a small set of deliberate hand shapes to ordinary mouse actions, so it works with the screen software already in the room. Pick a gesture to see what it does.

Animated illustration of the interaction design, not recorded system output. Close a fist and hold it to put SCRUB back to sleep.

Where it sits

A camera, SCRUB, and the software you already use.

SCRUB watches the clinician's hands through an ordinary webcam and speaks the language every computer already understands: mouse events. There is nothing to wear, nothing to touch and nothing to integrate into the imaging software itself.

Scroll sideways to see the full diagram.

Where SCRUB sits A gloved clinician inside the sterile field gestures toward a webcam. Video frames go to SCRUB, which finds the hand, steadies it and commits only deliberate gestures. SCRUB sends standard mouse events to the screen software, such as an imaging viewer, records or slides. The clinician sees the result without touching anything. the clinician sees the result; nothing is touched STERILE FIELD Clinicianscrubbed in, gloved Webcamordinary RGB camera SCRUB Findthe hand Steadythe motion Committhe gesture runs on the workstation Screen softwareimaging viewer, records, slides mid-air gestures video frames MOUSE EVENTS CURSOR · CLICK · RIGHT CLICK · SCROLL · DRAG The operating room is the long-term goal, not a claim: it would need clinical validation and regulatory clearance.

How it works

Find. Steady. Commit.

01

Find

A pose model built for gloved hands locates 21 hand joints in every video frame, on the same frames where trackers built for bare skin return nothing.

02

Steady

Single-frame estimates are noisy. Adaptive smoothing turns them into a cursor that holds still when the hand is still and keeps up when it moves.

03

Commit

A gesture state machine acts only on deliberate, held gestures, so a hand drifting through view never clicks. This is the classic Midas touch problem of gesture interfaces.

One frame, end to end mean of the full pipeline · single RTX 5090

SCRUB · 8.22 ms
headroom in one 30 fps camera frame · 33.3 ms
121.7 frames per secondp95 8.25 msp99 8.37 ms

Model architecture, training data and methods are intentionally not described here.

Results

From no signal to usable signal.

On real surgical footage, the off-the-shelf tracker does not degrade gracefully; it returns nothing. SCRUB recovers a usable signal from the same frames. That signal is partial: about a third of joints land within tolerance on the hardest real footage. It is far from clinical-grade, and it is the difference between a system that cannot work and one that can be engineered to.

0%→36%Joints located on real surgical footage, white gloves under OR lighting: off-the-shelf tracker vs SCRUB, 105 frames.
3.5×As many joints located across all 745 real gloved frames: 30.5% vs 8.7%.
+21.8ptsSCRUB's paired advantage on the gloved set, with a 95% confidence interval of 19.3 to 24.5 points.
8.2msPer frame, end to end: 121.7 frames per second on a single GPU.

Real gloved frames, by lighting share of joints located (PCK@20)

SCRUB Off-the-shelf tracker

Both systems were scored on identical frames against identical annotations from Surgical Hands, a public dataset of real operations (Louis et al. 2022), using a pre-registered paired image-level bootstrap with 10,000 resamples. A directional claim is made only where the 95% confidence interval excludes zero; every interval above does. The off-the-shelf tracker is MediaPipe Hands (Zhang et al. 2020).

What SCRUB does not claim

About

Waverly Rose Brim

SCRUB (Surgical Cross-Domain Robust Ungloved-to-Gloved Bridge) is research by Waverly Rose Brim. Waverly is earning a Master of Science in Artificial Intelligence at the Johns Hopkins Whiting School of Engineering (expected May 2027). The work began in EN.525.733 Deep Learning for Computer Vision at Johns Hopkins University in Spring 2026.

SCRUB is developed under ORITH, alongside VIGIL.

SCRUBThis page

Touchless screen control for gloved hands in sterile settings.

VIGIL ↗

A world-model verifier that tells surgeons when to supervise or intervene in autonomous robotic surgery.

Real surgical frame: Surgical Hands dataset (Louis et al., IJCARS 2022). Gesture photographs and the gloved-hand mosaic are SCRUB project imagery.