Control the screen without breaking scrub.
SCRUB is a hand tracker and gesture interface, in development, built for surgical gloves. It lets a scrubbed-in clinician move a cursor, click and scroll with mid-air gestures. Trackers built for bare skin return nothing on gloves; SCRUB recovers the hand.
- NO HAND off-the-shelf, on gloves
- TRACKED SCRUB, on the same gloves
- COMMITTED only deliberate gestures act
Why it matters
In a sterile field, every mouse is out of reach.
Once scrubbed in, a surgeon cannot touch a keyboard, mouse or touchscreen without breaking sterility. To scroll through imaging mid-procedure, they step away and re-scrub, or direct someone else to click for them. A review of 55 touchless interaction systems for surgery found that most were built to control image viewers, yet fewer than one in six had been tested clinically (Mewes et al. 2017).
Camera-based hand tracking looks like the answer, until the hands are gloved. In a clinical study, a widely used tracker detected hands 0% of the time on blue, purple and tan medical gloves (Kiefer et al. 2024). The gloves themselves are the problem.
SCRUB is a hand tracker and gesture interface designed for gloved hands, so a clinician can drive the screen without touching anything.
See it
One frame from a real operation.
A real cardiac procedure from the public Surgical Hands dataset: white surgical gloves under operating-room lighting. Green marks the annotated joints. Blue is SCRUB's estimate. An off-the-shelf tracker, run on the identical frame, returned no hand at all.
The problem underneath
Gloves erase the texture hand trackers depend on.
Skin carries fine detail: creases, nail beds, knuckles. Hand trackers learn to lock onto it. A surgical glove replaces all of that with a smooth, glossy, single-colour surface. That does not make the problem slightly harder; it makes it a different problem, and trackers trained on bare hands fall into a regime where they see nothing.
What the camera captures
The image's frequency content. Bare skin produces a crisp, structured pattern; a glove smears the energy diffusely.


Where the pose model looks
The model's confidence map. On bare skin it traces each finger; on a glove it collapses into a blur.


A representative bare and gloved hand, not the same person. Brighter means more energy (left) or more confidence (right).
Hands-free
Six gestures. No accidental clicks.
SCRUB maps a small set of deliberate hand shapes to ordinary mouse actions, so it works with the screen software already in the room. Pick a gesture to see what it does.
Animated illustration of the interaction design, not recorded system output. Close a fist and hold it to put SCRUB back to sleep.
Where it sits
A camera, SCRUB, and the software you already use.
SCRUB watches the clinician's hands through an ordinary webcam and speaks the language every computer already understands: mouse events. There is nothing to wear, nothing to touch and nothing to integrate into the imaging software itself.
Scroll sideways to see the full diagram.
How it works
Find. Steady. Commit.
Find
A pose model built for gloved hands locates 21 hand joints in every video frame, on the same frames where trackers built for bare skin return nothing.
Steady
Single-frame estimates are noisy. Adaptive smoothing turns them into a cursor that holds still when the hand is still and keeps up when it moves.
Commit
A gesture state machine acts only on deliberate, held gestures, so a hand drifting through view never clicks. This is the classic Midas touch problem of gesture interfaces.
One frame, end to end mean of the full pipeline · single RTX 5090
Model architecture, training data and methods are intentionally not described here.
Results
From no signal to usable signal.
On real surgical footage, the off-the-shelf tracker does not degrade gracefully; it returns nothing. SCRUB recovers a usable signal from the same frames. That signal is partial: about a third of joints land within tolerance on the hardest real footage. It is far from clinical-grade, and it is the difference between a system that cannot work and one that can be engineered to.
Real gloved frames, by lighting share of joints located (PCK@20)
SCRUB Off-the-shelf tracker
Both systems were scored on identical frames against identical annotations from Surgical Hands, a public dataset of real operations (Louis et al. 2022), using a pre-registered paired image-level bootstrap with 10,000 resamples. A directional claim is made only where the 95% confidence interval excludes zero; every interval above does. The off-the-shelf tracker is MediaPipe Hands (Zhang et al. 2020).
What SCRUB does not claim
- SCRUB is a research prototype. It is not a medical device and has not been cleared or clinically validated.
- Accuracy on real gloves is partial. About a third of joints land within tolerance on surgical footage, and localization error remains large.
- The real-glove evaluation uses recorded operations filmed from above, with instruments in hand. That is not a clinician gesturing toward a webcam; a task-matched gloved-webcam benchmark is the next required step.
- Real-footage results cover white surgical gloves. Other glove colours are not yet established on real data.
- The comparison is against one widely used off-the-shelf tracker on the same frames. The two systems define "detection" differently, so detection rates are not compared here.
- The gesture section is an illustration of the interaction design, not recorded output. The frequency figures use a representative bare and gloved hand, not the same person.
About
Waverly Rose Brim
SCRUB (Surgical Cross-Domain Robust Ungloved-to-Gloved Bridge) is research by Waverly Rose Brim. Waverly is earning a Master of Science in Artificial Intelligence at the Johns Hopkins Whiting School of Engineering (expected May 2027). The work began in EN.525.733 Deep Learning for Computer Vision at Johns Hopkins University in Spring 2026.
SCRUB is developed under ORITH, alongside VIGIL.
SCRUBThis page
Touchless screen control for gloved hands in sterile settings.
VIGIL ↗
A world-model verifier that tells surgeons when to supervise or intervene in autonomous robotic surgery.
Real surgical frame: Surgical Hands dataset (Louis et al., IJCARS 2022). Gesture photographs and the gloved-hand mosaic are SCRUB project imagery.