Making grab and drag feel predictable

Start with where the hand lands

The opening problem is simple to feel: pinching and grabbing a projected card is awkward. Before adding another gesture, the camera needs to map the hand to the correct position on the projected dashboard.

The calibration work behind this stage uses a pattern projected onto a clear desk. The camera observes known positions in that pattern, then the software calculates a map between camera coordinates and dashboard coordinates. That map is a homography: a way to relate two views of the same flat surface.

Why the calibration had been failing

The September 3 build notes identified several separate causes. The camera sometimes captured the dark reference before the projector had settled. A contrast check mixed up a brightness difference with an absolute brightness level. Some projected test squares fell over the rounded edge of the desk.

The fixes gave the image time to settle, tested the change in brightness directly, and moved the squares inward. Clearing the keyboard and mouse off the test area also mattered: a square landing on a raised object does not describe the desk surface.

Verify the interaction by using it

At 0:38 I demonstrate repeated grabs and drops. At 0:53 I switch to touch and drag. These are separate checks: locating a hand in the camera is only useful if a card starts moving when intended and stays where it is dropped.

The recorded fingertip check after the September 3 calibration had a median position error of 0.85 cm. That is a measurement from that calibration session, not a claim that every point in this video has the same error. Wide-lens distortion and corner accuracy still needed attention.

Where this episode leaves the build

The depth sensor has arrived, but I have not shown it improving this interaction. I describe more sensible card placement as the next step. Later layout and two-hand resizing changes should be read as follow-up work, rather than features already completed in this clip.