WHERE THIS TEST ENDED
The Mac prototype detects pinches and moves windows after recalibrating for an off-to-the-side camera. This is a Mac-control test; carrying a live window onto the projected desk is still a separate, unfinished step.
- Mac camera feed
- Vision framework hand tracking
- Four-corner calibration
- Pinch moves the window
How it was built
The Mac prototype, HandDesk, reads camera frames locally and detects hand landmarks using Apple's Vision framework. It maps the calibrated hand position onto the arrangement of the Mac's displays. With Accessibility permission, a pinch can move the selected window or act as a pointer drag; opening the hand releases it.
Four-corner calibration handles a camera placed to the left or right. I mark the corners of a comfortable hand-movement area, and the app maps that skewed shape back to the desktop rectangle. This compensates for a fixed camera angle within that plane. It does not make the camera see hidden fingers or provide full 3D pointing.
What this attempt established
The captions describe detecting a pinch, grabbing items and retesting after side-camera calibration. The matching September 7 project notes separately record my acceptance of basic Mac control and the calibration update. This is a Mac-control prototype. The opening idea of transferring a live window onto the projected desk is still a separate step in this clip.
Later follow-up
A separate live-window bridge was developed afterward. Its streaming, desk controls and connection problems belong in their own update; they are not demonstrated by this Mac hand-control test.
About this breakdown
Matched to this original TikTok, including its spoken captions. The result above describes this episode; later work is identified separately.
Breakdown checked September 6, 2026.