This is a submission for the Presage SmartSpectra SDK Challenge
What I Built
PleiadesSpectra is a browser extension that turns your camera into a remote control for any web page. Think of it as Tampermonkey for your face: small plain-JavaScript scripts decide what a smile, a nod or a lean of the head does on a site.
- Smile or nod to rate a service. A rating pop-up appears, the camera turns on for a few seconds, you smile, the five-star button is clicked, the camera turns off.
- Tilt your head to scroll a page, hands-free.
- Write a script with an AI agent. The agent can only propose a script. You see the code, the sites it runs on and what it reads from your face, and you press Install yourself.
A script is one file, no imports, no build step:
// ==Reaction== // @name Scroll by tilt // @match *://*/* // @camera always // @gestures tilt-down, tilt-up // ==/Reaction== reaction("scroll") .cooldown(400) .onGesture("tilt-down", () => scrollBy({ top: 400, behavior: "smooth" })) .onGesture("tilt-up", () => scrollBy({ top: -400, behavior: "smooth" })); The header is read without running any code, so the install window can say exactly which signals a script asks for. Beta, the mascot, stands in the corner with a red dot whenever the camera runs.
Demo
Code
PleiadesSpectra
Right now only the Chrome version works. The Firefox build is in the release too, but it has not been made to work yet.
Not in the browser stores yet: download the Chrome zip from the latest release and load it as in step 2
A browser extension that turns your camera into a remote control for any web page. Entry for the Presage SmartSpectra SDK Challenge. Small JavaScript scripts decide what a smile, a nod, a lean of the head, a pulse or a breath does on a site; the SmartSpectra SDK runs in a small local server, because it has no browser version.
camera → extension → ws://localhost:3001 → server (SmartSpectra SDK) → gestures and signals → script on the page -
apps/wxt: the browser extension (WXT, Chrome and Firefox). -
apps/server: the local server (Elysia on Bun) that hosts the SDK session…
Setup instructions (Docker Compose or Bun, then loading the extension in Chrome) are in the README. You need a SmartSpectra API key and a camera. A ready Chrome zip is in the latest release. Firefox: the build exists but does not work yet, please test in Chrome.
To try it as a judge: run docker compose up --build, load the extension, open the demo pages on 127.0.0.1:4010 (a blog), install the scripts from example/scripts, and use Spectra check in the popup to see what is recognised.
How It Responds to Human Signals
camera → extension → ws://localhost:3001 → server (SmartSpectra SDK) → gestures and signals → script on the page The SmartSpectra Node SDK has no browser version, so it runs in a small local server (Bun + Elysia). The extension sends camera frames to it over a local WebSocket (useCustomInput() + sendFrame()), and the server sends gestures and signals back, live.
Live Presage signals used:
-
expression(HAPPY, SURPRISE, ...) → thesmileandconfusedgestures. -
talking→ thetalkinggesture;blinking→ theblinksignal. -
face.landmarks→nod,shake,tilt-down,tilt-up,roll-left,roll-right.
Honest notes about the SDK: it has no gaze and no head pose, so head gestures are computed by me from the landmarks against a resting pose. It needs 25 fps or more (at 15 fps it raises kFrameRateTooLow). A pause in frames makes every sendFrame throw kTimestampGap, so the server stops the SDK session when the camera goes off and starts it again when it comes back. Blink detection is the least reliable signal and I treat it as one.
Prize Categories
Most Creative or Delightful Use. Pages already have buttons, scroll and rating pop-ups. PleiadesSpectra does not add a new site, it lets you decide what your face does on any site, and lets an AI agent draft that rule while you stay the one who approves it.
Consent and Data Handling
- Consent: the camera is off until a script that you installed needs it. The install window shows the script's code, its sites, its camera mode and the signals it reads from your face. While the camera runs, a red dot and Beta are shown on the page, and one click on the dot (or Turn off camera in the popup) pauses it. There is no hidden or background capture: a script with
@camera alwaysonly runs while a matching tab is open and in front. - Where video goes: only to the server on your own machine (
127.0.0.1:3001, ports are published on localhost only). It is processed in memory and never saved, never uploaded, never shared. - Storage and deletion: nothing from the camera is stored. The extension keeps your installed scripts and its settings in the browser's own extension storage; removing a script or the extension deletes them. Gestures and signals are used on the page and are not written down or sent anywhere by the example scripts.
- SDK telemetry: off by default (
SMARTSPECTRA_TELEMETRY). The SDK still contacts Presage to check the API key, as it must. - Optional AI agent: off until you press Connect to AI and give the agent a secret local URL. The agent's tools only read the script guide, validate and list scripts, and propose a script. None of them return camera frames or signal values, and the agent never installs, enables or deletes anything. Installing is always your click.
- Not a medical tool. No diagnosis, no treatment, no medical decisions.
- Scripts are page code: install only ones you trust.

