The first time I put a person in a scene, a friend asked why I'd made a zombie movie.

Fair question. The skin was pale grey. The eyes were stark white, ringed in black lashes. The lips were pink, like a doll's. When they walked, their feet slid along the ground like it was ice.

Riptide is my open source video editor. It runs in a browser tab and does 3D worlds as well as cuts, text and audio, so it needed people in those worlds. This is how they went from that to talking on camera. Watch it first:

Nothing here is one big trick. It's about twenty small ones, and every single one mattered. Here they are, in roughly the order I learned them.

1. Stop making people. Start from photos.

My first people were built by a script: a mesh, painted skin, painted clothes. Each fix made them a bit better and never good.

So I threw them out. The man and the woman are now two of Microsoft's Rocketbox avatars, which are MIT licensed. Their skin, clothes and hair come from photographs, with normal maps.

They needed some work before Riptide could use them. One script (tools/make_people_rocketbox.js) does it in the page itself:

  • turns them to face +Z, in metres
  • lets the arms down from the T pose
  • bakes the skin in that pose
  • rebuilds the skeleton with Mixamo's bone names

That last step meant every walk, pose and preset I'd already written kept working.

Lesson: if you're fighting your own art for weeks, the art is the problem.

2. The hair was the worst part, so I deleted it

Avatars like these have "hair cards": flat strips of hair texture floating over the scalp. Game engines draw them with sorted, soft transparency. A browser renderer without that draws them as dark strips and bald streaks.

I tried sorting them and failed. Then I noticed that under the cards, the scalp already has hair painted on it, photographed. I left the cards out, and the hairstyle came back on its own. You can see it in the side-on shots in the video.

3. Skin that light gets into

Real skin isn't paint. Light goes in, bounces around and comes back out a bit redder. That's why the edge of a shadow on a face looks warm instead of grey.

You don't need real subsurface scattering for most of that effect:

  • let the light bleed red a little past the shadow's edge
  • add a faint oily sheen
  • add tiny pores in the roughness so the highlight breaks up

Each part also gets its roughness from its own specular map. Lips and eyes shine, skin has a soft sheen, and hair and cloth stay matte. Before that, one roughness for the whole body made the face tint the hair pink.

4. Eyes are where "zombie" lives

The zombie look came almost entirely from the eyes:

  • White sclera is wrong. Real ones are ivory, a little warm.
  • Small irises look scared. Make them bigger, with a dark ring at the edge.
  • Real eyes are wet, so they get a clearcoat that catches the light.
  • And no black ring of lashes around the whole eye.

The brows got softer and the lips went from pink to a natural rose. Small changes, and the person stopped looking scared.

5. Blinking (more specific than you'd think)

A face that never blinks reads as dead within about three seconds. But random blinking looks wrong too. Here's what worked:

  • a blink every 2 to 6 seconds, now and then two in a row
  • it shuts in 0.07 s and opens in 0.12 s. Closing is faster than opening.
  • the upper lid comes most of the way down, and the lower lid comes up a little
  • it's seeded, so a render blinks at the same moments every time. Otherwise every export of the same video comes out different.

You can also close the eyes on purpose and keyframe it, so a slow blink is just two keyframes.

6. A walk built from the feet up

The old walk swung the legs from the hip and hoped for the best, so the feet slid. The new one starts where the person touches the world:

  • Planted feet that never slide. A foot on the ground stays exactly where it landed until it lifts.
  • Heel strike, then toe roll.
  • The hips rise, sway, turn and drop with each step.
  • The shoulders turn against the hips.
  • The head stays level while everything under it moves.
  • The arms swing a beat behind the legs, and an elbow bends more as its arm comes forward.
  • The fingers are half curled. Straight fingers are the uncanny valley's favourite tell.

Standing still needs motion too: the weight shifts from leg to leg, the chest breathes, and the head wanders a little. A person who stands perfectly still looks like a statue.

7. Lip sync from text, no audio model

This is the one people ask about. You type what a person says, and their mouth says it.

No speech model, no phoneme aligner. Just letters mapped to three numbers: how open the mouth is, how wide, and how round.

const VISEMES = { a: [.85, .35, 0], // jaw drops e: [.55, .7, 0], i: [.4, .85, 0], // corners spread o: [.65, 0, .85], u: [.35, 0, 1], // lips push forward m: [0, .1, 0], b: [0, .1, 0], p: [0, .1, 0], // lips meet f: [.15, .35, 0], v: [.15, .35, 0], }; 

The text plays at 14 letters a second, and a comma or full stop pauses for a quarter of a second. Each shape is eased into the next. The mouth is averaged over 35 ms either side of the current moment, because a real mouth can't jump. Those three numbers move the face's real bones:

  • the jaw drops up to 15°
  • the mouth corners spread for "ee" and draw in for "oo"
  • the lips push forward for round sounds and meet for m, b and p

The head nods a little while they talk, which turns out to matter as much as the mouth.

If there's a real voice on the timeline, you can point the person at it. Then the mouth opens as that track gets loud, and the text still shapes it. You get the timing from the audio and the shapes from the words.

Is it perfect? No. Is it good enough that nobody watching a 10-second clip notices? Yes.

8. The stuff nobody notices until it's wrong

  • Torn clothes. The clothes were a separate shell over the body, and a bent pose pushed skin through them. Now they're painted onto the body itself by region, so there's nothing to tear.
  • Bare toes through the shoes. Each foot now gets a solid shoe built around it.
  • A droopy eye. The irises pointed straight ahead instead of where each eye was actually looking, so one sat low. Aiming each iris along its own gaze fixed it.
  • Staircase hairlines, collars and cuffs. On a low-poly mesh, painted edges follow the triangles. Smoothing them into curves made the clothes look sewn instead of cut out.
  • Crowds walking on the spot. Each person was clamped inside their area, so on a long, narrow pavement most of them stood at the edge with their legs still walking. Now they walk back and forth along it. Embarrassing, and only obvious in the wide shot.
  • Blocky silhouettes. Smoothing the body one level gave rounder jaws, shoulders and heads.

What I'd tell anyone doing this

  1. Fix the eyes first. Everything else is a percentage. The eyes decide whether it's a person.
  2. Motion beats detail. A low-poly person who blinks and breathes looks more alive than a perfect statue.
  3. Make it repeatable. Every blink, sway and glance is seeded, so the same project renders the same frames every time. That's what makes it an editor and not a toy.
  4. Delete before you add. The two biggest wins were taking things away: my own people, and the hair cards.

Riptide is open source: github.com/Natuworkguy/Riptide. The video at the top was made in it. Drop a person into a 3D world, type what they say, and press play.

If you've fought the uncanny valley yourself, I'd love to hear the one fix that finally worked for you.