Background

A lot of what I read is still on paper: letters from the bank, school handouts, a recipe, a library book. I often want to listen to them instead, and I want to follow the words while I listen, so my eyes have something to hold on to.

So I built Outloud Reader. You take a photo of a printed page (or open a PDF, or paste text), the app reads it out loud, and the phrase being spoken is highlighted on screen. It runs on iOS and Android, and it can translate a page before reading it.

This post is about the reading screen, the part people look at for minutes at a time. It turned out to be the hardest screen in the app to get right.

What I wanted from it:

  • the current phrase highlighted, the current sentence easy to find, the rest of the page calm;
  • tap any word to jump the audio there;
  • a "just listen" mode with the text hidden, which you can still peek at;
  • Japanese, Chinese and Thai behaving like any other language.

What I tried

  • Word-by-word highlighting, like karaoke. It is still in the code as an option, but it was exhausting to watch. The highlight jumped several times a second, and on a phone the eye kept chasing it instead of reading.
  • A UIKit text view. That was the original plan, since it gives you text selection and word hit-testing for free. But the highlight needed to look a specific way, and I kept fighting the framework over small visual details.
  • A long press on the text itself to peek at hidden text. On a real device it often did nothing at all. More on that below.

How I did it

1. Highlight phrases, not words

The unit of highlighting is a phrase: a few words up to the next punctuation mark, at most four of them. The phrase sits on a soft yellow plate, the current sentence is in full black, and the rest of the page is greyed out.

Grouping is a simple pass over the words of a paragraph:

private static let phraseBreaks: Set<Character> = [",", ".", ";", ":", "!", "?", "—", "…"] var phraseID = 0 var wordsInPhrase = 0 for word in paragraph.words { word.phraseID = phraseID wordsInPhrase += 1 // A phrase ends on a punctuation mark or after four words. if let last = word.text.last, phraseBreaks.contains(last) || wordsInPhrase >= 4 { phraseID += 1 wordsInPhrase = 0 } }

Punctuation always stays attached to its word, so checking the last character is enough. One extra rule I added after the first tests: a quoted line of dialogue counts as its own sentence. Otherwise the whole "Come here," she said, "and look." turned black at once, and the "current sentence" covered half the page.

2. Every word is its own view

To make every word tappable, I dropped text views entirely. The paragraph is a custom SwiftUI Layout that wraps separate word views into lines, like a paragraph, except each word can be hit with a finger:

struct WordFlowLayout: Layout { var lineSpacing: CGFloat func sizeThatFits(proposal: ProposedViewSize, subviews: Subviews, cache: inout ()) -> CGSize { let width = proposal.replacingUnspecifiedDimensions().width return geometry.size(for: subviews.map { $0.sizeThatFits(.unspecified) }, availableWidth: width) } func placeSubviews(in bounds: CGRect, proposal: ProposedViewSize, subviews: Subviews, cache: inout ()) { let frames = geometry.frames(for: subviews.map { $0.sizeThatFits(.unspecified) }, availableWidth: bounds.width) for (index, frame) in zip(subviews.indices, frames) { subviews[index].place(at: CGPoint(x: bounds.minX + frame.minX, y: bounds.minY + frame.minY), proposal: ProposedViewSize(frame.size)) } } }

The line-breaking arithmetic lives in a separate plain struct (geometry), so I can unit-test it without rendering anything.

The highlight is just the background of each word in the phrase. That gave me a bug I didn't expect: thin white hairlines between the words of a highlighted phrase. The plates sat edge to edge, and the seams showed. The fix is to let neighbouring plates overlap a little:

.background { RoundedRectangle(cornerRadius: 6) .fill(isInCurrentPhrase ? palette.plate : .clear) // Neighbouring plates overlap, otherwise hairlines show between words. .padding(.vertical, -plateOverhang) .padding(.horizontal, -2) }

3. Languages without spaces

Splitting on whitespace works for English and fails for Japanese, Chinese and Thai. A whole sentence becomes one "word", and the highlight sits on a full line. Foundation already knows where the word boundaries are, through ICU:

var starts: [String.Index] = [] token.base.enumerateSubstrings(in: token.startIndex..<token.endIndex, options: [.byWords, .substringNotRequired]) { _, range, _, _ in starts.append(range.lowerBound) }

The pieces between those starts become words, and punctuation sticks to the word before it. Latin script never goes down this path, so well-known stays one word.

4. Press and hold to peek

In "just listen" mode the text is hidden behind a calm screen. To peek, you press and hold anywhere, and the text appears until you let go.

My first version put the long press on the text inside the scroll view. It failed in the worst way: you pressed the screen and nothing happened. The scroll view claimed the touch as soon as the finger moved a tiny bit and cancelled the press. The fix was to move the gesture to a separate layer over the scroll view, and to allow any amount of drift:

ListeningCover() .opacity(isPeeking ? 0 : 1) .contentShape(.rect) // A person holding the screen to read is reading, not aiming. .onLongPressGesture(minimumDuration: 0.2, maximumDistance: .infinity) { isPeeking = true } onPressingChanged: { pressing in if !pressing { isPeeking = false } }

While you peek, the layer stays in place and only becomes transparent. If you removed it mid-press, you would end the very gesture that is holding the text open.

What went wrong along the way

  • Pages counted by characters. I first split long texts into equal chunks of characters. With PDFs this was a mess: three real pages became six "pages" in the app, with tails of a few words and sentences cut in half. Now a photo or a PDF page is always one page, and only pasted text is split, into equal parts rather than "full pages plus a tail".
  • Photos of a whole spread. People photographed two pages at once, and the reading order went wrong. The guides now ask for one page at a time.
  • Numbers and names. "3/4", "Dr.", a year, or an English name in a Russian text can come out as noise. Every language needs its own rules, and the only real test is listening to real pages.
  • Too many screens before the first page. Every screen before you hear your first page cost people. The app has no account, and the first thing you do is take a photo of a page.

What improved

  • The highlight moves calmly, phrase by phrase, and you can find your place again after looking away.
  • Any word is a tap away from being the place the audio plays from.
  • Japanese, Chinese and Thai are highlighted word by word like everything else.
  • "Just listen" mode lets you glance at the text without changing modes.
  • A finished page plays offline and with the screen off, so the phone can stay in a pocket.

The app is on the App Store and Google Play, it reads 14 languages, and it is free to try. More at outloudreader.app.

If you've built a reading or karaoke-style screen, I'd love to hear how you handled highlighting. What did you highlight: words, phrases or whole lines?