When you talk to ChatGPT it feels like it is listening. I wanted to know what actually arrives on the other side, so I picked the simplest thing a voice carries besides words: a melody. I hummed Happy Birthday to ChatGPT, in every way the app lets me speak to it, and then ran the same question at scale through the voice models OpenAI sells to developers.
Two buttons, two different things
The microphone and the voice button do not do the same job. The small microphone is Dictation: OpenAI's help page describes it as turning a recording into editable text, which you then send like anything you typed. The other button opens Voice, a spoken conversation that today runs on a model called GPT-Live-1.1.
With Dictation the model only ever sees text, and a hum has no text. With Voice the sound itself reaches a model, so the question becomes what that model does with it.
Dictation wrote nothing. I hummed Happy Birthday twice, once as "hmmm" and once as "na na na", and both times the text box stayed empty. That is the expected outcome of a transcriber: there were no words to write down.
Voice told me it could not do it. I said I was going to hum a song and asked which one it was. Before I had hummed a note it answered, in Spanish, which is how I was speaking to it:
I'd love to, but right now I can't identify songs from humming or melodies. We're working on it and I'll be able to soon. If you like, tell me a line of the lyrics and we'll see if we can track it down together.
I tried again in a new chat, humming with no warning and then asking what song it was, and got the same answer. Then I sang "Cumpleaños feliz" with its words, and it replied at once that it was Happy Birthday.
So the product is honest about the limit. What I could not tell from the app is whether the model behind it hears a melody and is held back, or does not hear it well enough to try. For that I went to the models directly.
The same question, through the API
I tested three ways of getting a voice into an OpenAI model, all deployed on Azure OpenAI:
- Transcribe, then ask:
gpt-4o-mini-transcribewrites down the audio andgpt-6-luna, a text model, answers from the transcript. This is what Dictation does, and it is also the baseline that hears nothing but words. -
gpt-audio-1.5: a chat model that takes the audio file as input. -
gpt-realtime-2.1: the speech-to-speech model developers use to build voice agents.
None of these is GPT-Live, the model inside ChatGPT's Voice, which I could only test through the app. They are OpenAI's voice models as a developer gets them.
For recordings I used MTG-QBH, a research set of 118 clips of people singing or humming pop songs with no accompaniment. I took the first 20 seconds of each and asked every route which song it was. A text model graded the answers, and I checked its verdicts against a literal title match. From two transcripts per clip I sorted them into 76 sung with lyrics, 30 with no words at all, and 12 in between.
Songs named correctly on MTG-QBH. The route that only reads a transcript does as well as the two that receive the audio, and all three collapse when the words go away.
With lyrics, the route that only reads a transcript named 71 of 76 songs, the same as gpt-audio-1.5. gpt-realtime-2.1 named 74. The audio added nothing I could measure over the words. Without words, the three routes named 0, 1 and 1 of 30, and the single hit was "Ob-La-Di, Ob-La-Da", whose syllables are the title.
When gpt-realtime-2.1 did guess, it guessed "Hey Jude". I ran each wordless clip three times: 64 of its 90 answers were "unknown" and 19 were "Hey Jude", for hums of "Bohemian Rhapsody", "Smells Like Teen Spirit" or "Sweet Home Alabama". A "na na na" was enough to call it.
Maybe they hear it and cannot name it
The obvious objection is that naming a song is hard even if you hear the melody. So I made the task easy. MLEnd Hums and Whistles is a set of people humming and whistling eight film songs. I sampled 120 hums, 15 per song, from 95 different people, and gave the model the list of eight to choose from. Guessing blindly scores 12.5 %.
gpt-realtime-2.1 picked the right one in 11 of 120, or 9 %. The transcript-only baseline, which for most of these clips received an empty transcript, got 16 of 120. Hearing the hum was no better than not hearing it.
What gpt-realtime-2.1 chose for 120 hums when given the eight titles. It spread its answers over three of them whatever was hummed, which is what choosing without hearing the melody looks like.
Its choices did not follow the audio. Every song was hummed 15 times, yet the model answered Harry Potter 50 times, The Pink Panther 37 and Star Wars 32, and the other five songs once between them. Asked openly, with no list, it named 1 hum of 119.
My own voice, and a model that does hear it
I recorded myself so I could hold the song fixed and change one thing at a time. Four hums (Happy Birthday, Ode to Joy, Jingle Bells and The Beatles' "Dear Prudence"), Happy Birthday sung with its words, and one trick recording: the words of Happy Birthday sung to the tune of Jingle Bells.
My recordings. The three OpenAI routes fail on every hum and succeed as soon as there are words; a small model that only sees the melody's contour finds the three hums it has in its catalogue.
No route named any of the four hums; all of them named the song once it had words. The trick recording is the clearest result. I asked separately for the song the words belong to and the song the tune belongs to, and all seven answers said the tune was Happy Birthday. Asked directly about the melody, the models answered from the lyrics.
A small model that looks only at the melody had no trouble. hum2song is an open model that turns the pitch contour of a recording, how the note moves up and down over time, into a vector and compares it with the contours of known tunes. It runs on my laptop. I gave it a catalogue of 15 well-known melodies and my three hums that are in it, and it ranked the right tune first for each. Given the same list of 15 to choose from, gpt-realtime-2.1 picked the right one in 0 of 9 tries and gpt-audio-1.5 in 2 of 9; the transcript-only route, with nothing to go on, got 1 of 3. Three hums by one person are an example, not a measurement, but they show the information is in the recording.
What does get through
The voice models are not deaf to everything beyond words. On RAVDESS, where actors say the same sentence in different emotions, I used 48 clips across four emotions: gpt-audio-1.5 got the emotion in 37 and gpt-realtime-2.1 in 28 of 47, against 12 expected by chance. And when I sang two notes more than an octave apart, gpt-realtime-2.1 said which was higher all six times I asked.
So some of the sound arrives: enough for a coarse emotion or a wide jump in pitch, not enough to follow a melody. That is the precise version of the title. Tone of voice was something I only probed, and I will not claim more for it than those numbers.
What I take from it
If the words are the same, assume the model got the same message. That is the practical reading for anyone building on these models, a phone agent for customer service for example: a pipeline that transcribes first hands the model text and nothing else, and a speech-to-speech model hears more than that, but less than the conversation feels like.
This is the same shape I found with images in One Grey Level Out of 255: what a model perceives is not what I would perceive in its place, in either direction, and the only way to know is to measure.
The limits are worth stating. I tested ChatGPT itself by hand, a few times, in Spanish, and it declined rather than tried. The measurements are on three OpenAI models through the API, on the first 15 to 20 seconds of each recording, and the MTG-QBH clips come from only 17 singers. I did not test other companies' models.
ChatGPT says it is working on this, and it may be fixed by the time you read it. The interesting question is the one behind it: what would it take for a model like this to hear a tune, and whether handing it a tool that does is the shorter road. That is the next piece.
Originally published on javieraguilar.ai
Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.




