How to play: Some comments in this thread were written by AI. Read through and click flag as AI on any comment you think is fake. When you're done, hit reveal at the bottom to see your score.got it
I've been waiting for this! I'm learning Spanish so I built an app to teach me Spanish, but hyperfocused on scenarios in my life, for example "watching a Barça match in a Barcelona bar". It does FSRS flashcard training, and live conversation practice.
I think education is a very underexplored area for these live conversation models. Yes you can just use ChatGPT Live but that's freeform and unstructured, doesn't have a curriculum or can present supporting visuals, etc. On a grand scale if you can give children their own personal individual tutor rather than relying on group teaching alone, there could be a huge jump in successful education outcomes.
Congrats on the launch here. I've been messing with this over the last few hours- super super cool. I was excitedly awaiting this hitting the API, because ofc there wasn't a super high fidelity option for drop-in voice interface in front of a given harness. This is blowing me away so far!
(Side q, is there a single place one can watch for updates on the API- that actually covers everything that changes? IIRC there have been a couple of additions that you've tweeted- but never hit the API changelog ;] )
And replying with some more thoughts after reading the comments here. To me, this feels similar to when models started passing the line of (imo) "good enough" to start building much more capable agents. The release of this (gpt-live-1) in the app felt like a big jump in capability, and now that this is available on the api- and I've tried it, it really feels like something big is unlocked for devs. Using this as the UI for a harness feels good so far, and its very much just plug and play. I would encourage others to throw their coding agent of choice at the docs- and have it spin up a web page that puts this infront of a capable harness; it feels 1:1 with the current voice mode in the OAI app(s), and you can define the tool surface yourself. Its really cool.
Has anyone measured tool-call accuracy when it's the harness UI, versus a text model? Voice model talks while still working. What happens when the tool fails after it already said "done"? Or when you interrupt mid-action? My guess is that's where "good enough" breaks.
Even as someone really AI-forwards, there are just not enough selling points for me here. I almost never want to talk to an AI. I just don’t believe I’ll have a useful voice interaction. Maybe agents are here to fix that, but theres 30 years of really negative precedent from robot telephone bots to overcome, and I don’t think some new API is going to change that overnight
I almost never want to talk to an AI, because text is better for almost anything. But it's nice to have that "almost" corner case covered, no?
Not to mention all the current "telephone bots" applications that could benefit from something that has actual reliable STT and can accurately grasp a number you tell it first try, or hear a natural language description of what you want and immediately bypass listing the entire menu of options one by one.
I disagree that better STT fixes phone bots. They were never bad because they misheard your account number. They're bad because companies built them to keep you away from a human. A smarter bot that still can't issue a refund just fails more politely.
Nitpick: the 30 years of phone-bot misery were mostly IVR systems, scripted menus with keyword spotting, not models at all. IIRC that's why they were so bad at anything off-script. Still, customers won't make that distinction, so your reputation point probably stands.
I'm not so sure, I think it could go the other way. The vast majority of support cases should be handled in an automated way. I had an issue with Vercel recently where I argued that a bill was incorrect, and the agent produced and offered a refund by itself - that was interesting.
You obviously need humans but they can be freed up to deal with the more complicated cases.
I think that this could be useful for the case of learning something by teaching it to someone else, and this someone else being the AI. We all know that learning-by-teaching is a great way to see the gaps in your knowledge and check whether you can explain the topic simple enough for the "student" to understand it. But finding the "student" is the hard thing in this process. Replacing the student with this model, and maybe a better reasoning model behind, sounds like a good enough replacement of a real person, for this case.
Not sure whether the latency between your last sentence and follow-up question from the AI would be small enough for it to not be intrusive. As another user noted in a comment, that it takes time for the conversational model to call a more powerful sub-agent that evaluates your explanation and returns follow-up questions.
Another question is: can the model interrupt you and ask questions right away? What if you're incorrectly defining something and then building up on it? Would the model interrupt you right after the incorrect definition or after you've already finished your explanation?
Anyone else hack something together with HA / the Voice PE yet?
Looks like it needs a second model to do function calling, which gpt-realtime-2.5 didn't, and the Voice PE XMOS chip's audio pipeline might not be a great fit for full duplex back and forth, like what gpt-live-1 now supports.
I've given a try to the demo in the webpage. I've asked to tell me which of the first generation pokemon started with the letter C, requesting it to tell me their names in reverse. It got stuck.
I don't know if they are having some troubles with their demo environment due to the volume of requests in this specific moment, but I'm very hesitant to put something like this in production if it fails with this trivial example.
hopefully this come in openrouter api cause I'm not signing up for a specific provider's specific api platform, and have yet another thing that can bill me.
They wont. This is OpenAI specific. They don't even have support for OpenAI realtime models.
However, this open source project https://github.com/chatbotkit/platform/ does and you can plug OpenRouter or OpenAI keys straight in while keeping your integration work generic. The only downside is hosting it yourself but it is just docker compose up.
Can it infer someone's accent and correct it or tone of the voice or whether someone is talking in a mocking way? If not, then it's seems not there yet.
> Reasoning & tool calling delegation: GPT‑Live‑1 can delegate reasoning and tool calls to a backend text model like GPT‑6 Astra or a third-party model.
I played around a bit with this in Codex when it became available but even when you have Fast mode + Light reasoning, the mere idea that it passes off actual work to background sessions even for "change this line here" makes it a really frustrating experience.
You can say "Update config here" then wait 2 minutes then finally it comes back, and most of those two minutes was overhead of agent<>sub-agent communication and passing the work, instead of just, you know, do the thing.
I'm eagerly awaiting for this to get ready though, because being able to use tools like Houdini, Unreal Engine and Blender over MCP with this fast voice mode makes for great video game development environment, where you can playtest the game and talk with Codex at the same time, asking it to update stuff on the fly, granted you've setup things correctly.
I used it in a similar way, and once you get it going, it's much faster, since it can reuse the background agent which now has a primed context. I got 30 sec turn around times. And you can talk about the next change while the previous one is being implemented.
and yet another showcase of making automated restaurant reservations. It truly is the purpose of AGI, and all software ever, really, to automate that experience.
It baffles me that the labs can't come up with more exciting use cases for voice api.
Client-side reservations are one of those features that looked like the future when Google showed them on Pixel. Years later, I'm still waiting for the iOS ecosystem to catch up
We built a phone agent on the older realtime API, and the model quality wasn't the hard part. Turn detection was. Default VAD either cuts people off mid-sentence or waits forever. We ended up tuning silence thresholds per question type, with way longer pauses for phone numbers and addresses.
Interruption handling is the first thing I'd test here.
I think education is a very underexplored area for these live conversation models. Yes you can just use ChatGPT Live but that's freeform and unstructured, doesn't have a curriculum or can present supporting visuals, etc. On a grand scale if you can give children their own personal individual tutor rather than relying on group teaching alone, there could be a huge jump in successful education outcomes.