That https://api.openai.com/v1/decisions endpoint is notable because usually when OpenAI define an endpoint like that it ends up as a defecto standard for other providers.
The response to Jev should be the nail in the coffin over whether or not the AI business is a commodity market.
Out of no where Jev appeared as the next round of the price wars. Jev showed the value of System One models. A fast yes/no/confidence score not only is cheaper but also often all people want. Open source versions flood hugging face and now the big players are giving up a potentially big driver of output tokens to keep customers and race to the bottom price wise.
If I were OpenAI or Anthropic I’d be racing to make their products as sticky as possible bc ppl will flock to what’s cheapest otherwise.
A lot of places already subscribe to both. Indeed, in large enterprises it isn’t uncommon to simultaneously subscribe to Copilot, Claude, OpenAI, Cursor, AWS Bedrock, Gemini, etc — you might subscribe to different ones for different teams/projects/employees/etc, but often the approval by legal/IT/etc is generic not scoped to whoever is using it right now
If you are charged based on usage, you can “soft switch” between them really quickly.
It's a problem for the security, legal, and IT teams, but not that much for developers, unless they get really particular about their harness. On the other hand, these are the high dollar value accounts that providers want to keep; but if there's no reason to avoid switching, this market really will feel like a utility market (i.e. it'll be like switching ISPs or cell phone providers - annoying but fungible).
The AI companies want to differentiate and become something more than a commodity, even if it's as critical as a utility is.
Enterprises tend to be your largest customers. Especially when current stock values for tech companies are majority predicated on AI becoming a staple in everyday life.
Agree. Switching models with a keypress is for developer coding.
Jev and this decisions api are mostly useful for inference at scale in a workload where cost and latency matter… and that’s where evals become crucial. Could coding tools use it? Sure, but that’s probably a special case.
If you are a business dealing with with anything remotely sensitive then this is not so easy and you are basically forced to do business with a big player.
> If I were OpenAI or Anthropic I’d be racing to make their products as sticky as possible bc ppl will flock to what’s cheapest otherwise.
Hopefully people will flock to whatever product is making its mission to be commodity and the easiest to replace. Really don't want another free ingress, 100$/TB egress Cloud situation.
Ran my decisions evals (still rudimentary, less than 600 calls (UI component selection, chat charting, tag selection, PKM stuff)) on this via OpenRouter against Jev and Mercury Decide. Jev because it has replaced my mt0 efforts by sheer force of affordability (more importantly, the limits running on a MacBook Neo bring even after vocab pruning and quant insanity) and Mercury Decide because I do like dLLM efforts (and I'd like to use fewer model providers if possible).
Preliminary of course, but seems to be slower than Jev and similar to Mercury Decides latency, though not in growing linearly with the amount of input (346ms p50 and 860ms p95, (Mercury Decide also had some extremes up to 1,3s that were around 800ms today, likely preview related, it scaled far more consistently with size)), less "confidence" concerning my ambiguous UI component and response shape specific tasks (have very specific use cases for these models which Luna often fails to meet at 0.6 and lower), lead to a few failed calls which neither competitor had (4 vs 0 for both) and measured more expensive than Jev to boot by a factor of 3,1 times on average (Mercury Decide pricing I think is still unknown so no numbers there).
Basically slower, more expensive and less capable than Jev, roughly on par with Mercury Decide (provided, in my insane set of use cases and requirements that are a PKM focused Firefox fork with multiple infinite canvas using decision models to improve information synthesis from multiple sources).
Seems a bit undercooked overall and I'd rather frontier-labs don't jump on bandwagons until they can offer something competitive in price, performance or both. In fairness, though, I have yet to test image input, maybe that makes all the difference. Also, again, mine is unlikely to reflect everyones use case, so interested in seeing others results.
Didn't comment at the time, but having read up on Devday after the fact, there seems to have been a lot of that going around. Notion and GDocs, Jev, Muse, most seems to have been cloned from existing competitors (and despite infinite, ultrafast, ultra code tokens with unsandboxed Mega Astra not that amazing to boot).
Prefer less announcements, but focused and at a higher quality. Considering ChatGPT Atlas (their Chromium based browser) and its insanely fast death, I'd be skeptical to put much into any of these even if they were in some way an improvement over what is out there. Maybe focus on a fresh pre-train and some sandboxing improvements.
Well, Jev doesn't meet any real compliance requirements but OpenAI's models do. So even if Jev is faster, any customers with compliance needs will obviously pick OpenAI because they can't pick Jev out of necessity.
They can follow up in N weeks with a better one. Even if your eval is true and it’s worse, planting a flag makes sense. Some people will just use OAI because it’s OAI. No one will remember their week 2 evals in a few months.
I think releasing something like this makes sense even if it's underbaked, it's still very cheap, and if you have existing enterprise OpenAI relationship it's a lot easier to onboard something like this than set up a new Jev contract.
How these models play out is an open question but existing provider contracts and T&C are important for enterprise.
Good point, commercially, being an existing partner is always easier for adoption. Heck, why I'd like to get Mercury Decide to replace Jev myself, rather than one than two to work with.
Still surprised they even leveraged Luna for this. Given their resources in data, compute and manpower, would training a decision model from scratch take that much longer to not make sense given the cost, compute and performance advantages that would likely provide?
3 times more expensive at twice the latency with lower performance is a tough sell, though yeah, prior relationships will likely smooth some of those deficiencies over.
yeah and this isn't a long term solution, stand this up, see what value you get out of it, and in a few months you cans witch to whatever the best decision model is
i would be shocked if luna decides is less generally capable than jev. jev has failed to understand any novel domain I've given it. i have found that i use it only when "some data is better than no data"
Very task-dependent of course and mine are unique to say the least, so could see Luna being better in certain domains, even if my measurements have not shown that yet, happy for anyone to show otherwise.
For what it's worth, ran every task twice on each model, most were for some UI component synthesis and charting insanity that is a bit hard to explain, but some were simple tag selection, basic noul at threshold 60%. Essentially, whether to use the provided tag given the title of a browser tile:
Of course, tags can be a bit subjective, but in these cases, I'd argue the values provided by Jev were far more representative of my subjective assessment over Lunas. If SnP stuff on Bloomberg isn't investing, nothing is.
Goal for tagging is mainly a near instant, over writable, sane default provided to users in the background. Resolve the whole "I love using Notion/Obsidian/PKM software of your choice but spend 80% of my time just thinking about the ideal tag before starting to read" issue. Lunas output is not really helpful here.
All this is for extraordinarily simple decisions. Real world problems often are a lot more complex requiring highly structured outputs covering many output attributes and substructures, for which a conventional structured output via a documented schema is better. If instead you make twenty independent calls to a decisions API, you lose coherence among your twenty decisions. I think any hype surrounding decisions will be forgotten soon enough.
Since it is fast and understand images, I wonder if it can play video games. I have a harness setup for the LLM play EA FC but even the fastest LLMs are too slow for it. I need to try this with Decisions API
One of the examples on the docs page is it playing a video game. Doubt it’ll be able to run anything complex though. You’re simply trading accuracy for speed.
This rather didn't take long for OAI to create*, I remember people giving opinions and discussions that it won't take too long and that openAI should do it[0], so looks like they were right.
Interesting to see where all this leads us and if other major labs follow suit
Edit: decisions voice looks really interesting as well[1]
> The tulip became a luxury item and many varieties were introduced. The varieties were classified and the most sought-after, prized tulips were the streaked tulips, especially yellow or white streaks on a red or purple background. These flame-like tulips were highly sought after. Interestingly, the streaks or “flames” of the tulip petals were caused by a virus. The virus is the tulip breaking virus, or tulip mosaic virus.
I genuinely do not understand why anyone would pay OpenAI for this. Running something comparable to Jev is pretty trivial. The whole point of paying for ChatGPT is because OpenAI has a bunch of warehouses that can run a zillion-parameter model.
Running a decision model is way easier and much cheaper. Are they really just trying to capitalize on the hype here? It feels like they really have absolutely zero moat.
Yeah and OAI is twice as expensive as Jev, which is kind of my point. And more expensive than open models, which you don't necessarily have to host yourself. Pure bandwaggoning.
For my use case it will cost like $11 a month and we already have OpenaAI keys and accounts with billing in place. I don't want to run my own model infra and I don't want to get permission to set up an account with typesafe.ai
If you're in an enterprise that already has a procurement agreement with OpenAI, this means you don't have to onboard another vendor. Bucket platform strategy.
Depends on the quality of the results. These things are driven by text prompts. If it turns out the OpenAI one returns better quality results than open weight variants they'll be rewarded by the market.
Anyone using a decision model like this is going to have to spin up their own evals - these are far harder to vibe-check than regular text output LLMs.
There isn’t a moat in the sense of self hosting but you need a reason for people who don’t want that to stay on your platform. Customers save time and effort managing payments easier this way. However it’s a race to the bottom price wise.
Going to be all about branding and platform stickiness for OpenAI to make investors and creditors whole.
My opinion is similar, but for a different reason: every use case for decision models that I can think of, I don’t want the model to change in X weeks when the lab decides to “improve it” or “make it safer”.
Existing enterprise contracts? Data retention contracts (some have zero data retention contracts)? Staying with a single provider because it's easier to have everything in one place?
If you work for a company that has a 3 to 6 month onboarding period for new vendors and a lifetime commitment to maintain a whole bunch of vendor management horseshit for as long as that relationship exists, it makes a ton of sense.
Add in a bunch of model governance and oversight for anything you train yourself and it’s pretty much a slam dunk deal.
(I turned this all into a new llm plugin: https://github.com/simonw/llm-openai-decisions)
Out of no where Jev appeared as the next round of the price wars. Jev showed the value of System One models. A fast yes/no/confidence score not only is cheaper but also often all people want. Open source versions flood hugging face and now the big players are giving up a potentially big driver of output tokens to keep customers and race to the bottom price wise.
If I were OpenAI or Anthropic I’d be racing to make their products as sticky as possible bc ppl will flock to what’s cheapest otherwise.
If you are charged based on usage, you can “soft switch” between them really quickly.
The AI companies want to differentiate and become something more than a commodity, even if it's as critical as a utility is.
Jev and this decisions api are mostly useful for inference at scale in a workload where cost and latency matter… and that’s where evals become crucial. Could coding tools use it? Sure, but that’s probably a special case.
Hopefully people will flock to whatever product is making its mission to be commodity and the easiest to replace. Really don't want another free ingress, 100$/TB egress Cloud situation.
Preliminary of course, but seems to be slower than Jev and similar to Mercury Decides latency, though not in growing linearly with the amount of input (346ms p50 and 860ms p95, (Mercury Decide also had some extremes up to 1,3s that were around 800ms today, likely preview related, it scaled far more consistently with size)), less "confidence" concerning my ambiguous UI component and response shape specific tasks (have very specific use cases for these models which Luna often fails to meet at 0.6 and lower), lead to a few failed calls which neither competitor had (4 vs 0 for both) and measured more expensive than Jev to boot by a factor of 3,1 times on average (Mercury Decide pricing I think is still unknown so no numbers there).
Basically slower, more expensive and less capable than Jev, roughly on par with Mercury Decide (provided, in my insane set of use cases and requirements that are a PKM focused Firefox fork with multiple infinite canvas using decision models to improve information synthesis from multiple sources).
Seems a bit undercooked overall and I'd rather frontier-labs don't jump on bandwagons until they can offer something competitive in price, performance or both. In fairness, though, I have yet to test image input, maybe that makes all the difference. Also, again, mine is unlikely to reflect everyones use case, so interested in seeing others results.
Didn't comment at the time, but having read up on Devday after the fact, there seems to have been a lot of that going around. Notion and GDocs, Jev, Muse, most seems to have been cloned from existing competitors (and despite infinite, ultrafast, ultra code tokens with unsandboxed Mega Astra not that amazing to boot).
Prefer less announcements, but focused and at a higher quality. Considering ChatGPT Atlas (their Chromium based browser) and its insanely fast death, I'd be skeptical to put much into any of these even if they were in some way an improvement over what is out there. Maybe focus on a fresh pre-train and some sandboxing improvements.
That's a big motivator.
How these models play out is an open question but existing provider contracts and T&C are important for enterprise.
Still surprised they even leveraged Luna for this. Given their resources in data, compute and manpower, would training a decision model from scratch take that much longer to not make sense given the cost, compute and performance advantages that would likely provide?
3 times more expensive at twice the latency with lower performance is a tough sell, though yeah, prior relationships will likely smooth some of those deficiencies over.
For what it's worth, ran every task twice on each model, most were for some UI component synthesis and charting insanity that is a bit hard to explain, but some were simple tag selection, basic noul at threshold 60%. Essentially, whether to use the provided tag given the title of a browser tile:
Of course, tags can be a bit subjective, but in these cases, I'd argue the values provided by Jev were far more representative of my subjective assessment over Lunas. If SnP stuff on Bloomberg isn't investing, nothing is.Goal for tagging is mainly a near instant, over writable, sane default provided to users in the background. Resolve the whole "I love using Notion/Obsidian/PKM software of your choice but spend 80% of my time just thinking about the ideal tag before starting to read" issue. Lunas output is not really helpful here.
- Cost : It is the same for both scenarios $0.10 per 1M tokens
- Speed : decisions is 10x faster than responses API
- Quality : I guess if we compare with luna which is a pretty good model it itself, both will be at par
So essentially it has to do more with speed vs any other factor.
Open source classifier models you can run and train locally on CPU
Will this get folded into models / post training pipelines at some point and make them better at calibrated outputs?
[0] - https://news.ycombinator.com/item?id=49976996
Interesting to see where all this leads us and if other major labs follow suit
Edit: decisions voice looks really interesting as well[1]
[0]: https://news.ycombinator.com/item?id=49802161: OpenAI is well positioned to fast-follow Jev
[1]: https://developers.openai.com/api/docs/guides/decisions-voic...
Source: https://www.canr.msu.edu/news/tulip_mania_the_history_of_the...
What's old is new.
Running a decision model is way easier and much cheaper. Are they really just trying to capitalize on the hype here? It feels like they really have absolutely zero moat.
You could ask the same question about why anyone would rent a VPS. I can just run my own hardware, it's just a computer!
Buy vs rent is not just about what's possible, it's about what's economic.
Anyone using a decision model like this is going to have to spin up their own evals - these are far harder to vibe-check than regular text output LLMs.
Going to be all about branding and platform stickiness for OpenAI to make investors and creditors whole.
There are probably a lot more reasons.
Add in a bunch of model governance and oversight for anything you train yourself and it’s pretty much a slam dunk deal.