> Our first public model is Jev, available today in early access. Jev achieves similar levels of intelligence on System One tasks compared to existing LLMs, while being two orders of magnitude faster and more efficient.
Where is the 20-200x? misleading title.
Also I don't get it, is that decision tree for automation that can apply to generic problems?
I would love for things like this to be accessible via hubs like open router or AWS bedrock. It's hard to justify adding new model vendors directly with all the heightened concerns about privacy and security, but if bold new capabilities are added to a centralized already-vendor like AWS, technical people can adopt them without going through a whole compliance/purchasing/vendor review process. And an extra middleman tax is well worth it when the cost savings of the model itself can be one-two orders of magnitude.
This sounds good but so far all claims just sound like marketing terms. I'd love to see real proof. e.g. "RLCD" and "parallel sampling" have nothing to back it up.
also "70-500ms vs 3-329 seconds" are apples-to-oranges unless the LLM baseline is doing comparable work (e.g., long chain-of-thought). If Jev is skipping generation entirely for a narrow structured task, of course it's faster.
Nonetheless i want this to be true, so I'm looking forward to Jev
They have various benchmarks, e.g. how much time it takes them to do wikipedia page -> page games. Jev seems to take the same or fewer hops but in ~10x less time and for ~10x less money.
It's totally reasonable to compare against LLMs doing chain of thought if it gets comparable performance.
BTW it was not multi model playing doom, it was passing structured input and getting structured output. Its not what I thought: frames of video passed and real time game play.
It seems like the docs[0] are a better explanation? The comparison to llm tokens is kinda confusing.
It looks like the model takes as input a state (structured text? not sure if multi-modal) and a question (as a "Choice", "Score", or "Noul") with some additional augmentations possible. Then outputs the question's answers as appropriate (e.g. a choice, probabilities, confidence).
On the AI primer page, it looks like they do RLCD from a pre-trained base model?
I could see this being fantastic for classification tasks. Last year I shifted from using LLMs for bulk data classification tasks (1M transcripts) to generating embeddings and categorizing based on cosine similarity. It saved a ton of costs and time, but wasn't as accurate as LLMs. This seems like it can give me Terra-level classification ability with the cost/speed I need.
It could be used for coding if you gave it an AST.
If you work at TypeSafe please try this.
Side note: This is probably how LLMs would perform with better encoders and next-latent prediction, so eventually those will beat this architecture out. Still amazing though.
I've implemented tree-sitter in pi before, and while it works, I have no real proof it saves me tokens, or is more accurate. I think a better implementation is a model that's trained for AST's, not just "use tool, see what happens".
I'd love to do research on this when I have the time.
The doom video is also in the article itself (headline: "Doom").
I suppose this is the same video as the one from the parent comment, but I don't know for sure - I don't have a twitter account and the above link doesn't work for me.
Super intrigued by this - large scale automation using LLMs is quite annoying due to deprecation cycles of models from frontier labs and cost of running your own being prohibitive when you have a blend of them.
Signed up for the beta! :) would love to put this through some real-world shootouts against traditional LLMs to see where this type of model really excels.
I’m guessing it might be able to replace maybe 40-70% of LLM calls for a given pipeline depending on the business task, cutting the API costs on those calls by an order of magnitude.
Why did they pick the name System One? It's not really explained what "System One tasks" and "System One shaped queries" are. Things that need a fast response?
Does this imply it's a very small model? I couldn't find anything about the model itself.
> We deliberately chose not to publish performance against public benchmarks. In fact, we plan to only have one-off evals when we make product updates.
lol, I bet they would publish them if their score on those benchmarks were good.
> LLMS
> Strings / generated text. Strings are flexible and can be anything: chat responses, code, hallucinations, refusals, or even type-safe structured values. To be used by software, responses need to be parsed + validated. There is also always some risk that the AI goes off the rails.
> Jev
> Type-safe structured values. Possible outputs and structure are defined in advance. The model never makes type errors. All answers are accompanied with calibrated probabilities and confidence scores.
I mean, this isn't even remotely comparable to LLMs so why compare? Also, why are they bringing up AGI given there approach is so restrictive that what they're building literally cannot have the creativity required for AGI? The video is 100% marketing slop...
The bulk of the application of LLMs is that they generate reasonably reliable text which doesn't need to be defined in advanced. I'm sure there is a niche for this and congrats to the team, but please let's not hype this as if it's the next big thing in AI...
It looks like a specialized encoder-only(-ish) transformer with scalar and ordinal output heads. Acausal in effect, maybe? Probably not even autoregressive?
I'd use this as a tool an LLM can use for specialized tasks. It's not AI in itself.
Where is the 20-200x? misleading title.
Also I don't get it, is that decision tree for automation that can apply to generic problems?
I guess you can choose your provider still? But isn't the point that the lowest bidder is doing inference?
So, assuming this is not vaporware, this would raise the tide for everyone because it shows what's possible.
also "70-500ms vs 3-329 seconds" are apples-to-oranges unless the LLM baseline is doing comparable work (e.g., long chain-of-thought). If Jev is skipping generation entirely for a narrow structured task, of course it's faster.
Nonetheless i want this to be true, so I'm looking forward to Jev
It's totally reasonable to compare against LLMs doing chain of thought if it gets comparable performance.
It looks like the model takes as input a state (structured text? not sure if multi-modal) and a question (as a "Choice", "Score", or "Noul") with some additional augmentations possible. Then outputs the question's answers as appropriate (e.g. a choice, probabilities, confidence).
On the AI primer page, it looks like they do RLCD from a pre-trained base model?
[0]:https://docs.typesafe.ai/concepts/system-one
Either way, really cool and impressive.
>Think of Jev as a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out.
If you work at TypeSafe please try this.
Side note: This is probably how LLMs would perform with better encoders and next-latent prediction, so eventually those will beat this architecture out. Still amazing though.
I'd love to do research on this when I have the time.
That's what I was insinuating through "better encoder"; the model creating more efficient representations of ASTs using something like JEPA
> [them] Output tokens: FREE (too cheap to meter).
I'm very confused by this.
The doom demo is quite cool
https://x.com/completeskeptic/status/2099925687465570372
I suppose this is the same video as the one from the parent comment, but I don't know for sure - I don't have a twitter account and the above link doesn't work for me.
I can see the individual tweets in the browser while not signed in though.
I’m guessing it might be able to replace maybe 40-70% of LLM calls for a given pipeline depending on the business task, cutting the API costs on those calls by an order of magnitude.
Does this imply it's a very small model? I couldn't find anything about the model itself.
Yes, that’s the kind of attitude I want to see in these model releases
lol, I bet they would publish them if their score on those benchmarks were good.
I think we'll see a bunch of different architectures over the next five years.
edit: looks like a framer export where there is a text stroke being applied :|
> LLMS > Strings / generated text. Strings are flexible and can be anything: chat responses, code, hallucinations, refusals, or even type-safe structured values. To be used by software, responses need to be parsed + validated. There is also always some risk that the AI goes off the rails.
> Jev > Type-safe structured values. Possible outputs and structure are defined in advance. The model never makes type errors. All answers are accompanied with calibrated probabilities and confidence scores.
I mean, this isn't even remotely comparable to LLMs so why compare? Also, why are they bringing up AGI given there approach is so restrictive that what they're building literally cannot have the creativity required for AGI? The video is 100% marketing slop...
The bulk of the application of LLMs is that they generate reasonably reliable text which doesn't need to be defined in advanced. I'm sure there is a niche for this and congrats to the team, but please let's not hype this as if it's the next big thing in AI...
https://x.com/CompleteSkeptic/status/2099925682726002904
Reminds me of this: https://www.reddit.com/r/ITcrowd/comments/tg05j1/i_cant_beli...
I'd use this as a tool an LLM can use for specialized tasks. It's not AI in itself.