We have solved this problem by working with the providers to implement a prices and models API that we scrape, which is how we keep our marketplace up to date. It's been a journey; a year ago it was all happening through conversations in shared Slack channels!
The pricing landscape has become more complex as providers have introduced e.g. different prices for tokens depending on prompt length, caching, etc.
I do believe the right lens on this is actually the price per token by endpoint, not by model; there are fast/slow versions, thinking/non-thinking, etc. that can sometimes also vary by price.
The point of this comment is not to self promote, but we have put a huge amount of work into figuring all of this out, and have it all publicly available on OpenRouter (admittedly not in such a compact, pricing-focused format though!)
By endpoint, do you mean price by token by API shape? Perhaps my phrasing is even more confusing but that is how I see it. I.e. there are API "shapes" for which as long as the shape of the API is the same, my application can use it interchangeably with others. Other dimensions are quality, speed, acceptable error rates, etc., which naturally influence pricing.
First poster could have approach better too. Like "Cool site! I think I may see an error on one item?". Instead of going right to a 'wrong' angle as if all the data should be discredited. I get highly triggered by this too.
This is great, but as others have mentioned the UX problem is more complicated than this:
- for other models there are providers that serve the same model with different prices
- each provider optimizes for different parameters: speed, cost, etc.
- the same model can still be different quantizations
- some providers offer batch pricing (e.g., Grok API does not)
And there are plenty of other parameters to filter over- thinking vs. non-thinking, multi-modal or not, etc. not to even mention benchmarks ranking.
https://artificialanalysis.ai gives a blended cost number which helps with sorting a bit, but a blended cost model for input/output costs are going to change depending on what you're doing.
I'm still holding my breath for a site that has a really nice comparison UI.
but this is is not much user friendly unless you already know what models you want to compare. I would prefer I switch some toggles or make kind of a query what kind of models I'm looking for my use case and then sort by speed or price at the end e.g query:
I want model:
1) with audio input
2) minimum 50 tps speed
3) max price less than $1 input and less than $3 output
4) need to support english only or need to support polish etc.
sort by WER or some benchmark, dispaly charts etc.
edit:
extra bonus if I can tell how big typical my prompt will be like 20 seconds audio ant it will figure out how many tokens it will be because e.g. gemini 2.0 flash hide it very deep that its supposed to be 32 tokens per 1 second. Same hard to find how many tokens is for image input sometimes. Would be good where I can also attach some text or sample input for a query to do the calculation
Ideally write such prompt and it show me results or map this prompt to SQL that executes in your data so I can tweak SQL query on website. It doesn't have to be SQL can be some other simple but deterministic query language.
I think it would be very hard to make a fair comparison. Best you could do is probably make the trade-offs clear and let people make their own choices. I think it could be cool to make something like a token exchange where people put up their requirements, and then companies offer competing services that fit those requirements. Would be cool to let random people offer to their compute, but you would need to find a way to handle people lying about their capabilities or stealing data.
yeah I am going to add an experiment that runs everyday and the cost of that will be a column on the table. It will be something like summarize this article in 200 words and every model gets the same prompt + article
For me, and I suspect a lot of other HN readers, a comparison/benchmark on a coding task would be more useful. Something small enough that you can affordably run it every day across a reasonable range of coding focused models, but non trivial enough to be representative of day to day AI assisted coding.
One other idea - for people spending $20 or $200/month for AI coding tools, a monitoring service that tracks and alerts on detected pricing changes could be something worth paying for. I'd definitely subscribe at $5/month for something like that, and I'd consider paying more, possibly even talking work into paying $20 or $30 per month.
There was a time when it was unbelievably frustrating to navigate the bunch of marketing pages required to find the cost of a newly announced model, now I just look at OpenRouter to find pricing.
You can, almost, convert the number of nodes to gb of memory needed. For example, Deepseek-r1:7b needs about 7gb of memory to run locally.
Context window matters, the more context you need, the more memory you'll need.
If you are looking for AI devices at $2500, you'll probably want something like this [1]. A unified memory architecture (which will mean LPDDR5) will give you the most memory for the least amount of money to play with AI models.
I bought a Mac Mini M2Pro 32G 18 months ago for $1900. It is sufficient to run good up to and including 40B local models that are quantized.
When local models don’t cut it, I like Gemini 2.5 flash/pro and gemini-cli.
There are a lot of good options for commercial APIs and for running local models. I suggest choosing a good local and a good commercial API, and spend more time building things than frequently trying to evaluate all the options.
Are there any particular sources you found helpful to get started?
It's been a while since I checked out Mini prices. Today, $2400 buys an M4 Pro with all the cores, 64GB RAM, and 1TB storage. That's pleasantly surprising...
the local side of things with an $7,000 - $10,000 machine (512gb fast memory, cpu and disk) can almost reach parity with regard to text input and output and 'reasoning', but lags far behind for multimodal anything: audio input, voice output, image input, image output, document input.
there are no out the box solutions to run a fleet of models simultaneously or containerized either
so the closed source solutions in the cloud are light years ahead and its been this way for 15 months now, no signs of stopping
> The only place I am aware of is going to these provider's individual website pages to check the price per token.
Openrouter is a good alternative. Added bonus that you can also see where the open models come in, and can make an educated guess on the true cost / size of a model, and how likely it is it's currently subsidised.
A limitation though, at least the last time I checked, is that you only get a single provider returned per model. That’s fine for the major commercial models that all have the same pricing on each provider, but makes it hard to rely on for open source models, which tend to have many providers offering them at different price points (sometimes very different price points—like 5x or 10x difference).
Can you gather historical information as well? I did a bit of spelunking of the Wayback Machine to gather a partial dataset for OpenAI, but mine is incomplete. Future planning is well-informed by understanding the trends — my rough calculation was that within a model family, prices drop by about 40-80% per 12 months.
Yeah I am planning on setting up automatic scraping and just having my own database. Maybe could add historical data beyond as well but just gonna save all my own data for now
I've run into this a ton of times and these websites all kinda suck. Someone mentioned the OpenRouter /models endpoint in a sibling comment here, so I quickly threw this together just now. Please feel free to PR!
llm-pricing opus-4 -v
=== ANTHROPIC ===
Model: anthropic/claude-opus-4
Name: Anthropic: Claude Opus 4
Description: Claude Opus 4 is benchmarked as the world's best coding model, at time of release,
bringing sustained performance on complex, long-running tasks and agent workflows. It sets new
benchmarks in software engineering, achieving leading results on SWE-bench (72.5%) and
Terminal-bench (43.2%).
Pricing:
Input: $15.00 per 1M tokens
Output: $75.00 per 1M tokens
Cache Read: $1.50 per 1M tokens
Cache Write: $18.75 per 1M tokens
Per Request: $0
Image: $0.024
Context Length: 200000 tokens
Modality: text+image->text
Tokenizer: Claude
Max Completion Tokens: 32000
Moderated: true
That’s been the case for a lot of the models. As they release new models those models often include optimizations that bring runtime costs down. I don’t have the data to back it up but it’s felt like chip cycles. There is a new model that is better but more expensive. Then further iterations on that model being costs down.
KV caching is priced and managed quite differently between providers as well. Seeing as it becomes a huge chunk of the actual tokens used, wondering if there's an easy way to compare across providers.
Maybe I am blinded by my own use case, but I find the caching pricing and strategy (since different providers use a different implementation of caching as well as different pricing) to be a major factor rather than just the "raw" per token cost, and that is missing here, as well as on the Simon Willison site [1]. Do most people just not care / not use caching that much that it matters?
Nice! It will be good to also pull in leaderboard rankings and/or benchmarks for each of these models, so we understand capability perhaps from lmsys (not sure if there is a better source)
We are working on a similar problem, https://apiraces.com,
to personalize the cost calculation of your llm api use case,
We have uploaded mostly the openrouter api models, but trying to do it in a useful way to personalize calculation and comparison. If someone would like to test or have a demo, we will be glad for any feedback.
I'd love to see this data joined with common benchmarks, in order to see which models get you the most "bang for your buck", i.e. benchmark score / token cost
Does anyone have an API that maintains a list of all model versions for a provider? I hand-update OpenAI into a JSON file that I use for cost reporting in my apps (and in an npm package called llm-primitives).
Check out the source code to the vercel AI SDK. I’ve noticed that they broker calls out to various LLMs and then seem to return the cost as part of the response. So I’m thinking that this data could well be in there somewhere. Away from my desk right now so can’t check.
1. Pull some large tech company’s open source’ tools’ JS file
2. Extract an internal JSON blob that contains otherwise difficult information
3. Parse it and use what I need from within it for my tool
What tools / experiments out there exist to exercise these cheaper models to output more tokens / use more CoT tokens to achieve the quality of more expensive models?
eg, Gemini 2.5 flash / pro ratio is 1 1/3 for input, 1/8 for output... Surely there's a way to ask Flash to critique it's work more thoroughly to get to Pro level performance and still save money?
what's point of comparing token prices? especially for thinking models.
Just now I was testing the new Qwen3-thinking model. I've run the same prompt five times. The costs I got, sorted: 0.0143, 0.0288, 0.0321, 0.0389, 0.048 .
And this is for single model.
Also, in my experience, sonnet-4 is cheaper than gemini-2.5-pro, despite token costs being higher.
Love it! It's going on my toolbar. I face the same problem, constantly trying to hunt down the latest pricing which is often changing. I think it's great that you want to add more models and features, but maybe keep the landing page simple with a default filter that just shows the current content.
this is great, I've always wanted something like this, do you think you can add other model metadata, like api name (`gemini-2.5-pro`), context length, modalities, etc
They all tokenize a little differently so they are not exactly 1-1. However I plan on addressing this by having each model complete a test task and getting the actual price from each api + token count to make a real 1-1 comparison.
Are we really at a point already where we're treating tokens as a commodity? I certainly would not consider a token generated by Claude or Gemini to be of similar value to a token by Copilot, for example.
can we not just self host, expose things through VPN, and something that needs sharing with the world, then tunnel through some cloud server to keep the internal servers secure?
I am newly to this hobby, but would like to know more about what experienced person things and do.
tldr; low effort website that only contains 26 Google, OpenAI and Anthropic models and only input and output prices but no info about prompt cache and prompt cache prices. For a list of 473 models of 60+ providers with input, output, context, prompt caching and usage: https://openrouter.ai/models
(no affiliation)
Don't use it then. No need to shit on the effort someone has put into. Clearly it is a starting point which they seem keen to iterate on. If you can't say something kind, best not to say anything.
Plus maybe half an hour to verify and correct the prices and another few minutes for domain and hosting.
The author posted it himself, so why not spend an hour or two more and have a decent list with at least half a dozen providers and 100 models? In its current state it is just a mockup.
It is since 3 hours on the top of the front page, if the author had added one per minute to the .json, then it would now be 200 models.
Well, openrouter lists currently 473 models of dozens of providers. Three providers and 26 models is 2 minutes of Google search. I stand by my word to call that low effort.
We have solved this problem by working with the providers to implement a prices and models API that we scrape, which is how we keep our marketplace up to date. It's been a journey; a year ago it was all happening through conversations in shared Slack channels!
The pricing landscape has become more complex as providers have introduced e.g. different prices for tokens depending on prompt length, caching, etc.
I do believe the right lens on this is actually the price per token by endpoint, not by model; there are fast/slow versions, thinking/non-thinking, etc. that can sometimes also vary by price.
The point of this comment is not to self promote, but we have put a huge amount of work into figuring all of this out, and have it all publicly available on OpenRouter (admittedly not in such a compact, pricing-focused format though!)
https://github.com/tekacs/llm-pricing
[1] https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-fla...
edit: my bad I was wrong shouldnt have responded like this
Your website reports 0.30$ for input and that wouldn't make any sense as it would be priced the same as the bigger Flash model.
Put a really really bad taste in my mouth.
- for other models there are providers that serve the same model with different prices
- each provider optimizes for different parameters: speed, cost, etc.
- the same model can still be different quantizations
- some providers offer batch pricing (e.g., Grok API does not)
And there are plenty of other parameters to filter over- thinking vs. non-thinking, multi-modal or not, etc. not to even mention benchmarks ranking.
https://artificialanalysis.ai gives a blended cost number which helps with sorting a bit, but a blended cost model for input/output costs are going to change depending on what you're doing.
I'm still holding my breath for a site that has a really nice comparison UI.
Someone please build it!
We have a simple model comparison tool that is not-at-all-obvious to find on the website, but hopefully can help somewhat. E.g.
https://openrouter.ai/compare/qwen/qwen3-coder/moonshotai/ki...
I want model:
1) with audio input
2) minimum 50 tps speed
3) max price less than $1 input and less than $3 output
4) need to support english only or need to support polish etc.
sort by WER or some benchmark, dispaly charts etc.
edit:
extra bonus if I can tell how big typical my prompt will be like 20 seconds audio ant it will figure out how many tokens it will be because e.g. gemini 2.0 flash hide it very deep that its supposed to be 32 tokens per 1 second. Same hard to find how many tokens is for image input sometimes. Would be good where I can also attach some text or sample input for a query to do the calculation
Ideally write such prompt and it show me results or map this prompt to SQL that executes in your data so I can tweak SQL query on website. It doesn't have to be SQL can be some other simple but deterministic query language.
- An image will take 10x token on gpt-4o-mini vs gpt-4.
- On gemini 2.5 pro output token are token except if you are using structure output, then all character are count as a token each for billing.
- ...
Having the price per token is nice, but what is really needed is to know how much a given query / answer will cost you, as not all token are equals.
One other idea - for people spending $20 or $200/month for AI coding tools, a monitoring service that tracks and alerts on detected pricing changes could be something worth paying for. I'd definitely subscribe at $5/month for something like that, and I'd consider paying more, possibly even talking work into paying $20 or $30 per month.
Can you elaborate this? I don’t quite understand the difference.
https://ai.google.dev/gemini-api/docs/tokens
Are there any tutorials you can recommend for somebody interested in getting something running locally?
You can, almost, convert the number of nodes to gb of memory needed. For example, Deepseek-r1:7b needs about 7gb of memory to run locally.
Context window matters, the more context you need, the more memory you'll need.
If you are looking for AI devices at $2500, you'll probably want something like this [1]. A unified memory architecture (which will mean LPDDR5) will give you the most memory for the least amount of money to play with AI models.
[1] https://frame.work/products/desktop-diy-amd-aimax300/configu...
When local models don’t cut it, I like Gemini 2.5 flash/pro and gemini-cli.
There are a lot of good options for commercial APIs and for running local models. I suggest choosing a good local and a good commercial API, and spend more time building things than frequently trying to evaluate all the options.
It's been a while since I checked out Mini prices. Today, $2400 buys an M4 Pro with all the cores, 64GB RAM, and 1TB storage. That's pleasantly surprising...
there are no out the box solutions to run a fleet of models simultaneously or containerized either
so the closed source solutions in the cloud are light years ahead and its been this way for 15 months now, no signs of stopping
would probably need multiple models running in distinct containers, with another process coordinating them
- Off-peak pricing by DeepSeek
- Batch pricing by OpenAI and Anthropic
- Context window differentiated pricing by Google and Grok
- Thinking vs non-thinking token pricing by Qwen
- Input token tiered pricing by Qwen coder
I originally posted here: https://x.com/paradite_/status/1947932450212221427
Openrouter is a good alternative. Added bonus that you can also see where the open models come in, and can make an educated guess on the true cost / size of a model, and how likely it is it's currently subsidised.
A limitation though, at least the last time I checked, is that you only get a single provider returned per model. That’s fine for the major commercial models that all have the same pricing on each provider, but makes it hard to rely on for open source models, which tend to have many providers offering them at different price points (sometimes very different price points—like 5x or 10x difference).
https://github.com/tekacs/llm-pricing
--- ---[1] https://llm-prices.com/
[0] https://developers.googleblog.com/en/gemini-2-5-models-now-s...
The UI, however, is really clean and straight to the point. I like the interface, but miss the content
Mistral, Llama, Kimi, Qwen…?
Not a site if value unless it contains whole of the market.
We have uploaded mostly the openrouter api models, but trying to do it in a useful way to personalize calculation and comparison. If someone would like to test or have a demo, we will be glad for any feedback.
Here's the current version:
I would love to replace this with an API call.1. Pull some large tech company’s open source’ tools’ JS file 2. Extract an internal JSON blob that contains otherwise difficult information 3. Parse it and use what I need from within it for my tool
What tools / experiments out there exist to exercise these cheaper models to output more tokens / use more CoT tokens to achieve the quality of more expensive models?
eg, Gemini 2.5 flash / pro ratio is 1 1/3 for input, 1/8 for output... Surely there's a way to ask Flash to critique it's work more thoroughly to get to Pro level performance and still save money?
Just now I was testing the new Qwen3-thinking model. I've run the same prompt five times. The costs I got, sorted: 0.0143, 0.0288, 0.0321, 0.0389, 0.048 . And this is for single model.
Also, in my experience, sonnet-4 is cheaper than gemini-2.5-pro, despite token costs being higher.
- Filter by model "power" or price class. I want compare the mini models, the medium models, etc.
- I'd like to see a "blended" cost which does 80% input + 20% output, so I can quickly compare the overall cost.
Great work on this!
Should I use copilot pro in agent mode with sonnet 4, or is it cheaper to use claude with sonnet 4 directly?
You should also aggregate prices from Vertex and AWS Bedrock .
I am newly to this hobby, but would like to know more about what experienced person things and do.
https://claude.ai/share/20b36bd3-d817-4228-bc33-aa7c4910bc2b (the preview seems to only work in Chrome, for Firefox you have to download the html).
Plus maybe half an hour to verify and correct the prices and another few minutes for domain and hosting.
The author posted it himself, so why not spend an hour or two more and have a decent list with at least half a dozen providers and 100 models? In its current state it is just a mockup.
It is since 3 hours on the top of the front page, if the author had added one per minute to the .json, then it would now be 200 models.
I plan on adding cache prices and making the list more comprehensive