I've been part of a response team on a security incident and I really feel for them. However, this initial communication is terrible.
Something happened, we won't say what, but it was severe enough to notify law enforcement. What floors me is the only actionable advice is to "review environment variables". What should a customer even do with that advice? Make sure the variable are still there? How would you know if any of them were exposed or leaked?
The advice should be to IMMEDIATELY rotate all passwords, access tokens, and any sensitive information shared with Vercel. And then begin to audit access logs, customer data, etc, for unusual activity.
The only reason to dramatically overpay for the hosting resources they provide is because you expect them to expertly manage security and stability.
I know there is a huge fog of uncertainly in the early stages of an incident, but it spooks me how intentionally vague they seem to be here about what happened and who has been impacted.
Incidents like this are a good reminder of how concentrated our single points of failure have become in the modern web ecosystem. I appreciate the transparency in their disclosure so far, but it definitely makes you re-evaluate the risk profile of leaning entirely on fully managed PaaS solutions.
Huh, curiously; I'm on Arch Linux, crash happens in Google Chrome (147.0.7727.101) for me too, but not in Firefox (149.0.2) nor even in Chromium (147.0.7727.101).
I find it fun we're all reading a story how Vercel likely is compromised somehow, and managed to reproduce a crash on their webpage, so now we all give it a try. Surely could never backfire :)
Happens for me on an M4 Macbook Pro as well, and it happens when you mousedown, so doesn't even require a full click or execution of click listeners/navigation.
If you remove the href attribute "/home", it stops happening. If you add that attribute to a different link, that link now causes the crash.
Twice in a week for a ShinyHunters story! I'll keep saying this because it's interesting:
I was in federal prison with Sebastien Raoult of ShinyHunters. We were in the same unit and talked regularly. My takeaway from our conversations about technology/hacking/programming/whatever was that him and his contemporaries have an uncanny level of persistence to phish.
What Sebastien often described to me was trivial—scanning GitHub for API keys, using sendmail, and your good 'ol phishing page that did auth checks against the real thing. Nothing revolutionary is happening on the attacker's side here.
What has shifted is development processes and how we think about and perform development. I think we'll only see more and more of these kinds of incidents (not necessarily Vercel-scale) as more and more people-with-ideas have the means to blindly build.
The lack of details itself is telling enough. Whatever comes out will be no doubt PR sanitised and some bigger clumps of truth won't make it through the PR process.
We run on Vercel and I wonder if / how long before we're alerted about a leak. Quick look online suggests environment variables marked as sensitive are ok, but to which extent I wonder.
Ah, Theo with his vast insights and connections into everything. That man gets around, and his content is worth it's cost.
Theo's content boils down to the same boring formula.
1. Whatever buzzword headline is trending at the time
2. Immediate sponsored ad that is supposed to make you sympathize with Theo cause he "vets" his sponsors.
3. The man makes you listen to a "that totally happened" story that he somehow always involved himself personally.
4. Man serves you up an ad for his t3.chat and how it's the greatest thing in the world and how he should be paid more for his infinite wisdom.
5. A rag on Claude or OpenAI (whichever is leading at the time)
6. 5-10 minutes of paraphrasing an article without critical thought or analysis on the video topic.
I used to enjoy his content when he was still in his Ping era, but it's clear hes drunken the YT marketer kool-aid. I've moved on, his content gets recommend now and again, but I can't entertain his non-sense anymore.
I just wanted to chime in and say I think he is knowledgeable; he's not a con. I know you didn't say that, but people might have the impression he doesn't know what he's talking about. He does know, and I've learned quite a lot from him in the past.
However, since the LLM Cambria explosion, he has become very clickbaity, and his content has become shallow. I don't watch his videos anymore.
I don't watch his content, but I felt comfortable posting his link as I believe he's generally considered a reputable guy? His tweets sometimes come up in my for you tab and he seems reasonable and knowledgable generally? Maybe I'm wrong and shouldn't have linked to him as a source.
He's kind of like an LLM in that his content has the surface texture of something substantial, and sometimes it's backed by substance, yet it's often half-true or totally off the mark too. You'll notice if you're previously acquainted with what he's talking about, otherwise he seems to be as you described.
I don't think he's a bad guy or that he's trying to be misleading. I suspect he wants his content to actually carry value, but he produces too much for that to be possible. Primarily he's a performer, not a technologist.
> @theo: "I have reason to believe this is credible. If you are using Vercel, it’s a good idea to roll your secrets and env vars."
> @ErdalToprak: "And use your own vps or k3s cluster there’s no reason in 2026 to delegate your infra to a middle man except if you’re at AWS level needs"
> @theo: "This is still a stupid take"
lol, okay. Thanks for the insight, Theo, whoever you are.
Hey, I’m with you - I think social media needs to die specifically for this reason. I’m reminded of the term “snake oil” - it’s like the dawn of newspapers again.
This is why you pay a real provider for serious business needs, not an AWS reseller. Next.js is a fundamentally insecure framework, as server components are an anti-pattern full of magic leading to stuff like the below. Given their standards for framework security, it's not hard to believe their business' control plane is just as insecure (and probably built using the same insecure framework).
Next.js is the new PHP, but worse, since unlike PHP you don't really know what's server side and what's client side anymore. It's all just commingled and handled magically.
> Next.js is the new PHP, but worse, since unlike PHP you don't really know what's server side and what's client side anymore. It's all just commingled and handled magically.
Wasn't unheard of back in the day, that you leaked things via PHP templates, like serializing and adding the whole user object including private details in a Twig template or whatever, it just happened the other way around kind of. This was before a fat frontend and thin backend was the prevalent architecture, many built their "frontends" from templates with just sprinkles of JavaScript back then.
People say "Next.js is the new PHP" because it's the most popular and prominent tooling out there, and so by sheer number of available targets it's the one that comes up the most when things go wrong like this.
But there are more people trying to secure this framework and the underlying tools than there would be on some obscure framework or something the average company built themselves.
Also "pay a real provider", what does that mean? Are you again implying that the average company should be responsible for _more_ of their own security in their hosting stack, not less?
Most companies have _zero_ security engineers.. Using a vertically-integrated hosting company like Vercel (or other similar companies, perhaps with different tech stacks - this opinion has nothing to do with Next or Node) is very likely their best and most secure option based on what they are able to invest in that area.
PHP was so simple and easy to understand that anyone with a text editor and some cheap shared hosting could pick it up, but also low level enough that almost nothing was magically done for you. The result was many inexperienced developers making really basic mistakes while implementing essential features that we now take for granted.
Frameworks like Next.js take the complete opposite approach, they are insanely complex but hide that complexity behind layers and layers of magic, actively discouraging developers from looking behind the curtain, and the result is that even experienced developers end up shooting themselves in the foot by using the magical incantations wrong.
I recently got hit by a car on my bike. While I was starting the claim filing process the web portal for ICBC (British Columbia insurance) was acting a little funky / stalling / and then gave me a weird access error. Down at the bottom of the error page was a little grey underlined link that said “vercel”.
I’m not exactly surprised, but it seems like the unserious, ill-informed and lazy are taking over. There is absolutely zero reason why a large, essential public service should be overspending and running on an unnecessary managed service like vercel… yet, here we are.
Much as I want to rip on vercel, its clear that ai is going to lead to mass security breaches. The attack surface is so large, and ai agents are working around the clock. This is a new normal. Open source software is going to change, companies wont be running random repos off github anymore
Let that be the end of Microsoft. Was forced to use their shitty products for years, by corporate inertia and their free Teams and Azure licenses, first-dose-is-free, curse.
this like is saying email marketing is done better if you hand write every email. Thats true, but the hit rate is so low, that you are better off generating 1 million hyper personalized emails and firing them off into the ether
As someone who did the former for a couple years, “better off” is subjective and dependent on your business model, particularly for B2B. It’s a trade off like anything else. You may get more leads, but they may convert at a lower rate. Sending at that scale also increases your risk of email deliverability problems. Trashing your domain has more impacts than you’d think. In smaller, targeted markets it even can damage your business reputation and hurt future sales if done poorly; word gets around.
I disagree. Many humans are phishing in a different language than their native tongue, and LLMs are way better at sounding legit/professional than many of them. The best spear-phishing will still be humans, but AI definitely raises the bar.
Something happened, we won't say what, but it was severe enough to notify law enforcement. What floors me is the only actionable advice is to "review environment variables". What should a customer even do with that advice? Make sure the variable are still there? How would you know if any of them were exposed or leaked?
The advice should be to IMMEDIATELY rotate all passwords, access tokens, and any sensitive information shared with Vercel. And then begin to audit access logs, customer data, etc, for unusual activity.
The only reason to dramatically overpay for the hosting resources they provide is because you expect them to expertly manage security and stability.
I know there is a huge fog of uncertainly in the early stages of an incident, but it spooks me how intentionally vague they seem to be here about what happened and who has been impacted.
Clicking the Vercel logo at the top left of the page hard crashes my Chrome app. Like, immediate crash.
What an interesting bug.
I find it fun we're all reading a story how Vercel likely is compromised somehow, and managed to reproduce a crash on their webpage, so now we all give it a try. Surely could never backfire :)
If you remove the href attribute "/home", it stops happening. If you add that attribute to a different link, that link now causes the crash.
No crash.
Now I don't want to click that "Finish update" button.
https://x.com/theo/status/2045862972342313374
> I have reason to believe this is credible.
https://x.com/theo/status/2045870216555499636
> Env vars marked as sensitive are safe. Ones NOT marked as sensitive should be rolled out of precaution
https://x.com/theo/status/2045871215705747965
> Everything I know about this hack suggests it could happen to any host
https://x.com/DiffeKey/status/2045813085408051670
> Vercel has reportedly been breached by ShinyHunters.
https://t3.gg/
I was in federal prison with Sebastien Raoult of ShinyHunters. We were in the same unit and talked regularly. My takeaway from our conversations about technology/hacking/programming/whatever was that him and his contemporaries have an uncanny level of persistence to phish.
What Sebastien often described to me was trivial—scanning GitHub for API keys, using sendmail, and your good 'ol phishing page that did auth checks against the real thing. Nothing revolutionary is happening on the attacker's side here.
What has shifted is development processes and how we think about and perform development. I think we'll only see more and more of these kinds of incidents (not necessarily Vercel-scale) as more and more people-with-ideas have the means to blindly build.
https://x.com/theo/status/2045862972342313374?s=46
He also suggests in another post that Linear and GitHub could also be pwned?
Either way, hugops to all the SRE/DevOps out there, seems like it's going to be a busy Sunday for many.
> Here’s what I’ve managed to get from my sources:
>3. The method of compromise was likely used to hit multiple companies other than Vercel.
https://x.com/theo/status/2045870216555499636
To be fair journalists often do this too, eg. "[company] was breached, people within the company claim"
Theo's content boils down to the same boring formula. 1. Whatever buzzword headline is trending at the time 2. Immediate sponsored ad that is supposed to make you sympathize with Theo cause he "vets" his sponsors. 3. The man makes you listen to a "that totally happened" story that he somehow always involved himself personally. 4. Man serves you up an ad for his t3.chat and how it's the greatest thing in the world and how he should be paid more for his infinite wisdom. 5. A rag on Claude or OpenAI (whichever is leading at the time) 6. 5-10 minutes of paraphrasing an article without critical thought or analysis on the video topic.
I used to enjoy his content when he was still in his Ping era, but it's clear hes drunken the YT marketer kool-aid. I've moved on, his content gets recommend now and again, but I can't entertain his non-sense anymore.
However, since the LLM Cambria explosion, he has become very clickbaity, and his content has become shallow. I don't watch his videos anymore.
I don't think he's a bad guy or that he's trying to be misleading. I suspect he wants his content to actually carry value, but he produces too much for that to be possible. Primarily he's a performer, not a technologist.
> @ErdalToprak: "And use your own vps or k3s cluster there’s no reason in 2026 to delegate your infra to a middle man except if you’re at AWS level needs"
> @theo: "This is still a stupid take"
lol, okay. Thanks for the insight, Theo, whoever you are.
Hey, I’m with you - I think social media needs to die specifically for this reason. I’m reminded of the term “snake oil” - it’s like the dawn of newspapers again.
Next.js is the new PHP, but worse, since unlike PHP you don't really know what's server side and what's client side anymore. It's all just commingled and handled magically.
https://aws.amazon.com/security/security-bulletins/rss/aws-2...
Wasn't unheard of back in the day, that you leaked things via PHP templates, like serializing and adding the whole user object including private details in a Twig template or whatever, it just happened the other way around kind of. This was before a fat frontend and thin backend was the prevalent architecture, many built their "frontends" from templates with just sprinkles of JavaScript back then.
But there are more people trying to secure this framework and the underlying tools than there would be on some obscure framework or something the average company built themselves.
Also "pay a real provider", what does that mean? Are you again implying that the average company should be responsible for _more_ of their own security in their hosting stack, not less?
Most companies have _zero_ security engineers.. Using a vertically-integrated hosting company like Vercel (or other similar companies, perhaps with different tech stacks - this opinion has nothing to do with Next or Node) is very likely their best and most secure option based on what they are able to invest in that area.
PHP was so simple and easy to understand that anyone with a text editor and some cheap shared hosting could pick it up, but also low level enough that almost nothing was magically done for you. The result was many inexperienced developers making really basic mistakes while implementing essential features that we now take for granted.
Frameworks like Next.js take the complete opposite approach, they are insanely complex but hide that complexity behind layers and layers of magic, actively discouraging developers from looking behind the curtain, and the result is that even experienced developers end up shooting themselves in the foot by using the magical incantations wrong.
I’m not exactly surprised, but it seems like the unserious, ill-informed and lazy are taking over. There is absolutely zero reason why a large, essential public service should be overspending and running on an unnecessary managed service like vercel… yet, here we are.
What's your agenda here?
Let that be the end of Microsoft. Was forced to use their shitty products for years, by corporate inertia and their free Teams and Azure licenses, first-dose-is-free, curse.
AI agents have the benefit of working at scale, probably "better" used for mass targeting.
But I get your point.