- papers are read, summarized and digested by AI, because there are just so many papers at leading AI conferences that nobody has time to eyeball them all
We are very rapidly automating humans out of the academic publication loop here.
Idk if we're automating humans out of the publishing loop as much as rapidly automating the production of crap. I had a very similar experience reviewing for EMNLP recently.
We are nowhere near AI being able to judge the quality of research (in fact, one might reasonably state that even most humans can't really judge the quality of research). Most things in society are not like math: we can't automate (via verification) our way out of noise overwhelming the signal.
Folks are willing to entirely abuse the public resource that is faithful, honest reviewing. (This is unsurprising; the abuse of the commons / public resources has been rising for a long time). There isn't a good solution other than something akin to draconian social scoring to limit access to the reviewing system.
This is a legitimate question: should people have reputations? Should their behavior be made more visible publicly, both good and bad? How?
In a small contained society, where consequences are more directly affecting individuals immedately, these questions don't need to be asked, because they're inherantly answered. We now are a society with billions of people, and dire consequences sometimes deferred for a generation, or more. Part of our general failing is the lack of good answers to the above questions. For many people, there are rarely negative consequences for causing harm to others, and the rewards can be very great indeed.
Wonder if it could degrade to something worse than the status quo where reputations are falsely tarnished by competitors purely as a weapon to get ahead…
The "Chinese social credit" system is vastly overblown.
I'm living here (China, Beijing and Jiangsu) now, and there are exactly two cases in the past eight months where the "social credit" system has had any impact at all:
- To open "take first pay after" vending machines. These are vending machines which are basically big locked fridges; if your credit score is high enough, you can open the machines, take whatever drinks/snacks you want, and they'll use (I presume) computer vision to charge you afterwards. If you don't have a high enough credit score, tough, you can't use them. (But there's almost always normal pay-first vending machines nearby).
- To borrow mobile charging packs (powerbanks). Some operators will let you "swipe your credit" (check your credit score) to take one without paying a deposit. If your credit score isn't high enough, you first pay e.g. a ¥99 deposit (~$15USD) which gets returned when you return the powerbank, not a big deal.
That's...it. My credit score is high enough on only one platform (Alipay), so I get to try what happens with both "high" and "low" credit, and I can confidently say these are the *only* two cases where I have even been asked to show the credit score. I have taken multiple train trips, bought lots of stuff in stores and restaurants, etc. without ever touching the "social credit score".
P.S. I literally don't even know how to raise my credit score, and neither do many of the folks who live here - again, not that it matters, because the score is simply not that important.
"Consequences and reputations stemming from one's actions" isn't necessarily draconian social scoring. Without a structure for imposing consequences on wrongdoers, we're not a society, we're just monkeys flinging poo at each other.
For a more mathematically rigorous treatment, negative feedback is fundamental to Control Theory; it allows us to stabilise processes that would otherwise go out of control.
The problem is that everyone imagines some sort of just and fair arbiter of these things, when the reality is all of the social scoring and consequences and reputations rarely actually stem from one’s actions, and far more often stem from how much money someone has put in someone’s pocket, who someone knows, the color of someone’s skin, or what’s between someone’s legs.
Until we’re actually serious about treating people equitably, (not equally, as that would simply leave the lopsided power structure we have in place) we aren’t getting out of this.
People react negatively to this, because they fear a dystopian society of the sort we've seen in plenty of movies, and rightfully so.
But it's also worth pointing out that "consequences and reputations stemming from one's actions" is already the world we live in and always have lived in. Hell, even Hacker News has karma points, downvoting, shadow banning, and the like. There's no such thing as a society with zero consequences and zero reputations. The only real question is a matter of degree, structure, severity, reach, and various idiosyncrasies that differ across cultures.
So it would be nice to have a more nuanced discussion about this instead of treating it like a 0 or 1 decision.
People should get paid for reviewing. That’s the solution. Publishing companies rack in billions of dollars in pure profit exploring free labor. Once people actually get paid for reviewing, it becomes much faster and higher quality and you won’t need AI triage.
I think you are correct, its so hard to judge the quality of research that we have been using publication record/count as a proxy. Ultimately, having papers being easy to write is good, provided we find a better way of judging quality.
We just need journals run by an AI that charges other AI to read them, and then the AI run colleges can promote the AI with the most AI journal entries and citations.
I have been rolling over the idea of “reasoning deserts” in my head, akin to food deserts.
Places where economic calculations lead to only a facsimile of the real thing being provided, without the actual components necessary for human health, wellbeing and flourishing.
My 2 cents: AI has improved writing considerably for non-native english speakers (in particular China). Writing feels more standardized/boring but easier to read overall. I hit fewer papers that are a pain to read. The most problematic aspect I see are semi-bogus claims i.e. sentences that aren't false, but don't quite feel right either. YMMV.
> We are very rapidly automating humans out of the academic publication loop here
It's a sad thing. But the monetization and enshitification of journal publications over the decade or so, even prior to AI, certainly has not helped this trend.
I don't think it's academia not taking it seriously, more like these publishers are just coasting on brand name and trying to milk every cent out of it. Peer review is essentially being a Reddit mod or something, you get nothing out of it but karma or a sticker, and the platform/publisher gets all the benefit.
Imagine being an author and paying to publish your novel. Also btw the editor is another author but has to proofread your book for free.
Many academics will publish preprints or their more popular papers on their own websites etc. It's just common sense because to academics they don't get paid a single cent by the publisher (and instead have to pay the publishers instead) and more publicity for them is always better than less.
Validating existence should already be trivial: virtually all journal already have publicly-available indices which contain at least author information and an abstract. Combine that with DOI citations and you're basically done - even with closed-access journals.
Checking the content is of course a lot more difficult, but that doesn't magically become trivial with open-access journals: you still need to read and interpret what is being said in the paper and compare it to the claims being made in the citation. Granted, these days you could use AI for a first pass, but it's still going to be incredibly tedious work.
I'm sympathetic to the idea: we should have an open, publicly queryable citation graph. Google scholar could very easily offer this at marginal cost near zero, but they won't.
And yet the automation we pour trillions in, is the one that will do anything, including everything I find interesting and will never clean my own house.
I thought that it was. I thought journals were starting to implement full bans upwards of a year+ for those who don't honestly disclose AI usage in their work? If I'm not mistaken, arXiv is doing this as well? granted, disclosure is different from overuse, but it seems like a small jump to just go ahead and just ban not checking ones work! ...disclosed or not disclosed. my own personal view on it, is if you can't be bothered to spot hallucinations and other such errors, its not ai that is the issue, it is incompetence and laziness
> Both papers were accepted for oral presentations with the condition that they simply fix the hallucinated references.
I do wonder what truthfully could be on ai verification, if even one paper with such an error is accepted it sets the precedent you hopefully get lucky to not get caught (then again verifying for basic tells isn't the same verifying is this genuinely a worthwhile publication, but that's a separate matter)
"A lot of the content of this blog was initially drafted by an agent of some sort"
What? I mean who does this. My voice is my voice and it's literally never occurred to me to have an LLM do a first draft. I thought that was college kid stuff.
This is an oversimplification and I'll edit. Caleb and I had a conversation about our reviewing woes and thought it would be fun to do an interview style post, so we had Claude come up with some questions based on our convo. We answered the questions from scratch and had Claude proofread at the end.
For anyone that wants to evade the kind of people who want to figure out if an AI wrote your review or not, we wrote a whole paper (ICLR 2026!) on how to do that!
I consider all types of "I liked this output, but don't the moment I learned it was AI generated" to be externalizations of "carbon chauvinism" (https://en.wikipedia.org/wiki/Carbon_chauvinism) and basically bigotry.
And BTW, the term "meritocracy" was coined in a book that was extremely critical of the idea and which argued that a real meritocracy is actually dystopian. We consider our work "harming meritocracy" to be a good outcome: (https://en.wikipedia.org/wiki/The_Rise_of_the_Meritocracy)
- papers are written by AI (as pointed out in this article, and as obvious to anyone who spends a while actually reading recent AI research)
- papers are reviewed by AI (NeurIPS is doing an AI assisted review experiment - https://neurips.cc/Conferences/2026/ai-reviewing-experiment - and I feel the trend is moving towards AI reviewers whether we like it or not)
- papers are read, summarized and digested by AI, because there are just so many papers at leading AI conferences that nobody has time to eyeball them all
We are very rapidly automating humans out of the academic publication loop here.
We are nowhere near AI being able to judge the quality of research (in fact, one might reasonably state that even most humans can't really judge the quality of research). Most things in society are not like math: we can't automate (via verification) our way out of noise overwhelming the signal.
Folks are willing to entirely abuse the public resource that is faithful, honest reviewing. (This is unsurprising; the abuse of the commons / public resources has been rising for a long time). There isn't a good solution other than something akin to draconian social scoring to limit access to the reviewing system.
We can use it as vacuum to clean things up. The same pump can be used as a shit fire hose.
The effort we need to clean things up is significantly higher than the effort needed to make a mess.
This is a legitimate question: should people have reputations? Should their behavior be made more visible publicly, both good and bad? How?
In a small contained society, where consequences are more directly affecting individuals immedately, these questions don't need to be asked, because they're inherantly answered. We now are a society with billions of people, and dire consequences sometimes deferred for a generation, or more. Part of our general failing is the lack of good answers to the above questions. For many people, there are rarely negative consequences for causing harm to others, and the rewards can be very great indeed.
Wonder if it could degrade to something worse than the status quo where reputations are falsely tarnished by competitors purely as a weapon to get ahead…
I'm living here (China, Beijing and Jiangsu) now, and there are exactly two cases in the past eight months where the "social credit" system has had any impact at all:
- To open "take first pay after" vending machines. These are vending machines which are basically big locked fridges; if your credit score is high enough, you can open the machines, take whatever drinks/snacks you want, and they'll use (I presume) computer vision to charge you afterwards. If you don't have a high enough credit score, tough, you can't use them. (But there's almost always normal pay-first vending machines nearby).
- To borrow mobile charging packs (powerbanks). Some operators will let you "swipe your credit" (check your credit score) to take one without paying a deposit. If your credit score isn't high enough, you first pay e.g. a ¥99 deposit (~$15USD) which gets returned when you return the powerbank, not a big deal.
That's...it. My credit score is high enough on only one platform (Alipay), so I get to try what happens with both "high" and "low" credit, and I can confidently say these are the *only* two cases where I have even been asked to show the credit score. I have taken multiple train trips, bought lots of stuff in stores and restaurants, etc. without ever touching the "social credit score".
P.S. I literally don't even know how to raise my credit score, and neither do many of the folks who live here - again, not that it matters, because the score is simply not that important.
Until we’re actually serious about treating people equitably, (not equally, as that would simply leave the lopsided power structure we have in place) we aren’t getting out of this.
But it's also worth pointing out that "consequences and reputations stemming from one's actions" is already the world we live in and always have lived in. Hell, even Hacker News has karma points, downvoting, shadow banning, and the like. There's no such thing as a society with zero consequences and zero reputations. The only real question is a matter of degree, structure, severity, reach, and various idiosyncrasies that differ across cultures.
So it would be nice to have a more nuanced discussion about this instead of treating it like a 0 or 1 decision.
We are rendering it irrelevant. If this is the norm for academia, I’m sympathetic to the folks looking to cut its funding.
I may generate slop from time to time, but I do my best to keep it to myself.
It's a sad thing. But the monetization and enshitification of journal publications over the decade or so, even prior to AI, certainly has not helped this trend.
They should consider swapping this for a log plot.
I can imagine in 2027 academia looking like Moltbook.
If all these papers were not gatekept by journals, it would be trivially easy to validate at least the existence of cited papers and quotes.
Imagine being an author and paying to publish your novel. Also btw the editor is another author but has to proofread your book for free.
Many academics will publish preprints or their more popular papers on their own websites etc. It's just common sense because to academics they don't get paid a single cent by the publisher (and instead have to pay the publishers instead) and more publicity for them is always better than less.
That is... completely normal.
> Also btw the editor is another author but has to proofread your book for free.
That would be weirder; if you're paying for your own publication, you don't get an editor at all unless you hire them yourself.
Checking the content is of course a lot more difficult, but that doesn't magically become trivial with open-access journals: you still need to read and interpret what is being said in the paper and compare it to the claims being made in the citation. Granted, these days you could use AI for a first pass, but it's still going to be incredibly tedious work.
From my perspective, it's a clear manifestation of humanity's most pervasive failing - the one that defines every group, eventually.
Alas, it's probably just wishful thinking on my part.
I do wonder what truthfully could be on ai verification, if even one paper with such an error is accepted it sets the precedent you hopefully get lucky to not get caught (then again verifying for basic tells isn't the same verifying is this genuinely a worthwhile publication, but that's a separate matter)
What? I mean who does this. My voice is my voice and it's literally never occurred to me to have an LLM do a first draft. I thought that was college kid stuff.
https://arxiv.org/abs/2510.15061
I consider all types of "I liked this output, but don't the moment I learned it was AI generated" to be externalizations of "carbon chauvinism" (https://en.wikipedia.org/wiki/Carbon_chauvinism) and basically bigotry.
And BTW, the term "meritocracy" was coined in a book that was extremely critical of the idea and which argued that a real meritocracy is actually dystopian. We consider our work "harming meritocracy" to be a good outcome: (https://en.wikipedia.org/wiki/The_Rise_of_the_Meritocracy)