The official blog of the NeurIPS machine learning conference, covering review experiments, paper awards, and conference policies.
HN commenters link it for best paper award announcements and discussions of the peer review process, such as the consistency experiment and the emergent abilities paper.
The other factor preventing a fix is that people with no actual serious experience of academic publishing and peer review will defend these journals, because they still think that peer review acts like some kind of meaningful quality filter. But, it really doesn't.
Because someone is surely going to try to defend journals via peer review in this thread, I want to provide a counter to the arguments that journal peer review does much good. Also, since everyone knows that if you just go to a poor enough journal, you can be published, I am going to focus on the (IMO mostly false) claim that higher-profile journals are still doing a good thing here.
There are numerous studies showing that higher-profile journals in general have more retractions and research misconduct [1-2], lower research quality [3], in fact weaker statistical power and reliability [4], and that statistical reliability even in high prestige journals is still extremely poor overall [5]. Also, making it through peer review is highly random and dependent on who you get as a reviewer [6], or is just basically a coin toss even when looking at reviewer groups:
In 2014, 49.5% of the papers accepted by the first committee were rejected by the second (with a fairly wide confidence interval as the experiment included only 116 papers). This year, this number was 50.6%. We can also look at the probability that a randomly chosen rejected paper would have been accepted if it were re-reviewed. This number was 14.9%…
cperciva, if you're around, I'd be really curious to hear a bit more about the details behind your attempts to get "Cache Missing for Fun and Profit". I'm really surprised that it was so hard to find a home for it! Maybe a more general security venue like USENIX Sec, ACM CCS, or IEEE S&P would have been a better fit for something that was neither fish nor fowl? Just a couple years later in 2008 USENIX Sec published "Reverse-Engineering a Cryptographic RFID Tag" which is a personal favorite and similarly is a mix of low-level hardware architecture and cryptography.
There were two other shared CPU cache side channel attacks on cryptography in 2005 (DJB's "Cache-timing attacks on AES" [2] and Osvik/Shamir/Tromer's "Cache Attacks and Countermeasures: the Case of AES" [3]) so in some ways it feels like the time was right for the community to start finding these attacks -- which makes it all the more baffling to me that your work got rejected.
I'm also a bit dismayed at the advice not to work on anything too novel. I guess it's true in some ways (our Chaff Bugs [4] paper and an associated grant proposal got rejected for four years straight with comments like "the panel was uniformly skeptical that anyone would ever be willing to use this" [5]), but it feels from the inside that it just takes more work to convince people to take a chance on new ideas, and that the larger issue is just how much inherent randomness there is in the peer review system [6] (my advisor used to say it…
> paper reviewers are usually not supposed to look at the actual source code of the papers
Wait what? I haven't reviewed for ACL but most conferences don't say "don't look at the source code." They will say that reviewers are not required to look at it (as well as the appendix). But generally it just isn't uploaded. I do always look at the main method when it is there but talking to my peers and advisor, this is very uncommon[0]. My experience is that most reviewers do not spend more than an hour on a work and make an opinion within 15 minutes.
> Not sure what the best solution is, other than having the most "hyped" papers double verified by researchers on Twitter.
I'd say (as a start):
1) Get rid of the conference system. A zero-shot (maybe 1-shot if "rebuttal" is allowed) zero-sum system is just disastrous, especially at scale. There's high incentives to actually reject works you review for. A conference system has a binary outcome and the purpose is to reject 80% of papers based on a rather noisy metric of "top tier." A journal system is a back and forth where reviewers are trying to improve the paper. The purpose of the reviewers here is to determine if the idea is indeed good, and then if the paper meets the requirements or not and must explicitly state what needs to be changed for acceptance.
1.5) An actual rebuttal system could help alleviate some of these issues. Using OpenReview for a conversation between authors and reviewers is critical. A singular 1 page…
I really think there needs to be some deep reexamination of the current way we quanitfy the value of research output in the academic world. I don't have the time or energy to develop a full argument in favor of an alternative replacement, but I'll do my best to share a "cut down" version.
Some of the biggest problems with the current system are:
* Peer Review has its share of problems[1][2][3] that create horrible second order effects, especially among those pursuing PhDs[4]
* Elsevier, Nature, Science/AAAS and others perform rent-seeking behavior to the extent where I think it's worth asking whether they hinder the funding and dissemination of good science more than they help. As a personal aside, I always found it very off-putting that DeepMind regularly publishes in Nature in Science, despite the fact that outside of AlphaFold, their work often has little overlap with the readership that typically frequents these journals.
Personally, I am of the opinion that platforms like Semantic Scholar, arXiv and OpenReview are doing a better job of promoting open and transparent academic research with improved accessibility to both the public and the researchers doing good work.
Given the power of being mentored by great scientists, it makes sense to have filtering processes which concentrate great researchers in a small amount of schools. My point is that if there is too little oversight, these institutions become incentivized to all but encourage bad behavior in order to…
Hi! I'm the research lead for Anthropic's interpretability team, and was the decision maker for us publishing our papers web first and not doing traditional publications.
A few thoughts:
(1) As others have commented, I think peer review in ML is pretty widely accepted to be dysfunctional right now. I think most people who have published in ML conferences would agree. It's not unusual for early PhD students and sometimes even undergrads to review, and reviewers are overburdened to the point where they can carefully consider all their papers. Everything I've said so far is just anecdote and opinion though. A more objective test was the NeurIPS 2021 Consistency Experiment ( https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/ ) which found that if a paper was accepted by the conference, there was only a ~50% chance that a parallel review process would come to the same conclusion.
(2) Modern peer review is a relatively modern invention, arising in post-WW2 science as the scientific community grew dramatically, and there was a need for more systematized ways to make decisions about publication, funding, jobs, etc. Famously, Einstein was offended by one of his papers being sent for review. I don't think it's at all obvious that this transition has been good for science! I see lots of people writing papers for reviewers, rather than with the goal of doing the most impactful science they can.
(3) As background, I spent 5 years of my life running a…
Nevertheless this PoW is used by some conference venues now, in the case of a dispute against human authorship. NeurIPS does allow LLM use for "minor" polish like spelling, grammar, correcting non-native speakers. But you can't have the AI write the entire paper:
See https://blog.neurips.cc/2026/06/02/ai-generated-papers-in-the-neurips-2026-position-paper-track/
> Unfortunately, given the volume of submissions that appear non-compliant, relying on author declarations is insufficient.
...
> Authors whose submissions show significant AI involvement must provide an audit trail that clearly demonstrates that they complied with the policy. We expect that in future years this kind of audit trail will become a default.
The method uses Pangram, but that's somewhat arbitrary. The authors must provide the draft that was used to prompt the AI and show how the changes were made, which should be possible with a chat history. Where it doesn't work is if the authors claim they did not use AI, but otherwise it's quite hard to spoof. Edit history on its own isn't good enough because you could trivially have a program emulate keystrokes to type at a human speed into a document with tracked changes; the proof here accepts that AI was probably used and aims to determine to what extent.
I do think this goes back to if the text is high quality enough that humans don't notice, is that OK? Probably yes. What we don't want is poor quality writing from any source.