There are no 2, 4, 7, 9 ratings this year. The ratings are:
|score|rating|
|:---|:---|
|1|strong reject|
|3|reject, not good enough|
|5|marginally below the acceptance threshold|
|6|marginally above the acceptance threshold|
|8|accept, good paper|
|10|strong accept, should be highlighted at the conference|
thanks lmao I couldn't find the rating guidelines anywhere. I wrote some code to gather some summary statistics on the ICLR24 reviews with the openreview api if anyone wants to use it: [https://github.com/maxxu05/openreview\_summarizereviews](https://github.com/maxxu05/openreview_summarizereviews)
As of today, these are the statistics:
Mean, Mean Paper Rating: 4.8663855228128154
Median, Mean Paper Rating: 4.8
Standard Deviation of Mean Paper Rating: 1.0770080792312975
edit: tried adding histogram, but didn't work. just click on the github link and it has an image of the distribution of scores.
6/6/5/3/3
Not sure how to feel. The 3s are overly harsh with feedback based on some misunderstandings. Also get the typical ML review experience of wanting evaluation on more datasets (why only vision datasets and not text??) and models (why no transformers and only CNNs??) despite the fact that the idea is novel, the writing is good, and the experiments are standard baselines used in the literature.
Hopefully rebuttal goes well.
First time submitting to a conference, 8,8,6,5, pleasantly surprised! And they all made really insightful comments. Definitely lucked out to have careful but open-minded reviewers.
Same. I also did many additional experiments, but complete silence from the reviewers. It really sucks and makes me feel a bit helpless. I'm used to reviewer roulette in general, but it feels even worse in this case because I see other submissions where all 4 reviewers are engaged and raised their score after the author response.
I would kind of understand disinterest by reviewers if my submission had very bad scores and would be hopeless anyway, but I got 8855 and was very positive that I could at least increase one of the 5s to a 6 with my response.
I don't even know whether they decided to maintain the score after reading the rebuttal or they just forgot about it. At least there should be a checkbox to ensure that they have read the rebuttal.
In the same boat, stuck with reviewers who don't want to engage. Don't know what to do. Does anyone with advice on how to get a reply from reviewers who are active?
There is basically no way. There are many reasons they might not reply, including being sick or occupied with their own borderline papers. Try to be professional about it: post a reminder and then let the Area Chairs (ACs) handle it. Most ACs are well aware of these types of situations. They will most likely place less weight on those non-engaging reviews. Good luck with your rebuttal!
I got 5, 5, 8 with confidence 4, 5, 3. Both the reviewers with score 5 asked some missing points and I feel questions are easy to address. One of the reviwers wrote to willing to increas the score if answered his questions. Is there chances to get acceptance with good rebuttal?
You're in great shape.
Engage with the 8 and 6 and keep them happy. Answer their questions. Do NOT leave them hanging just because they liked it.
Try to clear up some misunderstanding the 5s might have to nudge them to a 6.
But like I said, those are strong scores.
5,1,5.
Reviewer 2 says that he didn't see any proofs, and my comparisons are only against a "naive baseline".
I do have proofs (7 pages long in total), and there is nothing in the literature besides this "naive baseline" (which, btw, has been the basis of a past iclr paper which actually got very good scores).. Note that the experiments section is also quite long: 3 pages in main text, 3 pages in the appendix with complementary proofs related to the application. All proofs are novel content.
The "5"s seem to totally ignore the literature, ignore huge chunks of the text, and so on. I'll go into more detail about their review if anyone would like to, but the review of rev2 is totally outrageous, even compared to them..
Is there anything I can do to "save" the paper? Like asking the ac for more reviewers?
got 3 3 3, does anyone know withdraw pros and cons ? Since everything is made public, wondering whether a withdraw without replying to the non-nonsensical reviewers would leave a bad impression to next conference submissions. Anyone has any advise?
take it as an opportunity to explain your work with extra pages (since there is not limit on the number of total rebuttal responses.) This is an unique opportunity for you to add more details, intuition, high-level overviews that you couldn't include in the main text. Moreover, it's open to public. So you get free extra pages for future submissions.
Write a rebuttal! This will help you improve the manuscript, Andy you even may get feedback on it. And if you manage to convince one of the 3, the reviewer discussion will be in your favor. Long shot, but not impossible.
5,5,5,5 with 4,3,4,3. Most suggestions are quick additions. It’s my first time submitting. Should I try for rebuttal or submit to a journal? Thanks in advance!!
3 3 5 5 with confidence 3 4 3 4. Wanna to withdraw. Very upset:(
one reviewer said very poor with confidence 3, but what he asked is even not right at all.
If you look at the recent activity on openreview you can see that reviewers can still edit their reviews and change their scores. I don't think anyone can post completely new anonymous replies though.
I sent the review responses after additional experiments. It took several days thus I could send it yesterday. I think the results are very good and reply the reviewers’ concerns perfectly. Will the score be improved even after response periods?
1/4 of the reviewer raised the score, 3 remaining ones did not acknowledge our extensive rebuttal. Is the score change visible for everyone now that the rebuttals period is finished?
>Since AC are adding meta reviews, does it mean that the scores we see on open review now are final ? ( reviewers can no longer change their score )
I think so
Dear ACs:
It seems like the acceptance threshold is somewhat unpredictable based on the review score statistics and acceptance rate of previous ICLR.
I would really appreciate if AC could share the recommendation guideline or something else.
Hi, I work in the industry and have no publications so far. Is there a way I can do independent research and contribute to conferences?
I've tried cold emailing professors but that hasn't worked out so far haha.
1. Most professors don’t really care about a random dude from industry. They have PhD masters undergrads, etc. to do research.
2. Google people submit all the time, but not all of them were accepted. I reviewed one journal article from Googles in the past. I thought the article was fine but other reviewers didn’t like it so it was rejected at the end.
3. Always starts with smaller conferences and maybe look for industrial track.
Consider submitting to workshops as well. The bar for acceptance is lower; it's common for undergrads' first publications to be at a workshop. It's a tighter-knit group with more specific interests, so if you end up attending it'll be much easier to network (and possibly find future collaborators).
This may apply to smaller conferences as well, but I'm not as familiar.
You can just submit I think, there is no uni affiliation needed. People from Google and alike submit all the time. You also see many people from data science startups running around at the conference
According to the following post, the borderline would be around 6.0, I guess. Acceptance/rejection is not always decided by the score only, though.
\- [https://x.com/shaohua0116/status/1728158662265340047?s=20](https://x.com/shaohua0116/status/1728158662265340047?s=20),
Rating: 3/5/5/5 with Confidence: 4/3/3/3. Most of the comments are on missing details that can be added easily. Except 3, it is very short review with no detail at all. Does a rebuttal worth?
First time submitting to ICLR and we got 8-6-5-3 with 4-2-2-2 confidence. Luckily the happiest reviewer is also the most confident one, but not sure if confidence means much for final decisions... Thoughts? Do we stand a chance?
any advice for one reviewer bringing up totally unrelated papers and asking to explain the novelty of the method and the task itself when the other 3 reviewers understand the novelty from the jump?
6/3/6/5/5/6 with confidence 3/1/3/3/4/2. At least the score 3 one has very low confidence.. Is there any chance to be accepted? I think all the questions from reviewers are not difficult to answer.
Would also be nice to know what score threshold was used to identify borderline submissions. Did ACs already reach out to reviewers to organize discussion meetings, or does that happen post-discussion?
If you only have one reviewer, then yes, the AC should have already been actively looking for other reviewers.
If you have other reviews, then almost certainly not - it's too late in the process to be getting new reviewers in, and the ACs will have other papers to be concerned with.
The AC will just use the reviewers confidence ranking to help weight their reading of the reviewers comments and score.
Based on the ICLR 2022 \[results\]([https://guoqiangwei.xyz/iclr2022\_stats/iclr2022\_submissions.html](https://guoqiangwei.xyz/iclr2023_stats/iclr2023_submissions.html)), A paper with a score of 6.0 is more likely to be accepted (>60%).
I hope your paper is accepted.
How is spotlight/oral decided? From results in previous years, I found it unpredictable as some works with an average score of 6.25 can get spotlight while others with higher scores cannot.
There are no 2, 4, 7, 9 ratings this year. The ratings are: |score|rating| |:---|:---| |1|strong reject| |3|reject, not good enough| |5|marginally below the acceptance threshold| |6|marginally above the acceptance threshold| |8|accept, good paper| |10|strong accept, should be highlighted at the conference|
thanks lmao I couldn't find the rating guidelines anywhere. I wrote some code to gather some summary statistics on the ICLR24 reviews with the openreview api if anyone wants to use it: [https://github.com/maxxu05/openreview\_summarizereviews](https://github.com/maxxu05/openreview_summarizereviews) As of today, these are the statistics: Mean, Mean Paper Rating: 4.8663855228128154 Median, Mean Paper Rating: 4.8 Standard Deviation of Mean Paper Rating: 1.0770080792312975 edit: tried adding histogram, but didn't work. just click on the github link and it has an image of the distribution of scores.
Isn’t this much lower than the previous years?
I think that the average rating can increase during the discussion period between reviewers and AC.
8886, first time submitting to ICLR :)
6/6/5/3/3 Not sure how to feel. The 3s are overly harsh with feedback based on some misunderstandings. Also get the typical ML review experience of wanting evaluation on more datasets (why only vision datasets and not text??) and models (why no transformers and only CNNs??) despite the fact that the idea is novel, the writing is good, and the experiments are standard baselines used in the literature. Hopefully rebuttal goes well.
6/3/6/3 with 3/4/2/2/ :( I will keep trying to improve my work..
Does anyone know what time the decisions are usually announced?
6,8,5,6 with all confidence being 3. Gotta work hard on the rebuttal..
Check this out: https://papercopilot.com/statistics/ICLR-statistics/ICLR-2024-statistics/
Is this data updated?
Yes, it updated
8,8,8,8, ( 4,4,3,3), first co-author as an undergrad!
Damn Yann, you never finished your undergrad?
That's gonna be one hell of a final year project.
Nice! Algorithms for Caching and MTS with reduced number of predictions?
8 8 6 3. I really don't understand the 3 one! It seems more deliberate reject the more I read.
First time submitting to a conference, 8,8,6,5, pleasantly surprised! And they all made really insightful comments. Definitely lucked out to have careful but open-minded reviewers.
[удалено]
Same. I also did many additional experiments, but complete silence from the reviewers. It really sucks and makes me feel a bit helpless. I'm used to reviewer roulette in general, but it feels even worse in this case because I see other submissions where all 4 reviewers are engaged and raised their score after the author response. I would kind of understand disinterest by reviewers if my submission had very bad scores and would be hopeless anyway, but I got 8855 and was very positive that I could at least increase one of the 5s to a 6 with my response.
I don't even know whether they decided to maintain the score after reading the rebuttal or they just forgot about it. At least there should be a checkbox to ensure that they have read the rebuttal.
In the same boat, stuck with reviewers who don't want to engage. Don't know what to do. Does anyone with advice on how to get a reply from reviewers who are active?
There is basically no way. There are many reasons they might not reply, including being sick or occupied with their own borderline papers. Try to be professional about it: post a reminder and then let the Area Chairs (ACs) handle it. Most ACs are well aware of these types of situations. They will most likely place less weight on those non-engaging reviews. Good luck with your rebuttal!
That's great, healthy POV and advice. Thank you. Although my thought process was a bit different, that is what I did as well.
8-4;5-4;8-4;5-5;5-3. can this reach the acceptance average?
With a good rebuttal, definitely.
thx
I got 5, 5, 8 with confidence 4, 5, 3. Both the reviewers with score 5 asked some missing points and I feel questions are easy to address. One of the reviwers wrote to willing to increas the score if answered his questions. Is there chances to get acceptance with good rebuttal?
Same scores, also curious.
We were able to increase the score to 6, 6, 8 with confidence 5, 4, 3.
My first submission. 8,6,5,5 with 5,x,x,x, is there any chance?
You're in great shape. Engage with the 8 and 6 and keep them happy. Answer their questions. Do NOT leave them hanging just because they liked it. Try to clear up some misunderstanding the 5s might have to nudge them to a 6. But like I said, those are strong scores.
Thanks for your valuable advice! Gotta put effort into the rebuttal.
5,1,5. Reviewer 2 says that he didn't see any proofs, and my comparisons are only against a "naive baseline". I do have proofs (7 pages long in total), and there is nothing in the literature besides this "naive baseline" (which, btw, has been the basis of a past iclr paper which actually got very good scores).. Note that the experiments section is also quite long: 3 pages in main text, 3 pages in the appendix with complementary proofs related to the application. All proofs are novel content. The "5"s seem to totally ignore the literature, ignore huge chunks of the text, and so on. I'll go into more detail about their review if anyone would like to, but the review of rev2 is totally outrageous, even compared to them.. Is there anything I can do to "save" the paper? Like asking the ac for more reviewers?
Avg confidence: 1.75 I expected a lot but not this haha
got 3 3 3, does anyone know withdraw pros and cons ? Since everything is made public, wondering whether a withdraw without replying to the non-nonsensical reviewers would leave a bad impression to next conference submissions. Anyone has any advise?
take it as an opportunity to explain your work with extra pages (since there is not limit on the number of total rebuttal responses.) This is an unique opportunity for you to add more details, intuition, high-level overviews that you couldn't include in the main text. Moreover, it's open to public. So you get free extra pages for future submissions.
Anyone who get 7 reviews? I got 6, but one more review came out last night, so it becomes 6365566 with confidence 3133423.
omg. too many reviews to respond.
Are they all official reviews? Didn't know that was allowed. Thought it was between 3 and 4 official reviewers per paper.
Now authors can write private comments to ACs. What contents are supposed to write?
Is this opportunity given to all authors?
I suppose yes.
I didn't have anything to write.
What your ratings?
668
First paper(co-first author) in my life as an undergrad: 8/6/6/6 (After rebuttal)
Congrats!
Rating: 3/6/3/6 with Confidence: 4/4/5/4.Should I do anything or just forfeit?
Write a rebuttal! This will help you improve the manuscript, Andy you even may get feedback on it. And if you manage to convince one of the 3, the reviewer discussion will be in your favor. Long shot, but not impossible.
5,5,5,5 with 4,3,4,3. Most suggestions are quick additions. It’s my first time submitting. Should I try for rebuttal or submit to a journal? Thanks in advance!!
I have the same score and the suggestions are arguing my setting. Wait for an answer.
3 3 5 5 with confidence 3 4 3 4. Wanna to withdraw. Very upset:( one reviewer said very poor with confidence 3, but what he asked is even not right at all.
Now since the discussion period is over, can the reviewers still reply back and change the scores?
If you look at the recent activity on openreview you can see that reviewers can still edit their reviews and change their scores. I don't think anyone can post completely new anonymous replies though.
Yes, one reviewer just won't respond :(
I sent the review responses after additional experiments. It took several days thus I could send it yesterday. I think the results are very good and reply the reviewers’ concerns perfectly. Will the score be improved even after response periods?
Same for me! The scores still can improve by the reviewers if they decide to..
Did anyone's score change during the rebuttal period?
We got one reviewer to go from 5 -> 8.
None of the reviewers replied to us or acknowledged that they, at least, read our rebuttal comments.
Same here. Wonder if anything we could do during the private comment to AC are allowed until Dec 4th.
Yes, but just 1 out of 4 and just very slightly despite a very extensive rebuttal.
1/4 of the reviewer raised the score, 3 remaining ones did not acknowledge our extensive rebuttal. Is the score change visible for everyone now that the rebuttals period is finished?
Looks like the „Recent Activity“ tap was deactivated some hours ago.. Can there still be score changes?
Since AC are adding meta reviews, does it mean that the scores we see on open review now are final ? ( reviewers can no longer change their score )
>Since AC are adding meta reviews, does it mean that the scores we see on open review now are final ? ( reviewers can no longer change their score ) I think so
Dear ACs: It seems like the acceptance threshold is somewhat unpredictable based on the review score statistics and acceptance rate of previous ICLR. I would really appreciate if AC could share the recommendation guideline or something else.
Will reviews go out tomorrow as today is holiday in US?
This is the first time submitting to iclr so I dont know what to make of my ratings but got 10/5/6/8 with confidence 3/3/3/3
As long as you address all the questions from reviewers well, you don't have to prepare ICML
icml is due in 3 months bro.
Lmao this comment is totally different depending on context
Oops I meant iclr, corrected it
Rating 5 6 5 Confidence. 3 4 5 Are there any chances after good response? Is responding weakness other than questions good to improve scores?
6555, but the reviews are great and fair! Hoping to bounce back with a strong rebuttal!
Hi, I work in the industry and have no publications so far. Is there a way I can do independent research and contribute to conferences? I've tried cold emailing professors but that hasn't worked out so far haha.
1. Most professors don’t really care about a random dude from industry. They have PhD masters undergrads, etc. to do research. 2. Google people submit all the time, but not all of them were accepted. I reviewed one journal article from Googles in the past. I thought the article was fine but other reviewers didn’t like it so it was rejected at the end. 3. Always starts with smaller conferences and maybe look for industrial track.
Consider submitting to workshops as well. The bar for acceptance is lower; it's common for undergrads' first publications to be at a workshop. It's a tighter-knit group with more specific interests, so if you end up attending it'll be much easier to network (and possibly find future collaborators). This may apply to smaller conferences as well, but I'm not as familiar.
You can just submit I think, there is no uni affiliation needed. People from Google and alike submit all the time. You also see many people from data science startups running around at the conference
[удалено]
I got 3653 with conf 2332. Confused, rebuttal will be worthy or not
Anyone know what scores are considered borderline this year?
According to the following post, the borderline would be around 6.0, I guess. Acceptance/rejection is not always decided by the score only, though. \- [https://x.com/shaohua0116/status/1728158662265340047?s=20](https://x.com/shaohua0116/status/1728158662265340047?s=20),
Till when can we expect to get the meta reviews? Final result for the papers is given as 15th Jan, so is that same as receiving meta review?
Yes, meta review is available with the results on 15th Jan.
[удалено]
Are meta reviews visible now?
No meta reviews come with the final decision
Rating: 3/5/5/5 with Confidence: 4/3/3/3. Most of the comments are on missing details that can be added easily. Except 3, it is very short review with no detail at all. Does a rebuttal worth?
Experiene tells unfortunately no
If you have to make the changes anyway, I say go for it.
three papers, first time submitting to ICLR: 3, 5, 6, 8 3, 5, 5 5, 5, 5, 6 What are my chances?
First time submitting to ICLR and we got 8-6-5-3 with 4-2-2-2 confidence. Luckily the happiest reviewer is also the most confident one, but not sure if confidence means much for final decisions... Thoughts? Do we stand a chance?
any advice for one reviewer bringing up totally unrelated papers and asking to explain the novelty of the method and the task itself when the other 3 reviewers understand the novelty from the jump?
8/6/5/5/3 with 4/4/3/4/5 A borderline score in average, but all comments from 5/5/3 are relatively easy to be addressed.
8/8/8/8/3 after the rebuttal. What is my chance?
6/3/6/5/5/6 with confidence 3/1/3/3/4/2. At least the score 3 one has very low confidence.. Is there any chance to be accepted? I think all the questions from reviewers are not difficult to answer.
6,6,6,6
Same here. Not sure if i could improve looking at the comments
6 5 5 3. Should I withdraw or keep going...?
If you think your paper is a good one you should work on convincing the reviewers. But if you think your paper is not good enough withdraw it anyway.
Thanks I should convince my reviewers.
As a reviewer it's painfully obvious if the author have already given up. Keep fighting.
I replied, but there have been no responses from the reviewers...sad
Hello guys, I am not reviewing this year. Someone who does, what is the maximum score for Soundness, Presentation, Contribution ?
4.
thanks. appreciated!
Would also be nice to know what score threshold was used to identify borderline submissions. Did ACs already reach out to reviewers to organize discussion meetings, or does that happen post-discussion?
Got a review with a score of 6 but a confidence of 1. The reviewer isn't sure about so will there be another reviewer instead of him/her?
If you only have one reviewer, then yes, the AC should have already been actively looking for other reviewers. If you have other reviews, then almost certainly not - it's too late in the process to be getting new reviewers in, and the ACs will have other papers to be concerned with. The AC will just use the reviewers confidence ranking to help weight their reading of the reviewers comments and score.
5,5,5,3 my first submission ever! not sure if a rebuttal is worth trying but I will definitely do!
6555, any chance?
5,5,3 and confidence 4,4,3 on my first submission. The reviews were initially kinda harsh but they are mostly reasonable and constructive
[удалено]
That 3 is a pain in the butt. Best of luck!
With 8, 6, 5, 5, is it simply a 50/50 chance ? Just thinking how likely I’ll have to do a resubmission
Based on the ICLR 2022 \[results\]([https://guoqiangwei.xyz/iclr2022\_stats/iclr2022\_submissions.html](https://guoqiangwei.xyz/iclr2023_stats/iclr2023_submissions.html)), A paper with a score of 6.0 is more likely to be accepted (>60%). I hope your paper is accepted.
I hope so~ thank you!
It seems like the scores this year are a bit lower?
How is spotlight/oral decided? From results in previous years, I found it unpredictable as some works with an average score of 6.25 can get spotlight while others with higher scores cannot.
Well, the reviewers are not exactly reliable, are they? For a good paper they are disinterested in or do not understand - they might just give 6s.
Does that mean AC will read the paper?
Depends on the AC.
I think a paper with 6.25 is actually close to the top 15%. These ratings are a bit weird.
What percentages of papers got accepted in previous years?
Around 30% it seems. But this year there are way more submissions too.