I wish this were organized according to suggested/logical reading order. For example, the paper introducing the attention mechanism probably ought to precede "attention is all you need".
Noting the theory papers on Kolmorogov complexity. For those not familiar, Ilya argues that the reason why neural networks generalize -- why they work at all -- is because they are effectively finding a simple description of their training data, converging down onto the limit of the Kolmorogov complexity. [1]
I was confused for a minute, I thought this was "top 30 papers by Ilya" and was then wondering why "Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton" is on the list.
> In additition, even though I have read the vast majority of the papers featured on the website, I have not read through each of the website's versions end to end.
Website's versions, as in - the actual text or the "explanations"? Either way this is a big red flag.
Author here. First year CS student at Trinity College Dublin. I Built this because when I was getting into reading research papers I ended up burning a ton of my Claude usage asking questions other people have probably already asked. The website is just a side project and definitely a WIP. Happy to answer questions or take PRs on GitHub.
As an aside, I've seen folks mention respecting reduced animation hints and such in the past and was always curious about this because I've never had any negative experiences with animations... until now!
Something about the animations on this site did my brain in while scrolling through the papers, and now I "get it."
I thought the actual 30 papers have never been disclosed. Do you have a source tying the recommendations back to Ilya, or did you come up with this list?
I wouldn't say so about Occam's Razor which is a heuristic.
The relationship between compression and intelligence, while not equal is definitely there. It looks like 3Blue1Brown is going to be doing some videos on this aspect.
The formatting of the articles on this website is bad. I've opened the first one and all the LaTeX formulas are messed up. The subscripts and superscripts are all flattened rendering the math hard to comprehend. Did the author actually try to read any of the articles?
>∏ plocal(x|z) = i p(xi|z,xWindowAround(i))
Images and tables are not rendered at all. What is the point of this? Just keep the links to arxiv and leave it at that, otherwise render the articles properly
Why on earth would you deliberately choose to do whatever the fuck it is you did with the scroll and the animations for each paper when scrolling through the landing page? What are those animations supposed to be? I use firefox but I also visited on chrome, and the page is even more broken there. Scroll doesn't "take" unless I scroll hard enough, otherwise it bounces back. But on chrome, at least, it seems like the animation for each paper is clearer - it's supposed to be animating the scale of the paper as you scroll to it.. but it seems that your background animation is lagging everything so much it just doesn't work.
I wish this were organized according to suggested/logical reading order. For example, the paper introducing the attention mechanism probably ought to precede "attention is all you need".
Noting the theory papers on Kolmorogov complexity. For those not familiar, Ilya argues that the reason why neural networks generalize -- why they work at all -- is because they are effectively finding a simple description of their training data, converging down onto the limit of the Kolmorogov complexity. [1]
[1] https://www.youtube.com/watch?v=AKMuA_TVz3A
I was confused for a minute, I thought this was "top 30 papers by Ilya" and was then wondering why "Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton" is on the list.
> In additition, even though I have read the vast majority of the papers featured on the website, I have not read through each of the website's versions end to end.
Website's versions, as in - the actual text or the "explanations"? Either way this is a big red flag.
Author here. First year CS student at Trinity College Dublin. I Built this because when I was getting into reading research papers I ended up burning a ton of my Claude usage asking questions other people have probably already asked. The website is just a side project and definitely a WIP. Happy to answer questions or take PRs on GitHub.
An option to disable animation and show the paper links in a simple list would be helpful.
Agree on the animation.
As an aside, I've seen folks mention respecting reduced animation hints and such in the past and was always curious about this because I've never had any negative experiences with animations... until now!
Something about the animations on this site did my brain in while scrolling through the papers, and now I "get it."
I think it'd be interesting to hear what you think the goal of the site is.
Is it just rehosting the list, plus a reformatted copy of the papers? I was hoping you'd have at least annotated them with what you'd learned?
Where did you get the list? AFAIK, list was never shared
Nice presentation of the list!
I'd recommend watching a few of his talks/podcasts before during reading these to get the overview and how all the bits in these works tie together.
https://www.dwarkesh.com/p/ilya-sutskever
https://simons.berkeley.edu/talks/ilya-sutskever-openai-2023...
https://www.dwarkesh.com/p/ilya-sutskever-2
I thought the actual 30 papers have never been disclosed. Do you have a source tying the recommendations back to Ilya, or did you come up with this list?
I think someone on Twitter made it up. It was also 40 papers, not 30. https://dallasinnovates.com/exclusive-qa-john-carmacks-diffe...
This list was made by some guy on twitter. https://x.com/keshavchan/status/1787861946173186062
It's unknown whether it has anything to do with Ilya Sutskever.
Is this meant to be read in order?
Kolmogorov Complexity looks interesting. It seems to formalize Occam’s Razor and the notion that intelligence = compression.
I wouldn't say so about Occam's Razor which is a heuristic.
The relationship between compression and intelligence, while not equal is definitely there. It looks like 3Blue1Brown is going to be doing some videos on this aspect.
Is there a way to download them all in one go?
https://colah.github.io/posts/2015-08-Understanding-LSTMs/
https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... https://papers.nips.cc/paper_files/paper/2012/file/c399862d3...
https://arxiv.org/pdf/1512.03385
https://arxiv.org/pdf/1511.07122
https://arxiv.org/pdf/1603.05027
https://arxiv.org/pdf/1409.2329
https://arxiv.org/pdf/1512.02595
https://arxiv.org/pdf/1409.0473
https://arxiv.org/pdf/1506.03134
https://arxiv.org/pdf/1706.03762
https://nlp.seas.harvard.edu/annotated-transformer/
https://arxiv.org/pdf/1410.5401
https://arxiv.org/pdf/1706.01427
https://arxiv.org/pdf/1806.01822
https://arxiv.org/pdf/1704.01212
https://arxiv.org/pdf/2001.08361
https://arxiv.org/pdf/1811.06965
https://www.cs.toronto.edu/~hinton/absps/colt93.pdf
https://arxiv.org/pdf/math/0406077
https://scottaaronson.blog/?p=762
https://arxiv.org/pdf/1405.6903
https://onlinelibrary.wiley.com/doi/10.1002/047174882X.ch14 https://github.com/Bladefidz/information-theory/blob/master/...
https://arxiv.org/pdf/1611.02731
https://www.vetta.org/documents/Machine_Super_Intelligence.p...
https://karpathy.github.io/2015/05/21/rnn-effectiveness/
https://cs231n.github.io/
The formatting of the articles on this website is bad. I've opened the first one and all the LaTeX formulas are messed up. The subscripts and superscripts are all flattened rendering the math hard to comprehend. Did the author actually try to read any of the articles?
>∏ plocal(x|z) = i p(xi|z,xWindowAround(i))
Images and tables are not rendered at all. What is the point of this? Just keep the links to arxiv and leave it at that, otherwise render the articles properly
Why on earth would you deliberately choose to do whatever the fuck it is you did with the scroll and the animations for each paper when scrolling through the landing page? What are those animations supposed to be? I use firefox but I also visited on chrome, and the page is even more broken there. Scroll doesn't "take" unless I scroll hard enough, otherwise it bounces back. But on chrome, at least, it seems like the animation for each paper is clearer - it's supposed to be animating the scale of the paper as you scroll to it.. but it seems that your background animation is lagging everything so much it just doesn't work.
Main page UX is terrible. If you go for quirky, fine, but I would not want to use it.
Yes, normally wouldn't ever say anything, but I could even read the text things were just flying around. (On firefox)
Indeed. I scoffed at your comment and went to the website. After scrolling a bit, I find myself having a mild headache and slight dizziness.
I would request the author to consider something that does not distract us from this educational and informative website ( I have bookmarked it ).
> "beginner friendly format" > looks inside > math