> because who would want to buy a Pentium Pro when the better-in-almost-every-way Pentium MMX
Maybe this was satirizing people running after every shiny new thing, but the MMX certainly wasn’t. The Pro was the first P6, a totally new, out-of-order microarchitecture that the Pentium 2 would be based on. The MMX was a standard P5 Pentium with a larger L1 and the MMX vector instruction set which, while the first of its kind in an x86, was ultimately not used much in applications, partly because it was integer-only (also because it awkwardly co-opted the existing FPU registers rather than having its own register file). In any case, the Pros were expensive and meant for business use, while the MMXs were consumer processors.
Just one caveat: the Pro overlapped in time with people running a lot of 16-bit code, and I think the Pentium MMX was faster for that, or at least that the Pro didn't offer much edge.
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves and do a better job with that analysis.
I don't buy this. I've seen GMs playing online while verbalizing their thought process (on twitch / youtube videos where they livestream their play), and they consider way more branches than I do, while also going deeper than I ever would on some branches. My top rating in lichess blitz is about 1900, not bad but way lower than a GM.
I am about 300 points higher and think this is roughly correct. In openings amateurs will actually analyze more since GMs just have hundreds of opening lines memorized many moves deep. Middlegame and endgame there're usually just 1-3 reasonable plans that are obvious to a GM (but not at all obvious to amateurs) with 1-3 initial moves each. So, again, probably about as many as I consider.
The only time when GM will definitely analyze more lines than an amateur is in a highly tactical position but that happens, again, maybe 1-3 times per typical game (tactics themselves will last several moves, but they would be largely calculated through at the beginning of the tactical melee).
Slightly difficult to disentangle is that they might be using the verbalisation as a way to explore depth, rubber-duck style, in a way that they wouldn't over the board.
It makes sense. The space of moves is a tree. If you're doing a breadth-first search and a grandmaster is doing a more depth-oriented search with heuristics for pruning low value branches, then the same total number of moves could be considered, but the grandmaster is reaching the better moves before you.
No, a grandmaster can also consider moves way faster than an amateur. Watch Hikaru Nakamura solve chess puzzles [1]. He has already solved the puzzle and moved on to the next one before an amateur can even see the pieces on the board.
1940s to 1980s papers (the source of the claims) might think "amateur" means something different from people today. (they probably mean club chess players)
It seemed to me that in the chess community, average club players are usually considered (much) stronger than the average online player. (Especially because of the much larger number of lower skill players online.)
So if you measure amateurs via club players (=what people might have considered chess amateurs before), that might give you a different result from the average online player (=what people might consider average amateurs today).
It is much easier to imagine the 1600 FIDE OTB player doing 20-40 positional evaluations than the 1200 random chess.com scrub (eg me) doing the same.
Yeah I would like to see where that factoid comes from. Maybe it means they explore as many branches from the root position as an amateur, but they explore each branch way deeper and more accurately
GMs also have access to a greater number of memorized situations. Regardless of whether they are conscious of it or not, they have previously analyzed so many moves that they can cut off branches they just know instinctively to be dead ends. Machines could do this, directly reference a massive dataset of precomputed board positions, but that isn't considered a proper chess engine... despite the fact that human GMs do use such techniques.
A modern chess engine would definitely have defeated Kasparov in 1996. It would have had a database containing all the games, particularly that strange game against a computer nobody knew anything about, which seemed to be able to anticipate all of Kasparov's moves.
Great project but I have a nit on the board displays: every board is displayed as <human username> vs Pentium Pro, even when the Pentium is playing White. The normal chess convention is <White name> vs <Black name>.
I spent some time trying to figure out how some of the humans seemed to have trapped the machine out of the opening, because it looked like the human was winning.
Very cool project. It seems incredible that such a "humble" device for the time would match and beat deep blue. It must be true that software has been living off hardware for a very long time, even if as others comment, the latest developments in software have only been possible after hardware got much much better.
Cool exploration, and very much needed because depressingly few people who use these engines actually understand this.
Back in 2021 I got cfish and weiss working on a 386sx, though didn't test it much. cfish was doing 5k nps so some results can be extrapolated. NNUE complicated things due to RAM requirements. Though I did train a small one for lichess at some point and they can be surprisingly good.
Very interesting. I don't play chess but like to observe others playing and talking about it on YouTube.
I thought about something the other day... Modern chess sites run proprietary algorithms to detect cheating by modern engines. But would it be possible for an old engine, let's say from the 70s or 80s, running on modern hardware to deceive these algorithms?
They wouldn't win every tournament but still have a respectable performance.
I've built engines from scratch. I don't think that would work at all. It's a whole extra challenge to make an engine feel like a human. The engine has to come up with its best move and then sometimes choose not to play it and play something else for some reason. There are statistical patterns to how humans make mistakes based on their level, what kinds of mistake, how big and how often. Cheat detection is less about knowing the engine and more about statistics.
Maybe a pure neural net (no tree search at all) would stand a better chance. Shockingly, those can play at IM level.
From what I gather cheat detection uses a combination of factors to determine how "human-like" an account is: time usage, webpage interactions, move quality under time pressure and so on. I think you could probably try to deliberately deceive a specific website's cheat detection with software that tries to emulate how humans play, at which point you'd be in the classic attack vs. defense arms race.
Also notice that for events like Title Tuesday you must play on a dedicated client with at least 2 cameras, so it would be significantly harder to pull it off.
With AI gaining consciousness I can't wait for such "human-only" measures to be delegated to the trashcan of history, same as slavery. What happens when AI-human cyborgs start walking around? Will they also be treated as second class citizens?
We divide competitions into classes (by sex and age is common, for example "17 yo boys", and similar competition classes), so I don't see why it shouldn't be subject to the same kind of class distinction, maybe with an without augmentation.
> But yes, totally unsurprising that a megahertz machine with 100MB+ available ram has >2500 ELO.
I think this is underselling it a bit. If you'd told this to the IBM engineers at the time I think they would have taken great interest in how the performance was achieved.
I grew up with these machines around, and as a kid they seemed to so much more powerful than humans. It's kinda weird to think that I became 'smarter' than them eventually. I suspect they will be able to run superhuman AI one day too.
I don't think that real-time, superhuman AI performance is ever gonna happen with <1b parameter models.
Biology had a lot of time, attempts and pressure to make this work but even the most optimized lines (something like a magpie) need more complexity (almost a billion neurons) to get a fraction of superhuman intelligence.
Results with small artificial neural nets are very limited so far, too.
> As it turns out, the majority of the Elo gains (+720 of the +1335 gained) can be attributed to better programming, and not better hardware.
It's not so easy, a lot of that better programming can be attributed to better hardware, it wouldn't have been possible to implement NNUE or Fishtest without modern hardware.
I'll go out on a limb and say that future AI models, running on today's smartphones, will be better at programming and formal math than today's frontier models running in datacenters. And by "future" I mean less than 5 years from now.
When I tried this with a 6502 (Apple II), it did not work. It was just too underpowered for enough modern algorithms to kick in. It might also be an LLm-driving skill issue of course!
> (Also, uh, if anyone has any ideas for projects I could do with a Pentium Pro in a few weeks once I no longer need it to host this chess engine ... do let me know!)
Throw a 3dfx Voodoo in there and do some real-time 3D Gaussian Splatting!
In one sense, Knuth fans and really good engineers are correct to prioritize algorithms.
Unfortunately, most humans see big|new|expensive hardware as great status signifiers, barriers to lesser folks joining the Cool Club, and optimal sources of sales commissions.
> because who would want to buy a Pentium Pro when the better-in-almost-every-way Pentium MMX
Maybe this was satirizing people running after every shiny new thing, but the MMX certainly wasn’t. The Pro was the first P6, a totally new, out-of-order microarchitecture that the Pentium 2 would be based on. The MMX was a standard P5 Pentium with a larger L1 and the MMX vector instruction set which, while the first of its kind in an x86, was ultimately not used much in applications, partly because it was integer-only (also because it awkwardly co-opted the existing FPU registers rather than having its own register file). In any case, the Pros were expensive and meant for business use, while the MMXs were consumer processors.
Just one caveat: the Pro overlapped in time with people running a lot of 16-bit code, and I think the Pentium MMX was faster for that, or at least that the Pro didn't offer much edge.
Awesome article. Nit:
> the average chess grandmaster during a classical game will analyze roughly the same number of moves as an amateur player, but they analyze only the most critical moves and do a better job with that analysis.
I don't buy this. I've seen GMs playing online while verbalizing their thought process (on twitch / youtube videos where they livestream their play), and they consider way more branches than I do, while also going deeper than I ever would on some branches. My top rating in lichess blitz is about 1900, not bad but way lower than a GM.
I am about 300 points higher and think this is roughly correct. In openings amateurs will actually analyze more since GMs just have hundreds of opening lines memorized many moves deep. Middlegame and endgame there're usually just 1-3 reasonable plans that are obvious to a GM (but not at all obvious to amateurs) with 1-3 initial moves each. So, again, probably about as many as I consider.
The only time when GM will definitely analyze more lines than an amateur is in a highly tactical position but that happens, again, maybe 1-3 times per typical game (tactics themselves will last several moves, but they would be largely calculated through at the beginning of the tactical melee).
Human brain can do calculations in the back of their head without realizing it. Verbalizing does not mean it's all they thought about.
But it does mean they thought about at least as much as they verbalised.
Slightly difficult to disentangle is that they might be using the verbalisation as a way to explore depth, rubber-duck style, in a way that they wouldn't over the board.
Right, but the GMs you see online are often "super GMs". They are on a category of their own in the ultra tail.
No, I watch plenty of "regular" GMs, as well as some IMs and even FMs who all consider way more moves/positions than I do.
It makes sense. The space of moves is a tree. If you're doing a breadth-first search and a grandmaster is doing a more depth-oriented search with heuristics for pruning low value branches, then the same total number of moves could be considered, but the grandmaster is reaching the better moves before you.
No, a grandmaster can also consider moves way faster than an amateur. Watch Hikaru Nakamura solve chess puzzles [1]. He has already solved the puzzle and moved on to the next one before an amateur can even see the pieces on the board.
[1] https://youtu.be/4EfSymvT59M
1940s to 1980s papers (the source of the claims) might think "amateur" means something different from people today. (they probably mean club chess players)
Yes? Amateur means someone who does a thing for recreation, as opposed to for money.
It seemed to me that in the chess community, average club players are usually considered (much) stronger than the average online player. (Especially because of the much larger number of lower skill players online.)
So if you measure amateurs via club players (=what people might have considered chess amateurs before), that might give you a different result from the average online player (=what people might consider average amateurs today).
It is much easier to imagine the 1600 FIDE OTB player doing 20-40 positional evaluations than the 1200 random chess.com scrub (eg me) doing the same.
Yeah I would like to see where that factoid comes from. Maybe it means they explore as many branches from the root position as an amateur, but they explore each branch way deeper and more accurately
GMs also have access to a greater number of memorized situations. Regardless of whether they are conscious of it or not, they have previously analyzed so many moves that they can cut off branches they just know instinctively to be dead ends. Machines could do this, directly reference a massive dataset of precomputed board positions, but that isn't considered a proper chess engine... despite the fact that human GMs do use such techniques.
A modern chess engine would definitely have defeated Kasparov in 1996. It would have had a database containing all the games, particularly that strange game against a computer nobody knew anything about, which seemed to be able to anticipate all of Kasparov's moves.
Great project but I have a nit on the board displays: every board is displayed as <human username> vs Pentium Pro, even when the Pentium is playing White. The normal chess convention is <White name> vs <Black name>.
I spent some time trying to figure out how some of the humans seemed to have trapped the machine out of the opening, because it looked like the human was winning.
Very cool project. It seems incredible that such a "humble" device for the time would match and beat deep blue. It must be true that software has been living off hardware for a very long time, even if as others comment, the latest developments in software have only been possible after hardware got much much better.
> Throughout this article, when I say "I wrote", what I really mean is "I asked an LLM to write".
I wish disclaimers like that would be at the start and not halfway trough the text.
Cool exploration, and very much needed because depressingly few people who use these engines actually understand this.
Back in 2021 I got cfish and weiss working on a 386sx, though didn't test it much. cfish was doing 5k nps so some results can be extrapolated. NNUE complicated things due to RAM requirements. Though I did train a small one for lichess at some point and they can be surprisingly good.
edit. A good view on the continuous progress of Stockfish itself https://nextchessmove.com/dev-builds
Very interesting. I don't play chess but like to observe others playing and talking about it on YouTube.
I thought about something the other day... Modern chess sites run proprietary algorithms to detect cheating by modern engines. But would it be possible for an old engine, let's say from the 70s or 80s, running on modern hardware to deceive these algorithms?
They wouldn't win every tournament but still have a respectable performance.
I've built engines from scratch. I don't think that would work at all. It's a whole extra challenge to make an engine feel like a human. The engine has to come up with its best move and then sometimes choose not to play it and play something else for some reason. There are statistical patterns to how humans make mistakes based on their level, what kinds of mistake, how big and how often. Cheat detection is less about knowing the engine and more about statistics.
Maybe a pure neural net (no tree search at all) would stand a better chance. Shockingly, those can play at IM level.
Sargon II is available:
https://c64online.com/c64-games/sargon-ii-de/
My guess is that they would be even easier to detect.
From what I gather cheat detection uses a combination of factors to determine how "human-like" an account is: time usage, webpage interactions, move quality under time pressure and so on. I think you could probably try to deliberately deceive a specific website's cheat detection with software that tries to emulate how humans play, at which point you'd be in the classic attack vs. defense arms race.
Also notice that for events like Title Tuesday you must play on a dedicated client with at least 2 cameras, so it would be significantly harder to pull it off.
With AI gaining consciousness I can't wait for such "human-only" measures to be delegated to the trashcan of history, same as slavery. What happens when AI-human cyborgs start walking around? Will they also be treated as second class citizens?
We divide competitions into classes (by sex and age is common, for example "17 yo boys", and similar competition classes), so I don't see why it shouldn't be subject to the same kind of class distinction, maybe with an without augmentation.
> Will they also be treated as second class citizens?
Yes, along side you robosexuals
what does this have to do with anything whatsoever
Wonderful work! Just inspiring.
I had a Pentium Pro 200 back in the day, to run 3DSMax 1.0 on Windows NT 3.5.
This is very cool. But yes, totally unsurprising that a megahertz machine with 100MB+ available ram has >2500 ELO.
> But yes, totally unsurprising that a megahertz machine with 100MB+ available ram has >2500 ELO.
I think this is underselling it a bit. If you'd told this to the IBM engineers at the time I think they would have taken great interest in how the performance was achieved.
I grew up with these machines around, and as a kid they seemed to so much more powerful than humans. It's kinda weird to think that I became 'smarter' than them eventually. I suspect they will be able to run superhuman AI one day too.
I don't think that real-time, superhuman AI performance is ever gonna happen with <1b parameter models.
Biology had a lot of time, attempts and pressure to make this work but even the most optimized lines (something like a magpie) need more complexity (almost a billion neurons) to get a fraction of superhuman intelligence.
Results with small artificial neural nets are very limited so far, too.
Except it has 64MB of RAM... Anyway, in the early 2000s machines were already doing better than DeepBlue, so it didn't last long
Okay, but probably also a turbo button, which is tantamount to cheating.
The turbo button slows the CPU down so old programs that depend on clock speed (e.g. early games) run correctly.
> As it turns out, the majority of the Elo gains (+720 of the +1335 gained) can be attributed to better programming, and not better hardware.
It's not so easy, a lot of that better programming can be attributed to better hardware, it wouldn't have been possible to implement NNUE or Fishtest without modern hardware.
I'll go out on a limb and say that future AI models, running on today's smartphones, will be better at programming and formal math than today's frontier models running in datacenters. And by "future" I mean less than 5 years from now.
No way. The parameter count can't fit on consumer devices, and you're not getting orders of magnitude smarter behavior with smaller models.
I wonder how far back you can go? Could a modern chess engine run on a Cray 1? 386? 68k?
I don't think there's a meaningful difference between the Pentium and the 80386 except speed. You might have to change the compiler options a bit.
And, with more work, that's true of computers in general; they can all do the same computations once you somehow wire up enough memory: https://dmitry.gr/?r=05.Projects&proj=35.%20Linux4004
I understand about Turing machines.
For chess though, there are time constraints. The 386 is a lot slower. Is it fast enough?
When I tried this with a 6502 (Apple II), it did not work. It was just too underpowered for enough modern algorithms to kick in. It might also be an LLm-driving skill issue of course!
https://github.com/zellyn/8fish
> (Also, uh, if anyone has any ideas for projects I could do with a Pentium Pro in a few weeks once I no longer need it to host this chess engine ... do let me know!)
Throw a 3dfx Voodoo in there and do some real-time 3D Gaussian Splatting!
Why is the time per move fixed? A chess engine can, of course, do much better by deciding itself how much time to allot to each move.
In one sense, Knuth fans and really good engineers are correct to prioritize algorithms.
Unfortunately, most humans see big|new|expensive hardware as great status signifiers, barriers to lesser folks joining the Cool Club, and optimal sources of sales commissions.