> The Jeopardy! match with Watson was essentially rigged due to the computer’s superhuman reaction time.
From many anecdotes I've heard from previous Jeopardy contestants, buzzer technique is more than half the battle. That's part of the reason Ken Jennings was able to go on winning as long as he did - he continuously refined and developed better intuition for when to buzz.
Many contestants (a large portion of whom likely participated in college Trivia Bowls and were seasoned trivia buffs) could answer the vast majority of Jeopardy clues with relative ease.
The reason for this is that Jeopardy, in order to have household appeal, cannot legitimately engage in an increasingly difficult arms race for its questions - it has to have a capped ceiling in terms of clue difficulty.
> Recently, however, that long-held truism has been shaken: The show has started releasing statistics about buzzer attempts, revealing that some contestants, especially those in the game’s upper echelons, ring in much more often than others. It’s not that the buzzer doesn’t matter—it just matters much less than was previously believed, a shake-up that has reelevated the importance of subject-matter expertise.
Difficult questions are fun, there's a reason the UK show "Only Connect" does well. Yeah, I'm not going to know what's going on when the question relies on knowing the difference between Finnegans Wake and Finnegan's Wake, but when you list decreasingly obscure eliminated Google products I know what's going on by clue three even if many contestants only even recognize the fourth clue.
Among HN readers I'm guessing "Control" and "Shift" already set people on the correct path before they see "Alt" or "Option" but a lot of word focused people doing such a quiz will be studying the words - counting vowels, thinking about meanings in other languages before they realise it's about keyboards.
Jeopardy already puts a small spin on things with the reverse format, they could work on that harder to deliver a more challenging quiz that's still getting the audience.
There were two astounding takeaways when Watson beat Ken and Brad:
1. We've advanced the state of the art in terms of being able to reason and parse complex language to a level the general population wasn't aware of.
2. Beating humans on the buzzer, of all things, feels intrinsically unfair and there's no graceful way to account for that.
Watson's ability to interpret, correctly, what most of the Jeopardy! "answers" were looking for was a sea change. It really was as big as Deep Blue, the crossing of a frontier we collectively thought would likely be much further in the future, and it was here now: complex clues, difficult to reason out context, and the ability to search its corpus of knowledge and assign a confidence value and then produce accurate results... it was mind blowing.
It's funny because, in a pinch, ChatGPT will answer essentially every Jeopardy! clue correctly every single time now, owing to the specific nature of generative transformers and the advent of attention, specifically. But Watson was pre "attention is all you need!" It was blood, sweat, and tears, and markov chains, and all kinds of craziness. It was a huge swing, and the fact that it actually worked was remarkable.
But back to the buzzer: most people sitting at home probably don't realize that the buzzer is locked out from buzzing in until the host (then Alex, now Ken) completes the read of the clue. At that point, an off-stage producer presses a switch to unlock the buzzers. So prospective players spend their weeks before their tape date spamming a retractable pen in front of the TV trying to nail the timing from the cadence of the host's voice, and hoping they can get it right. Because getting it wrong penalizes you: if you attempt to buzz in before the lock is lifted, your buzzer is individually locked for another 250ms or so. Another producer has a screen that monitors the buzzer activity so they can see if players are habitually buzzing in too early and getting locked, or to monitor for malfunctions, and it's wild to watch it light up like a Christmas tree in the 100 or so last milliseconds of the clue.
This is why you see players spam the buzzer: if they're locked out, the best thing they can do is keep attempting to get back in until the lock lifts. This is also why on rare occasions, you'll see all three players spamming and no one actually getting buzzed in: everyone locked.
So how do you translate this to a computer? The answer was basically to create a makeshift "finger" that would activate the buzzer, but would consistently buzz in at exactly 28ms after the lock cleared... every single time (so long as its confidence was high enough).
I think it's fair to say that there's nothing, at _all_, remarkable about a computer beating a human on a buzzer. It's a signal loop: wait for lock to clear, receive signal lock has cleared, activate buzzer 28ms later. Every device with a stopwatch or timer or clock is more accurate than a human attempting the same timing exercise. It's a solved problem, and not an interesting one. And it disproportionately took the wind out of the sails from the human competitors, who are both the most world-class, best-on-the-planet Jeopardy! players, who were losing by milliseconds to that signal loop.
It's still remarkable Watson was able to successfully answer so many clues correctly and end up winning, but the buzzer aspect was an (intractable) bummer.
If I recall, Watson was fed the whole text of the answer as soon as the other contestants could see it. Personally, I thought it would be fairer to have it do speech-to-text and/or OCR to level the playing field.
But I suppose these constraints are just targeting the machine's input mechanism and not its actual reasoning ability once the answer is read. I'm curious how Watson at the time could handle a particular category that Ken dominated: "Initials to Roman Numerals to Numbers" [1]
I think it's fair to say that there's nothing, at _all_, remarkable about a computer beating a human on a buzzer.
As anyone who has experienced early video game consoles can attest --- even the <2MHz CPU of a NES is far more than fast enough to beat a human at reaction time, which is what makes it all the more impressive when humans do things like https://news.ycombinator.com/item?id=38841080
>It's funny because, in a pinch, ChatGPT will answer essentially every Jeopardy! clue correctly every single time now
“Answering Jeopardy questions” is one of those tasks where being close to correct is not good enough. A system that hallucinates plausible-sounding answers would be a kind of awful approach to creating software that’s meant to 100% a game like Jeopardy
I'm really curious how this kind of writing will be perceived in the coming years. It's "good" long form writing but a lot of that also feels like journalistic padding for the sake of the form, as if you asked an LLM to "write in the style of a long form journalist."
Easy example:
"...dismissed as “disappointing,” “unfair,” and—my favorite—“a gimmick”..."
Is it really important for me to know Clare McNear's favourite? A couple of years ago I wouldn't have thought anything of it but now I'm so sickened constantly by LLMs adding so much needless cruft to everything they write I'm reading this thinking "just tell me the story, for the love of humanity!"
I’m seeing hyperreality in real time. You complain that an article of long form journalism appears to be the sort of thing that might be produced by asking an LLM to create it?
Doesn’t that mean it is true to form? Can you not imagine that some people like reading stories written by others?
Human long-form writing will be perceived as the top of value.
What is more valued today, IKEA furniture or hand-made natural wood furniture? A mass-produced Timex quartz watch or a hand-made mechanical watch? A painting or an ink-jet print of a painting?
https://archive.is/p2hG5
From the article:
> The Jeopardy! match with Watson was essentially rigged due to the computer’s superhuman reaction time.
From many anecdotes I've heard from previous Jeopardy contestants, buzzer technique is more than half the battle. That's part of the reason Ken Jennings was able to go on winning as long as he did - he continuously refined and developed better intuition for when to buzz.
Many contestants (a large portion of whom likely participated in college Trivia Bowls and were seasoned trivia buffs) could answer the vast majority of Jeopardy clues with relative ease.
The reason for this is that Jeopardy, in order to have household appeal, cannot legitimately engage in an increasingly difficult arms race for its questions - it has to have a capped ceiling in terms of clue difficulty.
That is covered in the article:
> Recently, however, that long-held truism has been shaken: The show has started releasing statistics about buzzer attempts, revealing that some contestants, especially those in the game’s upper echelons, ring in much more often than others. It’s not that the buzzer doesn’t matter—it just matters much less than was previously believed, a shake-up that has reelevated the importance of subject-matter expertise.
> in order to have household appeal
Difficult questions are fun, there's a reason the UK show "Only Connect" does well. Yeah, I'm not going to know what's going on when the question relies on knowing the difference between Finnegans Wake and Finnegan's Wake, but when you list decreasingly obscure eliminated Google products I know what's going on by clue three even if many contestants only even recognize the fourth clue.
Among HN readers I'm guessing "Control" and "Shift" already set people on the correct path before they see "Alt" or "Option" but a lot of word focused people doing such a quiz will be studying the words - counting vowels, thinking about meanings in other languages before they realise it's about keyboards.
Jeopardy already puts a small spin on things with the reverse format, they could work on that harder to deliver a more challenging quiz that's still getting the audience.
There were two astounding takeaways when Watson beat Ken and Brad:
1. We've advanced the state of the art in terms of being able to reason and parse complex language to a level the general population wasn't aware of.
2. Beating humans on the buzzer, of all things, feels intrinsically unfair and there's no graceful way to account for that.
Watson's ability to interpret, correctly, what most of the Jeopardy! "answers" were looking for was a sea change. It really was as big as Deep Blue, the crossing of a frontier we collectively thought would likely be much further in the future, and it was here now: complex clues, difficult to reason out context, and the ability to search its corpus of knowledge and assign a confidence value and then produce accurate results... it was mind blowing.
It's funny because, in a pinch, ChatGPT will answer essentially every Jeopardy! clue correctly every single time now, owing to the specific nature of generative transformers and the advent of attention, specifically. But Watson was pre "attention is all you need!" It was blood, sweat, and tears, and markov chains, and all kinds of craziness. It was a huge swing, and the fact that it actually worked was remarkable.
But back to the buzzer: most people sitting at home probably don't realize that the buzzer is locked out from buzzing in until the host (then Alex, now Ken) completes the read of the clue. At that point, an off-stage producer presses a switch to unlock the buzzers. So prospective players spend their weeks before their tape date spamming a retractable pen in front of the TV trying to nail the timing from the cadence of the host's voice, and hoping they can get it right. Because getting it wrong penalizes you: if you attempt to buzz in before the lock is lifted, your buzzer is individually locked for another 250ms or so. Another producer has a screen that monitors the buzzer activity so they can see if players are habitually buzzing in too early and getting locked, or to monitor for malfunctions, and it's wild to watch it light up like a Christmas tree in the 100 or so last milliseconds of the clue.
This is why you see players spam the buzzer: if they're locked out, the best thing they can do is keep attempting to get back in until the lock lifts. This is also why on rare occasions, you'll see all three players spamming and no one actually getting buzzed in: everyone locked.
So how do you translate this to a computer? The answer was basically to create a makeshift "finger" that would activate the buzzer, but would consistently buzz in at exactly 28ms after the lock cleared... every single time (so long as its confidence was high enough).
I think it's fair to say that there's nothing, at _all_, remarkable about a computer beating a human on a buzzer. It's a signal loop: wait for lock to clear, receive signal lock has cleared, activate buzzer 28ms later. Every device with a stopwatch or timer or clock is more accurate than a human attempting the same timing exercise. It's a solved problem, and not an interesting one. And it disproportionately took the wind out of the sails from the human competitors, who are both the most world-class, best-on-the-planet Jeopardy! players, who were losing by milliseconds to that signal loop.
It's still remarkable Watson was able to successfully answer so many clues correctly and end up winning, but the buzzer aspect was an (intractable) bummer.
If I recall, Watson was fed the whole text of the answer as soon as the other contestants could see it. Personally, I thought it would be fairer to have it do speech-to-text and/or OCR to level the playing field.
But I suppose these constraints are just targeting the machine's input mechanism and not its actual reasoning ability once the answer is read. I'm curious how Watson at the time could handle a particular category that Ken dominated: "Initials to Roman Numerals to Numbers" [1]
[1] https://youtu.be/PsQ_mT5YSvg?si=8L9uKJj1hpYU_kuf
I think it's fair to say that there's nothing, at _all_, remarkable about a computer beating a human on a buzzer.
As anyone who has experienced early video game consoles can attest --- even the <2MHz CPU of a NES is far more than fast enough to beat a human at reaction time, which is what makes it all the more impressive when humans do things like https://news.ycombinator.com/item?id=38841080
>It's funny because, in a pinch, ChatGPT will answer essentially every Jeopardy! clue correctly every single time now
“Answering Jeopardy questions” is one of those tasks where being close to correct is not good enough. A system that hallucinates plausible-sounding answers would be a kind of awful approach to creating software that’s meant to 100% a game like Jeopardy
ChatGPT would have to just always buzz since generating text is too slow for it to come up with an answer before buzzing.
I still think its when they went ultra pedantic about the pronunciation of Solzhenitsyn.
I thought Jeopardy!’s most controversial moment was when Ken Jennings used the word “ho”.
[flagged]
I'm really curious how this kind of writing will be perceived in the coming years. It's "good" long form writing but a lot of that also feels like journalistic padding for the sake of the form, as if you asked an LLM to "write in the style of a long form journalist."
Easy example:
"...dismissed as “disappointing,” “unfair,” and—my favorite—“a gimmick”..."
Is it really important for me to know Clare McNear's favourite? A couple of years ago I wouldn't have thought anything of it but now I'm so sickened constantly by LLMs adding so much needless cruft to everything they write I'm reading this thinking "just tell me the story, for the love of humanity!"
I’m seeing hyperreality in real time. You complain that an article of long form journalism appears to be the sort of thing that might be produced by asking an LLM to create it?
Doesn’t that mean it is true to form? Can you not imagine that some people like reading stories written by others?
Human long-form writing will be perceived as the top of value.
What is more valued today, IKEA furniture or hand-made natural wood furniture? A mass-produced Timex quartz watch or a hand-made mechanical watch? A painting or an ink-jet print of a painting?