I'm tired of these posts; LLMs are good for happy-path demos, that's it. And even then, their success rate depends on the prompter already knowing the answer!
Literally any out-of-distribution project in which I used LLMs lead to catastrophic failure. The models can't "see" stuff outside their training data.
I legitimately can’t tell if you’re being serious. It kind of seems like you might be trying to parody LLM detractors that will never admit to their usefulness. If you’re serious, why choose to say so in this post, which includes hard evidence that you’re wrong?
You should already know what to ask to extract the answer OpenAI claims gpt-5.2-pro gave them.
Then you should be lucky to get an answer that makes sense.
Then you should already know how to verify the model's response.
Only after all these steps should you cherry-pick the one-in-a-million successful response to feature on your website.
And finally, you should prove that the answer didn't already exist in the training data. It's highly likely that the problem was solved before and the model picked that up. I have yet to see a genuinely novel discovery these models can produce.
* I'm an LLM researcher, but that doesn't mean I should close my eyes to the unjustified hype around language models.
According to the post, this result was first derived for gluons in a previous paper. That paper was provided to the model as context, and then the model was asked to derive an analogous result for gravitons, which presumably has not been done before. The authors claim it would have taken "considerable time" for human experts to derive the graviton result.
I don't see any reason to believe that this exact problem was solved before in the training data, but it's definitely an incremental result based on a very similar problem that the model had seen before.
>It's highly likely that the problem was solved before and the model picked that up.
If you can demonstrate that, I would put it to Strominger and his colleagues, and I imagine they would be obligated to cite your contribution in the peer-reviewed publication.
> If you can demonstrate that, I would put it to Strominger and his colleagues, and I imagine they would be obligated to cite your contribution in the peer-reviewed publication.
There's one little problem: OpenAI isn't actually open and doesn't reveal which dataset they used for training.
Do you understand the purported result, and the verification? I don't, but I'm confident that Andrew Strominger wouldn't have agreed to put his name on this if he didn't think it was correct and interesting.
The human authors have positions at the Institute for Advanced Study (Einstein's old institution), Vanderbilt, Harvard (Strominger) and Cambridge in the UK.
If you have to gauge this by the reputation of the experts involved in it as I do, that seems like a good list to me.
I'm tired of these posts; LLMs are good for happy-path demos, that's it. And even then, their success rate depends on the prompter already knowing the answer!
Literally any out-of-distribution project in which I used LLMs lead to catastrophic failure. The models can't "see" stuff outside their training data.
I legitimately can’t tell if you’re being serious. It kind of seems like you might be trying to parody LLM detractors that will never admit to their usefulness. If you’re serious, why choose to say so in this post, which includes hard evidence that you’re wrong?
> which includes hard evidence that you’re wrong?
You should already know what to ask to extract the answer OpenAI claims gpt-5.2-pro gave them.
Then you should be lucky to get an answer that makes sense.
Then you should already know how to verify the model's response.
Only after all these steps should you cherry-pick the one-in-a-million successful response to feature on your website.
And finally, you should prove that the answer didn't already exist in the training data. It's highly likely that the problem was solved before and the model picked that up. I have yet to see a genuinely novel discovery these models can produce.
* I'm an LLM researcher, but that doesn't mean I should close my eyes to the unjustified hype around language models.
According to the post, this result was first derived for gluons in a previous paper. That paper was provided to the model as context, and then the model was asked to derive an analogous result for gravitons, which presumably has not been done before. The authors claim it would have taken "considerable time" for human experts to derive the graviton result.
I don't see any reason to believe that this exact problem was solved before in the training data, but it's definitely an incremental result based on a very similar problem that the model had seen before.
>It's highly likely that the problem was solved before and the model picked that up.
If you can demonstrate that, I would put it to Strominger and his colleagues, and I imagine they would be obligated to cite your contribution in the peer-reviewed publication.
> If you can demonstrate that, I would put it to Strominger and his colleagues, and I imagine they would be obligated to cite your contribution in the peer-reviewed publication.
There's one little problem: OpenAI isn't actually open and doesn't reveal which dataset they used for training.
This shouldn't prevent anyone from finding and reporting a similar pre-existing result in the literature.
Do you understand the purported result, and the verification? I don't, but I'm confident that Andrew Strominger wouldn't have agreed to put his name on this if he didn't think it was correct and interesting.
The human authors have positions at the Institute for Advanced Study (Einstein's old institution), Vanderbilt, Harvard (Strominger) and Cambridge in the UK.
If you have to gauge this by the reputation of the experts involved in it as I do, that seems like a good list to me.