Yeah it already helps me too with finetuning small models for image stuff.
Im not telling it what to do, i ask it for ar eport/analysis and research and read its article and tell it what to implement based on 'feelings' and it works.
Limit is hardware and tokens and an AI Researcher not having hardware or tokens is probably not what they refer to.
Alone normal researchers were quite handicaped: converting data from tool a to python library x, understanding how tools work, doing basic google research etc.
I find this as well for models that make it to production.
On a more fun note, I also find just trying some hair brained idea is now extremely cheap - so why not? I used to have to weigh up spending a week of effort on something likely not to work and now I can now get spit out test in 20 minutes.
Not trying to discredit the premise but I do wonder how much this is from vertical scaling of model intelligence in weights vs the sheer amount of capital being thrown at training and horizontal scaling of agent swarms etc.
Anecdotally it seems like the majority of continued uplift comes from models doing more and exploring more of the search space, and we are getting better at giving them embarrassingly parallel problems to Hoover up with infinite token spend.
I grok that you're not trying to discredit anything, or anything like that but what are you trying to do, if I may ask?
What information are you trying to either ask for or share with us?
The original article is pretty clear, agent swarms and token spend are AI automating AI, and the RSP attribution condition specifically excludes your capital thrown at training concern?
I guess my point is that the scaling is coming from capital explosion and it isn’t guaranteed that this will continue scaling as it did (the impending hardware and infrastructure crunch basically guarantees this).
The article is pretty strongly projecting this trend to continue but hasn’t acknowledged this factor.
> the impending hardware and infrastructure crunch basically guarantees this
I'm not sure I follow. Are you saying a hardware shortage is going to restrict AI development? Because it seems to me like it would just push us towards an even greater focus on efficiency, which is still pushing the Pareto Frontier.
i'm also wondering whether there are confounding factors like more domain experts and talents joining the AI research field because there are more interesting problems to be solved (or other 2nd order effects of AI models being more powerful).
Another data point: look at the number of submissions to Neurips, ICLR, etc. Now, I agree that a submission does not by itself imply groundbreaking resesarch, but the fact that people are able to try out/vet ideas much faster now makes a difference.
In my own work, I've found that it's much faster to brainstorm with a frontier model like Opus. I'll have an idea and be able to get a working implementation and try it out on toy problems within O(minutes), instead of O(weeks) earlier.
Trying out problems and making tangible progress towards AGI are entirely different categories. Likewise what’s changed in terms of paper submissions is also the amount of slop. Most reviewers say that they haven’t seen a lot more papers that are good but rather they’ve seen a lot more papers and almost all of them are terrible slop that’s desk rejected. This is happening in OSS too where a lot of projects are closing the gate at the onslaught of terrible contributions.
I do agree. AI isn't "free" the way silicon valley products are.
Therefore it cannot avoid taxes the way they did before (using shell companies to route the money through tax havens). It'll be expensive, every government will double the price or more, and you're selling to consumers, and the products will be at the mercy of every government where they try to sell AI and other services.
People don't get how serious this is. If AI doubles revenue but drives silicon valley firms' tax rates from 5% to 10%, it's a loss.
Add to that that it's something silicon valley firms have never done before.
The very same report also found that AI could not be deployed outside of a narrow domain of verifiable rewards … so take it with grains, or mountains of salt. It’s also extremely unclear how the productivity measures are calculated. They surely don’t seem to translate into economic output. Antrophic has seen its valuation corrected by 50% and possibly more… so yeah. This all sounds very fishy.
As long as we believe, we can imagine it is true. All seeing Oracle: how do you command me? Praise his name, bringer of many miracles - he who definitely can't be manipulated from the backend to provide biased information that suites a person's malicious purpose. Ya totally.
"The industry is not prepared to handle this safely"
After the election, titles like this just irritate me. Every piece written about AI progress is:
"The machines taking jobs, flooding the internet with sludge and poisoning our online experiences, are getting better even faster! Isn't this incredible? Yeah but sorry we also think it's only riskier than ever to deploy these systems at scale. That being said, we're still going to do it, go public and take institutional money from people's retirement funds, destroy the job market, and make 3000 Anthropic employees and 500 investors really rich. There wasn't anything we could do besides write a lot of ink on it and be shocked we lost every local data center referendum."
Yeah it already helps me too with finetuning small models for image stuff.
Im not telling it what to do, i ask it for ar eport/analysis and research and read its article and tell it what to implement based on 'feelings' and it works.
Limit is hardware and tokens and an AI Researcher not having hardware or tokens is probably not what they refer to.
Alone normal researchers were quite handicaped: converting data from tool a to python library x, understanding how tools work, doing basic google research etc.
I find this as well for models that make it to production.
On a more fun note, I also find just trying some hair brained idea is now extremely cheap - so why not? I used to have to weigh up spending a week of effort on something likely not to work and now I can now get spit out test in 20 minutes.
Not trying to discredit the premise but I do wonder how much this is from vertical scaling of model intelligence in weights vs the sheer amount of capital being thrown at training and horizontal scaling of agent swarms etc.
Anecdotally it seems like the majority of continued uplift comes from models doing more and exploring more of the search space, and we are getting better at giving them embarrassingly parallel problems to Hoover up with infinite token spend.
I grok that you're not trying to discredit anything, or anything like that but what are you trying to do, if I may ask?
What information are you trying to either ask for or share with us?
The original article is pretty clear, agent swarms and token spend are AI automating AI, and the RSP attribution condition specifically excludes your capital thrown at training concern?
I guess my point is that the scaling is coming from capital explosion and it isn’t guaranteed that this will continue scaling as it did (the impending hardware and infrastructure crunch basically guarantees this). The article is pretty strongly projecting this trend to continue but hasn’t acknowledged this factor.
> the impending hardware and infrastructure crunch basically guarantees this
I'm not sure I follow. Are you saying a hardware shortage is going to restrict AI development? Because it seems to me like it would just push us towards an even greater focus on efficiency, which is still pushing the Pareto Frontier.
METR always says what BigAI wants to hear. They are not independent in any way.
i'm also wondering whether there are confounding factors like more domain experts and talents joining the AI research field because there are more interesting problems to be solved (or other 2nd order effects of AI models being more powerful).
Another data point: look at the number of submissions to Neurips, ICLR, etc. Now, I agree that a submission does not by itself imply groundbreaking resesarch, but the fact that people are able to try out/vet ideas much faster now makes a difference.
In my own work, I've found that it's much faster to brainstorm with a frontier model like Opus. I'll have an idea and be able to get a working implementation and try it out on toy problems within O(minutes), instead of O(weeks) earlier.
Trying out problems and making tangible progress towards AGI are entirely different categories. Likewise what’s changed in terms of paper submissions is also the amount of slop. Most reviewers say that they haven’t seen a lot more papers that are good but rather they’ve seen a lot more papers and almost all of them are terrible slop that’s desk rejected. This is happening in OSS too where a lot of projects are closing the gate at the onslaught of terrible contributions.
I do agree. AI isn't "free" the way silicon valley products are.
Therefore it cannot avoid taxes the way they did before (using shell companies to route the money through tax havens). It'll be expensive, every government will double the price or more, and you're selling to consumers, and the products will be at the mercy of every government where they try to sell AI and other services.
People don't get how serious this is. If AI doubles revenue but drives silicon valley firms' tax rates from 5% to 10%, it's a loss.
Add to that that it's something silicon valley firms have never done before.
The very same report also found that AI could not be deployed outside of a narrow domain of verifiable rewards … so take it with grains, or mountains of salt. It’s also extremely unclear how the productivity measures are calculated. They surely don’t seem to translate into economic output. Antrophic has seen its valuation corrected by 50% and possibly more… so yeah. This all sounds very fishy.
As long as we believe, we can imagine it is true. All seeing Oracle: how do you command me? Praise his name, bringer of many miracles - he who definitely can't be manipulated from the backend to provide biased information that suites a person's malicious purpose. Ya totally.
Or maybe Anthropic got more user data to train on when they launched Fable without ZDR. That could explain large Opus 5.5 improvements.
Could be a mix of it but I thought it was RL.
Perhaps scaling to fable for the first time also gave Opus a push (distillation?)
That doesn't line up with their claimed divergence point, which was pre-Mythos.
"The industry is not prepared to handle this safely"
After the election, titles like this just irritate me. Every piece written about AI progress is:
"The machines taking jobs, flooding the internet with sludge and poisoning our online experiences, are getting better even faster! Isn't this incredible? Yeah but sorry we also think it's only riskier than ever to deploy these systems at scale. That being said, we're still going to do it, go public and take institutional money from people's retirement funds, destroy the job market, and make 3000 Anthropic employees and 500 investors really rich. There wasn't anything we could do besides write a lot of ink on it and be shocked we lost every local data center referendum."
Emacs is dramatically accelerating Emacs research