This refers specifically to "AI Frontiers", a research group within Microsoft Research that focuses on fundamental advances in AI. There is no end in sight to real AI frontiers.
Perhaps "The End of Microsoft Research's AI Frontiers Group"
would be a less ambiguous title.
Yeah, I went down the rabbit hole scratching my head the whole time. This is about the group at Microsoft Research that was disbanded or reorged into what is now Microsoft Research.
To the author: 100% need this kind of thinking. It’s a balancing act. Historically though, dedicated labs haven’t had the best track record at commercializing their tech. The skillset differs vastly from running a product-focused business.
That said, even within this narrow scope - there are areas ripe for research.
Some of my particular interests include:
(1) Latent thinking/reasoning, which could arguably be more efficient but also opaque is one
(2) Separation of “intelligence” from knowledge - ie. keep the ability to reason like a frontier model and make tool calls (calling the ‘oracle’ or searching the web) but remove all parameters associated with niche knowledge/facts.
These are a couple off the top of my head that are very much aligned to scaling.
The transformer based approach is seen as a highway to race down in terms of training and building better models with minimal organizational distractions allowed while every company puts its foot down on this highway. Exploratory research into other AI methods is highly discouraged for now and organizations are encouraging transformer based scale out and iteration as a top priority.
As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models. They got someone willing to scale the current method out. Essentially the messaging it "Foot down, there's a straight highway ahead don't divert".
I have the view this makes sense too. Current top models are mere (mere in big datacenter terms) trillions of parameters right now. We definitely haven't hit the ceiling and everyone's noticing "Hey this Opus 5.5 with the increased parameters seems much better than the last one". Well that's a trillion parameter model. We'll soon start hitting 10's of trillions and 100's of trillions of parameter models. Unless there's a ceiling which nobody expects they should keep scaling just as they have been. I really do agree that it'd be insane to veer off this highway with exploratory research at this particular point in time. Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.
> As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models.
Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.
Especially when a company pays billions to OpenAI, any activity that does not do glorious dance in praise and support of the investment is an act of disloyalty.
To me, all signs point to US over investment in hardware capacity to support the "frontier" when in reality the next major breakthroughs are likely radical optimizations that end in increased consumer training and inference efficiency.
I'm not sure how that, combined with open models inching up on frontier intelligence, is compatible with these massive AI IPO valuations?
It seems like closed model AI demand has a lot of contingencies that are ultimately fragile and possibly short sighted.
And gp's comment is relevant to the particular problems the guy (John Langford) cites in his comment about leaving. IE, that research first winds being only about scaling and then winds-up ignored because "just scale" can be an easier answer.
This refers specifically to "AI Frontiers", a research group within Microsoft Research that focuses on fundamental advances in AI. There is no end in sight to real AI frontiers.
Perhaps "The End of Microsoft Research's AI Frontiers Group" would be a less ambiguous title.
Yeah, I went down the rabbit hole scratching my head the whole time. This is about the group at Microsoft Research that was disbanded or reorged into what is now Microsoft Research.
To the author: 100% need this kind of thinking. It’s a balancing act. Historically though, dedicated labs haven’t had the best track record at commercializing their tech. The skillset differs vastly from running a product-focused business.
That said, even within this narrow scope - there are areas ripe for research.
Some of my particular interests include:
(1) Latent thinking/reasoning, which could arguably be more efficient but also opaque is one
(2) Separation of “intelligence” from knowledge - ie. keep the ability to reason like a frontier model and make tool calls (calling the ‘oracle’ or searching the web) but remove all parameters associated with niche knowledge/facts.
These are a couple off the top of my head that are very much aligned to scaling.
This is true in every large company right now.
The transformer based approach is seen as a highway to race down in terms of training and building better models with minimal organizational distractions allowed while every company puts its foot down on this highway. Exploratory research into other AI methods is highly discouraged for now and organizations are encouraging transformer based scale out and iteration as a top priority.
As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models. They got someone willing to scale the current method out. Essentially the messaging it "Foot down, there's a straight highway ahead don't divert".
I have the view this makes sense too. Current top models are mere (mere in big datacenter terms) trillions of parameters right now. We definitely haven't hit the ceiling and everyone's noticing "Hey this Opus 5.5 with the increased parameters seems much better than the last one". Well that's a trillion parameter model. We'll soon start hitting 10's of trillions and 100's of trillions of parameter models. Unless there's a ceiling which nobody expects they should keep scaling just as they have been. I really do agree that it'd be insane to veer off this highway with exploratory research at this particular point in time. Foot down, get to 10 trillion parameters, see if the scaling holds, if so off to 100 trillion parameters. If the scaling continues to hold we'll be asking the AI to do the exploratory research for us soon enough.
> As an example Meta literally fired it's previously head of AI who held views that transformers won't scale. He wanted to veer off the highway and heavily fund exploratory new types of models.
Almost all statements Yann LeCun made back then (LLMs won't be able to do this or that) turned out to be wrong. They were right to fire him.
Especially when a company pays billions to OpenAI, any activity that does not do glorious dance in praise and support of the investment is an act of disloyalty.
To me, all signs point to US over investment in hardware capacity to support the "frontier" when in reality the next major breakthroughs are likely radical optimizations that end in increased consumer training and inference efficiency.
I'm not sure how that, combined with open models inching up on frontier intelligence, is compatible with these massive AI IPO valuations?
It seems like closed model AI demand has a lot of contingencies that are ultimately fragile and possibly short sighted.
The article is about a guy leaving Microsoft and the disbanding of an internal team called "AI Frontiers".
And gp's comment is relevant to the particular problems the guy (John Langford) cites in his comment about leaving. IE, that research first winds being only about scaling and then winds-up ignored because "just scale" can be an easier answer.
Lack of focus and long term planning at MS
Microsoft is a really shitty place to work right now. There are a hundred stories like this all over the company like this.