Also your dignity and sense of purpose, I would add. I think the consequences of these shenanigans are going to be way more wide reaching than what we can currently comprehend. I pray for us all.
This IMO is the real importance of deepseek, GLM, and other free weight models. You might not have the hardware to run something decent locally, but at least with hundreds of API providers they can all compete on whether they are reading the logs.
It’s incredible really that the big providers are hoovering up all the plain text of people describing every situation they find themselves in, at depth, not in googles wildest dreams did they have such detailed dossiers on every individua.
The population metrics of how people were thinking (to what propaganda are they most susceptible to?)
to control people you must have feedback on how effective your campaigning is, how many minds are you changing?
So opting out of OpenAI/Anthropic/Zuckerbucks’ data mining is all well and good but it is a population level problem of centralized data collection by the state, lest we forget room 641A!
They can say “unequivocally” they didn’t train on it but how long are they keeping it though? Anthropic went to five years in 2025 for anyone who allows training. All bets are off sounds about right.
That's unique connection of certain ideas that someone discovered - which LLM had access to - let it then repeat it and freely cross the bridge whose existence it had previously been unaware of. Inevitable (it's not Memex). Big steal on multiple levels including credits - it is how it works.
It's scary to think if too many scientists might be trying to save $20/month and use the free AI plans.
Or worse, not understand how this piece of technology is not snapshot based like the software before it, and instead is evolving constantly, and what impact that has on their base assumptions.
If only we had a technology that could classify the topic, tone, and sophistication of arbitrary natural-language conversations; you know, to sort the proofs from the furries…
Training the LLM on logs might make little sense, but that doesn't mean the LLM can't get access to those logs outside of training. Companies have been gobbling up your data left and right, why wouldn't these companies do the same and then feed it to their LLM for a specific use case?
I think Anthropic et al. would have very little difficulty determining exactly who their users are, so it would probably be trivial to focus any training of this sort on individuals known to be doing cutting edge research, as well as their colleagues, postdocs, grad students etc. Not saying it is happening, but they certainly have the capacity to make a system to skip all the prompts they don’t care about and leave just the prompts they do care about.
Are you kidding? It's clear that frontier labs are desperate to ingest any human text they can find. They're paying people to find rare books and digitize them. They're vacuuming up Reddit threads. And in their own products, they make it difficult to turn off conversation logging unless you pay.
I have zero difficulty accepting that they're using conversation logs, probably anonymized and then chopped so that they can claim it's no longer user data and that it can't be tracked back to a specific chat. And yes, sometimes, a single piece of text is enough to move the needle, especially when it comes to some exotic, sparse topics.
Of course it is. The people here who claim that there are too many prompts that would pollute the training data (deliberately) overlook that AI can classify the value of prompts quite well.
Even the free Google AI continues the conversation much longer and asks followup questions if you are serious.
Prompts of scientists will be forwarded to a data labeler and be used for training.
These prompts of course include the use case of converting a colleague's work to another file format using AI, which mathematicians always proudly announce. So they leak other people's ideas, too.
The goal of NSF grants aren't really to fund research into things that cure cancer, they're primarily to fund the education pipeline needed to train young scientists. When those students graduate, they tend to get jobs in the industry where they can advance commercial solutions that often have significant societal benefit.
If you're working with an AI to solve a mathematical problem, presumably it is contributing ideas. I think the scientists might be confusing the LLM's ideas with their own.
AI saves data in its network from which it can be (partially) recovered. It's lossy compression at its core. So I think you are the one who got it wrong, confusing people's work and ideas with the output of an LLM.
The point of science and math is to discover truths about the world, not to have a "career" being a "scientist". They should "steal ideas", uploading documents to a company that they are going to store for 100 years is what you're doing when you use a model on someone else's servers. As long as they say who they stole it from there shouldn't be any issue.
AI is the product and process of stealing all your ideas. The purpose of the thing is what it does. You are training your replacement.
Also your dignity and sense of purpose, I would add. I think the consequences of these shenanigans are going to be way more wide reaching than what we can currently comprehend. I pray for us all.
I have a feeling detaching your dignity and sense of purpose from something an AI can easily steal might be a net good for most people.
This IMO is the real importance of deepseek, GLM, and other free weight models. You might not have the hardware to run something decent locally, but at least with hundreds of API providers they can all compete on whether they are reading the logs.
It’s incredible really that the big providers are hoovering up all the plain text of people describing every situation they find themselves in, at depth, not in googles wildest dreams did they have such detailed dossiers on every individua.
The population metrics of how people were thinking (to what propaganda are they most susceptible to?)
to control people you must have feedback on how effective your campaigning is, how many minds are you changing?
So opting out of OpenAI/Anthropic/Zuckerbucks’ data mining is all well and good but it is a population level problem of centralized data collection by the state, lest we forget room 641A!
this conversation may be reviewed for safety!
This is like when the price of things increases because they are in your shopping cart.
They can say “unequivocally” they didn’t train on it but how long are they keeping it though? Anthropic went to five years in 2025 for anyone who allows training. All bets are off sounds about right.
That's unique connection of certain ideas that someone discovered - which LLM had access to - let it then repeat it and freely cross the bridge whose existence it had previously been unaware of. Inevitable (it's not Memex). Big steal on multiple levels including credits - it is how it works.
It's scary to think if too many scientists might be trying to save $20/month and use the free AI plans.
Or worse, not understand how this piece of technology is not snapshot based like the software before it, and instead is evolving constantly, and what impact that has on their base assumptions.
I think there's a market for a fundamentally privacy focused AI hosting company that just hosts Open models. I think Proton has something like this.
If someone's thinks their ideas are so precious, they should be using the privacy focused ones.
It makes no sense that the LLMs could be solving problems based on prompts given by individual scientists
Are they implying that chat history of a single person is being taken an used to skew post-training results?
Why would it only be for Navier-Stokes that we see this having an effect
There are 1 million times more prompts being processed for stuff like furry roleplay than unsolved scientific problems
If only we had a technology that could classify the topic, tone, and sophistication of arbitrary natural-language conversations; you know, to sort the proofs from the furries…
It's actually probably the case the proofs are coming from the furies.
Training the LLM on logs might make little sense, but that doesn't mean the LLM can't get access to those logs outside of training. Companies have been gobbling up your data left and right, why wouldn't these companies do the same and then feed it to their LLM for a specific use case?
I think Anthropic et al. would have very little difficulty determining exactly who their users are, so it would probably be trivial to focus any training of this sort on individuals known to be doing cutting edge research, as well as their colleagues, postdocs, grad students etc. Not saying it is happening, but they certainly have the capacity to make a system to skip all the prompts they don’t care about and leave just the prompts they do care about.
Are you kidding? It's clear that frontier labs are desperate to ingest any human text they can find. They're paying people to find rare books and digitize them. They're vacuuming up Reddit threads. And in their own products, they make it difficult to turn off conversation logging unless you pay.
I have zero difficulty accepting that they're using conversation logs, probably anonymized and then chopped so that they can claim it's no longer user data and that it can't be tracked back to a specific chat. And yes, sometimes, a single piece of text is enough to move the needle, especially when it comes to some exotic, sparse topics.
Of course it is. The people here who claim that there are too many prompts that would pollute the training data (deliberately) overlook that AI can classify the value of prompts quite well.
Even the free Google AI continues the conversation much longer and asks followup questions if you are serious.
Prompts of scientists will be forwarded to a data labeler and be used for training.
These prompts of course include the use case of converting a colleague's work to another file format using AI, which mathematicians always proudly announce. So they leak other people's ideas, too.
Your final paragraph refers to something that's about to be published on the web. So the AI will train on it anyway.
Imagine : scientists might have to work with each other to avoid using AI out of fear of stealing.
Something like the movie "The Most Unknown", where scientists spend time explaining their problems to others scientists in other/adjacent fields.
Tax payers wonder: are scientists stealing my money to patent tax funded research?
The goal of NSF grants aren't really to fund research into things that cure cancer, they're primarily to fund the education pipeline needed to train young scientists. When those students graduate, they tend to get jobs in the industry where they can advance commercial solutions that often have significant societal benefit.
If you have to ask, the answer is yes. Betteridge's Law.
If you're working with an AI to solve a mathematical problem, presumably it is contributing ideas. I think the scientists might be confusing the LLM's ideas with their own.
AI saves data in its network from which it can be (partially) recovered. It's lossy compression at its core. So I think you are the one who got it wrong, confusing people's work and ideas with the output of an LLM.
The point of science and math is to discover truths about the world, not to have a "career" being a "scientist". They should "steal ideas", uploading documents to a company that they are going to store for 100 years is what you're doing when you use a model on someone else's servers. As long as they say who they stole it from there shouldn't be any issue.
They will not say who they stole it from, and more importantly they will not pay them. Do you seriously think that scientists should work unpaid?
Hey they’re getting subsidized tokens to access the mother superior intelligence, how else can they afford to use it??