When people refute the idea that "LLM is the new compiler from natural language to code" the common reason is that LLM is non-deterministic but compiler is deterministic.
But the determinism isn't the real distinction. LLM inference can be made deterministic (temparaute zero, etc.). And multi-threaded compiler can be non-dterministic.
And human developers are also non-deterministic. There is always the nondeterminism even before AI.
I think the real difference is how leaky the abstraction is. If you just need to understand language instead of compiler internal to write correct code, the compiler is non-leaky abstrction. If you rely on compiler to do auto SIMD optimization, and have to do hacks to trigger the optimization, compiler is leaky abstraction.
For AI, if you don't ever need to read the code and AI writes perfectly bug-free program that perfectly follows instruction, then AI is a non-leaky abstraction, but that's not the case for now. To make code maintainable and debuggable you still need to read the code if the project is complex enough.
> And human developers are also non-deterministic. There is always the nondeterminism even before AI.
Pointing out humans are not deterministic is irrelevant to the comparison. In the original example of LLMs replacing compilers, the deterministic compiler is being replaced, not the non-deterministic human. In the original example, we move from non-deterministic human -> sourced code -> deterministic compiler, to non-deterministic human -> sourced prompt(?) -> non-deterministic LLM. The important part is the sourced code into a deterministic final step so your build artifact doesn't change.
You can point out you can change temperature to zero or what ever, but then you don't actually get the build artifact you wanted in the first place.
Yeah, it's not the non-determinism; it's that they don't do what I say. I can't imagine writing a nontrivial program without, at a minimum, reading and editing the pieces as they're being produced. People aren't complaining about non-determinism; they're complaining about the results being non-deterministically sometimes wrong.
That actually would be significantly better, because then you could work out under what conditions they were wrong and correct for it. But being stochastically wrong means you have to check everything, because you never know when the errors will creep in.
> I think the real difference is how leaky the abstraction is. If you just need to understand language instead of compiler internal to write correct code, the compiler is non-leaky abstrction. If you rely on compiler to do auto SIMD optimization, and have to do hacks to trigger the optimization, compiler is leaky abstraction.
Autovectorization is not a programming model!
I forgot who put me on that article - I think it might've been Horace He - but that is a pretty big idea in idea space for me.
> LLMs, and their respective harnesses, are actors in a system. Code is just an output.
This tracks well with my experiences integrating AI into a software dev pipeline. Sure the output of an LLM is not deterministic in the same sense as a compiler (at least when the LLM has the typical configuration). However, that just means the LLM should not be solely responsible for the final product of your pipeline.
I'm so tired of the endless stream of articles that a.) don't know what "non-determinism" means in Computer Science and b.) don't really understand how LLMs work.
"Non-determinism" (i.e. the "N" in NP-Complete) does not mean "random", it means "following all paths at once" (there's a really elegant way to understand the List monad as representing non-deterministic computing). It's such a fundamental (and delightful) concept in computer science that refusing to understand what it means is a tragedy related to the loss of technical language itself.
But, even if we just sub "non-deterministic" for "inconsistent" or "random" or "stochastic" or whatever better term you come up with. Statements like the following just seem to not understand the basic architecture of LLMs:
> Given the same input prompt, a model's output is not guaranteed to be identical on every run, at least under normal running conditions.
"Normal running conditions"? The reproducibility of the output is 100% a property of how you choose to sample from an LLM. There are virtually endless ways you can define your sampler and whether or not the same result is reproducible is a fairly minor aspect of all the other choices you make when sampling.
At the end of the day, each step of the LLM gives you a distribution over next tokens, and you can do with that what you like.
In the end the article almost becomes a tautology: the output of an LLM represents a probability distribution over all the outputs it believes are valid with some variation around an expected output. Yes, that's literally what an LLM is.
The concept of determinism is useful in computer-science contexts other than automata theory. See, e.g., https://en.wikipedia.org/wiki/Deterministic_algorithm. It's not incorrect to use the term to refer to those things.
> One of my fundamental objections to LLM-driven software development was that model output is non-deterministic.
Model outputs are deterministic. Your API provider may be injecting untunable amounts of randomness, but if you're not at the whim of an opaque cloud service, your model can be as deterministic as you please.
The unease that you describe isn't about models not being deterministic (because they are), it's that generating outputs from LLMs is an inherently inductive process (operating backwards by extrapolating from observations and examples) rather than a deductive process (operating forwards from careful logical reasoning founded on axiomatic principles). Deductive processes are more reliable, predictable, and analyzable than inductive processes, but they're also much harder to produce for many classes of problems.
When people refute the idea that "LLM is the new compiler from natural language to code" the common reason is that LLM is non-deterministic but compiler is deterministic.
But the determinism isn't the real distinction. LLM inference can be made deterministic (temparaute zero, etc.). And multi-threaded compiler can be non-dterministic.
And human developers are also non-deterministic. There is always the nondeterminism even before AI.
I think the real difference is how leaky the abstraction is. If you just need to understand language instead of compiler internal to write correct code, the compiler is non-leaky abstrction. If you rely on compiler to do auto SIMD optimization, and have to do hacks to trigger the optimization, compiler is leaky abstraction.
For AI, if you don't ever need to read the code and AI writes perfectly bug-free program that perfectly follows instruction, then AI is a non-leaky abstraction, but that's not the case for now. To make code maintainable and debuggable you still need to read the code if the project is complex enough.
> And human developers are also non-deterministic. There is always the nondeterminism even before AI.
Pointing out humans are not deterministic is irrelevant to the comparison. In the original example of LLMs replacing compilers, the deterministic compiler is being replaced, not the non-deterministic human. In the original example, we move from non-deterministic human -> sourced code -> deterministic compiler, to non-deterministic human -> sourced prompt(?) -> non-deterministic LLM. The important part is the sourced code into a deterministic final step so your build artifact doesn't change.
You can point out you can change temperature to zero or what ever, but then you don't actually get the build artifact you wanted in the first place.
Yeah, it's not the non-determinism; it's that they don't do what I say. I can't imagine writing a nontrivial program without, at a minimum, reading and editing the pieces as they're being produced. People aren't complaining about non-determinism; they're complaining about the results being non-deterministically sometimes wrong.
And if they were deterministically sometimes wrong, that wouldn't be better.
That actually would be significantly better, because then you could work out under what conditions they were wrong and correct for it. But being stochastically wrong means you have to check everything, because you never know when the errors will creep in.
> I think the real difference is how leaky the abstraction is. If you just need to understand language instead of compiler internal to write correct code, the compiler is non-leaky abstrction. If you rely on compiler to do auto SIMD optimization, and have to do hacks to trigger the optimization, compiler is leaky abstraction.
Autovectorization is not a programming model!
I forgot who put me on that article - I think it might've been Horace He - but that is a pretty big idea in idea space for me.
the same way calculators replaced mental sums. not because it was sometimes faster, it was definitely more deterministic.
> LLMs, and their respective harnesses, are actors in a system. Code is just an output.
This tracks well with my experiences integrating AI into a software dev pipeline. Sure the output of an LLM is not deterministic in the same sense as a compiler (at least when the LLM has the typical configuration). However, that just means the LLM should not be solely responsible for the final product of your pipeline.
I'm so tired of the endless stream of articles that a.) don't know what "non-determinism" means in Computer Science and b.) don't really understand how LLMs work.
"Non-determinism" (i.e. the "N" in NP-Complete) does not mean "random", it means "following all paths at once" (there's a really elegant way to understand the List monad as representing non-deterministic computing). It's such a fundamental (and delightful) concept in computer science that refusing to understand what it means is a tragedy related to the loss of technical language itself.
But, even if we just sub "non-deterministic" for "inconsistent" or "random" or "stochastic" or whatever better term you come up with. Statements like the following just seem to not understand the basic architecture of LLMs:
> Given the same input prompt, a model's output is not guaranteed to be identical on every run, at least under normal running conditions.
"Normal running conditions"? The reproducibility of the output is 100% a property of how you choose to sample from an LLM. There are virtually endless ways you can define your sampler and whether or not the same result is reproducible is a fairly minor aspect of all the other choices you make when sampling.
At the end of the day, each step of the LLM gives you a distribution over next tokens, and you can do with that what you like.
In the end the article almost becomes a tautology: the output of an LLM represents a probability distribution over all the outputs it believes are valid with some variation around an expected output. Yes, that's literally what an LLM is.
The concept of determinism is useful in computer-science contexts other than automata theory. See, e.g., https://en.wikipedia.org/wiki/Deterministic_algorithm. It's not incorrect to use the term to refer to those things.
> One of my fundamental objections to LLM-driven software development was that model output is non-deterministic.
Model outputs are deterministic. Your API provider may be injecting untunable amounts of randomness, but if you're not at the whim of an opaque cloud service, your model can be as deterministic as you please.
The unease that you describe isn't about models not being deterministic (because they are), it's that generating outputs from LLMs is an inherently inductive process (operating backwards by extrapolating from observations and examples) rather than a deductive process (operating forwards from careful logical reasoning founded on axiomatic principles). Deductive processes are more reliable, predictable, and analyzable than inductive processes, but they're also much harder to produce for many classes of problems.