When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison.
[1] I specifically mean those who are able to hold very big complex ideas and systems in their head, and reason about them, which seems to be an important talent for mathematicians.
"you could have..." is among the top insulting phrases used by maths-adjacent people. Others in that league are "it should now be obvious...", "it's abundantly clear...", "it can be easily shown that...", "this is nothing but..." etc.
The rest of us reading this are like, holy batman, what the fuck was that?!
You could have your own hacker news, it's just a textbox, a bunch of tables and headings! Once you add these, it'll be abundantly clear that you also need a database. It should now be obvious that you also need a user system and it can be easily shown that needs a backend. Admin tools, tests, statistics, performance checks and so on can easily be derived from such backend.
Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation.
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.
Math notation ultimately is pseudo code just with mostly single letter variables and many operators that are encoded purely by position thus not even requiring a symbol.
Remember that the oft-used e^x is actually an infinite series, even writing it out in summation form would be quite verbose given its frequency in many equations.
You know its a doozy when the author writes a disclaimer at the top saying that bra-ket notation was chosen in order to make the algorithm and data structures clearer.
All of this is true, but I don't believe and I want to be wrong about this that there is something in this notation that starts at an ELI 5 level and gently guides you to physicist level expertise. I all but majored in math (deriving back prop was trivial once it was clear it was the chain rule as one example) but I have never been able to keep bra ket notation straight in my head for the more exotic operations. Einsteinian notation on the other hand is a few minutes of furled brows and then all is clear.
It is what has separated me from being able to code just about anything on a GPU and being known for some of that work and coming up with a better way to run ab initio quantum chemistry on them.
It truly has been my Waterloo for many years. So make me wrong.
Yeah, bra-ket is arbitrary tensors (inner and outer multiplication) rather than the nice 4D of space-time (with derivatives).
I will say that seeing transformers written this way gives me a bit more intuition for what is going on (being able to identify correct equations), but there's enough complexity in actual transformer implementations, that it still feels like I'm fooling myself.
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
At first I felt bad about not having come up with this solution. But then I realized I have problems with writing binary search by myself in JS and immediately felt better.
Now way I could have come up with Kimi Delta Attention.
Lots of linear algebra codes are actually “easy to write” in a way. It isn’t like conventional CS where you are always going a bunch of recursive nonsense going on. There should be mathematical relationships between all of the variables, there are well implemented libraries for the common mathematical concepts, and it is rare to need to go more than a couple loops deep (anything more complex than that should get shunted off into a library anyway).
No, you couldn't have. There are plenty of ML innovations that when push comes to shove only depend on having access to more compute, but this is one of the worst examples I've ever seen.
I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody anywhere near ML knew perfectly well why NOT to try that: because you just don't have the compute to fully unroll an LSTM. From that point attention is optimized (they key-query mechanic). The big innovation is not so much the mechanism itself but realizing the parallelize-ability of it.
It's sort of like if one would today make the "improvement" to attention to replace they key-query-value mechanic by just dropping it while making the entire context the latent space. That will outperform attention, nearly guaranteed. It'll also make even Google's cluster networks meltdown. Attention is one of those innovations that came mostly from realizing you had better hardware than everybody else and asking yourself how to use it. It's still quite the accomplishment, they had to get it working. But nobody else was really capable of making this leap.
I agree 100%. This field is not amenable to progress from people with a pen sitting in a corner proving theorems. The math is mostly uncertain vibes and to test it you need millions of dollars of compute. Smart loners just can't.
What? Little old me! Well, then, let's have a look...
> (First paragraph)
> A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode,
∣
q
⟩
∣q⟩ is a column vector,
⟨
k
∣
⟨k∣ is a row vector,
⟨
k
∣
q
⟩
⟨k∣q⟩ is a number, and
∣
v
⟩
⟨
k
∣
∣v⟩⟨k∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space.
If your aim is to truly 'get started' with ML then hardware is absolutely not a bottleneck (either local or cloud).
Remember that ML is much more than LLMs. Even modern day LLMs can be quantized to a point where they can run on local hardware although their capabilities won't be as impressive.
I would recommend looking into some of Andrej Karpathy's videos if you want a grasp of the basics.
He has a master's degree in physics from Oxford. Also there is a toggle to normal notation. Well, CS notation. I'm not a fan of transpose marks everywhere. I like an even more mathematics notation.
I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps.
Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.
Inference (the real time computation) and training (the computation you do when developing a model) are deeply tied by automatic differentiation.
In a way, you train by repeatedly inferring (forward), calculating a loss (how much your model sucks with its current predictions) and then improving the model by differentiating.
Take a look at Karpathy's micrograd repo to become more familiar with the process.
When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison.
[1] I specifically mean those who are able to hold very big complex ideas and systems in their head, and reason about them, which seems to be an important talent for mathematicians.
after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.
"you could have..." is among the top insulting phrases used by maths-adjacent people. Others in that league are "it should now be obvious...", "it's abundantly clear...", "it can be easily shown that...", "this is nothing but..." etc.
The rest of us reading this are like, holy batman, what the fuck was that?!
Answer with:
You could have your own hacker news, it's just a textbox, a bunch of tables and headings! Once you add these, it'll be abundantly clear that you also need a database. It should now be obvious that you also need a user system and it can be easily shown that needs a backend. Admin tools, tests, statistics, performance checks and so on can easily be derived from such backend.
Also the proof is so trivial that it’s left to the reader.
This one hurts the most, esp. in fields you're not familiar.
Well, at least these days an actually trivial (to a domain expert) proof can be delegated to a frontier model...
Right next to "Learn More" by software UI designers.
I don't even know most words they used in the paper haha
The notation looks complex, but underneath it's all just adding and multiplying.
Nothing complex
Grand Theft Auto VI looks complex, but underneath it's all just ones and zeros and NAND
"I invented a new algorithm"
"New algorithm, or fmadd?"
"... fmadd."
as is practically all of transformer maths if you squint hard enough...
Relevant xkcd: https://www.explainxkcd.com/wiki/index.php/2501:_Average_Fam...
Yeah, pretty sure the “you” in “you could have” is a different “you” than “we”.
Imagine reading the title again in the voice of the Asian Father Meme.
"You could have come up with Kimi Delta Attention, but you didn't, did you."
Not even close for me too.
I thought I was the only one.
Thank goodness. There are dozens of us.
dozens!!!
possibly scores!!!
Scores? More like "grosses"
They lost me just describing the notation lol
Same!
Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation.
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.
Math notation ultimately is pseudo code just with mostly single letter variables and many operators that are encoded purely by position thus not even requiring a symbol.
Remember that the oft-used e^x is actually an infinite series, even writing it out in summation form would be quite verbose given its frequency in many equations.
math notation doesn't bias towards English language understanding like pseudocode
You know its a doozy when the author writes a disclaimer at the top saying that bra-ket notation was chosen in order to make the algorithm and data structures clearer.
One of the more annoying parts of my physics study was getting used to the new matrix multiplication notation they came up with every semester.
Raising and lowering operators for summation notation are the beginner tools for covariant derivatives of the metric tensor.
Christoffel symbols are where it's at, if you need to write out the Ricci tensor. The more constrained the space the more concise the notation can be.
Note that MechE tensor notation has an even more compact (eigen) form for principal stresses.
All of this is true, but I don't believe and I want to be wrong about this that there is something in this notation that starts at an ELI 5 level and gently guides you to physicist level expertise. I all but majored in math (deriving back prop was trivial once it was clear it was the chain rule as one example) but I have never been able to keep bra ket notation straight in my head for the more exotic operations. Einsteinian notation on the other hand is a few minutes of furled brows and then all is clear.
It is what has separated me from being able to code just about anything on a GPU and being known for some of that work and coming up with a better way to run ab initio quantum chemistry on them.
It truly has been my Waterloo for many years. So make me wrong.
Yeah, bra-ket is arbitrary tensors (inner and outer multiplication) rather than the nice 4D of space-time (with derivatives).
I will say that seeing transformers written this way gives me a bit more intuition for what is going on (being able to identify correct equations), but there's enough complexity in actual transformer implementations, that it still feels like I'm fooling myself.
A visualized tutorial: https://snowchord.com/blog/linear-attention-visualized/
I really liked the ket notation. I was aprehensive at first, but it makes operations much more clear.
I would have liked some refresher on some variables though (like d_k in quadratic attention).
LLM written for sure:
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
Ya, probably started with asking for a buzzy title.
This is what you get when you prompt claude to avoid –
Side note, before you ask: yes, bra-ket notation is called like that because of the brackets.
https://en.wikipedia.org/wiki/Bra-ket_notation
At first I felt bad about not having come up with this solution. But then I realized I have problems with writing binary search by myself in JS and immediately felt better.
Now way I could have come up with Kimi Delta Attention.
Lots of linear algebra codes are actually “easy to write” in a way. It isn’t like conventional CS where you are always going a bunch of recursive nonsense going on. There should be mathematical relationships between all of the variables, there are well implemented libraries for the common mathematical concepts, and it is rare to need to go more than a couple loops deep (anything more complex than that should get shunted off into a library anyway).
The toggle is really useful. Liked it!!
I like the math vs physics toggle.
No, you couldn't have. There are plenty of ML innovations that when push comes to shove only depend on having access to more compute, but this is one of the worst examples I've ever seen.
I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody anywhere near ML knew perfectly well why NOT to try that: because you just don't have the compute to fully unroll an LSTM. From that point attention is optimized (they key-query mechanic). The big innovation is not so much the mechanism itself but realizing the parallelize-ability of it.
It's sort of like if one would today make the "improvement" to attention to replace they key-query-value mechanic by just dropping it while making the entire context the latent space. That will outperform attention, nearly guaranteed. It'll also make even Google's cluster networks meltdown. Attention is one of those innovations that came mostly from realizing you had better hardware than everybody else and asking yourself how to use it. It's still quite the accomplishment, they had to get it working. But nobody else was really capable of making this leap.
I agree 100%. This field is not amenable to progress from people with a pen sitting in a corner proving theorems. The math is mostly uncertain vibes and to test it you need millions of dollars of compute. Smart loners just can't.
replace they key-query-value mechanic by just dropping it while making the entire context the latent space.
What do you mean by this? Like concatenating all token embeddings into one large vector?
Loved this incremental evolution, things gets way more understandable...usually xD
>You Could Have Come Up With Kimi Delta Attention
What? Little old me! Well, then, let's have a look...
> (First paragraph)
> A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣ q ⟩ ∣q⟩ is a column vector, ⟨ k ∣ ⟨k∣ is a row vector, ⟨ k ∣ q ⟩ ⟨k∣q⟩ is a number, and ∣ v ⟩ ⟨ k ∣ ∣v⟩⟨k∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space.
Hmm... Guess not!
I love that they let you switch to a more common q'k notation!
Where do linear algebra folks go to get started with ML stuff? It seems pretty easy but the hardware is expensive.
I think Karpathys nn zero to hero is a good starting point. And you can experiment on small networks using pretty normal hardware.
Huh?
If your aim is to truly 'get started' with ML then hardware is absolutely not a bottleneck (either local or cloud).
Remember that ML is much more than LLMs. Even modern day LLMs can be quantized to a point where they can run on local hardware although their capabilities won't be as impressive.
I would recommend looking into some of Andrej Karpathy's videos if you want a grasp of the basics.
what is a linear algebra folk?
Its greek to me.
If I could, I'd be working for one of the labs and commanding a seven-figure salary. :)
Relevant XKCD
https://xkcd.com/2501/
https://www.youtube.com/watch?v=p0CEOkoSsgA
why are you using braket notation?
He has a master's degree in physics from Oxford. Also there is a toggle to normal notation. Well, CS notation. I'm not a fan of transpose marks everywhere. I like an even more mathematics notation.
And a PhD in Quantum Computing! I'm a physicist so a fan of bra-ket tbh
I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps.
Is it really one big computation f(g(h(x)))?
Yes.
Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.
Inference (the real time computation) and training (the computation you do when developing a model) are deeply tied by automatic differentiation.
In a way, you train by repeatedly inferring (forward), calculating a loss (how much your model sucks with its current predictions) and then improving the model by differentiating.
Take a look at Karpathy's micrograd repo to become more familiar with the process.
What's your distinction between real time vs multiple steps? All computation is done in steps.
Is it all one big computation? Its turtles all the way down.
It's all vibes.
Hmm. Hmmm. Hmm. HMMM. Hmm.
Yep! I know some of these words.