Inspired by this article, I tried to read some tutorials on Forth. My question is whether concatenative languages are AI-coding friendly. Apart from the training data availability, the question is also whether LLMs can correctly understand long flows of concatenated operations. Any ideas?
I've experimented with it and after some initial skepticism, I've been impressed -- because a simple Forth is implemented relatively concisely, it can get a "full" picture of the language by reading the implementation. The issue I've encountered is that it tends to treat forth as a single specified programming language, using non-existent words in some specific forth. With coaxing, it can correct this, though. I may write something up, but I am interested if anyone else has experimented with this -- culturally, Forth programmers tend to be very anti-AI (and I feel a bit 'icky' about my experiments)
They can produce idioms that resemble the flow of Forth code but when asked to produce a working algorithm, they get lost very quickly because there's a combination of reading "backwards" (push order) and forwards (execution order) needed to maintain context. At any time a real Forth program may inject a word into the stack flow that completely alters the meaning of following words, so reading and debugging Forth are nearly the same thing - you have to walk through the execution step by step unless you've intentionally made patterns that will decouple context - and when you do, you've also entered into developing syntax and the LLM won't have training data on that.
I suggest using Rosetta Code as a learning resource for Forth idioms.
Then you get to definitions like ": open ( string -- handle 1 | 0) ... ;" which describes returning algebraic type Maybe Handle unboxed on the stack. Algebraic types are fun, they can easily represent Peano arithmetic and get us into the realm Goedel incompleteness theorem very quickly.
Or you can deduce signature for EXEC EXEC sequence. EXEC's stack effect can be described as ( \alpha (\alpha -- \beta) -- \beta), where \greekletter is a placeholder for a stack part of arbitrary length. Notice that this type comment has nested brackets and does not adhere to Forth stack-effect comment convention.
When I thought about this more than fifteen years ago, I've got at least two equally valid types for the EXEC EXEC: one where xt at top of stack consumes all its input and leaves no output ( \alpha (\alpha -- \gamma) \beta (\beta -- ) -- \gamma) and when first EXEC produces something for second to execute upon ( \alpha \beta (\beta -- \gamma (\alpha \gamma -- \theta) -- \theta).
One can argue that second type of EXEC EXEC subsume first one, if greek-letter-named-stack-parts are allowed to be empty.
Still it shows that typing Forth, at the very least, needs unification on the Peano's arithmetic level, implementing deduction from length zero to unbounded length.
So, in my opinion, for LLM to dependably combine typed Forth/concatenative definitions, it needs to call external tool like Prolog to properly deduce type(s) of the sequence of Forth's (or concatenative language's) definitions.
This is a Prolog system to learn programs in polynomial time. For one example, it can one-shot-learn a grammar, without being "trained" on millions of samples.
So, should one use a LLM that either needs a paid access or just slow to run, or freely go where "old school" systems like Eurisco [1] and Cyc went?
Eurisco demonstrated superhuman abilities in 1982-83. It also demonstrated knowledge transfer at the time, where rules from VLSI place-and-route algorithms were used to design winning Traveler TCS fleet.
phreda4 has been doing cool stuff with ColorForth-likes for ages and for some reason barely gets any attention for it. Always brings a smile to my face to see it submitted here
Inspired by this article, I tried to read some tutorials on Forth. My question is whether concatenative languages are AI-coding friendly. Apart from the training data availability, the question is also whether LLMs can correctly understand long flows of concatenated operations. Any ideas?
I've experimented with it and after some initial skepticism, I've been impressed -- because a simple Forth is implemented relatively concisely, it can get a "full" picture of the language by reading the implementation. The issue I've encountered is that it tends to treat forth as a single specified programming language, using non-existent words in some specific forth. With coaxing, it can correct this, though. I may write something up, but I am interested if anyone else has experimented with this -- culturally, Forth programmers tend to be very anti-AI (and I feel a bit 'icky' about my experiments)
They can produce idioms that resemble the flow of Forth code but when asked to produce a working algorithm, they get lost very quickly because there's a combination of reading "backwards" (push order) and forwards (execution order) needed to maintain context. At any time a real Forth program may inject a word into the stack flow that completely alters the meaning of following words, so reading and debugging Forth are nearly the same thing - you have to walk through the execution step by step unless you've intentionally made patterns that will decouple context - and when you do, you've also entered into developing syntax and the LLM won't have training data on that.
I suggest using Rosetta Code as a learning resource for Forth idioms.
Any concatenative program can be reduced to a rho type, and AI are pretty good about combining properly typed abstractions.
Then you get to definitions like ": open ( string -- handle 1 | 0) ... ;" which describes returning algebraic type Maybe Handle unboxed on the stack. Algebraic types are fun, they can easily represent Peano arithmetic and get us into the realm Goedel incompleteness theorem very quickly.
Or you can deduce signature for EXEC EXEC sequence. EXEC's stack effect can be described as ( \alpha (\alpha -- \beta) -- \beta), where \greekletter is a placeholder for a stack part of arbitrary length. Notice that this type comment has nested brackets and does not adhere to Forth stack-effect comment convention.
When I thought about this more than fifteen years ago, I've got at least two equally valid types for the EXEC EXEC: one where xt at top of stack consumes all its input and leaves no output ( \alpha (\alpha -- \gamma) \beta (\beta -- ) -- \gamma) and when first EXEC produces something for second to execute upon ( \alpha \beta (\beta -- \gamma (\alpha \gamma -- \theta) -- \theta).
One can argue that second type of EXEC EXEC subsume first one, if greek-letter-named-stack-parts are allowed to be empty.
Still it shows that typing Forth, at the very least, needs unification on the Peano's arithmetic level, implementing deduction from length zero to unbounded length.
So, in my opinion, for LLM to dependably combine typed Forth/concatenative definitions, it needs to call external tool like Prolog to properly deduce type(s) of the sequence of Forth's (or concatenative language's) definitions.
And here we enter a realm interesting in itself.
Here it is: https://github.com/stassa/louise
This is a Prolog system to learn programs in polynomial time. For one example, it can one-shot-learn a grammar, without being "trained" on millions of samples.
So, should one use a LLM that either needs a paid access or just slow to run, or freely go where "old school" systems like Eurisco [1] and Cyc went?
[1] https://en.wikipedia.org/wiki/Eurisko
Eurisco demonstrated superhuman abilities in 1982-83. It also demonstrated knowledge transfer at the time, where rules from VLSI place-and-route algorithms were used to design winning Traveler TCS fleet.
phreda4 has been doing cool stuff with ColorForth-likes for ages and for some reason barely gets any attention for it. Always brings a smile to my face to see it submitted here
That is a great looking tutorial. Can't wait to try it. Thanks!