I see mixed comments,
so let me add some praise.
I am one of countless, who match
his intro-filter:
repeatedly hearing 'enlightened'
people lament that the vast masses don't "get" lisp and FP,
and repeatedly attempting/failing to pick up the red string myself.
background - I am a computer science major with 30+ years experience. I did do a mandatory class of 'implement your own lisp' many eons ago.
It just never really 'clicked' for me.
I do, by accident, assimilation and lazyness,employ
FP style designs in my software.
And I guess fp techniques gradually rub off on me from e.g. javascript, lambdas,closures, and map-filter-reduce.
in particular, lambdas are useful to me.
But I am one of the guys who
continue to read the "let me tell you what monads really are", and every time I fall off the bicycle.
So, well, I appreciated this 'Xfor 5year olds" :-)
It sounded of interest to me, but I read it and closed the tab within a page or so as it wandered off into tech arcana. Shame. There may be an interesting idea in here but it's phrased in terms I think few will be able to follow and understand.
I did not finish it but I saw no mention of the lambda calculus or of currying, both of which -- from my very meagre understanding -- seem directly relevant to what I understood to be the core point, which seems to be about anonymous functions.
I don't think you're missing much. Yeah, the main point seems to be that if your language has closures, you suddenly can express a lot of things that were out of reach before. Not a new insight. But there's another point I think that is hinted at on the topic of control abstractions. Or at least I'm reminded of the topic. It's better and more succinctly and explicitly talked about in an early chapter of the free book Patterns of Software: https://dreamsongs.com/Files/PatternsOfSoftware.pdf
The extra point might be that more languages should facilitate defining your own control abstractions just as they support defining your own data abstractions. Functions are one way of making data abstractions, but languages often provide multiple ways. Closures are one way of doing a type of control abstraction (involving such things as delayed or multiple evaluation), but there are other ways too. For some reason we see value and a need for defining our own data abstractions, but not so much for control abstractions, even though (according to the book) once they were often co-designed, like Fortran's arrays and DO loop. And for some reason even in the few languages that do support making your own control abstractions, like Lisp, you'll still find users who disapprove of doing so, claiming all you need are the standard existing methods like looping, map/reduce style functions, and some non-local exits.
which is allegedly about programming with macros but I'd say 80% of the time he implements something with closures and then makes a macro-based implementation that peforms better. That 80% can be done in Python and the other 20% you wouldn't want to do in Python because Python already has those features... And if you wanted to implement meta-objects in Python you would do it Pythonically.
Graham unfortunately doesn't work any examples that involve complex transformations on the expression trees because these are hard and if you want to work that hard you're better off looking at the Dragon book.
You can work almost all the examples in Norvig's Common Lisp book
Currying is done automatically in Haskell but not in Lisp. If you wanted currying in Lisp you could write it, but Lisp programmers don't depend on or talk about currying as much as Haskell programmers do.
In languages like Java (or C) you can build S-expression like structures like so
Variable<Integer> x = newVariable();
Expression<Integer> = add(x,literal(5));
x.set(15);
System.out.println(eval(x)) // prints "20"
and it is not that hard to either serialize these to code or run them in a tree-walking interpreter where quote() and eval() imply an extended language where you can write functions that work on Expression<Expression<X>>. Type erasure causes some problems in Java that make you sometimes write a type you shouldn't have to and you do have to unerase types in method names which is a little ugly but it works.
I did some experiments towards this to convince myself it would work
had I really kept at it I would have bootstrapped by developing a ferocity0 which was sufficient to write a code generator that could generate stubs for the Java stdlib + a persistent collections library and then write a ferocity1 in ferocity0, and if necessary ferocity(N+1) in ferocityN until it supported "all" of Java, though "all" might have omitted some features like "var" that are both sugar and use type inference that ferocity would struggle with -- if you need sugar in this system you implement it with metaprogramming.
The idea is that certain projects would benefit from balls-to-the-walls metaprogramming and the code compression you get would compensate for the code getting puffed up. My guess is a lot of people would see it as an unholy mating of the worst of Java and Common Lisp. However, I'm certain it would be good for writing code generators.
The solution, which I often seen in practice, is to eventually write code generators, which is what Lisp macros are, after all. I've seen it in C and wrote a big piece about it that was posted here some time ago[1], about the extra tools, code generators, special formats and standards employed and needed to make up for C's deficiencies (in respect to meta-programming, at least).
Everywhere I see code generators it means a feature is lacking in the main language used for the project. Then you bring in other tools to make up for that deficiency. Only, usually, we don't call that deficiency, since we are used to things being that way. It is called day-to-day business. I think that's what I've tried to convey in the article.
I don't see code generation as a bad smell at all.
At my job we use the JooQ code generator which is well integrated with maven and either IntellJ IDEA or Eclipse so autcompletion "just works". In modern Java you can pack up a code generator as a maven plugin [1] and put something in your POM that runs the generator. It's easy. There are other ways to hook the compiler too, see the controversial
Lisp does come closest to a "language construction set" that lets you bend a language to your will. I think a compiler could be built for a language that looks more conventional that would be just as malleable, maybe even more malleable, but a generation of system programmers were traumatized by slow C++ builds and want to have nothing to do with a compiler which could be slow, even if you could make up for the slowness by having dramatically less code.
[1] A maven plugin is just a Java class which can do everything in the ordinary Java way which gets dependencies injected by the maven runtime. It's a common rookie mistake to try to solve problems by writing XML. I mean, if you can write a POM that makes existing plugins do what you want go right ahead, but if you can't just write your own plugin.
> I don't see code generation as a bad smell at all.
Well, exactly!
> At my job we use the JooQ code generator...
And in Lisp one would use the...Lisp code generator. That is, the macro. And the beauty of it is that it doesn't work with pure strings but 'understands', parses, manipulates the code as expressions in its own language. That is, it has at its disposal the entire language for manipulating those expressions.
And I think that is one of the "aha" moments. At least, it was for me.
When you realize the reason for having those code generators, regardless of the project and language, in the first place. That is, something missing from the language. Some extra feature that can't be implemented. Some solution that works really close to 99%, but not beyond. Something that one would like to express, but can't. Some piece of code that you want to be parameterized like you would a function, some piece of code that you want to use in multiple places but you don't want to write the same boilerplate or copy/paste it all over the place with the risk that when you modify something, you'll need to modify in all those places. Or some piece of code that you want to be auto-generated when you build/deploy/etc.
The examples are countless. The world of meta-programming offers enough of them. The article gives the control statements as an example, as a hint to build the appetite, as is suggested in the intro, in fact.
I have done something exactly like this in production for a system that turned natural language into SQL. This was pre-LLM, so we had models that produced intent and keywords as structured output and we had to turn it into queries for several backends.
The project didn't work out for a variety of reasons, but technically it was beautiful: it produced query plans that in many cases were identical to those from the queries analysts wrote by hand.
So yeah, I accidentally wrote a compiler. Does it still count?
JooQ isn't everybody's taste but I use it for my job and I think it's great particularly in that you can reuse expressions and write generators for complex queries. We have a powerful search interface that combines full-text with other kinds of queries ("Is about topic T", "Data was collected between S and E") that is beautiful. I think it's funny how JooQ has that lispy f(a,b) style (no accident it is like ferocity) and how Sqlalchemy is really fluent and takes advantage of operator overloading.
I find it is a real challenge to come up with a good title. On the one hand it should, probably, convey to the potential reader something about the contents of the article, on the other hand it should be something to differentiate it from the rest of the articles published on the same subject.
I like those that read something like a punch-line, that come across as something different that just a summary of the article. But these maybe work best for literature, prose, movies, etc.
I see mixed comments, so let me add some praise. I am one of countless, who match his intro-filter: repeatedly hearing 'enlightened' people lament that the vast masses don't "get" lisp and FP, and repeatedly attempting/failing to pick up the red string myself.
background - I am a computer science major with 30+ years experience. I did do a mandatory class of 'implement your own lisp' many eons ago. It just never really 'clicked' for me. I do, by accident, assimilation and lazyness,employ FP style designs in my software. And I guess fp techniques gradually rub off on me from e.g. javascript, lambdas,closures, and map-filter-reduce. in particular, lambdas are useful to me. But I am one of the guys who continue to read the "let me tell you what monads really are", and every time I fall off the bicycle. So, well, I appreciated this 'Xfor 5year olds" :-)
Previously:
https://news.ycombinator.com/item?id=28851992
https://news.ycombinator.com/item?id=44359454
No comments on any of them.
It sounded of interest to me, but I read it and closed the tab within a page or so as it wandered off into tech arcana. Shame. There may be an interesting idea in here but it's phrased in terms I think few will be able to follow and understand.
I did not finish it but I saw no mention of the lambda calculus or of currying, both of which -- from my very meagre understanding -- seem directly relevant to what I understood to be the core point, which seems to be about anonymous functions.
I don't think you're missing much. Yeah, the main point seems to be that if your language has closures, you suddenly can express a lot of things that were out of reach before. Not a new insight. But there's another point I think that is hinted at on the topic of control abstractions. Or at least I'm reminded of the topic. It's better and more succinctly and explicitly talked about in an early chapter of the free book Patterns of Software: https://dreamsongs.com/Files/PatternsOfSoftware.pdf
The extra point might be that more languages should facilitate defining your own control abstractions just as they support defining your own data abstractions. Functions are one way of making data abstractions, but languages often provide multiple ways. Closures are one way of doing a type of control abstraction (involving such things as delayed or multiple evaluation), but there are other ways too. For some reason we see value and a need for defining our own data abstractions, but not so much for control abstractions, even though (according to the book) once they were often co-designed, like Fortran's arrays and DO loop. And for some reason even in the few languages that do support making your own control abstractions, like Lisp, you'll still find users who disapprove of doing so, claiming all you need are the standard existing methods like looping, map/reduce style functions, and some non-local exits.
There's closures and there's being able to transform the expression tree.
Graham's On Lisp is a really interesting book
https://paulgraham.com/onlisptext.html
which is allegedly about programming with macros but I'd say 80% of the time he implements something with closures and then makes a macro-based implementation that peforms better. That 80% can be done in Python and the other 20% you wouldn't want to do in Python because Python already has those features... And if you wanted to implement meta-objects in Python you would do it Pythonically.
Graham unfortunately doesn't work any examples that involve complex transformations on the expression trees because these are hard and if you want to work that hard you're better off looking at the Dragon book.
You can work almost all the examples in Norvig's Common Lisp book
https://www.amazon.com/Paradigms-Artificial-Intelligence-Pro...
in Python and today Norvig would advocate that you do.
Currying is done automatically in Haskell but not in Lisp. If you wanted currying in Lisp you could write it, but Lisp programmers don't depend on or talk about currying as much as Haskell programmers do.
The core point, to me, seemed to be about limiting factors in language extension. To allow something like:
Where the various parameters are lazily evaluated. Or like: Where frazzle, frozzle, and frizzle are position-independent keyword variables.Allowing those in C would require a modicum of effort, while other languages make these kinds of syntax extension fairly easy.
In languages like Java (or C) you can build S-expression like structures like so
and it is not that hard to either serialize these to code or run them in a tree-walking interpreter where quote() and eval() imply an extended language where you can write functions that work on Expression<Expression<X>>. Type erasure causes some problems in Java that make you sometimes write a type you shouldn't have to and you do have to unerase types in method names which is a little ugly but it works.I did some experiments towards this to convince myself it would work
https://github.com/paulhoule/ferocity/blob/main/ferocity0/sr...
had I really kept at it I would have bootstrapped by developing a ferocity0 which was sufficient to write a code generator that could generate stubs for the Java stdlib + a persistent collections library and then write a ferocity1 in ferocity0, and if necessary ferocity(N+1) in ferocityN until it supported "all" of Java, though "all" might have omitted some features like "var" that are both sugar and use type inference that ferocity would struggle with -- if you need sugar in this system you implement it with metaprogramming.
The idea is that certain projects would benefit from balls-to-the-walls metaprogramming and the code compression you get would compensate for the code getting puffed up. My guess is a lot of people would see it as an unholy mating of the worst of Java and Common Lisp. However, I'm certain it would be good for writing code generators.
The solution, which I often seen in practice, is to eventually write code generators, which is what Lisp macros are, after all. I've seen it in C and wrote a big piece about it that was posted here some time ago[1], about the extra tools, code generators, special formats and standards employed and needed to make up for C's deficiencies (in respect to meta-programming, at least).
Everywhere I see code generators it means a feature is lacking in the main language used for the project. Then you bring in other tools to make up for that deficiency. Only, usually, we don't call that deficiency, since we are used to things being that way. It is called day-to-day business. I think that's what I've tried to convey in the article.
[1] https://news.ycombinator.com/item?id=41066544
I don't see code generation as a bad smell at all.
At my job we use the JooQ code generator which is well integrated with maven and either IntellJ IDEA or Eclipse so autcompletion "just works". In modern Java you can pack up a code generator as a maven plugin [1] and put something in your POM that runs the generator. It's easy. There are other ways to hook the compiler too, see the controversial
https://projectlombok.org/
Lisp does come closest to a "language construction set" that lets you bend a language to your will. I think a compiler could be built for a language that looks more conventional that would be just as malleable, maybe even more malleable, but a generation of system programmers were traumatized by slow C++ builds and want to have nothing to do with a compiler which could be slow, even if you could make up for the slowness by having dramatically less code.
[1] A maven plugin is just a Java class which can do everything in the ordinary Java way which gets dependencies injected by the maven runtime. It's a common rookie mistake to try to solve problems by writing XML. I mean, if you can write a POM that makes existing plugins do what you want go right ahead, but if you can't just write your own plugin.
> I don't see code generation as a bad smell at all.
Well, exactly!
> At my job we use the JooQ code generator...
And in Lisp one would use the...Lisp code generator. That is, the macro. And the beauty of it is that it doesn't work with pure strings but 'understands', parses, manipulates the code as expressions in its own language. That is, it has at its disposal the entire language for manipulating those expressions.
And I think that is one of the "aha" moments. At least, it was for me. When you realize the reason for having those code generators, regardless of the project and language, in the first place. That is, something missing from the language. Some extra feature that can't be implemented. Some solution that works really close to 99%, but not beyond. Something that one would like to express, but can't. Some piece of code that you want to be parameterized like you would a function, some piece of code that you want to use in multiple places but you don't want to write the same boilerplate or copy/paste it all over the place with the risk that when you modify something, you'll need to modify in all those places. Or some piece of code that you want to be auto-generated when you build/deploy/etc.
The examples are countless. The world of meta-programming offers enough of them. The article gives the control statements as an example, as a hint to build the appetite, as is suggested in the intro, in fact.
I have done something exactly like this in production for a system that turned natural language into SQL. This was pre-LLM, so we had models that produced intent and keywords as structured output and we had to turn it into queries for several backends. The project didn't work out for a variety of reasons, but technically it was beautiful: it produced query plans that in many cases were identical to those from the queries analysts wrote by hand. So yeah, I accidentally wrote a compiler. Does it still count?
Two good examples of "builders to SQL" are
https://www.jooq.org/
and
https://www.sqlalchemy.org/
JooQ isn't everybody's taste but I use it for my job and I think it's great particularly in that you can reuse expressions and write generators for complex queries. We have a powerful search interface that combines full-text with other kinds of queries ("Is about topic T", "Data was collected between S and E") that is beautiful. I think it's funny how JooQ has that lispy f(a,b) style (no accident it is like ferocity) and how Sqlalchemy is really fluent and takes advantage of operator overloading.
At least "the dead C" was a nice pun :D
I find it is a real challenge to come up with a good title. On the one hand it should, probably, convey to the potential reader something about the contents of the article, on the other hand it should be something to differentiate it from the rest of the articles published on the same subject.
I like those that read something like a punch-line, that come across as something different that just a summary of the article. But these maybe work best for literature, prose, movies, etc.