Pipes was ahead of its time. Built a scraper and aggregator back in 2008-2010, fetching content from over 2000 sources, pumping all of it into a Drupal website. The amount of regex transformations and other things I had in that pipeline - is in hindsight - more complex, larger scale, and more convoluted that any other nifi or airflow pipelines I built later in my career. And all of it was free. What was possible was limited only by one's creativity!
It is surely only a matter of time until someone says 'yes, but node based editors are crap, you should do everything in code'. This has been discussed ad nauseum in HN, so I have tried to summarise the arguments here:
Having spent a log of time with visual programming and having also loved yahoo pipes, for me, it's time to find a new paradigm.
Last year, I came up with Breadboard Programming[1] which is visual programming using Node-RED but with a different perspective, that of a breadboard[2]. The point is take another perspective on programming and to find new parallels to how electronics are created using breadboards.
Breadboarding is prototyping but with the intention of throwing everything away or iterating by simply copying and pasting the entire code base and then throwing it away. The difference to typical software prototyping is that the prototype can easily be copied and iterated on. Normally prototypes mutate into the final product and then remain those mutated monsters that we were going to refactor before they went into production.
Seems to me that breadboard coding would be a like a spreadsheet where dependencies between cells are visible at a glance and could be "rewired" in a way that's easier than hand editing the formula text
I'm not sure if I've ever seen a rigorous comparison between visual and text based editing before, and I'm really interested.
I disagree with some of your points:
- Higher level abstractions - I don't think that's a usage of high level abstraction I'm familiar with, or else it's not true. You can have very very high level abstractions in text code, you can have very low level abstractions in visual programming (max msp). This also goes for the point about optimization in text languages.
- Less hidden state - this really depends on the visual programming language. The geometry node editor in Blender has tons of hidden state - what are the inputs to your program, what are the outputs, how is it applied, etc.
- Less configuration - I have no idea, but my guess is you're thinking of something like Rust where library A uses image type X and library B uses image type Y and you need to write code to convert between X and Y yourself. I don't think this is inherent to visual programming - it's just a consequence of having a large, complex ecosystem with independently evolving parts.
- No syntax to remember - There's still syntax, it's just visual syntax. What does this squiggle mean? What does this color mean? What about this box around things? And unlike text, where you can just write what you see, even if you know what something is with visual programming you might not know how to replicate it (which menu button, what conditions are required, what selection do I have to make, etc), google it, etc.
- Looping and recursion - This is a failing point in visual programming languages I'm familiar with, but I don't think there's anything stopping a visual programming language from offering functions, looping, and recursion.
- Optimization - I touched on this above, but higher level abstraction makes _more_ opportunity for compilers to optimize. Specifically, in visual programming languages, there's no inherent concept of a "sequence" of calculations as there are in imperative languages, so compilers can freely parallelize.
Then: errors and run-time feedback - yes, this is true of most visual programming languages and is super great. It's equally true for many non-visual programming languages (e.g. rust).
The point about discoverability is good, and I agree. Just knowing what you can do in a programming language is very hard (the first steps). Generally in visual programming languages all the parameters are shown right there on the screen.
Here are my extra advantages to visual programming languages:
- I think it's good to make a distinction between control flow graphs and data flow graphs. IMO representing state machines in text based languages is pretty bad and understanding complex call flows etc can be very difficult. Most visual programming languages are purely data flow, but I think a control flow graph programming language would be very good at this.
- Another advantage of visual programming languages is you don't need to come up with names in order to reuse values. Having to name things isn't a joke, and seeing lines and arrows is easier to grok then trying to 1. understand what a complex name means (and maybe forcing yourself to ignore it if the name is bad) and 2. remembering that name so you can mentally connect uses.
That said, there's no need to force these to be mutually exclusive. You could mix them (allow text within the graph, or refer to the graph from text, allow making some files graphs or some text, etc), or even freely convert between them (luna lang, which appears to be thoroughly dead atm but here's some remnants: https://github.com/Luna-Tensorflow/luna).
>I don't think that's a usage of high level abstraction I'm familiar with
The 'primitives' you are presented with are generally at a lot higher level of abstraction in a visual language ('join' node vs '+'). Obviously you can build any level of abstraction using these primitives.
>The geometry node editor in Blender has tons of hidden state - what are the inputs to your program, what are the outputs, how is it applied, etc
I'm not overly familar with Blender. But aren't the inputs and outputs shown as connections? if so, they aren't hidden in the same way they are in code.
>There's still syntax, it's just visual syntax.
You don't have to remember the name of the function, you usually just have to drag it from a list of functions and there is no list of arguments to remember either.
>I don't think there's anything stopping a visual programming language from offering functions, looping, and recursion.
Both are possible in principal.
>Most visual programming languages are purely data flow, but I think a control flow graph programming language would be very good at this.
Trying to mix the two in one environment sounds like a recipe for a lot of additional complexity.
>Another advantage of visual programming languages is you don't need to come up with names in order to reuse values
Good point.
>That said, there's no need to force these to be mutually exclusive.
Indeed. Most visual programming environments include scripting nodes, to get the best of both worlds.
I expect I can drag and drop nodes to solve a problem faster than you can vibe-code a solution. Plus a node-based solution is likely to be more maintainable if your aren't a coder.
Let's see! I'm actually working on a node-based programming LLM paradigm for an app I'm building.
The idea is that you can write the 'nodes' in plain english rather than pre-written blocks, and then the arrows indicate the flow but don't need to absolutely encode everything (closer to a flow chart). The process of writing the flow chart helps define and document the business logic, and the flows are totally clear because everything is encoded into a state machine.
Thanks! It's in a very narrow domain (Handset UI flows in Warehouse Management Systems) so i'm hoping that helps, but it's very domain specific and certainly not anything mainstream.
In a world where node based editors and code are also equivalent to LLMs, it's not super clear to me that the future will not be nicely visualized and understandable nodes (generated by the LLM to explain things to me and to get guidence) kept in sync with the codebase.
I was an avid user of Pipes and blogged a bit about the experience of using it to build an aggregated set of feeds from various employee blogs to feed into our company site back in 2009 [0]. It holds a special place in my memory alongside early internet greats like del.icio.us [1]
I do a fair amount with n8n now, and this feels like n8n way way way before. I generally hate the term "ahead of its time", but I think this might apply.
Pipes was ahead of its time. Built a scraper and aggregator back in 2008-2010, fetching content from over 2000 sources, pumping all of it into a Drupal website. The amount of regex transformations and other things I had in that pipeline - is in hindsight - more complex, larger scale, and more convoluted that any other nifi or airflow pipelines I built later in my career. And all of it was free. What was possible was limited only by one's creativity!
> And all of it was free.
It's important to take the perspective that the Silicon Valley isn't an incubator of technology, but of business models.
Yahoo Pipes being free is exactly what killed them. Without a sustainable business model, they could not last.
Absolutely loved Yahoo Pipes. Created my own last year - https://www.mashups.io
Very nice! Love the clean UI.
cool! self-hostable?
It is surely only a matter of time until someone says 'yes, but node based editors are crap, you should do everything in code'. This has been discussed ad nauseum in HN, so I have tried to summarise the arguments here:
https://successfulsoftware.net/2024/01/16/visual-vs-text-bas...
Having spent a log of time with visual programming and having also loved yahoo pipes, for me, it's time to find a new paradigm.
Last year, I came up with Breadboard Programming[1] which is visual programming using Node-RED but with a different perspective, that of a breadboard[2]. The point is take another perspective on programming and to find new parallels to how electronics are created using breadboards.
Breadboarding is prototyping but with the intention of throwing everything away or iterating by simply copying and pasting the entire code base and then throwing it away. The difference to typical software prototyping is that the prototype can easily be copied and iterated on. Normally prototypes mutate into the final product and then remain those mutated monsters that we were going to refactor before they went into production.
[1]: https://blog.openmindmap.org/blog/breadboard-programming
[2]: https://en.wikipedia.org/wiki/Breadboard
Seems to me that breadboard coding would be a like a spreadsheet where dependencies between cells are visible at a glance and could be "rewired" in a way that's easier than hand editing the formula text
Definitely not :) i would definitely avoid drawing parallels to spreadsheets rather parallels to the electronics.
I explicitly draw parallels to wires because of the relation to flow based programming.
I'm not sure if I've ever seen a rigorous comparison between visual and text based editing before, and I'm really interested.
I disagree with some of your points:
- Higher level abstractions - I don't think that's a usage of high level abstraction I'm familiar with, or else it's not true. You can have very very high level abstractions in text code, you can have very low level abstractions in visual programming (max msp). This also goes for the point about optimization in text languages.
- Less hidden state - this really depends on the visual programming language. The geometry node editor in Blender has tons of hidden state - what are the inputs to your program, what are the outputs, how is it applied, etc.
- Less configuration - I have no idea, but my guess is you're thinking of something like Rust where library A uses image type X and library B uses image type Y and you need to write code to convert between X and Y yourself. I don't think this is inherent to visual programming - it's just a consequence of having a large, complex ecosystem with independently evolving parts.
- No syntax to remember - There's still syntax, it's just visual syntax. What does this squiggle mean? What does this color mean? What about this box around things? And unlike text, where you can just write what you see, even if you know what something is with visual programming you might not know how to replicate it (which menu button, what conditions are required, what selection do I have to make, etc), google it, etc.
- Looping and recursion - This is a failing point in visual programming languages I'm familiar with, but I don't think there's anything stopping a visual programming language from offering functions, looping, and recursion.
- Optimization - I touched on this above, but higher level abstraction makes _more_ opportunity for compilers to optimize. Specifically, in visual programming languages, there's no inherent concept of a "sequence" of calculations as there are in imperative languages, so compilers can freely parallelize.
Then: errors and run-time feedback - yes, this is true of most visual programming languages and is super great. It's equally true for many non-visual programming languages (e.g. rust).
The point about discoverability is good, and I agree. Just knowing what you can do in a programming language is very hard (the first steps). Generally in visual programming languages all the parameters are shown right there on the screen.
Here are my extra advantages to visual programming languages:
- I think it's good to make a distinction between control flow graphs and data flow graphs. IMO representing state machines in text based languages is pretty bad and understanding complex call flows etc can be very difficult. Most visual programming languages are purely data flow, but I think a control flow graph programming language would be very good at this.
- Another advantage of visual programming languages is you don't need to come up with names in order to reuse values. Having to name things isn't a joke, and seeing lines and arrows is easier to grok then trying to 1. understand what a complex name means (and maybe forcing yourself to ignore it if the name is bad) and 2. remembering that name so you can mentally connect uses.
That said, there's no need to force these to be mutually exclusive. You could mix them (allow text within the graph, or refer to the graph from text, allow making some files graphs or some text, etc), or even freely convert between them (luna lang, which appears to be thoroughly dead atm but here's some remnants: https://github.com/Luna-Tensorflow/luna).
>I don't think that's a usage of high level abstraction I'm familiar with
The 'primitives' you are presented with are generally at a lot higher level of abstraction in a visual language ('join' node vs '+'). Obviously you can build any level of abstraction using these primitives.
>The geometry node editor in Blender has tons of hidden state - what are the inputs to your program, what are the outputs, how is it applied, etc
I'm not overly familar with Blender. But aren't the inputs and outputs shown as connections? if so, they aren't hidden in the same way they are in code.
>There's still syntax, it's just visual syntax.
You don't have to remember the name of the function, you usually just have to drag it from a list of functions and there is no list of arguments to remember either.
>I don't think there's anything stopping a visual programming language from offering functions, looping, and recursion.
Both are possible in principal.
>Most visual programming languages are purely data flow, but I think a control flow graph programming language would be very good at this.
Trying to mix the two in one environment sounds like a recipe for a lot of additional complexity.
>Another advantage of visual programming languages is you don't need to come up with names in order to reuse values
Good point.
>That said, there's no need to force these to be mutually exclusive.
Indeed. Most visual programming environments include scripting nodes, to get the best of both worlds.
I actually like node based editors but LLMs are the nail in the coffin.
Visual programming just doesn't make sense in a world where "low-code" users can safely be assumed to be using llms
I expect I can drag and drop nodes to solve a problem faster than you can vibe-code a solution. Plus a node-based solution is likely to be more maintainable if your aren't a coder.
Let's see! I'm actually working on a node-based programming LLM paradigm for an app I'm building.
The idea is that you can write the 'nodes' in plain english rather than pre-written blocks, and then the arrows indicate the flow but don't need to absolutely encode everything (closer to a flow chart). The process of writing the flow chart helps define and document the business logic, and the flows are totally clear because everything is encoded into a state machine.
I am unconvinced that natural language is the best way to describe a data transformation problem.
My personal sense is business logic which involves processes with lots of steps rather than data transformation.
I’m unconvinced that code is the best way to describe business logic, so here we are! :)
(Code is probably the best way to encode business logic, but most users can’t read it or edit it, which makes it bad)
Good luck with your project. It is a tough problem you're solving, but hopefully your idea has wings.
Thanks! It's in a very narrow domain (Handset UI flows in Warehouse Management Systems) so i'm hoping that helps, but it's very domain specific and certainly not anything mainstream.
In a world where node based editors and code are also equivalent to LLMs, it's not super clear to me that the future will not be nicely visualized and understandable nodes (generated by the LLM to explain things to me and to get guidence) kept in sync with the codebase.
I was an avid user of Pipes and blogged a bit about the experience of using it to build an aggregated set of feeds from various employee blogs to feed into our company site back in 2009 [0]. It holds a special place in my memory alongside early internet greats like del.icio.us [1]
- [0] https://blog.davemo.com/posts/2009-04-06-yahoo-pipes-at-vend...
- [1] https://en.wikipedia.org/wiki/Delicious_(website)
Man I miss delicious
I'm surprised I hadn't heard of Pipes.
I do a fair amount with n8n now, and this feels like n8n way way way before. I generally hate the term "ahead of its time", but I think this might apply.
related:
December 2023 - 129 comments - https://news.ycombinator.com/item?id=38650878
Reminds me of an early form of IFTTT
I miss Yahoo Pipes every day of my life.
What about Node-RED?
Isn‘t n8n a substitute of sorts?
There was some other tool also, I guess called Dapper.
A similar experience is Apache Nifi for the curious.
Previously: https://news.ycombinator.com/item?id=38650878
That's a wildly self-indulgent and navel gazing post with more name dropping than you would think for a barely used product.