Very interesting article. As a developer that has focused on writing more code than the infrastructure and tooling, I can definitely see the benefit to a product existing that allows everything to work together more seamlessly because on the surface everything we need to do it already exists. Something like glue code we could drop in or like you mentioned the whole integrated environment. I didn't realize how specific and difficult it was to do something like this, maybe it's not that we should have deployable option, but that technology could get to a point to where maybe implementing and deploying this could be worth it as you could get more value than effort it takes to implement on a per company basis so companies besides Google and other big tech companies can operate like this.
This article made me laugh and cry at the same time. I've spent so much time in my career trying to make things nice and seamless only to see my own team members throw out my careful work for something newer, shinier, and shittier. I was on a team that used Bazel. Like two devs took the time to figure out how it worked, everybody else just either worked around it or complained endlessly about it.
I am now thoroughly convinced that software engineers, if there is currently no snake whispering in their ear to throw away the paradise garden they're been handed on high, will find a way to do it anyway. Coders will, to the last, prefer self-inflicted misery over the heaven they've been gifted for free.
And if you don't believe me, let me tell you we already have a fully-vertically integrated tool stack, a whole family of them. It's called Smalltalk, it's been around since the 70s, and modern variants of course exist. You can build stuff in it today, and thoroughly enjoy your computing life as a result.
The second you turn your head though, your fellow teammates will conspire to replatform onto Go or Rust or NodeJS or GitHub Actions and make everything miserable again.
Don't buy the nonsense that vertical integration is hard. It's not. You just hire really sharp folks, get them excited about the idea, and they do all the hard integration work, then you release it to the community and let them build on it. Rails was like this back in the 2010s, there was this golden age where everything just worked. Then all of a sudden javascript took over the web world like cancer and the web stopped being fun.
It's not that it's hard, what it is is brittle. A vertically-integrated stack, by its very nature, cannot survive forces that jostle it in the horizontal direction. And coders are too afraid of falling behind that they end up fetishizing any new idea that comes along, no matter how daft. Javascript on the server?!?! Your solution to js's, let's call them problems, is gradual typing? That snake's never gonna run out of ears to whisper in, lemme tell ya.
Integrated toolsets can, have, and are still being done. You can use them now. But you don't want it. Even if you do, nobody you work with will trust it or keep it after you leave. And so companies have no motive anymore to sell them to you. Microsoft themselves stopped trying after 2019.
The Rails experience of 15 years ago is still achievable, with Rails, today. It's even better if you can believe it. You just have to throw out the notion of SPA frameworks and be willing to actually learn how HTML / CSS works. I read an article here a month ago about how nobody's willing to do that.
The deficiency in web standards that SPA frameworks were invented to resolve have all been fixed, there's nothing they offer anymore that you can't do without them. But they hang over the neck of the web world like an albatross.
Meh, I think its just hubris to ignore your users (in this case the rest of the team) and tell yourself they're just dumb for not using your clearly perfect system.
There is a slight amount of NIH you have to fight, but for the most part, if a process is truly seamless, that NIH will get pushed to a different part of the stack that's more front of mind.
The real issue is that you're trivializing needs that live outside your purview and because of that you can't fathom why a js dev who wants a server to do something would want js on the server.
This kind of rigid, singular view of operational workflows based on precomposed automations not only constantly break but also inevitably introduce huge inefficiences.
yes! couldn't agree more with your long post.
Especially this part:
"(This is exacerbated when components of the automation require internal-only tooling—the poor data scientist now needs to go read through a bunch of half-written, out-of-date documentation about tools they simply don't care about to do a task that is not a core responsibility for them.)"
"Vertical integration" in my experience has just been turning a group of simple tools into a complex monolith that no one understands and is extremely difficult to debug
Very interesting article. As a developer that has focused on writing more code than the infrastructure and tooling, I can definitely see the benefit to a product existing that allows everything to work together more seamlessly because on the surface everything we need to do it already exists. Something like glue code we could drop in or like you mentioned the whole integrated environment. I didn't realize how specific and difficult it was to do something like this, maybe it's not that we should have deployable option, but that technology could get to a point to where maybe implementing and deploying this could be worth it as you could get more value than effort it takes to implement on a per company basis so companies besides Google and other big tech companies can operate like this.
You can at least get (3) for low/no cost with pre-commit hooks running locally and in CI.
The problem with hooks is they’re not enforced (in git) so you need to run them in CI.
This article made me laugh and cry at the same time. I've spent so much time in my career trying to make things nice and seamless only to see my own team members throw out my careful work for something newer, shinier, and shittier. I was on a team that used Bazel. Like two devs took the time to figure out how it worked, everybody else just either worked around it or complained endlessly about it.
I am now thoroughly convinced that software engineers, if there is currently no snake whispering in their ear to throw away the paradise garden they're been handed on high, will find a way to do it anyway. Coders will, to the last, prefer self-inflicted misery over the heaven they've been gifted for free.
And if you don't believe me, let me tell you we already have a fully-vertically integrated tool stack, a whole family of them. It's called Smalltalk, it's been around since the 70s, and modern variants of course exist. You can build stuff in it today, and thoroughly enjoy your computing life as a result.
The second you turn your head though, your fellow teammates will conspire to replatform onto Go or Rust or NodeJS or GitHub Actions and make everything miserable again.
Don't buy the nonsense that vertical integration is hard. It's not. You just hire really sharp folks, get them excited about the idea, and they do all the hard integration work, then you release it to the community and let them build on it. Rails was like this back in the 2010s, there was this golden age where everything just worked. Then all of a sudden javascript took over the web world like cancer and the web stopped being fun.
It's not that it's hard, what it is is brittle. A vertically-integrated stack, by its very nature, cannot survive forces that jostle it in the horizontal direction. And coders are too afraid of falling behind that they end up fetishizing any new idea that comes along, no matter how daft. Javascript on the server?!?! Your solution to js's, let's call them problems, is gradual typing? That snake's never gonna run out of ears to whisper in, lemme tell ya.
Integrated toolsets can, have, and are still being done. You can use them now. But you don't want it. Even if you do, nobody you work with will trust it or keep it after you leave. And so companies have no motive anymore to sell them to you. Microsoft themselves stopped trying after 2019.
Luckily I work with laravel and that's a spiritual successor to rails.
The development experience is almost always really smooth and there are more and more tools to further smoothen that experience every day.
There are definitely better tools out there but given how the web ecosystem functions, it could be much worse.
The Rails experience of 15 years ago is still achievable, with Rails, today. It's even better if you can believe it. You just have to throw out the notion of SPA frameworks and be willing to actually learn how HTML / CSS works. I read an article here a month ago about how nobody's willing to do that.
The deficiency in web standards that SPA frameworks were invented to resolve have all been fixed, there's nothing they offer anymore that you can't do without them. But they hang over the neck of the web world like an albatross.
Meh, I think its just hubris to ignore your users (in this case the rest of the team) and tell yourself they're just dumb for not using your clearly perfect system.
There is a slight amount of NIH you have to fight, but for the most part, if a process is truly seamless, that NIH will get pushed to a different part of the stack that's more front of mind.
The real issue is that you're trivializing needs that live outside your purview and because of that you can't fathom why a js dev who wants a server to do something would want js on the server.
I just could not disagree more.
This kind of rigid, singular view of operational workflows based on precomposed automations not only constantly break but also inevitably introduce huge inefficiences.
I posted a longer comment on lobste.rs: https://lobste.rs/s/azpsqe/vertical_integration_is_only_thin...
yes! couldn't agree more with your long post. Especially this part: "(This is exacerbated when components of the automation require internal-only tooling—the poor data scientist now needs to go read through a bunch of half-written, out-of-date documentation about tools they simply don't care about to do a task that is not a core responsibility for them.)"
"Vertical integration" in my experience has just been turning a group of simple tools into a complex monolith that no one understands and is extremely difficult to debug
I like the points made in the article. I hate this overloading of the term.