These days it seems to be deeply unpopular though: the normal pattern is superficial Loyalty, followed by a quick dash to Exit as soon as something slightly better pops up. Anybody even attempting to Voice and actually change the organization is laughed off as hopelessly naive, especially if they're junior.
In a bit related way I've been trying to push the idea of "engineers need to be grumpy."
Not so much that we need to not be happy and not enjoy our lives, but that our job is to find problems and fix them. In that setting, being "grumpy" is recognizing the problems. If you're dogfooding your product, you should be aware of its strengths and faults. Fixing the faults gives clear direction that allows you to make your products better. You don't have to reach perfection, such a thing does not even exist. Instead we do this iteratively, as the environment is dynamic, just as is the customer's needs.
I would say that this is loyalty to the company and the product, though not loyal to the politics. It is clearly loyal to the product, which we as engineers are in charge of creating. But it is also loyal to the company because the things we build are the very foundation of the company itself. Being loyal to politics may keep you your job, but it is a short term solution that reinforces a culture of politics itself.
To managers: don't dissuade engineers who raise issues or complaints. These are not "no" in the language of engineers, they are "yes, but". Encourage those conversations because that's how we resolve the issues and build the things. The managerial role is to help those conversations not get stuck or too heated. Your job is to help maintain engineers' passions because that's what pushes the product, and consequently the business (and consequently your success), forward. But that passion is fragile. STEM is a creative endeavor through and through. If not given time to "play around" and try new things then that passion dies. When that passion dies your innovation turns lazy and your only goal is to make something bland like a thinner phone for the 7th year in a row.
the "being grumpy" or being able to recognize problems is why during technical interviews I always ask the candidate to name their biggest complaint about a technology they know well/love/mentioned earlier. This to me is a great signal especially for mid seniority engineers that they are aware that things could improve
"Exit and voice themselves represent a union between economic and political action. Exit is associated with Adam Smith's invisible hand, in which buyers and sellers are free to move silently through the market, constantly forming and destroying relationships. Voice, on the other hand, is by nature political and at times confrontational."
In national and international politics we see a lot of "Voice" (in these terms) but I feel they are often ineffective because they are blunted by threshold effects, i.e. the resultant change is a softmax with temperature near zero.
People don't realise and utilise the power they hold by means of "Exit", especially as wealthy consumers who engage in these debates, who have more consumption and hence more influence as it's a "one dollar, one vote" system rather "one person, one vote". See for example how effective the response to Jimmy Kimmel was. The same thing was effective against Apartheid in the 1980s and can be applied through things like the BDS Movement.
Exit applies a much more continuous pressure where the effect of each actor is cumulative whereas Voice is more discontinous and requires a critical mass.
Also, as our Voices are becoming increasingly silenced in online discourse, either through cancelling or self-censorship, Exiting becomes one of our last avenues to effect change.
Thanks for the book rec, seems like it'd be worth reading.
In general it is not worth pursuing changes to an organization, because in some sense the you need the organization more than they need you, and stepping on the toes of too many people with unpopular opinions is just gonna make you look like an idiot, or in the worst case, get you fired. Therefore, people can only rationally ascribe to the the capitalistic view of "vote with your wallet" or "if you don't like your job, just leave."
It's always interesting to read opinions that are diametrically opposed to your own, especially when they come from people you have deep disagreements with (in this case, it's not the poster, but his inspiration, Drew DeVault who I am philosophically not aligned with).
Maybe the philosophical disagreements stem from different life experiences, as I started my career at a Big Tech, and learned the hard way that no, I could NOT change the organization I was a part of. It was the C-suite's way (and they served at the pleasure of the Board and the shareholders), or it was the highway.
So I took the highway, started my own business, and we're small and obscure and it follows the principles I want it to follow and it makes me money, and as far as I'm concerned that's awesome. It's the good version of capitalism and if we're smart enough we might very well beat the old company I walked away from in some manner in a free market one day.
My main point is that a company is going to follow the interests of its board and its shareholders. I suspect that whatever Drew did did NOT change the organization at Linode. I suspect it always followed the interests of its shareholders and still does, and what he really did was improve the efficiency of the asset they owned. Not to throw shade on someone who makes things better within their organization. But either you own it or you don't in which case you merely serve at the pleasure of the ones who do. I think more talented people should go off and own their own thing.
> My main point is that a company is going to follow the interests of its board and its shareholders.
Why do you think these are misaligned?
Fundamentally the company's success depends on the product, right? Sure, the shareholders are what needs to be appeased but they set their price on the excitement of what's to come. We can abuse this by creating artificial hype but do you not think that growth is better sustained with hype over worthwhile new products?
The investors are fickle and do not have the interest of the company in mind. They only have that interest so long as they hold their shares and they couldn't care less when they sell. But a business should not think in the moment, but the trend over time. Those investors will come and go and we want them to. You don't do that by appeasing the small subset of investors that invest now, especially those who are looking to get in and out as fast as possible. If the investors are the "real customer" then how do you sustain them in the long term? Certainly this is highly aligned with sustaining the people actually buying the products.
This seems highly related to Pournelle's Iron Law of Bureaucracy[0]. Those who voice are the first group, devoted to the goals of the organization. Those who care about the quarter and no further are the second. Loyal to the organization. Worse, loyal only while it suits them. And that's no loyalty at all.
Even if this is possible, it is seldom worth it. If you look around and find deep disagreements with your coworkers, the right answer is typically to find another cohort of people to work with, not to fight an uphill battle every day. You will burn out. Just move on.
I guess this is only possible at engineering focused organizations that value technical excellence and also requires that one person be right enough in most cases to expend their social capital to advocate for the engineering changes that they want to see.
As a counterexample, I worked at a company with an extremely bureaucratic release process involving multiple levels of reviews from stakeholders, people manually monitoring a system after a release, and a policy of performing deployments only at nights, all indicators of the lack of confidence in the engineering processes of the organization.
While company management talked a lot about faster releases, “falling behind in the age of AI”, and the like, they also loved their processes and would rather keep it as to them it was a sign of meticulousness and quality. I hated it, but I don’t see how anyone, even the people who carried far more importance than me could have changed it, even though they’d acknowledge that it was slow and could do with more automation in private discussions.
Do you have an example of a large organization that deploys in the middle of the business day that hasn’t had a catastrophic failure? I dont think “deploying after hours” is a sign of lack of confidence in engineering, it’s just basic common sense not to disrupt the people paying your bills because it might be slightly less convenient for a small subsets of your employees.
People always point to Facebook but they literally constantly have issues, it’s just that nobody dies when the like button glitches on grandma’s feed.
I feel like a lot of people have this mistaken impression that they don't need to invest in engineering processes because there's a "downtime" during which they can make a deployment. However, large companies don't have this luxury because their application is being used all the time, so they'd usually do some sort of blue/green, canary or cellular deployment where the alarm/metric thresholds can be utilized towards stopping further traffic propagation and/or a rollback.
I also see that people are just generally unwilling to invest in an integration test suite, which can be run on a staging environment before the deployment, which would also catch lots of these issues. At a smaller scale, you can also run a lightweight integration test with test data on accounts that you control that runs just before you release the new version, similar to a canary, which is something I wanted to pursue there, but by that time I had decided to leave.
Note that "inconvenience" is not a concern for me, all organizations maintaining external applications have the concept of oncall. And any large organization, at scale, will have failures, it's just that Facebook has gotten good at mitigating them.
If you have properly tuned canary releases and sufficiently large scale, it's effectively safe to release at any time, because any failures will be caught by the 1% stage.
A neat story, but it can be a bad trait too. Someone who can’t take no for an answer can be incredibly toxic and forcing a change by incessant badgering until everyone gives in is not the right way to go about this. I’m not saying that was the case in this particular instance, but it happens all too often.
The author admits at the bottom of the article that he actually drove people to quit by pushing his opinions. So, yeah, be careful following this example.
There is not enough nuance here to really understand the friction that this would have created but I've found the more conducive environments that engage and value healthy debate are made up of individuals that have the "strong opinions loosely held" mindset. This, on the other hand, sounds like stubbornness and wearing everyone down but again a lot gets lost in the details so I'll opt to not jump to conclusions.
> if you can find the logic and the will to do it.
This is important. Both logic and will are required. If only one of the 2 exists the impact can be limited if any at all. Broadly speaking, mostly, people have the logic but not the "will" in a sense that latter gets diluted by factors like ego, seniority, org lag etc.
Work in a large organization. Though it's a special industry in terms of being heavily regulated.
One person can push over time to change management opinions on their scope of work and what the policy is.
Junior engineers become seniors and leads over a decade or longer. If you're jumping every 2 years to a different company, yes you won't be changing much.
Why would you work your ass off trying to do this for a likely meager raise and a pat on the back? No thanks, I’ll take my sanity instead. Keep this energy for your own projects. Do NOT use it to make someone else rich.
I know I’ll get some whataboutisms of people who work for places that give good raises for good work. Great for you but you’re in the minority.
> there are plenty of companies with people who will make a good faith effort to do what makes sense ... I always like working for these companies
Everyone likes working for these companies, but this requires a mature work culture. The people with seniority have to be competent and experienced enough to distinguish good ideas and not abuse their position for personal gain.
In my experience these good places tend to be midsized companies. In contrast, just about any team you land in at a big tech company is going to be siloed off with one or more psychopaths at the helm who never "spent enough time in the trenches" to understand what healthy management looks like. The same is true at a startup. Those are workplaces mired in politics precisely because nobody in charge knows what good sense is and they do everything in their power to make sure anyone with a good idea is silenced or bullied out.
There aren’t many situations where expecting everyone to explain everything in every detail is correct, but there are some.
Many of those situations where it is OK are down at the foundational level of the internet itself, which is what linode and Drew DeVault were concerned with back in the day.
An example today I’m wrestling with is TLS interception (valid) vs protecting against TLS man in the middle attacks. It’s tough to get people to see it’s an either or situation, they truly are mutually exclusive.
Unless, we walk together through every painstaking detail to reach the necessary conclusion together.
I'm confused by this take. Explaining things in full detail shouldn't be a chore for a reasonably competent leader.
It's just basic due diligence, and it's worth reviewing details when these topics come up. Maybe the new ideas aren't always fully baked, but they may have a point. Regular discussion is just part of the job.
This is the "Voice" option of the Exit/Voice/Loyalty model:
https://en.wikipedia.org/wiki/Exit,_Voice,_and_Loyalty
These days it seems to be deeply unpopular though: the normal pattern is superficial Loyalty, followed by a quick dash to Exit as soon as something slightly better pops up. Anybody even attempting to Voice and actually change the organization is laughed off as hopelessly naive, especially if they're junior.
In a bit related way I've been trying to push the idea of "engineers need to be grumpy."
Not so much that we need to not be happy and not enjoy our lives, but that our job is to find problems and fix them. In that setting, being "grumpy" is recognizing the problems. If you're dogfooding your product, you should be aware of its strengths and faults. Fixing the faults gives clear direction that allows you to make your products better. You don't have to reach perfection, such a thing does not even exist. Instead we do this iteratively, as the environment is dynamic, just as is the customer's needs.
I would say that this is loyalty to the company and the product, though not loyal to the politics. It is clearly loyal to the product, which we as engineers are in charge of creating. But it is also loyal to the company because the things we build are the very foundation of the company itself. Being loyal to politics may keep you your job, but it is a short term solution that reinforces a culture of politics itself.
To managers: don't dissuade engineers who raise issues or complaints. These are not "no" in the language of engineers, they are "yes, but". Encourage those conversations because that's how we resolve the issues and build the things. The managerial role is to help those conversations not get stuck or too heated. Your job is to help maintain engineers' passions because that's what pushes the product, and consequently the business (and consequently your success), forward. But that passion is fragile. STEM is a creative endeavor through and through. If not given time to "play around" and try new things then that passion dies. When that passion dies your innovation turns lazy and your only goal is to make something bland like a thinner phone for the 7th year in a row.
the "being grumpy" or being able to recognize problems is why during technical interviews I always ask the candidate to name their biggest complaint about a technology they know well/love/mentioned earlier. This to me is a great signal especially for mid seniority engineers that they are aware that things could improve
Very interesting, thank you!
The following stood out to me:
"Exit and voice themselves represent a union between economic and political action. Exit is associated with Adam Smith's invisible hand, in which buyers and sellers are free to move silently through the market, constantly forming and destroying relationships. Voice, on the other hand, is by nature political and at times confrontational."
In national and international politics we see a lot of "Voice" (in these terms) but I feel they are often ineffective because they are blunted by threshold effects, i.e. the resultant change is a softmax with temperature near zero.
People don't realise and utilise the power they hold by means of "Exit", especially as wealthy consumers who engage in these debates, who have more consumption and hence more influence as it's a "one dollar, one vote" system rather "one person, one vote". See for example how effective the response to Jimmy Kimmel was. The same thing was effective against Apartheid in the 1980s and can be applied through things like the BDS Movement.
Exit applies a much more continuous pressure where the effect of each actor is cumulative whereas Voice is more discontinous and requires a critical mass.
Also, as our Voices are becoming increasingly silenced in online discourse, either through cancelling or self-censorship, Exiting becomes one of our last avenues to effect change.
Thanks for the book rec, seems like it'd be worth reading.
In general it is not worth pursuing changes to an organization, because in some sense the you need the organization more than they need you, and stepping on the toes of too many people with unpopular opinions is just gonna make you look like an idiot, or in the worst case, get you fired. Therefore, people can only rationally ascribe to the the capitalistic view of "vote with your wallet" or "if you don't like your job, just leave."
It's always interesting to read opinions that are diametrically opposed to your own, especially when they come from people you have deep disagreements with (in this case, it's not the poster, but his inspiration, Drew DeVault who I am philosophically not aligned with).
Maybe the philosophical disagreements stem from different life experiences, as I started my career at a Big Tech, and learned the hard way that no, I could NOT change the organization I was a part of. It was the C-suite's way (and they served at the pleasure of the Board and the shareholders), or it was the highway.
So I took the highway, started my own business, and we're small and obscure and it follows the principles I want it to follow and it makes me money, and as far as I'm concerned that's awesome. It's the good version of capitalism and if we're smart enough we might very well beat the old company I walked away from in some manner in a free market one day.
My main point is that a company is going to follow the interests of its board and its shareholders. I suspect that whatever Drew did did NOT change the organization at Linode. I suspect it always followed the interests of its shareholders and still does, and what he really did was improve the efficiency of the asset they owned. Not to throw shade on someone who makes things better within their organization. But either you own it or you don't in which case you merely serve at the pleasure of the ones who do. I think more talented people should go off and own their own thing.
Fundamentally the company's success depends on the product, right? Sure, the shareholders are what needs to be appeased but they set their price on the excitement of what's to come. We can abuse this by creating artificial hype but do you not think that growth is better sustained with hype over worthwhile new products?
The investors are fickle and do not have the interest of the company in mind. They only have that interest so long as they hold their shares and they couldn't care less when they sell. But a business should not think in the moment, but the trend over time. Those investors will come and go and we want them to. You don't do that by appeasing the small subset of investors that invest now, especially those who are looking to get in and out as fast as possible. If the investors are the "real customer" then how do you sustain them in the long term? Certainly this is highly aligned with sustaining the people actually buying the products.
This seems highly related to Pournelle's Iron Law of Bureaucracy[0]. Those who voice are the first group, devoted to the goals of the organization. Those who care about the quarter and no further are the second. Loyal to the organization. Worse, loyal only while it suits them. And that's no loyalty at all.
[0] https://jerrypournelle.com/reports/jerryp/iron.html
Even if this is possible, it is seldom worth it. If you look around and find deep disagreements with your coworkers, the right answer is typically to find another cohort of people to work with, not to fight an uphill battle every day. You will burn out. Just move on.
If you can’t change your company, change your company.
I guess this is only possible at engineering focused organizations that value technical excellence and also requires that one person be right enough in most cases to expend their social capital to advocate for the engineering changes that they want to see.
As a counterexample, I worked at a company with an extremely bureaucratic release process involving multiple levels of reviews from stakeholders, people manually monitoring a system after a release, and a policy of performing deployments only at nights, all indicators of the lack of confidence in the engineering processes of the organization.
While company management talked a lot about faster releases, “falling behind in the age of AI”, and the like, they also loved their processes and would rather keep it as to them it was a sign of meticulousness and quality. I hated it, but I don’t see how anyone, even the people who carried far more importance than me could have changed it, even though they’d acknowledge that it was slow and could do with more automation in private discussions.
Do you have an example of a large organization that deploys in the middle of the business day that hasn’t had a catastrophic failure? I dont think “deploying after hours” is a sign of lack of confidence in engineering, it’s just basic common sense not to disrupt the people paying your bills because it might be slightly less convenient for a small subsets of your employees.
People always point to Facebook but they literally constantly have issues, it’s just that nobody dies when the like button glitches on grandma’s feed.
I feel like a lot of people have this mistaken impression that they don't need to invest in engineering processes because there's a "downtime" during which they can make a deployment. However, large companies don't have this luxury because their application is being used all the time, so they'd usually do some sort of blue/green, canary or cellular deployment where the alarm/metric thresholds can be utilized towards stopping further traffic propagation and/or a rollback.
I also see that people are just generally unwilling to invest in an integration test suite, which can be run on a staging environment before the deployment, which would also catch lots of these issues. At a smaller scale, you can also run a lightweight integration test with test data on accounts that you control that runs just before you release the new version, similar to a canary, which is something I wanted to pursue there, but by that time I had decided to leave.
Note that "inconvenience" is not a concern for me, all organizations maintaining external applications have the concept of oncall. And any large organization, at scale, will have failures, it's just that Facebook has gotten good at mitigating them.
Most sufficiently large companies don’t have the luxury of „after hours“ because they have customers in every time zone.
If you have properly tuned canary releases and sufficiently large scale, it's effectively safe to release at any time, because any failures will be caught by the 1% stage.
A neat story, but it can be a bad trait too. Someone who can’t take no for an answer can be incredibly toxic and forcing a change by incessant badgering until everyone gives in is not the right way to go about this. I’m not saying that was the case in this particular instance, but it happens all too often.
The author admits at the bottom of the article that he actually drove people to quit by pushing his opinions. So, yeah, be careful following this example.
There is not enough nuance here to really understand the friction that this would have created but I've found the more conducive environments that engage and value healthy debate are made up of individuals that have the "strong opinions loosely held" mindset. This, on the other hand, sounds like stubbornness and wearing everyone down but again a lot gets lost in the details so I'll opt to not jump to conclusions.
> if you can find the logic and the will to do it.
This is important. Both logic and will are required. If only one of the 2 exists the impact can be limited if any at all. Broadly speaking, mostly, people have the logic but not the "will" in a sense that latter gets diluted by factors like ego, seniority, org lag etc.
Work in a large organization. Though it's a special industry in terms of being heavily regulated. One person can push over time to change management opinions on their scope of work and what the policy is. Junior engineers become seniors and leads over a decade or longer. If you're jumping every 2 years to a different company, yes you won't be changing much.
it's actually easier to start over than change the culture of a large organization.
you can't even change the culture in a restaurant without replacing every single person.
It comes with being named DeVault of all things...
sorry but
1. who is Drew DeVault?
2. what change did he bring to Linode?
I lost the context after 'the same title "Developer"'
Yeah, no.
If it's not part of my job, and neither is my own company, it's not my problem.
Why would you work your ass off trying to do this for a likely meager raise and a pat on the back? No thanks, I’ll take my sanity instead. Keep this energy for your own projects. Do NOT use it to make someone else rich.
I know I’ll get some whataboutisms of people who work for places that give good raises for good work. Great for you but you’re in the minority.
> there are plenty of companies with people who will make a good faith effort to do what makes sense ... I always like working for these companies
Everyone likes working for these companies, but this requires a mature work culture. The people with seniority have to be competent and experienced enough to distinguish good ideas and not abuse their position for personal gain.
In my experience these good places tend to be midsized companies. In contrast, just about any team you land in at a big tech company is going to be siloed off with one or more psychopaths at the helm who never "spent enough time in the trenches" to understand what healthy management looks like. The same is true at a startup. Those are workplaces mired in politics precisely because nobody in charge knows what good sense is and they do everything in their power to make sure anyone with a good idea is silenced or bullied out.
A typical cowboy - start breaking things and quit with a nice achievement in the CV before it all goes down
One of those that thinks everyone owes them explaining everything to them in every detail or its not "right".
There aren’t many situations where expecting everyone to explain everything in every detail is correct, but there are some.
Many of those situations where it is OK are down at the foundational level of the internet itself, which is what linode and Drew DeVault were concerned with back in the day.
An example today I’m wrestling with is TLS interception (valid) vs protecting against TLS man in the middle attacks. It’s tough to get people to see it’s an either or situation, they truly are mutually exclusive.
Unless, we walk together through every painstaking detail to reach the necessary conclusion together.
The TLS issue mentioned can be more easily conceptualized if you view the root CA lists as "The people you're OK with MITM-ing you".
And then whether your trust in the browser vendor coalition to push back against and punish even accidental CA malfeasance are reasonable.
I feel like I've been in the same position as you. I ended up not being able to convince those who mattered so I left. Hope you have better luck!
I'm confused by this take. Explaining things in full detail shouldn't be a chore for a reasonably competent leader.
It's just basic due diligence, and it's worth reviewing details when these topics come up. Maybe the new ideas aren't always fully baked, but they may have a point. Regular discussion is just part of the job.