I use this example when I speak about and teach devops trainings.
I call it the migration sandwich. (Nothing to do with the cube rule).
A piece of bread isn't a sandwich and a single migration in a tool like alembic isn't a "sandwich" either. You have a couple layers of bread with one or several layers of toppings and it's not a sandwich until it's all done.
People get a laugh out of the "idiot sandwich meme" and we always have a good conversation about what gnarly migrations people have seen or done (72+ hours of runtime, splitting to dozens or more tables and then reconstructing things, splitting things out to safely be worked on in the expanded state for weeks, etc).
I had never heard it called "expand and contract" before reading this article a few years ago.
To me is just a blue-green type deployment for schemas. You have an old and a new thing, you split and merge as traffic replays to the new thing and shows that its viable and not breaking, you swap over as you can.
Is there any easy way to implement this pattern in AWS RDS deployments where we need to deploy multiple times a day and need it to be done in few minutes?
In my experience, this process typically spans multiple deploys. I would say the key insight that I have taken away from decades of applying this approach, is that data migrations need to be done in an __eventually consistent__ approach, rather than as an all-or-nothing, stop-the-world, global transaction or transformation.
Indeed, this pattern, in particular, is extremely useful in environments where you are trying to making changes to one part of a system while multiple deploys are happening across the entire system, or where you are dealing with a change that requires a large number of clients to be updated where you don't have direct control of those clients or they operate in a loosely-connected fashion.
So, regardless of AWS RDS as your underlying database technology, plan to break these steps up into individual deployment steps. I have, in fact, done this with systems deployed over AWS RDS, but also with systems deployed to on-prem SQL Server and Oracle, to nosql systems (this is especially helpful in those environments), to IoT and mobile systems, to data warehouse and analysis pipelines, and on and on.
I’m confused. I thought Expand and Contract was about mutating an existing schema, adding columns and tables, not creating a full replacement schema. But maybe I misunderstood?
What’s in the article, I know as the Strangler Fig Pattern.
> For column-level changes, this often means adding new columns to a table that have the characteristics you want while leaving the current columns as-is.
I think what makes it confusing is that their diagrams depict a completely separate schema, but what they describe is really just altering the existing schema.
> What’s in the article, I know as the Strangler Fig Pattern.
Strangler fig pattern is mostly concerned with migrating from an old software to a new software, from example from a monolith to microservices. But I guess you can also apply it to database schemas.
This is the model we used at the SaaS I worked for a decade ago. It worked great to allow for smooth, zero-downtime upgrades across a fleet of thousands of DB servers serving tens of thousands of app servers and millions of active users.
I use this example when I speak about and teach devops trainings.
I call it the migration sandwich. (Nothing to do with the cube rule).
A piece of bread isn't a sandwich and a single migration in a tool like alembic isn't a "sandwich" either. You have a couple layers of bread with one or several layers of toppings and it's not a sandwich until it's all done.
People get a laugh out of the "idiot sandwich meme" and we always have a good conversation about what gnarly migrations people have seen or done (72+ hours of runtime, splitting to dozens or more tables and then reconstructing things, splitting things out to safely be worked on in the expanded state for weeks, etc).
I had never heard it called "expand and contract" before reading this article a few years ago.
What does everyone else call these?
To me is just a blue-green type deployment for schemas. You have an old and a new thing, you split and merge as traffic replays to the new thing and shows that its viable and not breaking, you swap over as you can.
This is just the natural solution that falls out if you want to change a schema with no downtime. I always just called it “dual writing”.
Is there any easy way to implement this pattern in AWS RDS deployments where we need to deploy multiple times a day and need it to be done in few minutes?
In my experience, this process typically spans multiple deploys. I would say the key insight that I have taken away from decades of applying this approach, is that data migrations need to be done in an __eventually consistent__ approach, rather than as an all-or-nothing, stop-the-world, global transaction or transformation.
Indeed, this pattern, in particular, is extremely useful in environments where you are trying to making changes to one part of a system while multiple deploys are happening across the entire system, or where you are dealing with a change that requires a large number of clients to be updated where you don't have direct control of those clients or they operate in a loosely-connected fashion.
So, regardless of AWS RDS as your underlying database technology, plan to break these steps up into individual deployment steps. I have, in fact, done this with systems deployed over AWS RDS, but also with systems deployed to on-prem SQL Server and Oracle, to nosql systems (this is especially helpful in those environments), to IoT and mobile systems, to data warehouse and analysis pipelines, and on and on.
Expand Contract from Fowler's bliki
https://martinfowler.com/bliki/ParallelChange.html
Expand the interface contract and then contract the interface contract? ;)
I’m confused. I thought Expand and Contract was about mutating an existing schema, adding columns and tables, not creating a full replacement schema. But maybe I misunderstood?
What’s in the article, I know as the Strangler Fig Pattern.
What you describe is what they describe as well:
> For column-level changes, this often means adding new columns to a table that have the characteristics you want while leaving the current columns as-is.
I think what makes it confusing is that their diagrams depict a completely separate schema, but what they describe is really just altering the existing schema.
> What’s in the article, I know as the Strangler Fig Pattern.
Strangler fig pattern is mostly concerned with migrating from an old software to a new software, from example from a monolith to microservices. But I guess you can also apply it to database schemas.
This is the model we used at the SaaS I worked for a decade ago. It worked great to allow for smooth, zero-downtime upgrades across a fleet of thousands of DB servers serving tens of thousands of app servers and millions of active users.