> A data warehouse on the other hand is an OLAP database and is optimized to work on columns
A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together disparate datasets to build a unified model for analytics, you’ve got yourself a data warehouse no matter what system it lives on.
It's a good all-round primer, well written.
Would love to hear more about larger-than-memory tasks and running local Dask clusters. I processed many-a-dataset that way that would normally make pandas choke.
Now I'll be thinking of "L" in ETL as "Land" and not "Load".
Although the article doesn't propose that but uses a lot of "Land" terminology.
"Load" => "load where? or FROM where?" - ambiguous
"Land" => "land where?" - clear
> A data warehouse on the other hand is an OLAP database and is optimized to work on columns
A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together disparate datasets to build a unified model for analytics, you’ve got yourself a data warehouse no matter what system it lives on.
It's a good all-round primer, well written. Would love to hear more about larger-than-memory tasks and running local Dask clusters. I processed many-a-dataset that way that would normally make pandas choke.
This was great, thanks for writing it up. Even as someone in the data space for a long time, I learned quite a bit.
Apache Avro has 2 encodings: binary AND json.
Are you sure? I can't find any reference indicating that.
Update: Huh, TIL https://avro.apache.org/docs/%2B%2Bversion%2B%2B/specificati...
Now I'll be thinking of "L" in ETL as "Land" and not "Load". Although the article doesn't propose that but uses a lot of "Land" terminology. "Load" => "load where? or FROM where?" - ambiguous "Land" => "land where?" - clear
always understood "land" as the raw data layer though, i.e only the bronze layer in the article.
Data is the new oil