There’s also an excellent paper called “Fair Benchmarking Considered Difficult:
Common Pitfalls In Database Performance Testing”.
Benchmarks are genuinely hard. Modern systems and hardware are notoriously complex and have different behaviors across runs and with minor tweaks to performance settings.
Yep, absolutely... and in the end benchmarks aren't going to sell a database anyways. Reliability and provable correctness and ease of integration are probably going to win in the end. It's a dubious marketing edge to lead with.
PostgreSQL itself is remarkably slow in many configurations and leaves a lot of performance on the table due to its architecture and assumptions. But it works and people trust it. So they use it.
The SpacetimeDB launch video came up on my YouTube feed - and I was surprised the video didn't go at all into how it was implemented. I assumed it was proprietary magic, but then I was surprised to find it was opensource. That confused me more - to get the kind of semantics they were talking about I had assumed that it was pretty novel and if it was open source they should be leading with that.
Kind of bummed to see its essentially 2015-era React Flux in Rust around a mutex.
I watched a few of their talks and while I'm somewhat impressed by the results, it's not exactly surprising to me when you make the trade-offs they're making that you can get those numbers. Most applications cannot make those trade-offs and thus cannot ever hope to see those numbers. The lack of rigor in database benchmarking when they were talking on any of the videos I saw, was an obvious indicator of something to be weary of.
"Because the system is, well, a hash table with a lock in front of it."
This actually surprises me? I was a very early employee/contractor working on this product but left after 3 months. But this wasn't the shape it had when I was there. It would have been better described as a CoW tree of various sorts, which lets reads and writes advance on separate transactions and reconciles at commit with what I recall was fairly minimal lock-holding time.
When I was there there was no global lock being held while a WASM "reducers" was executing. They operated on their own separate transaction in their own thread.
I haven't looked at their source since -- and I really don't want to because I work on similar things I don't want IP contamination as last I looked they had some sort of weird license -- but there's been .. 3 years? of development on it since and I know Tyler well enough and he hired smart-ish people.
I had a lot of criticisms of what he was making and the way it was being made... but ehh....I would want to confirm in source that this claim is accurate, it makes me a bit suspicious of the accuracy and fairness of the rest of TFA.
Always a fun time to bring up my favorite writeups in database benchmarking: https://questdb.com/blog/lies-damn-lies-and-database-benchma...
There’s also an excellent paper called “Fair Benchmarking Considered Difficult: Common Pitfalls In Database Performance Testing”.
Benchmarks are genuinely hard. Modern systems and hardware are notoriously complex and have different behaviors across runs and with minor tweaks to performance settings.
Yep, absolutely... and in the end benchmarks aren't going to sell a database anyways. Reliability and provable correctness and ease of integration are probably going to win in the end. It's a dubious marketing edge to lead with.
PostgreSQL itself is remarkably slow in many configurations and leaves a lot of performance on the table due to its architecture and assumptions. But it works and people trust it. So they use it.
Or if you're in enterprise, whichever company's sales rep some executive has locked you into while 10 drinks in at a fancy dinner.
The SpacetimeDB launch video came up on my YouTube feed - and I was surprised the video didn't go at all into how it was implemented. I assumed it was proprietary magic, but then I was surprised to find it was opensource. That confused me more - to get the kind of semantics they were talking about I had assumed that it was pretty novel and if it was open source they should be leading with that.
Kind of bummed to see its essentially 2015-era React Flux in Rust around a mutex.
I watched a few of their talks and while I'm somewhat impressed by the results, it's not exactly surprising to me when you make the trade-offs they're making that you can get those numbers. Most applications cannot make those trade-offs and thus cannot ever hope to see those numbers. The lack of rigor in database benchmarking when they were talking on any of the videos I saw, was an obvious indicator of something to be weary of.
"Because the system is, well, a hash table with a lock in front of it."
This actually surprises me? I was a very early employee/contractor working on this product but left after 3 months. But this wasn't the shape it had when I was there. It would have been better described as a CoW tree of various sorts, which lets reads and writes advance on separate transactions and reconciles at commit with what I recall was fairly minimal lock-holding time.
When I was there there was no global lock being held while a WASM "reducers" was executing. They operated on their own separate transaction in their own thread.
I haven't looked at their source since -- and I really don't want to because I work on similar things I don't want IP contamination as last I looked they had some sort of weird license -- but there's been .. 3 years? of development on it since and I know Tyler well enough and he hired smart-ish people.
I had a lot of criticisms of what he was making and the way it was being made... but ehh....I would want to confirm in source that this claim is accurate, it makes me a bit suspicious of the accuracy and fairness of the rest of TFA.