A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more.
That's why I built a performance challenge for the PHP community
The goal of this challenge is to parse 100 million rows of data with PHP, as efficiently as possible. The challenge will run for about two weeks, and at the end there are some prizes for the best entries (amongst the prize is the very sought-after PhpStorm Elephpant, of which we only have a handful left).
Microbenchmarks are very different from optimizing performance in real applications in wide use though, they could do great on this specific benchmark but still have no clue about how to actually make something large like Wordpress to perform OK out of the box.
> A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds
That's a huge improvement! How much was low hanging fruit unrelated to the PHP interpreter itself, out of curiosity? (E.g. parallelism, faster SQL queries etc)
- Cursor based pagination
- Combining insert statements
- Using database transactions to prevent fsync calls
- Moving calculations from the database to PHP
- Avoiding serialization where possible
A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more.
That's why I built a performance challenge for the PHP community
The goal of this challenge is to parse 100 million rows of data with PHP, as efficiently as possible. The challenge will run for about two weeks, and at the end there are some prizes for the best entries (amongst the prize is the very sought-after PhpStorm Elephpant, of which we only have a handful left).
I hope people will have fun with it :)
Pitch this to whoever is in charge of performance at Wordpress.
A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.
Microbenchmarks are very different from optimizing performance in real applications in wide use though, they could do great on this specific benchmark but still have no clue about how to actually make something large like Wordpress to perform OK out of the box.
exec(‘c program that does the parsing’);
Where do I get my prize? ;)
The FAQ states that solutions like FFI are not allowed because the goal is to solve it with PHP :)
> A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds
That's a huge improvement! How much was low hanging fruit unrelated to the PHP interpreter itself, out of curiosity? (E.g. parallelism, faster SQL queries etc)
Almost all, actually. I wrote about it here: https://stitcher.io/blog/11-million-rows-in-seconds
A couple of things I did:
- Cursor based pagination - Combining insert statements - Using database transactions to prevent fsync calls - Moving calculations from the database to PHP - Avoiding serialization where possible
Aren’t these optimizations less about PHP, and more about optimizing how your using the database.
> Also, the generator will use a seeded randomizer so that, for local development, you work on the same dataset as others
Except that the generator script generates dates relative to time() ?
Side note - I wasn't aware that there is active collectors scene for Elephpants, awesome!
https://elephpant.me/
Elephpants should be for second and third place. First place should be the double-clawed hammer.
How large is a sample 100M row file in bytes? (I tried to run the generator locally but my php is not bleeding-edge enough)
Around 7GB
Awesome. I’ll be following this. I’ll probably learn a ton.