100M-Row Challenge with PHP
github.com
100M-Row Challenge with PHP
1–10 of 104 posts
Re: 100M-Row Challenge with PHP
#2A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more.
That's why I built a performance challenge for the PHP community
The goal of this challenge is to parse 100 million rows of data with PHP, as efficiently as possible. The challenge will run for about two weeks, and at the end there are some prizes for the best entries (amongst the prize is the very sought-after PhpStorm Elephpant, of which we only have a handful left).
I hope people will have fun with it :)
Re: 100M-Row Challenge with PHP
#3Awesome. I’ll be following this. I’ll probably learn a ton.
Re: 100M-Row Challenge with PHP
#4Side note - I wasn't aware that there is active collectors scene for Elephpants, awesome!
Re: 100M-Row Challenge with PHP
#5A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…
> A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds
That's a huge improvement! How much was low hanging fruit unrelated to the PHP interpreter itself, out of curiosity? (E.g. parallelism, faster SQL queries etc)
Re: 100M-Row Challenge with PHP
#6How large is a sample 100M row file in bytes? (I tried to run the generator locally but my php is not bleeding-edge enough)
Re: 100M-Row Challenge with PHP
#7A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…
exec(‘c program that does the parsing’);
Where do I get my prize? ;)
Re: 100M-Row Challenge with PHP
#8A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…
exec(‘c program that does the parsing’); Where do I get my prize? ;)
The FAQ states that solutions like FFI are not allowed because the goal is to solve it with PHP :)
Re: 100M-Row Challenge with PHP
#9A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…
> A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds That's a huge improvement! How much was low hanging fruit unrelated to the PHP interpreter itself, out of curiosity? (E.g. parallelism, faster SQL queries etc)
Almost all, actually. I wrote about it here: https://stitcher.io/blog/11-million-rows-in-seconds
A couple of things I did:
- Cursor based pagination - Combining insert statements - Using database transactions to prevent fsync calls - Moving calculations from the database to PHP - Avoiding serialization where possible
Re: 100M-Row Challenge with PHP
#10How large is a sample 100M row file in bytes? (I tried to run the generator locally but my php is not bleeding-edge enough)
Around 7GB