Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

100M-Row Challenge with PHP

github.com

Post reply on HN
Print view View original

11–20 of 104 posts

Re: 100M-Row Challenge with PHP

#11

Post by Tade0 »

brentroose wrote: post #2

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

Pitch this to whoever is in charge of performance at Wordpress.

A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.

Re: 100M-Row Challenge with PHP

#13

Post by tveita »

> Also, the generator will use a seeded randomizer so that, for local development, you work on the same dataset as others

Except that the generator script generates dates relative to time() ?

Re: 100M-Row Challenge with PHP

#15

Post by tiffanyh »

brentroose wrote: post #9
gib444 wrote: post #5

Earlier quoted context omitted.

> A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds That's a huge improvement! How much was low hanging fruit unrelated to the PHP interpreter itself, out of curiosity? (E.g. parallelism, faster SQL queries etc)

Almost all, actually. I wrote about it here: https://stitcher.io/blog/11-million-rows-in-seconds A couple of things I did: - Cursor based pagination - Combining insert statements - Using database transactions to prevent fsync calls - Moving calculations from the database to PHP - Avoiding serialization where possible

Aren’t these optimizations less about PHP, and more about optimizing how your using the database.

Re: 100M-Row Challenge with PHP

#16

Post by embedding-shape »

Tade0 wrote: post #11
brentroose wrote: post #2

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

Pitch this to whoever is in charge of performance at Wordpress. A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.

Microbenchmarks are very different from optimizing performance in real applications in wide use though, they could do great on this specific benchmark but still have no clue about how to actually make something large like Wordpress to perform OK out of the box.

Re: 100M-Row Challenge with PHP

#18

Post by hu3 »

tiffanyh wrote: post #15
brentroose wrote: post #9

Earlier quoted context omitted.

Almost all, actually. I wrote about it here: https://stitcher.io/blog/11-million-rows-in-seconds A couple of things I did: - Cursor based pagination - Combining insert statements - Using database transactions to prevent fsync calls - Moving calculations from the database to PHP - Avoiding serialization where possible

Aren’t these optimizations less about PHP, and more about optimizing how your using the database.

It's still valid as as example to the language community of how to apply these optimizations.

Re: 100M-Row Challenge with PHP

#19

Post by swasheck »

tiffanyh wrote: post #15
brentroose wrote: post #9

Earlier quoted context omitted.

Almost all, actually. I wrote about it here: https://stitcher.io/blog/11-million-rows-in-seconds A couple of things I did: - Cursor based pagination - Combining insert statements - Using database transactions to prevent fsync calls - Moving calculations from the database to PHP - Avoiding serialization where possible

Aren’t these optimizations less about PHP, and more about optimizing how your using the database.

in all my years doing database tuning/admin/reliability/etc, performance have overwhelmingly been in the bad query/bad data pattern categories. the data platform is rarely the issue

Re: 100M-Row Challenge with PHP

#20

Post by kpcyrd »

brentroose wrote: post #8
user3939382 wrote: post #7

Earlier quoted context omitted.

exec(‘c program that does the parsing’); Where do I get my prize? ;)

The FAQ states that solutions like FFI are not allowed because the goal is to solve it with PHP :)

What about using the filesystem as an optimized dict implementation?
Post reply on HN