Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

100M-Row Challenge with PHP

github.com

Post reply on HN
Print view View original

41–50 of 104 posts

Re: 100M-Row Challenge with PHP

#41

Post by nicoburns »

onion2k wrote: post #38
brentroose wrote: post #2

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. When people say leetcode interviews are pointless I might share a link to this post. If that sort of optimization is possible there is a structures and algorithms problem in the background somewhere.

I find that these kind of optimizations are usually more about technical architecture than leetcode. Last time I got speedups this crazy the biggest win was reducing the number of network/database calls. There were also optimisations around reducing allocations and pulling expensive work out of hot loops. But leetcode interview questions don't tend to cover any of that.

They tend to be about the implementation details of specific algorithms and data structures. Whereas the important skill in most real-world scenarios would be to understand the trade-offs between different algorithms and data structures so that you pick an appropriate off-the-shelf implementation to use.

Re: 100M-Row Challenge with PHP

#42

Post by rkozik1989 »

Tade0 wrote: post #11
brentroose wrote: post #2

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

Pitch this to whoever is in charge of performance at Wordpress. A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.

Much like anything else your performance is going to vary a lot based on architecture of implementation. You really shouldn't deploying anything into production without some kind of caching. Whether that's done in the application itself or with memcached/redis or varnish or OPcache.

Re: 100M-Row Challenge with PHP

#43

Post by csjh »

Obligatory DuckDB solution:

> duckdb -s "COPY (SELECT url[20:] as url, date, count(*) as c FROM read_csv('data.csv', columns = { 'url': 'VARCHAR', 'date': 'DATE' }) GROUP BY url, date) TO 'output.json' (ARRAY)"

Takes about 8 seconds on my M1 Macbook. JSON not in the right format, but that wouldn't dominate the execution time.

Re: 100M-Row Challenge with PHP

#44

Post by idoubtit »

_yvnd wrote: post #29
semiquaver wrote: post #26

Are they just confused about what characters require escaping in JSON strings or is PHP weirder than I remember? { "\/blog\/11-million-rows-in-seconds": { "2025-01-24": 1, "2026-01-24": 2 }, "\/blog\/php-enums": { "2024-01-24": 1 } }

That's the default output when using json_encode with the JSON_PRETTY_PRINT flag in php.

> That's the default output when using json_encode with the JSON_PRETTY_PRINT flag in php.

JSON_PRETTY_PRINT is irrelevant. Escaping slashes is the default behavior of json_encode(). To switch it off, use JSON_UNESCAPED_SLASHES.

Re: 100M-Row Challenge with PHP

#45

Post by LollipopYakuza »

nicoburns wrote: post #41
onion2k wrote: post #38

Earlier quoted context omitted.

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. When people say leetcode interviews are pointless I might share a link to this post. If that sort of optimization is possible there is a structures and algorithms problem in the background somewhere.

I find that these kind of optimizations are usually more about technical architecture than leetcode. Last time I got speedups this crazy the biggest win was reducing the number of network/database calls. There were also optimisations around reducing allocations and pulling expensive work out of hot loops. But leetcode interview questions don't tend to cover any of that. They tend to be about the implementation detail…

I agree. The "advanced" leetcode is about those last % of optimization. But when network latency is involved in a flow, it is usually the most obvious low hanging fruit.

Re: 100M-Row Challenge with PHP

#46

Post by idoubtit »

semiquaver wrote: post #26

Are they just confused about what characters require escaping in JSON strings or is PHP weirder than I remember? { "\/blog\/11-million-rows-in-seconds": { "2025-01-24": 1, "2026-01-24": 2 }, "\/blog\/php-enums": { "2024-01-24": 1 } }

The weirdness is partly in JSON . In the JSON spec, the slash (named "solidus" there) is the only character that can be written plainly or prefixed with a backslash (AKA "reverse solidus").

See page 4, section 9 of the latest ECMA for JSON: https://ecma-international.org/wp-content/uploads/ECMA-404_2...

Re: 100M-Row Challenge with PHP

#47

Post by LoganDark »

rkozik1989 wrote: post #42
Tade0 wrote: post #11

Earlier quoted context omitted.

Pitch this to whoever is in charge of performance at Wordpress. A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.

Much like anything else your performance is going to vary a lot based on architecture of implementation. You really shouldn't deploying anything into production without some kind of caching. Whether that's done in the application itself or with memcached/redis or varnish or OPcache.

> You really shouldn't deploying anything into production without some kind of caching.

Citation needed? You only need cache if a render is expensive to produce.

Re: 100M-Row Challenge with PHP

#48

Post by tuetuopay »

onion2k wrote: post #38
brentroose wrote: post #2

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. When people say leetcode interviews are pointless I might share a link to this post. If that sort of optimization is possible there is a structures and algorithms problem in the background somewhere.

Well leetcode asks you to implement the data structure, not how and when to use which data structure. I don’t need to know how to implement a bloom filter on a whiteboard off the top of my head to know when to use it.

Re: 100M-Row Challenge with PHP

#49

Post by edoceo »

tosti wrote: post #37
swasheck wrote: post #19

Earlier quoted context omitted.

in all my years doing database tuning/admin/reliability/etc, performance have overwhelmingly been in the bad query/bad data pattern categories. the data platform is rarely the issue

The worst offenders I've seen were looping over a shitty ORM

And that is true across languages.

Re: 100M-Row Challenge with PHP

#50

Post by almosthere »

monkey_monkey wrote: post #21
Tade0 wrote: post #11

Earlier quoted context omitted.

Pitch this to whoever is in charge of performance at Wordpress. A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.

That's often a skill issue.

skill issue being they only know php
Post reply on HN