Live data from Hacker News

100M-Row Challenge with PHP

github.com

51–60 of 104 posts

Re: 100M-Row Challenge with PHP

#51
post #37

Earlier quoted context omitted.

in all my years doing database tuning/admin/reliability/etc, performance have overwhelmingly been in the bad query/bad data pattern categories. the data platform is rarely the issue

The worst offenders I've seen were looping over a shitty ORM

hey don’t forget, that shitty ORM also empowers you to write beautiful, fluent code that, under the hood, generates a 12-way join that brings down your entire database.

Re: 100M-Row Challenge with PHP

#53

Earlier quoted context omitted.

Almost all, actually. I wrote about it here: https://stitcher.io/blog/11-million-rows-in-seconds A couple of things I did: - Cursor based pagination - Combining insert statements - Using database transactions to prevent fsync calls - Moving calculations from the database to PHP - Avoiding serialization where possible

Aren’t these optimizations less about PHP, and more about optimizing how your using the database.

PHP is kind of like C. It can be very fast if you do things right, and it gives you more than enough rope to tie yourself in knots.

Making your application fast is less about tuning your runtime and more about carefully selecting what you do at runtime.

Runtime choice does still matter, an environment where you can reasonably separate sending database queries and receiving the result (async communication) or otherwise lets you pipeline requests will tend to have higher throughput, if used appropriately, batching queries can narrow the gap though. Languages with easy parallelism can make individual requests faster at least while you have available resources. Etc.

A lot of popular PHP programs and frameworks start by spending lots of time assembling a beautiful sculpture of objects that will be thrown away at the end of the request. Almost everything is going to be thrown away at the end of the request; making your garbage beautiful doesn't usually help performance.

Re: 100M-Row Challenge with PHP

#54
post #11

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

Pitch this to whoever is in charge of performance at Wordpress. A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.

Are you talking about a new, empty WordPress instance running the default theme? Because if so, that doesn't match my anecdotal experience.

If you're talking about a WordPress instance with arbitrary plugins running an arbitrary theme, then sure — but that's an observation about those plugins and themes, not core.

As someone who has to work with WordPress, I have all kinds of issues with it, but "20 seconds to load core with caching disabled" isn't one of them.

Re: 100M-Row Challenge with PHP

#55
post #28

Earlier quoted context omitted.

They probably mean "Should look like the output of json_encode($data, JSON_PRETTY_PRINT)". Which most PHP devs would be familiar with.

It sounds plausible, but they really need to spell out exactly what the formatting requirements are, because it can make a huge difference in how efficiently you can write the json out.

It's a challenge for PHP programmers. I imagine the relevant people would recognise that format.

Re: 100M-Row Challenge with PHP

#56
post #40
post #22

Earlier quoted context omitted.

this is never going to be faster because it requires syscalls

The time you lose at the syscall boundary you may be able to win back during much shorter GC pauses.

Some suggestions that are already in the PR list disable GC.

https://www.php.net/manual/en/function.gc-disable.php

Re: 100M-Row Challenge with PHP

#57

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

Using a language that is 100x slower than naive native programs to do a "speed challenge" is like spending your entire day speed walking to run errands when you can just learn how to drive a car.

Re: 100M-Row Challenge with PHP

#58
post #43

Obligatory DuckDB solution: > duckdb -s "COPY (SELECT url[20:] as url, date, count(*) as c FROM read_csv('data.csv', columns = { 'url': 'VARCHAR', 'date': 'DATE' }) GROUP BY url, date) TO 'output.json' (ARRAY)" Takes about 8 seconds on my M1 Macbook. JSON not in the right format, but that wouldn't dominate the execution time.

This log in one of the PR:s claims a 5.4s running time on some Mac.

https://github.com/tempestphp/100-million-row-challenge/pull...

Re: 100M-Row Challenge with PHP

#59
post #29

Earlier quoted context omitted.

That's the default output when using json_encode with the JSON_PRETTY_PRINT flag in php.

> That's the default output when using json_encode with the JSON_PRETTY_PRINT flag in php. JSON_PRETTY_PRINT is irrelevant. Escaping slashes is the default behavior of json_encode(). To switch it off, use JSON_UNESCAPED_SLASHES.

Ah, my bad. Thanks for the correction, TIL!

Re: 100M-Row Challenge with PHP

#60
post #11

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

Pitch this to whoever is in charge of performance at Wordpress. A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.

Wordpress is something that I cannot believe hasn't been displaced by a service that uses a separate application for editing and delivery.

It seems like something like vercel/cloudflare could host the content-side published as a worker for mostly-static content from a larger application and that would be more beneficial and run better with less risk, for that matter. Having the app editing and auth served from the same location is just begging for the issues WP and plugins have seen.

Post reply on HN