Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

100M-Row Challenge with PHP

github.com

Post reply on HN
Print view View original

71–80 of 104 posts

Re: 100M-Row Challenge with PHP

#71

Post by slopinthebag »

ge96 wrote: post #61
brentroose wrote: post #2

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

5 days to 30 seconds? What kind of factor/order of magnitude is that damn What takes 5 days to run

One query per column per row

Re: 100M-Row Challenge with PHP

#72

Post by slopinthebag »

onion2k wrote: post #38
brentroose wrote: post #2

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. When people say leetcode interviews are pointless I might share a link to this post. If that sort of optimization is possible there is a structures and algorithms problem in the background somewhere.

Do you think they achieved that performance optimisation with a networked service because they switched from insertion sort to quicksort?

Re: 100M-Row Challenge with PHP

#73

Post by lossyalgo »

rectang wrote: post #54
Tade0 wrote: post #11

Earlier quoted context omitted.

Pitch this to whoever is in charge of performance at Wordpress. A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.

Are you talking about a new, empty WordPress instance running the default theme? Because if so, that doesn't match my anecdotal experience. If you're talking about a WordPress instance with arbitrary plugins running an arbitrary theme, then sure — but that's an observation about those plugins and themes, not core. As someone who has to work with WordPress, I have all kinds of issues with it, but "20 seconds to load c…

Can concur. I bought a plugin a few years ago after using the free version for many years, wanting to support the devs for making such a useful plugin. I installed it on a few sites, and left my PC running overnight with a tab open to the plugin and woke up the next day to a lovely rebooted Windows (I hate how default Windows behavior after BSOD is a reboot with ZERO indication that it crashed or if it was an update that rebooted). Re-opened all my tabs, and queue the same waking up the next day to a freshly rebooted Windows, which made me suspicious. I assumed at that point it must have been a BSOD, so dug into Windows event logs, eventually realizing it was Firefox. Restored tabs yet again, left browser open over night, while installing more and more debugging tools for Firefox, none of which helped me track down the culprit. What pissed me off the most was that Firefox even allowed a process to consume > 30GB of RAM and cause my PC to crash! I finally caught it one night after > 10 BSODs - the tab had been open for 20+ hours, and right as it started to spiral out of control and my PC was about to crash, as programs were starting to error out and Windows was madly paging things to disk. I got lucky, and was able to open about:memory to see the culprit - this plugin had some kind of memory leak that wasn't noticeable but then suddenly went nuts. I emailed the devs multiple times with the full debug output, and was ignored for weeks until finally they responded, which pissed me off even more having finally paid for the pro version, only to be greeted with this. The free version didn't seem to have this issue either, which was like an extra slap in the face.

Naked Wordpress is plenty fast, but as soon as you start adding sketchy plugins and Themes, things can spiral out of control.

Re: 100M-Row Challenge with PHP

#74

Post by slopinthebag »

thinkingtoilet wrote: post #33
Xeoncross wrote: post #23

This is why I jumped from PHP to Go, then why I jumped from Go to Rust. Go is the most battery-included language I've ever used. Instant compile times means I can run tests bound to ctrl/cmd+s every time I save the file. It's more performant (way less memory, similar CPU time) than C# or Java (and certainly all the scripting languages) and contains a massive stdlib for anything you could want to do. It's what scripti…

It's almost comical how often bring up Rust. "Here's a fun PHP challange!" "Let's talk about Rust..."

I mean, it's kinda like complaining that people are mentioning excavators on your "how I optimised digging a massive ditch with teaspoons" post.

Re: 100M-Row Challenge with PHP

#75

Post by hparadiz »

slopinthebag wrote: post #72
onion2k wrote: post #38

Earlier quoted context omitted.

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. When people say leetcode interviews are pointless I might share a link to this post. If that sort of optimization is possible there is a structures and algorithms problem in the background somewhere.

Do you think they achieved that performance optimisation with a networked service because they switched from insertion sort to quicksort?

I did the same thing in PHP before. Issue was a foreach over a foreach with a search. The fix was to build the result set as you populate the main array of objects. Basically as you add stuff to the array if there's a value that matches you throw it into another "results" array for any cardinality (unique value) that exists. Since in PHP objects are always just pointers your results arrays are relatively painless. Just a series of int32 integers basically. Then when you need an answer your result is instant. I ended up getting a 80-90% speed up. This is not just a php thing either. Folks end up doing this type of stuff in every language.

Re: 100M-Row Challenge with PHP

#76

Post by AlienRobot »

shimman wrote: post #68
devmor wrote: post #66

Earlier quoted context omitted.

As someone who built full ecommerce websites on wordpress over 15 years ago, I can tell you exactly why it hasn't been replaced - the plugin/theme ecosystem. There are tens of thousands of plugins and themes to make a Wordpress website do whatever you want and look however you want, either for free or a very low fee. You have to replace that entire ecosystem for the same price to replace Wordpress. No matter how many…

Maybe it's just my poor imagination but how many plugins are truly unique to WP that you can't find on other CMSs? The only ones that come to mind would be those plugins that help connect to various B2B or B2C workflows, is that where the gold is mostly found?

WP essentially lets plugins do anything they want. The plugins are just scripts that register callbacks to events. WP calls events on BASICALLY EVERY FUNCTION. This is without exaggeration. I don't remember the exact names right now, but if you have a function like wp_get_title that gets the title of a post, there will be a "get_title" event that can modify which title is returned. So for every function first the data is computed using the default WP way, then plugins are allowed to discard all that work and replace it without something else entirely. There are events for deciding the canonical URL, for deciding the description of a post, for deciding whether RSS links will be displayed or not (the callback just returns true or false), etc.

In other words, every property can be modified through global event callbacks. Some events are called very early in the whole pipeline that let plugins just render whatever they want (e.g. render custom XML sitemaps).

Re: 100M-Row Challenge with PHP

#78

Post by chrismarlow9 »

I don't have time to put together a submission but I'm willing to bet you can use this:

https://github.com/kjdev/php-ext-jq

And replicate this command:

jq -R ' [inputs | split(",") | {url: .[0], date: .[1] | split("T")[0]}] | group_by(.url) | map({ (.[0].url): ( map(.date) | group_by(.) | map({(.[0]): length}) | add ) }) | add ' And it will be faster than anything you can do in native php

Edit: I'm assuming none of the urls have a comma with this but it's more about offloading it through an extension, even if you custom built it

Re: 100M-Row Challenge with PHP

#79

Post by Retr0id »

chrismarlow9 wrote: post #78

I don't have time to put together a submission but I'm willing to bet you can use this: https://github.com/kjdev/php-ext-jq And replicate this command: jq -R ' [inputs | split(",") | {url: .[0], date: .[1] | split("T")[0]}] | group_by(.url) | map({ (.[0].url): ( map(.date) | group_by(.) | map({(.[0]): length}) | add ) }) | add ' And it will be faster than anything you can do in native php Edit: I'm assuming none of t…

The rules exclude FFI etc.

Re: 100M-Row Challenge with PHP

#80

Post by tiffanyh »

toast0 wrote: post #53
tiffanyh wrote: post #15

Earlier quoted context omitted.

Aren’t these optimizations less about PHP, and more about optimizing how your using the database.

PHP is kind of like C. It can be very fast if you do things right, and it gives you more than enough rope to tie yourself in knots. Making your application fast is less about tuning your runtime and more about carefully selecting what you do at runtime. Runtime choice does still matter, an environment where you can reasonably separate sending database queries and receiving the result (async communication) or otherwis…

Would love to read more stories by you toast0 on things you've optimized in the past (given the huge scale you've worked on). Lessons learned, etc. I always find your comments super interesting :)
Post reply on HN