Live data from Hacker News

100M-Row Challenge with PHP

github.com

91–100 of 104 posts

Re: 100M-Row Challenge with PHP

#91

Earlier quoted context omitted.

Much like anything else your performance is going to vary a lot based on architecture of implementation. You really shouldn't deploying anything into production without some kind of caching. Whether that's done in the application itself or with memcached/redis or varnish or OPcache.

Either you use a slow language and deal with caching or you use a fast language and just put Cloudflare/Bunny/etc in front.

Aren't CF, Bunny, etc CDNs and therefore cache?

Re: 100M-Row Challenge with PHP

#92
post #87

I took a quick look, the dependency on php 8.5 is mildly irritating, even Ubuntu 26.04 isn't lined up to ship with that version, it's on 8.4.11. You mention in the README that the goal is to run things in a standard environment, but then you're using a near bleeding edge PHP version that people are unlikely to be using? I thought I'd just quickly spin up a container and take a look out of interest, but now it looks l…

Those are quite good:

https://launchpad.net/~ondrej/+archive/ubuntu/php

Anyway, whatever you write in an earlier PHP version is likely to work on future versions. PHP has remarkable BC.

If you're just experimenting, might as well start in the browser:

https://alganet.github.io/phasm/

Not all extensions available there, but it has the essentials.

Re: 100M-Row Challenge with PHP

#93
post #84

Earlier quoted context omitted.

Would love to read more stories by you toast0 on things you've optimized in the past (given the huge scale you've worked on). Lessons learned, etc. I always find your comments super interesting :)

Well on the subject of PHP, I think I've got a nice story. The more recent one is about Wordpress. One day, I had this conversation: Boss: "will the blog stay up?" toast0: "yeah, nobody goes to the blog, it's no big deal" Boss: "they will" toast0: "oh, ummmm we can serve a static index.html and that should work" Later that day, he posted https://blog.whatsapp.com/facebook I took a snapshot to serve as index.html and…

Re: PHP vs a rendered index.html … your story brings back fond memories of my college days (around 2001–2002).

I was a full-time student but also worked for the university’s “internet group.” We ran a homegrown PHP CMS (this was before WordPress/Movable Type), and PHP still felt pretty new. Perl was everywhere, but I was pushing PHP because I’d heard Yahoo had started using it.

Around then, the university launched its first online class registration system. Before that it was all phone/IVR. I warned our team lead the web server would melt down on registration day because every student would be hammering refresh at 9am to get the best class times and professors. He brushed it off, so I pre-rendered the login page as a static index.html and dropped it in the web root.

He noticed, got mad (he had built the CMS and was convinced it could handle the load), and deleted my pre-rendered index.html. So young and dumb me wrote a cron job that pinged the site every few minutes, and if it looked down, it copied my static index.html back into the web directory. Since Apache would serve index.html ahead of PHP, it became an instant fallback page.

Sure enough, at 9am the entire university website went down. Obviously orders of magnitude less scale than your FB story (and way less exciting of an event), but for my small university it was brief moment panic. But my little cron job kicked in and at least kept the front door standing.

While I’m not in active day to day development anymore, I do still work in tech and think a lot about ways to avoid computation. And something I’ve learned a lot from reading your posts over the years and my own personal experiences is just how big you can scale when you can architect in a way that “just pushes bits” (eg “index.html”) as opposed to computes/transforms/renders something … and I’m not sure you can ever really learn that expect through real world experience.

Regarding your links, I’ve seen you post about 1 before and have read about it - it looks very cool. I don’t recall seeing 2 or 3 before and look forward to reading more about those. Thanks as always for your insights!

Re: 100M-Row Challenge with PHP

#94
You should say in the output formatting rules that the pages should be output in the order that the pages are in the input file. Currently it only specifies the order of the visits within the entry for each page.

Re: 100M-Row Challenge with PHP

#95
post #88

What's a decent time for this? I was curious what it would take if I approached it the way I do with most CSV transformation tasks that I'm only intending to do once: use Unix command line tools such as cut, sed, sort, and uniq to do the bulk of the work, and then do something in whatever scripting language seems appropriate to put the final output in whatever format is needed. The first part, using this command [1],…

You can check the benchmarks here: https://github.com/tempestphp/100-million-row-challenge/blob...

A "good" run seems to be around 20-40s mark.

Re: 100M-Row Challenge with PHP

#96

Earlier quoted context omitted.

Sorry, but it's honestly just a lot of our journeys. Started on scripting languages like PHP/Ruby/Lua (self-taught) or Java/VB/C#/Python (collage) and then slowly expanded to other languages as we realized we were being held back by our own tools. Each new language/relationship makes you kick yourself for putting up with things so long.

I understand that but there's a time and a place. Rust has nothing to do with this. 100% of the people on this site understand that this challenge can be done faster in C, or Rust, or whatever. This is a PHP challenge. Perhaps we could discuss the actual submission as opposed to immediately derailing it.

> I understand that but there's a time and a place.

Dude, this is a website where a bunch of developer nerds congregate and talk shop. They're fine, this is the same kind of shit that's been happening across these kinds of sites for decades.

Re: 100M-Row Challenge with PHP

#97
post #84

Earlier quoted context omitted.

Well on the subject of PHP, I think I've got a nice story. The more recent one is about Wordpress. One day, I had this conversation: Boss: "will the blog stay up?" toast0: "yeah, nobody goes to the blog, it's no big deal" Boss: "they will" toast0: "oh, ummmm we can serve a static index.html and that should work" Later that day, he posted https://blog.whatsapp.com/facebook I took a snapshot to serve as index.html and…

Re: PHP vs a rendered index.html … your story brings back fond memories of my college days (around 2001–2002). I was a full-time student but also worked for the university’s “internet group.” We ran a homegrown PHP CMS (this was before WordPress/Movable Type), and PHP still felt pretty new. Perl was everywhere, but I was pushing PHP because I’d heard Yahoo had started using it. Around then, the university launched it…

> Regarding your links, I’ve seen you post about 1 before and have read about it - it looks very cool. I don’t recall seeing 2 or 3 before and look forward to reading more about those. Thanks as always for your insights!

So #1 now has dist connection stuff as of a few hours ago. Not super obvious, but you can load two (or more) nodes and call nodes() and see they're connected. Dist connection opens up lots of neat possibilities... but I do need to add an obvious application so it's like actually neat instead of just potentially neat.

#2 is a pretty neat way to diagnose path mtu problems. And I've been seeing people use it and link to it on networking forums all over, even forums in other languages. Which is pretty awesome. Maybe a few links in forums over the past year, but it's always cool to see people using stuff I built mostly for me. :)

#3 is like I dunno, probably not that useful, I think you could do a lot of similar stuff already, but it felt like a tool that was missing... but I also got some feedback that maybe there's other ways to do it already too, so shrug. But pxe booting is always fun.

Re: 100M-Row Challenge with PHP

#98
post #54

Earlier quoted context omitted.

Are you talking about a new, empty WordPress instance running the default theme? Because if so, that doesn't match my anecdotal experience. If you're talking about a WordPress instance with arbitrary plugins running an arbitrary theme, then sure — but that's an observation about those plugins and themes, not core. As someone who has to work with WordPress, I have all kinds of issues with it, but "20 seconds to load c…

Can concur. I bought a plugin a few years ago after using the free version for many years, wanting to support the devs for making such a useful plugin. I installed it on a few sites, and left my PC running overnight with a tab open to the plugin and woke up the next day to a lovely rebooted Windows (I hate how default Windows behavior after BSOD is a reboot with ZERO indication that it crashed or if it was an update…

A memory leak should not be able to BSOD your install. Something else is wrong with your machine, too.

Re: 100M-Row Challenge with PHP

#99

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

Fun challenge, but running the benchmark on Apple hardware is a weird decision as Apple doesn't even have server hardware. Would make much more sense to run it on a dedicated Linux box as that is more accessible and more realistic.

Re: 100M-Row Challenge with PHP

#100
post #11

A month ago, I went on a performance quest trying to optimize a PHP script that took 5 days to run. Together with the help of many talented developers, I eventually got it to run in under 30 seconds. This optimization process with so much fun, and so many people pitched in with their ideas; so I eventually decided I wanted to do something more. That's why I built a performance challenge for the PHP community The goal…

Pitch this to whoever is in charge of performance at Wordpress. A Wordpress instance will happily take over 20 seconds to fully load if you disable cache.

I’ve long since abandoned WP but this seems like an SQL resource issue as it bogs up against the oom reaper dealing with no swap. WordPress is like a mid level V6 Kia with all the options and a huge aftermarket.
Post reply on HN