Scaling PHP Book: I will teach you to scale PHP to millions of users • Hacker News Board

Re: Scaling PHP Book: I will teach you to scale PHP to millions of users

#71

Post by devgutt » Tue, Jun 05, 2012, 10:49 PM UTC

snorkel wrote: post #38

Here's a short list: 1. Cache the output at the edges: Use Varnish or other reverse proxy cache. 2. Cache byte code: Use APC or XCache PHP opcode cache. 3. Cache and minimize database I/O: reduce database touches using memcached, redis, file caches, and application-level caches (ie. global vars) 4. Do event logging in local files, not to the database: Make all write operations as simple and fast as possible, any data…

What your opinion about the use of PHP frameworks like Cake PHP, Yii, Symfony, etc?

Re: Scaling PHP Book: I will teach you to scale PHP to millions of users

#72

Post by riledhel » Tue, Jun 05, 2012, 11:18 PM UTC

stevencorona wrote: post #55

qeorge wrote: post #46

Earlier quoted context omitted.

I'm expecting the book to go much further than this. Your list is a fine starting point, but its fairly basic and lacks specifics. I want to know about the weird things that happen when you push your PHP stack to really ridiculous limits. I think that's what this book will offer. If it ends up being stuff like "use APC! minimize trips to the DB!" I would be very surprised and disappointed.

I agree- good starting point, but the book will be much more in depth. It's going to include a fair amount of high-level design, identifying how and where to scale, weird issues you run into (fun fact: we don't use APC @ Twitpic), solutions to PHP anti-scaling quirks (you can't set sub-1s mySQL timeouts, for instance, although I think this is due to libmysql, not PHP), lots of case studies, scaling design patterns, a…

What was your experience with APC? You still use some opcode cache, right?

Re: Scaling PHP Book: I will teach you to scale PHP to millions of users

#73

Post by nolok » Tue, Jun 05, 2012, 11:27 PM UTC

devgutt wrote: post #71

snorkel wrote: post #38

Here's a short list: 1. Cache the output at the edges: Use Varnish or other reverse proxy cache. 2. Cache byte code: Use APC or XCache PHP opcode cache. 3. Cache and minimize database I/O: reduce database touches using memcached, redis, file caches, and application-level caches (ie. global vars) 4. Do event logging in local files, not to the database: Make all write operations as simple and fast as possible, any data…

What your opinion about the use of PHP frameworks like Cake PHP, Yii, Symfony, etc?

Using PHP for a 30+M registered users website, I'm using a modified Code Igniter for the base framework, and specific Zend Framework's modules on some functionality where they provide good libraries.

As the parent said, ultimately cpu cycle on your web servers don't matter much, you will be limited by IO everywhere. Knowing how to properly use redis' data types and handle multi-level cache invalidation will be a much more valuable asset than using a framework you might not enjoy the most / be the most fluent with just because it's a wee bit faster.

Even in terms of actual php's calculation, your main strong point is figuring out what you can remove from the page processing and put in a daemon instead. "If the user wants to do action A, what is the minimal amount of things I can do to make him believe that it has been done", do that in your controllers, and put all the actual hard work in an event processing queue that doesn't have an user waiting for it to answer back.

Re: Scaling PHP Book: I will teach you to scale PHP to millions of users

#74

Post by nolok » Tue, Jun 05, 2012, 11:32 PM UTC

rlander wrote: post #62

dangrossman wrote: post #50

Earlier quoted context omitted.

> It's 2012 for god's sake, you can rent a 32GB server for less than $100. Where, in the US, can I rent a 32GB server for less than $100?

Hetzner[1]. I never said "in the US", but even in the US you can get a pretty good deal: 16GB for $79/month[2]. [1] http://www.hetzner.de/en/hosting/produkte_rootserver/ex4s [2] http://www.honelive.com/xml/#new-york-city-dedicated-servers

Hetzner doesn't use ECC ram modules. Might not be important if you use it for a memcached server, or it might be a disaster in other cases.

Re: Scaling PHP Book: I will teach you to scale PHP to millions of users

#75

Post by rlander » Tue, Jun 05, 2012, 11:43 PM UTC

dangrossman wrote: post #66

rlander wrote: post #62

Earlier quoted context omitted.

Hetzner[1]. I never said "in the US", but even in the US you can get a pretty good deal: 16GB for $79/month[2]. [1] http://www.hetzner.de/en/hosting/produkte_rootserver/ex4s [2] http://www.honelive.com/xml/#new-york-city-dedicated-servers

No, you didn't, but we all knew you were talking about Hetzner. One company, selling consumer-grade desktop parts unfit for use in a production server, in one country, does not lend credibility to statements like you made. Neither does the NYC site with no info, no SLA, and again, desktop parts. It's something like saying salary isn't important to startups because labor costs in China are pennies per hour. It's 2012…

You can split hairs all you like, Dan, but you're still missing the point: scalability is only an issue for a fraction of the entrepreneurs/engineers who worry about it.

Re: Scaling PHP Book: I will teach you to scale PHP to millions of users

#76

Post by josephmoniz » Wed, Jun 06, 2012, 1:02 AM UTC

riledhel wrote: post #72

stevencorona wrote: post #55

Earlier quoted context omitted.

I agree- good starting point, but the book will be much more in depth. It's going to include a fair amount of high-level design, identifying how and where to scale, weird issues you run into (fun fact: we don't use APC @ Twitpic), solutions to PHP anti-scaling quirks (you can't set sub-1s mySQL timeouts, for instance, although I think this is due to libmysql, not PHP), lots of case studies, scaling design patterns, a…

What was your experience with APC? You still use some opcode cache, right?

If you use a persistent PHP stack like mongrel2 + photon an opcode cache is just a waste of a php module.

http://www.photon-project.com/

Re: Scaling PHP Book: I will teach you to scale PHP to millions of users

#77

Post by FuzzyDunlop » Wed, Jun 06, 2012, 1:15 AM UTC

DanielN wrote: post #61

snorkel wrote: post #38

Here's a short list: 1. Cache the output at the edges: Use Varnish or other reverse proxy cache. 2. Cache byte code: Use APC or XCache PHP opcode cache. 3. Cache and minimize database I/O: reduce database touches using memcached, redis, file caches, and application-level caches (ie. global vars) 4. Do event logging in local files, not to the database: Make all write operations as simple and fast as possible, any data…

I'm curious why you suggest avoiding ORMs. I've been debating this for a while and while I like the accessibility and abstraction that ORMs provide in Django and Rails I'm not familiar enough with Doctrine, DB_DataObject or the other PHP options to really have formed my own opinions yet. Is this a language specific suggestion or something broader? I would love to hear a deeper debate on the subject.

It could be as simple as the performance loss incurred by an ORM is greater than the benefit of using it. You have PDO with PHP, which can be used quite adequately as an ORM itself (you can fetch a tailored SQL query into a class as opposed to working with basic arrays).

I'd argue that, in some cases, an ORM is only used as a reaction to the separation of concerns. Writing SQL mixes up your languages, so why not try to build that SQL in the language you're working with? Or maybe SQL is considered too difficult.

I quite like having full control over a 'raw' query without requiring an ORM's opinion on how it should be built.

Re: Scaling PHP Book: I will teach you to scale PHP to millions of users

#78

Post by roryokane » Wed, Jun 06, 2012, 1:21 AM UTC

executive wrote: post #64

rlander wrote: post #44

This fetishism with scaling... to me it's just procrastination. It feels like work because you're doing something technical but, in the end, you're adding very little to your product. Yesterday I had a meeting with a potential customer and I hated it. I hate to try to explain my SaaS software to non-technical people who treat me like some 17-year-old webmaster. I'd much rather be refactoring Clojure code. But I got o…

What is a 32GB server?

He means a web server with 32 gigabytes of memory. Citation: those are the stats of the server he links in his comment at http://news.ycombinator.com/item?id=4071035.

Re: Scaling PHP Book: I will teach you to scale PHP to millions of users

#79

Post by danudey » Wed, Jun 06, 2012, 1:39 AM UTC

jakejake wrote: post #65

rlander wrote: post #44

This fetishism with scaling... to me it's just procrastination. It feels like work because you're doing something technical but, in the end, you're adding very little to your product. Yesterday I had a meeting with a potential customer and I hated it. I hate to try to explain my SaaS software to non-technical people who treat me like some 17-year-old webmaster. I'd much rather be refactoring Clojure code. But I got o…

scale is one of those problems that sneaks up on you (well unless you're an instant hit site I suppose) In all of my cases dealing with scale issues it's been due to the DB growing very large in size to the point where you can't use the same techniques that you're used to using. That usually happens in combination with more concurrent users. Depending on how large and how organized your code is, things can get very u…

One of the issues I've run into in the past was a MySQL problem where their system was creating tons of temporary tables, and because of the massive amount of traffic they were getting, it was killing their database server because it was choking out the disk's I/O, and the server was at 80% iowait most of the time because the disk cache was paging things in and out like crazy.

I went back and forth with them about optimizing the app, but it was apparently some huge labyrinthine monstrosity. They insisted that they didn't have the resources to do any of the significant rewrites that it would require to fix the app to do proper queries (or at least, not enough to be worthwhile).

Eventually I gave up. /tmp was mounted onto a separate partition, so I disabled ext3 journalling and set commit=30 so that it only sync'ed to the disk every 30 seconds. Since no temporary tables lasted that long, the VFS layer never wrote to the disk if it didn't have to. /tmp became an in-memory cache, and CPU use dropped to 5%.

Optimizing isn't about a checklist, it's about looking at the system that you have, understanding what it's doing and why, and understanding how the other systems around it behave so that you can resolve the issue. Moving onto another database server wouldn't have helped them. Moving onto a RAID would have reduced the impact, but their load didn't scale linearly so they'd hit their limit in a few months anyway.

Re: Scaling PHP Book: I will teach you to scale PHP to millions of users

#80

Post by ams6110 » Wed, Jun 06, 2012, 2:01 AM UTC

debaserab2 wrote: post #45

rlander wrote: post #44

This fetishism with scaling... to me it's just procrastination. It feels like work because you're doing something technical but, in the end, you're adding very little to your product. Yesterday I had a meeting with a potential customer and I hated it. I hate to try to explain my SaaS software to non-technical people who treat me like some 17-year-old webmaster. I'd much rather be refactoring Clojure code. But I got o…

Performance isn't a problem until it is, and then when it is a problem, it's bigger than any other problem in the world because you have nothing to sell if it does not operate. I always thought I was being wise by not doing any premature optimization, but after a few lessons learned the hard way I certainly factor in performance to the design of software before I build now. Scalability is not a "feature" tacked on at…

Part of being a good developer is knowing where the scalability bottlenecks are likely to be as you are developing software, and making intelligent decisions about the algorithms, data structures, and architectures you use. I would never advise a developer to not "factor in" performance or future scalability needs at all. The problem is when you spend significant extra effort building highly scalable architectures that you don't need now and may (likely) never need.