91–100 of 166 posts
This article cites its source as a talk by Twitter VP Raffi Krikorian. 38 minutes of video, audio, and slides are at http://www.infoq.com/presentations/Twitter-Timeline-Scalabil...
Thank you for sharing it.... I was wondering about the video.
It goes into more depth about how they handle timelines.
> Your home timeline sits in a Redis cluster and has a maximum of 800 entries. Wow, that's pretty cool. Congrats to antirez - it must be a nice feeling knowing that your software powers such a big system!
Redis is probably the most useful tool powering the internet after nginx. It really is an amazing piece of engineering.
Earlier quoted context omitted.
They create big, complex, fragile architectures because they started with simple, off-the-shelf architectures that completely fell over at scale. I dunno how long you've been on HN, but around 2007-2008 there were a bunch of HighScalability articles about Twitter's architecture. Back then it was a pretty standard Rails app where when a Tweet came in, it would do an insert into a (replicated) MySQL database, then at r…
>They create big, complex, fragile architectures because they started with simple, off-the-shelf architectures that completely fell over at scale. No, they fell over at "shit we're hitting the limits of our hardware, lets re-architect everything instead of buying bigger hardware". Rather than buy 1000 shitty $2000 servers, buy 2 good $1,000,000 servers. I know it is not fad-compliant, but it does in fact work.
Earlier quoted context omitted.
Redis is probably the most useful tool powering the internet after nginx. It really is an amazing piece of engineering.
I have not used nginx, and have never heard of it spoken about in such glowing terms. Can you maybe compare/contrast it with apache?
nginx does an excellent job as a reverse proxy for applications, there are many configurations where nginx acts as a load balancer, and serves static content, and everything else is passed off to a real application. It's also useful if you're running on a tiny VPS with very little RAM.
However, Apache has a few features that nginx lacks, like embedding languages into it like mod_php does, and per directory configuration, in that directory.
They have a lot of RAM. Dang.
Not as much as I would have thought. 2TB can be one server: http://www.supermicro.co.uk/products/system/5U/5086/SYS-5086... An expensive server for sure... but it's just one server.
Earlier quoted context omitted.
The "fanout" or "wasteful duplication of a single message 30 million times" is only required because they are using tiny underpowered hardware to begin with. The approach that they claim can't possibly work actually does work. You just can't do it on a $2000 "server".
The messages are not repeated in fanout, just the ids. I guess you would know that if you actually the links.
While I'm inclined to favour a fanout approach, a "full" fanout can easily be incredibly costly on the write side.
Earlier quoted context omitted.
I have not used nginx, and have never heard of it spoken about in such glowing terms. Can you maybe compare/contrast it with apache?
The biggest difference is that nginx uses an evented I/O, and handles many connections per thread/process, Apache can only handle one at a time. This allows it to to use far less memory per connection, and lets it perform reasonably under very high loads. It also has very low latency, even for small loads, and is relatively easy to install and configure on most platforms. nginx does an excellent job as a reverse prox…
Earlier quoted context omitted.
No, you buy another 2 servers at the same price, because performance continues to increase incredibly quickly, and what you got $200,000 2 years ago is now half the speed of what $200,000 gets you. >When they scale horizontally across thousands of commodity machines, then knowledge of their problem domain becomes encoded in the scaling decisions they make and stays internal to the company. Or to put it another way: "…
When two of your 2,000 servers die, your load balancers etc kick in and route around the problem. When two of your two servers die, you ... um, well, you lose money and reputation. Quickly.
- Servers that phone home; sometimes the first you know of a potential problem is engineers at your door come to service your server.
- Hot swappable RAID'ed RAM.
- Hot swappable CPU's, with spares, and OS support for moving threads of CPU's that are showing risk factors for failure.
- Hot swappable storage where not just the disks are hot swappable, but whole disk bays, and even trays of hot swappable RAID controllers etc.
- Redundant fibre channel connections to those raid controllers from the rest of the system.
- Redundant network interfaces and power supplies (of course, even relatively entry level servers offers that these days).
In reality, once you go truly high end, you're talking about multiple racks full of kit that effectively does a lot of the redundancy we tend to try to engineer into software solutions either at the hardware level, or abstracted from you in software layers your application won't normally see (e.g. a typical high end IBM system will set aside a substantial percentage of CPU's as spares and/or for various offload and management purposes; IBM's "classic" "Shark" storage system used two highly redundant AIX servers as "just" storage controllers hidden behind SCSI or Fibre Channel interfaces, for example).
You don't get some server where a single component failure somewhere takes it down. Some of these vendors have decades of designing out single points of failure in their high end equipment.
Some of these systems have enough redundancy that you could probably fire a shotgun into a rack and still have decent odds that the server survives with "just" reduced capacity until your manufacturers engineers show up and asks you awkward questions about what you were up to.
In general you're better off looking at many of those systems as highly integrated clusters rather than individual servers, though some fairly high end systems actually offer "single system image" clustering (that is, your monster of a machine will still look like a single server from the application point of view even in the cases where the hardware looks more like a cluster, though it may have some unusual characteristics such as different access speeds to different parts of memory).
Earlier quoted context omitted.
I'm not sure what you are trying to say. My entire point was that you can buy a single server that is more powerful than their entire cluster. You appear to agree, but think that is a problem?
Massively parallel single-machine supercomputers are still, essentially, distributed systems on the inside. You still have to use many of the same techniques to avoid communicating all-to-all. If you treat such a system as a flat memory hierarchy, your application will fall down.