121–130 of 133 posts
That's porn for database geeks: 100 million pageviews per day, 300k requests per second against Redis. The way they use MySQL is also interesting IMHO: they populate a relational database in order to be able to build new indexes in the Redis side, using the relational DB for the stuff it is best at, generating new "views" of the data easily. (Relational DBs are also good to do a zillion more things of course.)
Earlier quoted context omitted.
You know your software is successful when it's used by porn sites :D
I remember in the 90's having a conversation with my friend. We were discussing Linux vs Windows for web hosting. My friend said "tell me if you can find a single porn site that's hosted on Windows." He was right, I couldn't find any.
Earlier quoted context omitted.
I remember in the 90's having a conversation with my friend. We were discussing Linux vs Windows for web hosting. My friend said "tell me if you can find a single porn site that's hosted on Windows." He was right, I couldn't find any.
They do exist. I run a .NET developer tools company and had a customer purchase our data access products to use with several of their porn sites. Never did manage to see the final product though!
That's porn for database geeks: 100 million pageviews per day, 300k requests per second against Redis. The way they use MySQL is also interesting IMHO: they populate a relational database in order to be able to build new indexes in the Redis side, using the relational DB for the stuff it is best at, generating new "views" of the data easily. (Relational DBs are also good to do a zillion more things of course.)
100M page views per day is 1157 page views per second. I wonder how that's being blown up by a factor of 300? I suppose if each video's metadata causes a redis lookup and there are 300 videos displayed per page that would do it, but you'd think you could batch those lookups. It seems a fairly inefficient. Either that or they have some background jobs causing redis lookups that aren't a result of page views.
There were 76 requests made as observed from the net tab in Firebug. The majority of these requests seemed to be for those video images. Note that I'm using an adblocker and refuse to check out the site without it.
Note also that they said 300k queries. There could be constantly-running background jobs doing conversion, pulling those preview frames, and crunching numbers on the metadata.
Earlier quoted context omitted.
Nothing fancy actually. We have 3 dedicated storage servers (two in the same DC as our web server, one in germany), with lots and lots of disk space. We then SCP converted videos to them and serve them trough nginx (supports seeking).
Hey, thank you so much for the information. I hope you don't mind to share a little bit more (it's okay if you don't reply due to IP or whatnot, I'll understand). How do you handle where to put the files on which disks? do you write custom software or there's some sort of off-the-shelf software to do this? The reason I asked about this is because it looked like a few people I talked to seem to prefer to use Distribut…
Like I said, nothing fancy but it works without any issues (apart from a small delay when you have to copy the file to 2 or 3 different servers (redundancy) before it becomes available, but it's not a big problem but I have few ideas how to solve that if it becomes one :)).
Earlier quoted context omitted.
Nothing fancy actually. We have 3 dedicated storage servers (two in the same DC as our web server, one in germany), with lots and lots of disk space. We then SCP converted videos to them and serve them trough nginx (supports seeking).
Hey, thank you so much for the information. I hope you don't mind to share a little bit more (it's okay if you don't reply due to IP or whatnot, I'll understand). How do you handle where to put the files on which disks? do you write custom software or there's some sort of off-the-shelf software to do this? The reason I asked about this is because it looked like a few people I talked to seem to prefer to use Distribut…
Earlier quoted context omitted.
You would imagine wrong. I'm running a video sharing page (think FB&youtube mashed together with a dash of blogger) with about 800k unique/month users and it's no picnic.
I apparently did NOT say that it's picnic. And I don't see how your reply is related to my comment. Does your site have enough social features? Like follow, walls... And is your site 100% redis too?
What I'm saying is that there is more than it meets the eye.
Earlier quoted context omitted.
Redis can run anywhere from 50-200K IOPs, on a single core, dependent on CPU; you can find benchmarks at the following: * http://code.google.com/p/redis/wiki/Benchmarks * http://redis.io/topics/benchmarks At the point where multi-core sharding/ring setups are involved, you begin to have NIC/network saturation. It's feasible to achieve 300K on a single box.
The OP actually addresses this. They're doing some form of clustering and had to add more nodes due to network saturation.
I love how they use sInterStore as a replacement for a fat SQL query.
Can someone explain why there's no Ordered Map in Redis? Basically to achieve and ordered index, one has to use Sorted Set and a classic Hash Map, and pair these two when ordered data are needed, which is an extra level of indirection and wasted space. Am I missing something?
Earlier quoted context omitted.
Hey, thank you so much for the information. I hope you don't mind to share a little bit more (it's okay if you don't reply due to IP or whatnot, I'll understand). How do you handle where to put the files on which disks? do you write custom software or there's some sort of off-the-shelf software to do this? The reason I asked about this is because it looked like a few people I talked to seem to prefer to use Distribut…
We split files into two levels of folders. For instance if a video has a filename "somethingsomethingXY" it will go into "videos/y/x/" folder. Then it's up to our sys admins to mount those folders to whichever disk they want. So from the app's point of view it doesn't matter where the file actually is. And we backup everything every couple of hours. Like I said, nothing fancy but it works without any issues (apart fr…
Wish you all the best to hit and solve that problem (meaning you'll be successful by that time I'm guessing (^_^)).