I Broke Justin.tv • Hacker News Board

Re: I Broke Justin.tv

#41

Post by ROFISH » Wed, Jul 20, 2011, 2:14 AM UTC

wildmXranat wrote: post #26

freebsd wrote: post #20

I followed Dreamhack's streams on Justin.tv and it was really an awful experience for me, with several downtime right into the live performances and such. I do remember someone from the staff coming into the overcrowded channels and expressing their astonishment feelings toward the fact that the chat was actually standing and surviving the load. Besides that, some phrases they translated into my language are just pre…

I subscribed to the NASL stream and paid $25 and suffered a similar awful experience related to video quality. Any questions on how to fix it on their chat was met with anger from mods, with answers not to spam even though I asked once or twice the most, and now my comments are getting downvoted in this thread. I really liked Justin.tv and had intended on paying for more streams, but it seems like a big joke to me as…

I've had nothing but great experiences from the Justin.tv/Twitch.tv team regarding their tech. NASL and Dreamhack both just had fairly crappy streaming setups and equipment, in my opinion.

Re: I Broke Justin.tv

#42

Post by justin_vanw » Wed, Jul 20, 2011, 3:11 AM UTC

ammon wrote: post #21

emmett wrote: post #13

Earlier quoted context omitted.

1) We've started introducing tests - as you grow I think they're essential. I would say though, not all bugs can be caught by tests (and in fact, I once went over the last 10 bugs we had fixed and discovered that none of them could have been caught by any testing framework). Here are some kinds of bugs you can't catch with testing: - The code assumes that an external service is highly available - The page looks glitc…

For what it's worth, the payment servers were one of the only parts of our system that did have (manual) tests in place at that time (because it took 24 hours to do a restart). The bug in question was in DB timeout code, and all tests passed (until several weeks later, when the DB had problems). Now, clearly coverage was insufficient, and I do agree that adding more testing (as we now have) is a good idea. But some b…

Every hotshot thinks they don't need tests until they try writing tests for a while, then they usually become evangelists.

My shock and amazement comes from the 24 hours to restart your payments system? Whaaaaat? If that is correct you guys have far, far more issues than not having tests.

Re: I Broke Justin.tv

#43

Post by steven_h » Wed, Jul 20, 2011, 1:00 PM UTC

ROFISH wrote: post #41

wildmXranat wrote: post #26

Earlier quoted context omitted.

I subscribed to the NASL stream and paid $25 and suffered a similar awful experience related to video quality. Any questions on how to fix it on their chat was met with anger from mods, with answers not to spam even though I asked once or twice the most, and now my comments are getting downvoted in this thread. I really liked Justin.tv and had intended on paying for more streams, but it seems like a big joke to me as…

I've had nothing but great experiences from the Justin.tv/Twitch.tv team regarding their tech. NASL and Dreamhack both just had fairly crappy streaming setups and equipment, in my opinion.

When I watched NASL the first day everything played great for the majority of the day then my stream started randomly locking up for no reason, just the video, the audio would continue to play then the stream would catch up a few seconds later.

I'm on a high end gaming PC with oodles of RAM / SSD / i7 on a 50 down 10 up connection so I don't think that watching a stream in 1080p would tax my computer too much.

I also don't think it was the NASL team's fault because the same thing happens on other streams periodically.

It seems to be a bandwidth issue from justin.tv to the end user but it could be anything, all I know is that it is quite annoying.

The next few days everything worked great again.

Re: I Broke Justin.tv

#44

Post by emmett » Wed, Jul 20, 2011, 4:11 PM UTC

justin_vanw wrote: post #42

ammon wrote: post #21

Earlier quoted context omitted.

For what it's worth, the payment servers were one of the only parts of our system that did have (manual) tests in place at that time (because it took 24 hours to do a restart). The bug in question was in DB timeout code, and all tests passed (until several weeks later, when the DB had problems). Now, clearly coverage was insufficient, and I do agree that adding more testing (as we now have) is a good idea. But some b…

Every hotshot thinks they don't need tests until they try writing tests for a while, then they usually become evangelists. My shock and amazement comes from the 24 hours to restart your payments system? Whaaaaat? If that is correct you guys have far, far more issues than not having tests.

How do you restart a system which holds open long-lived connections without kicking everyone off?

If you want to be kind to your users, you stop accepting connections on a new server, wait a period of time (the longer the better), and then restart. It's a matter of how many users you're willing to disrupt for how quickly you can restart the system.

The canonical example at Justin.tv is the video system: people broadcast for days to weeks at a time. If you restart the server they're connected to, their stream will be disrupted (even if the auto-reconnect works).

We have a separate system that handles most of the complexity which is stateless, but sometimes you need to restart the actual connection-holding-daemon itself. How would you suggest doing that without disrupting service?