Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Broke Justin.tv

blog.socialcam.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: I Broke Justin.tv

#21

Post by ammon »

emmett wrote: post #13
jrockway wrote: post #3

Finally, a few months into the job, I wrote “break” where I meant “continue,” and caused our payments servers to shutdown one-by-one over the course of several weeks. So, they don't write tests at justin.tv, and they don't do automatic deployment? Sounds like a great place to work at...

1) We've started introducing tests - as you grow I think they're essential. I would say though, not all bugs can be caught by tests (and in fact, I once went over the last 10 bugs we had fixed and discovered that none of them could have been caught by any testing framework). Here are some kinds of bugs you can't catch with testing: - The code assumes that an external service is highly available - The page looks glitc…

For what it's worth, the payment servers were one of the only parts of our system that did have (manual) tests in place at that time (because it took 24 hours to do a restart). The bug in question was in DB timeout code, and all tests passed (until several weeks later, when the DB had problems). Now, clearly coverage was insufficient, and I do agree that adding more testing (as we now have) is a good idea. But some bugs are always going to slip past. Code reviews are what could have caught this bug (which we also now take more seriously).

Re: I Broke Justin.tv

#22

Post by richcollins »

emmett wrote: post #13
jrockway wrote: post #3

Finally, a few months into the job, I wrote “break” where I meant “continue,” and caused our payments servers to shutdown one-by-one over the course of several weeks. So, they don't write tests at justin.tv, and they don't do automatic deployment? Sounds like a great place to work at...

1) We've started introducing tests - as you grow I think they're essential. I would say though, not all bugs can be caught by tests (and in fact, I once went over the last 10 bugs we had fixed and discovered that none of them could have been caught by any testing framework). Here are some kinds of bugs you can't catch with testing: - The code assumes that an external service is highly available - The page looks glitc…

Unit and Integration Tests don't always catch everything but cluster immune systems do a pretty good job on the important stuff:

When someone wanted to do a deployment, we had a completely automated system that we called the cluster immune system. This would deploy the change incrementally, one machine at a time. That process would continually monitor the health of those machines, as well as the cluster as a whole, to see if the change was causing problems. If it didn't like what was going on, it would reject the change, do a fast revert, and lock deployments until someone investigated what went wrong.

http://www.startuplessonslearned.com/2008/11/five-whys.html

Re: I Broke Justin.tv

#23

Post by abstractbill »

richcollins wrote: post #22
emmett wrote: post #13

Earlier quoted context omitted.

1) We've started introducing tests - as you grow I think they're essential. I would say though, not all bugs can be caught by tests (and in fact, I once went over the last 10 bugs we had fixed and discovered that none of them could have been caught by any testing framework). Here are some kinds of bugs you can't catch with testing: - The code assumes that an external service is highly available - The page looks glitc…

Unit and Integration Tests don't always catch everything but cluster immune systems do a pretty good job on the important stuff: When someone wanted to do a deployment, we had a completely automated system that we called the cluster immune system. This would deploy the change incrementally, one machine at a time. That process would continually monitor the health of those machines, as well as the cluster as a whole, t…

Monitoring tends to be far more useful than tests for web apps. And despite how advanced this system sounds, it's usually far less work to monitor a web app than it is to test it.

Re: I Broke Justin.tv

#25

Post by emmett »

wildmXranat wrote: post #19
emmett wrote: post #9

Earlier quoted context omitted.

We use almost 100% OSX at the office (with smatterings of Ubuntu) so I have to say: yes, we have a lot of happy OSX users. If anything it's the IE users on Windows who should be annoyed. I watch from home on my macbook air constantly and from work on an old mac mini, so I don't think it's a general OSX issue. What kind of hardware are you on? Which OS version?

Explain why my streams at 720p and up are unplayable on my top end macbook air 10.6 . I have asked how to fix it via Justin.tv support and had no response. Honestly, I would not give a shit, but since I paid $25 bucks, I would expect to see what I pay for... uh ?

That's very strange. Maybe it's a flash version issue? Because I watch NASL all the time on my air (which sounds like the same hardware as yours but maybe slightly worse) and it runs great.

Well, flash drains my battery like nothing else, but that's just flash video. Same thing happens to me on YouTube.

Re: I Broke Justin.tv

#26

Post by wildmXranat »

freebsd wrote: post #20

I followed Dreamhack's streams on Justin.tv and it was really an awful experience for me, with several downtime right into the live performances and such. I do remember someone from the staff coming into the overcrowded channels and expressing their astonishment feelings toward the fact that the chat was actually standing and surviving the load. Besides that, some phrases they translated into my language are just pre…

I subscribed to the NASL stream and paid $25 and suffered a similar awful experience related to video quality. Any questions on how to fix it on their chat was met with anger from mods, with answers not to spam even though I asked once or twice the most, and now my comments are getting downvoted in this thread. I really liked Justin.tv and had intended on paying for more streams, but it seems like a big joke to me as far I can see.

Re: I Broke Justin.tv

#27

Post by wildmXranat »

emmett wrote: post #25
wildmXranat wrote: post #19

Earlier quoted context omitted.

Explain why my streams at 720p and up are unplayable on my top end macbook air 10.6 . I have asked how to fix it via Justin.tv support and had no response. Honestly, I would not give a shit, but since I paid $25 bucks, I would expect to see what I pay for... uh ?

That's very strange. Maybe it's a flash version issue? Because I watch NASL all the time on my air (which sounds like the same hardware as yours but maybe slightly worse) and it runs great. Well, flash drains my battery like nothing else, but that's just flash video. Same thing happens to me on YouTube.

Well, me too. I paid for the NASL stream and couldn't watch it. Surely, you can understand that throwing $25 away without getting the service kind of sucks. To contrast it against an HD stream from gomtv, these GSL games stream/re-play archives beautifully without any stutters.

So I want to be a fan, a paying fan, but it's kind of a bummer that I can't.

Re: I Broke Justin.tv

#28

Post by emmett »

wildmXranat wrote: post #26
freebsd wrote: post #20

I followed Dreamhack's streams on Justin.tv and it was really an awful experience for me, with several downtime right into the live performances and such. I do remember someone from the staff coming into the overcrowded channels and expressing their astonishment feelings toward the fact that the chat was actually standing and surviving the load. Besides that, some phrases they translated into my language are just pre…

I subscribed to the NASL stream and paid $25 and suffered a similar awful experience related to video quality. Any questions on how to fix it on their chat was met with anger from mods, with answers not to spam even though I asked once or twice the most, and now my comments are getting downvoted in this thread. I really liked Justin.tv and had intended on paying for more streams, but it seems like a big joke to me as…

We are always working really hard on quality of service for video. For example, note the button we have added to report video problems.

The biggest problem here is measuring quality - none of the obvious metrics really correlate.

That said, sorry you had a bad experience and we will strive to make sure in the future you have a good one.

Re: I Broke Justin.tv

#29

Post by emmett »

wildmXranat wrote: post #26
freebsd wrote: post #20

I followed Dreamhack's streams on Justin.tv and it was really an awful experience for me, with several downtime right into the live performances and such. I do remember someone from the staff coming into the overcrowded channels and expressing their astonishment feelings toward the fact that the chat was actually standing and surviving the load. Besides that, some phrases they translated into my language are just pre…

I subscribed to the NASL stream and paid $25 and suffered a similar awful experience related to video quality. Any questions on how to fix it on their chat was met with anger from mods, with answers not to spam even though I asked once or twice the most, and now my comments are getting downvoted in this thread. I really liked Justin.tv and had intended on paying for more streams, but it seems like a big joke to me as…

We are always working really hard on quality of service for video. For example, note the button we have added to report video problems.

The biggest problem here is measuring quality - none of the obvious metrics really correlate.

That said, sorry you had a bad experience and we will strive to make sure in the future you have a good one.

Re: I Broke Justin.tv

#30

Post by jdeseno »

timr wrote: post #12
abstractbill wrote: post #7

Earlier quoted context omitted.

There seems to be a general assumption that writing [unit?] tests is always the "right thing" to do, even at a startup, and no matter what the code in question does. In my experience that's absolutely not the case.

Just to be clear: Bill's opinion on this isn't universally shared at JTV. When you're two guys in a garage, unit tests probably slow you down. But at some point, they begin to dramatically speed things up -- working on someone other person's code becomes orders of magnitude easier when there's a comprehensive test suite to validate your changes. The problem is that at some point between first employee and a dozen (or…

Having been at JTV, looking back it almost seems insane to try to do as much work as we we're without good coverage. If it matters, it should get a test.

I think a lot of people also see tests as slowing things down but, they also provide a lot of value in documenting intentions, make upgrades easier, etc.

Post reply on HN