Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Broke Justin.tv

blog.socialcam.com

Post reply on HN
Print view View original

11–20 of 44 posts

Re: I Broke Justin.tv

#11

Post by emmett »

shadowmatter wrote: post #5
jrockway wrote: post #3

Finally, a few months into the job, I wrote “break” where I meant “continue,” and caused our payments servers to shutdown one-by-one over the course of several weeks. So, they don't write tests at justin.tv, and they don't do automatic deployment? Sounds like a great place to work at...

That's what caught my eye too. I like how it's soon followed by "Pushing code fast and often does have a cost, but the benefit in productivity is well worth it." That's certainly true if it's mostly _tested_ code, but there is nothing productive about spending a day trying to find and fix a bug like this in production. Been there, done that. I'm trying to bootstrap a startup now, and if it fails to gain traction then…

We're not the wild west we once were, but neither are we a paragon of testing. When I started Justin.tv, I had 1.5 years of software development experience, never on a team larger than 2 developers; I've now been at it for almost 7 years and yeah I don't think I'd do everything the same from the beginning.

Here are some things we now do at Justin.tv because of experiences like the one you mentioned:

- Everything gets code reviewed

- We have started to introduce tests (sorry Bill, they're really quite helpful in the right circumstance)

- We have extensive monitoring for all systems in case something DOES slip through

- We have fully automated deployment (and rollback)

There's a cost to all of these things, either in setup time or in constant maintenance. When you have few users, maybe they're not worth it - as you grow they become essential.

Re: I Broke Justin.tv

#12

Post by timr »

abstractbill wrote: post #7
jrockway wrote: post #3

Finally, a few months into the job, I wrote “break” where I meant “continue,” and caused our payments servers to shutdown one-by-one over the course of several weeks. So, they don't write tests at justin.tv, and they don't do automatic deployment? Sounds like a great place to work at...

There seems to be a general assumption that writing [unit?] tests is always the "right thing" to do, even at a startup, and no matter what the code in question does. In my experience that's absolutely not the case.

Just to be clear: Bill's opinion on this isn't universally shared at JTV.

When you're two guys in a garage, unit tests probably slow you down. But at some point, they begin to dramatically speed things up -- working on someone other person's code becomes orders of magnitude easier when there's a comprehensive test suite to validate your changes.

The problem is that at some point between first employee and a dozen (or more) engineers, you start to wish you'd spent a bit more time on testing code. Writing those tests from scratch becomes "a project", and it only gets done with herculean effort.

Re: I Broke Justin.tv

#13

Post by emmett »

jrockway wrote: post #3

Finally, a few months into the job, I wrote “break” where I meant “continue,” and caused our payments servers to shutdown one-by-one over the course of several weeks. So, they don't write tests at justin.tv, and they don't do automatic deployment? Sounds like a great place to work at...

1) We've started introducing tests - as you grow I think they're essential. I would say though, not all bugs can be caught by tests (and in fact, I once went over the last 10 bugs we had fixed and discovered that none of them could have been caught by any testing framework). Here are some kinds of bugs you can't catch with testing:

- The code assumes that an external service is highly available

- The page looks glitchy in IE

- If you use that flash api, computers with an nvidia graphics card will show a green screen

- If you open and close a socket quickly, flash will hard-crash the entire browser...sometimes

- Under peak load, when a vacuum is running, this new query will hose the DB

That said, it sounds like this bug COULD have been caught by testing (though I don't know the details and can't be sure). And tests can be darn useful, when used in the right circumstance. But in the hierarchy of important things for running a production website, here's what I would list in priority order:

- Extensive monitoring and alerting

- Detailed, easily accessible logging

- Code review required and taken seriously

- Hiring good developers

- Automated deployment and rollback

- Integration testing

- Unit testing

We've been working our way down the hierarchy :-)

2) We have 100% automated deployment and rollback, and a neat internal tool called Brigade for managing it that I really should get around to open sourcing some day...

Re: I Broke Justin.tv

#14

Post by mikeryan »

abstractbill wrote: post #7
jrockway wrote: post #3

Finally, a few months into the job, I wrote “break” where I meant “continue,” and caused our payments servers to shutdown one-by-one over the course of several weeks. So, they don't write tests at justin.tv, and they don't do automatic deployment? Sounds like a great place to work at...

There seems to be a general assumption that writing [unit?] tests is always the "right thing" to do, even at a startup, and no matter what the code in question does. In my experience that's absolutely not the case.

While I'm not a unit test nazi by any means, if you don't have a QA and deployment process in place to catch things like this unit, and functional tests may be time well spent.

Re: I Broke Justin.tv

#15

Post by abstractbill »

emmett wrote: post #11
shadowmatter wrote: post #5

Earlier quoted context omitted.

That's what caught my eye too. I like how it's soon followed by "Pushing code fast and often does have a cost, but the benefit in productivity is well worth it." That's certainly true if it's mostly _tested_ code, but there is nothing productive about spending a day trying to find and fix a bug like this in production. Been there, done that. I'm trying to bootstrap a startup now, and if it fails to gain traction then…

We're not the wild west we once were, but neither are we a paragon of testing. When I started Justin.tv, I had 1.5 years of software development experience, never on a team larger than 2 developers; I've now been at it for almost 7 years and yeah I don't think I'd do everything the same from the beginning. Here are some things we now do at Justin.tv because of experiences like the one you mentioned: - Everything gets…

sorry Bill, they're really quite helpful in the right circumstance

I don't think I've ever claimed they're never useful ;) But I do still believe that, out of the things you listed, they have the lowest utility for the amount of effort involved.

Personally, I would do fully-automated deploy and rollback first, closely followed by monitoring, then code reviews, and then if problems were still slipping through the net I'd tell people to start writing unit tests.

Re: I Broke Justin.tv

#16

Post by steve-howard »

abstractbill wrote: post #7
jrockway wrote: post #3

Finally, a few months into the job, I wrote “break” where I meant “continue,” and caused our payments servers to shutdown one-by-one over the course of several weeks. So, they don't write tests at justin.tv, and they don't do automatic deployment? Sounds like a great place to work at...

There seems to be a general assumption that writing [unit?] tests is always the "right thing" to do, even at a startup, and no matter what the code in question does. In my experience that's absolutely not the case.

And as a lack of testing endangered the payment system, I'd say it would have been better for them to have some tests in place.

Re: I Broke Justin.tv

#17

Post by dfxm12 »

rglover wrote: post #10

Cool to learn that a bigger site tosses programmers into the ring from the start. What's even better is that they don't have meltdowns over mistakes and make sure every event is a chance to learn and improve rather than be concerned about losing one's job. There should be more of this in the dev community. I think I'd be much more comfortable in an environment like this where I'm being challenged as opposed to spendi…

There is a middle ground where you work with some controls in place that don't bring the whole production system down. Fat fingering something (as described in the article) can happen no matter what your experience level is. More than showing that they treat new hires well, I think this reflects poorly on their dev environment...

Re: I Broke Justin.tv

#18

Post by wildmXranat »

9999 wrote: post #8
wildmXranat wrote: post #6

I would like to see how many happy OSX users are on Justin tv. It feels like they didn't test for usability at all. I asked them how it could be possible to code a player that is capable of playing HD flash at 1fps. Yes - 1 f p s, whereas other sites can play it fluidly(1). It's as if breaking stuff is their motto and answering user questions on how to solve it or explain why it happens is not. Overall, their product…

You're assuming that they actually have real control over player performance on your particular hardware setup, when in fact they have almost no control over it. Go ask Adobe and Apple why flash video (especially anything that isn't h.264) playback occasionally (definitely if you're talking about pre-10.6) sucks on OS X.

Why is your reading comprehension so bad? I said other sites play HD streams very well. I've got a high end Macbook air 10.6 for your information, but they already should know that since I asked their support about it and failed to get a response.

Re: I Broke Justin.tv

#19

Post by wildmXranat »

emmett wrote: post #9
wildmXranat wrote: post #6

I would like to see how many happy OSX users are on Justin tv. It feels like they didn't test for usability at all. I asked them how it could be possible to code a player that is capable of playing HD flash at 1fps. Yes - 1 f p s, whereas other sites can play it fluidly(1). It's as if breaking stuff is their motto and answering user questions on how to solve it or explain why it happens is not. Overall, their product…

We use almost 100% OSX at the office (with smatterings of Ubuntu) so I have to say: yes, we have a lot of happy OSX users. If anything it's the IE users on Windows who should be annoyed. I watch from home on my macbook air constantly and from work on an old mac mini, so I don't think it's a general OSX issue. What kind of hardware are you on? Which OS version?

Explain why my streams at 720p and up are unplayable on my top end macbook air 10.6 . I have asked how to fix it via Justin.tv support and had no response. Honestly, I would not give a shit, but since I paid $25 bucks, I would expect to see what I pay for... uh ?

Re: I Broke Justin.tv

#20

Post by freebsd »

I followed Dreamhack's streams on Justin.tv and it was really an awful experience for me, with several downtime right into the live performances and such.

I do remember someone from the staff coming into the overcrowded channels and expressing their astonishment feelings toward the fact that the chat was actually standing and surviving the load.

Besides that, some phrases they translated into my language are just preposterous, usually it's just better off to do not translate a site at all if you manage to do it in such a way.

I did read all the comments where they've pointed out their many improvements, i hope it will get better and better since a platform like that has great potential.

Post reply on HN