Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

List of YC companies, ordered by response time from EC2

blamestella.com

Post reply on HN
Print view View original

21–30 of 41 posts

Re: List of YC companies, ordered by response time from EC2

#21

Post by ldng »

delano wrote: post #9
onteria wrote: post #3

The only problem is that if you look at the fastest response time, it's because it's simply text indicating that they're working on the project. No navigation or anything. The difference in content based on the expectations of what the services provide makes it difficult to grasp what one can get out of this.

Ya that's true. I'm not sure yet how to handle placeholder sites. I could separate them based on a page size threshold but that might be more confusing.

Maybe you should ponderate by total received content weight. It's not perfect but still give you a clue. I mean, not just page size but images and css as well.

Re: List of YC companies, ordered by response time from EC2

#24

Post by mayank »

delano wrote: post #15
emmett wrote: post #14

Earlier quoted context omitted.

So basically, anyone hosting their site on EC2 will appear to have a fantastic response time, compared to anyone else. Which makes the test mostly useless for comparison between startups (though it could be interesting/valid data for a single startup over time, which is what pingdom sells)

I'm just one guy so I have to start somewhere :] Sometime soon though, it will be possible to select the location.

Unfortunately, I think the parent commenter is right -- your numbers are (currently) meaningless. Statistically, your sample size for estimating response time is 1 for each of those sites. You need at least a dozen or so monitor sites, preferably distributed in the same way that Internet users are, for anything meaningful information to be gained. Ideally, you'd also want to distribute your monitors taking peering agreements of the ISP, etc. into account for a truly representative picture of response times.

As it stands, you're doing a disservice to startups that aren't on EC2, for no good reason. You solution "works" on a technical level, but the data is statistically meaningless.

Re: List of YC companies, ordered by response time from EC2

#26

Post by delano »

mayank wrote: post #24
delano wrote: post #15

Earlier quoted context omitted.

I'm just one guy so I have to start somewhere :] Sometime soon though, it will be possible to select the location.

Unfortunately, I think the parent commenter is right -- your numbers are (currently) meaningless. Statistically, your sample size for estimating response time is 1 for each of those sites. You need at least a dozen or so monitor sites, preferably distributed in the same way that Internet users are, for anything meaningful information to be gained. Ideally, you'd also want to distribute your monitors taking peering ag…

Combining monitoring data from multiple locations would give a more accurate response time overall. But as a site owner, that number is not very useful because it's not actionable. For example, if response times to my site become slow from say, Dallas or Ireland, there's is nothing I can do about it other than be aware that it's happening.

Monitoring from a single location is interesting because it gives a metric for how well my application is performing.

Re: List of YC companies, ordered by response time from EC2

#27

Post by delano »

ldng wrote: post #21
delano wrote: post #9

Earlier quoted context omitted.

Ya that's true. I'm not sure yet how to handle placeholder sites. I could separate them based on a page size threshold but that might be more confusing.

Maybe you should ponderate by total received content weight. It's not perfect but still give you a clue. I mean, not just page size but images and css as well.

That's a good idea. I could sort on a score based on response time and page size.

Update: I added a score-based sorting (bytes/response time). Is this interesting or confusing?

https://www.blamestella.com/group/ycombinator?sort=score

Re: List of YC companies, ordered by response time from EC2

#28

Post by mayank »

delano wrote: post #26
mayank wrote: post #24

Earlier quoted context omitted.

Unfortunately, I think the parent commenter is right -- your numbers are (currently) meaningless. Statistically, your sample size for estimating response time is 1 for each of those sites. You need at least a dozen or so monitor sites, preferably distributed in the same way that Internet users are, for anything meaningful information to be gained. Ideally, you'd also want to distribute your monitors taking peering ag…

Combining monitoring data from multiple locations would give a more accurate response time overall. But as a site owner, that number is not very useful because it's not actionable. For example, if response times to my site become slow from say, Dallas or Ireland, there's is nothing I can do about it other than be aware that it's happening. Monitoring from a single location is interesting because it gives a metric for…

Certainly, but I would say that there are very few Internet startups that cater to a single (or even a few) geographical locations. Your numbers currently indicate quality of service for customers in the same hosting block as Amazon EC2.

If you have enough monitors and spread them out well (which shouldn't be too hard, since you could just sign up for Perl/PHP enabled hosting accounts in different locations), you can still diagnose where the slow geographical regions are.

In short, having more data can only give you a more accurate picture, depending on how you analyze it.

Re: List of YC companies, ordered by response time from EC2

#29

Post by jrosoff »

The look and feel of your site is great! Nice job! I agree with other commenters about the need for richer data. For example, just picking on the "top performing" site in your list (skysheet.com), here's the yottaa report on that site:

http://www.yottaa.com/url/skysheet-com-4d2ca049038ade0c05000...

Yottaa notes a similar reachability time as stella does in the "Reachability > Washington" metric. But response time performance is significantly worse at other locations.

Also, it's important to track not just response time of the server, but actual browser performance. If you look at the Page Load Time metric, you'll see that even when the server responds pretty quickly, the actual browser experience is significantly worse than a simple HTTP Client.

Overall, I like the site a lot. If you could get some more accurate metrics, it'd be great. Keep going!

BTW> Here at Yottaa we're beta-ing an API that would let you run tests and access all of our browser and low level data. Would you be interested in getting some better data from us? Contact me : jrosoff AT yottaa DOT com

Post reply on HN