Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So, that was a bummer

blog.foursquare.com

Post reply on HN
Print view View original

81–90 of 93 posts

Re: So, that was a bummer

#81

Post by lemming »

harryh wrote: post #61
dotBen wrote: post #52

Earlier quoted context omitted.

The down voting on my previous reply is sad but I probably shouldn't be too surprised. The problem of being way too technical for a mainstream audience is a problem many people on Hacker News seem to have, and so no wonder many would disagree with me. It's silly for someone on Hacker News to say "well I thought the level of detail was fine" - of course you would, like the rest of us you're a technical geek. The point…

Interesting. Thx very much fo the feedback. I'm sure you understand it's a hard balance to strike between technical detail and ease of understanding. Will strive to make things a bit more on the "ease of understanding" side next time. Also considering starting a separate engineering blog where it would probably be appropriate to go into more detail for those that are interested. -harryh

For what it's worth, I liked the original much better than the proposed replacement. People aren't idiots, if you give them a good explanation they appreciate it even if they don't fully understand all the details or implications. Think of it like going to the doctor - if I have something wrong with me, I want my doctor to explain it to me in a way I can understand, not just tell me that I have something generic wrong.

Re: So, that was a bummer

#82

Post by dhimes »

ankimal wrote: post #65

Firstly, as a free user you really shouldn't be asking for HA. (I m assuming their paid customer are kicking some butt as I speak .. or maybe not .. the site is back up :)). However, as a business you really want to give ALL your customers HA. Its not just a reputation thing, its a "we love you all equally" kinda attitude. As for MongoDB, we ve been using in production for small insignificant things. FWIW, they have…

HA?

Re: So, that was a bummer

#83

Post by allwein »

dhimes wrote: post #82
ankimal wrote: post #65

Firstly, as a free user you really shouldn't be asking for HA. (I m assuming their paid customer are kicking some butt as I speak .. or maybe not .. the site is back up :)). However, as a business you really want to give ALL your customers HA. Its not just a reputation thing, its a "we love you all equally" kinda attitude. As for MongoDB, we ve been using in production for small insignificant things. FWIW, they have…

HA?

High Availability

Re: So, that was a bummer

#84

Post by JabavuAdams »

jscore wrote: post #78

I'm so happy they're back up, here I was thinking the world would come to a screeching halt when people cannot check-in to places. Seriously though, are they THAT important?

No, they're not, but they have to dance the dance and say / do the right things if they want to keep their users and investors happy.

Re: So, that was a bummer

#85

Post by JabavuAdams »

harryh wrote: post #61
dotBen wrote: post #52

Earlier quoted context omitted.

The down voting on my previous reply is sad but I probably shouldn't be too surprised. The problem of being way too technical for a mainstream audience is a problem many people on Hacker News seem to have, and so no wonder many would disagree with me. It's silly for someone on Hacker News to say "well I thought the level of detail was fine" - of course you would, like the rest of us you're a technical geek. The point…

Interesting. Thx very much fo the feedback. I'm sure you understand it's a hard balance to strike between technical detail and ease of understanding. Will strive to make things a bit more on the "ease of understanding" side next time. Also considering starting a separate engineering blog where it would probably be appropriate to go into more detail for those that are interested. -harryh

Maybe you don't need to find a balance. Provide a "more (technical) info" link after the general-user description.

Re: So, that was a bummer

#86

Post by dowskitest »

dmytton wrote: post #43

This posting in the MongoDB mailing list provides more detail from the developers: http://groups.google.com/group/mongodb-user/browse_thread/th...

Sounds like a side effect of relying on MMAP (and not doing compaction).

"Basically, the issue is that if data migrates to a new shard, there is no re-compaction yet in the old shard of the old collection. So there could be small empty spots throughout it which were migrated out, and if the objects are small, there is no effective improvement in RAM caching immediately after the migration." - Dwight Merriman (at the link in the parent).

"The kernel is able to swap/load 4k pages. For a page to be idle from the point of view of the kernel and its LRU algorithm, what is needed is that there are no memory accesses in the whole page for some time."

-antirez from http://antirez.com/post/what-is-wrong-with-2006-programming....

Re: So, that was a bummer

#87

Post by spuz »

waxman wrote: post #21

The site just went down again, awkwardly enough, only moments after they published their post-mortem on yesterday's outage. Clearly, as their blog post indicates, they were unable to trace the root problem. To me, the worst feeling in the world as a developer is when there's a major bug in your production site, and you can't figure out exactly why it happened. Then even after you get the site working there's that pit…

After this past week, I know exactly what you mean...

Re: So, that was a bummer

#88

Post by sabat »

pvg wrote: post #53
sabat wrote: post #12

I like the NoSQL approach as an option. But we should keep in mind: operationally, these databases/stores are comparatively new, and don't have the years and years of use that would help find and solve problems like this. It reminds me of Ebay's 3-day downtime in 1999 -- based on an Ebay mistake and an Oracle bug. Although Oracle had been around for a while in 1999, OLTP was still new, and, hence the bug. I'm not bla…

in 1999, OLTP was still new, and, hence the bug. No, OLTP was not at all new in 1999. OLTP probably means something other than what you think it means.

Online Transaction Processing. I meant specifically on the web, where life is more chaotic than in traditional environments. I suppose I could have written "web OLTP". By "new", I mean less than 10 years old.

Re: So, that was a bummer

#89

Post by sabat »

blantonl wrote: post #32
sabat wrote: post #12

I like the NoSQL approach as an option. But we should keep in mind: operationally, these databases/stores are comparatively new, and don't have the years and years of use that would help find and solve problems like this. It reminds me of Ebay's 3-day downtime in 1999 -- based on an Ebay mistake and an Oracle bug. Although Oracle had been around for a while in 1999, OLTP was still new, and, hence the bug. I'm not bla…

I'm unable to come up with any reference to a 3 day outage regarding Oracle and Ebay in 1999. Can you provide more info on this - I'm very interested to see what happened. Edit: I found this reference to the 22hr outage that occurred, and I remember this outage, but I don't ever remember it being a 3 days outage. http://www.internetnews.com/ec-news/article.php/137251/Cost-...

I remember it being 3 days. Could be faulty memory.

Re: So, that was a bummer

#90

Post by sabat »

blantonl wrote: post #32
sabat wrote: post #12

I like the NoSQL approach as an option. But we should keep in mind: operationally, these databases/stores are comparatively new, and don't have the years and years of use that would help find and solve problems like this. It reminds me of Ebay's 3-day downtime in 1999 -- based on an Ebay mistake and an Oracle bug. Although Oracle had been around for a while in 1999, OLTP was still new, and, hence the bug. I'm not bla…

I'm unable to come up with any reference to a 3 day outage regarding Oracle and Ebay in 1999. Can you provide more info on this - I'm very interested to see what happened. Edit: I found this reference to the 22hr outage that occurred, and I remember this outage, but I don't ever remember it being a 3 days outage. http://www.internetnews.com/ec-news/article.php/137251/Cost-...

Here's a link to a Forbes article about it. You're right: my memory was distorted. The outage was only 22 hours. (I'm sure it felt like three days to the Ebay admins at the time.)

Fun fact: the "Steve Abatangle" quoted in the article is yours truly, and the author of the piece is Dan Lyons, now AKA Fake Steve Jobs.

http://www.forbes.com/forbes/1999/0726/6402238a.html

Post reply on HN