Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Three Months to Scale NewsBlur

blog.newsblur.com

Post reply on HN
Print view View original

111–120 of 146 posts

Re: Three Months to Scale NewsBlur

#111

Post by TheTaytay »

Sam, congrats!

For a task a parallelizable as fetching feeds, I can't recommend Picloud.com enough. You can ssh into a box, provision it however you want, save that image off, and then run arbitrary commands/scripts on instances of that image, on the CPU type you want, and pay only for the seconds of usage you have. (Their "s1" core type is $0.04/hour: http://www.picloud.com/pricing/ ) They also have a system that lets you mount the same shared "drive" from multiple instances at the same time, so if you're doing file-based stuff, it's easy.

The thing I like about it is that running another instance of your script on a new machine (or 2k of them) is trivial. No need to wait to provision a new VPS. Starting and stopping jobs is fast, so scaling up/down is fast.

(I'm swear I'm not affiliated with them. I'm using it for a side project at the moment and have been so excited about it, I want to spread the word.)

Re: Three Months to Scale NewsBlur

#112

Post by MikeAmelung »

laureny wrote: post #99
conesus wrote: post #20

Earlier quoted context omitted.

This is something I'll be working on soon enough. There are many, many dependencies (not only for the web app, but there are three DBs that you have to have installed as prerequisites - mongo, postgres/mysql, and redis). The real problem with setting up your own instance of NewsBlur is that you'll have to do your own feed fetching. This is effectively what you're paying for when you pay for NewsBlur. Let me break it…

> So you have to run 10 processes in parallel, and that's already beyond the capabilities of most single machines. Er... really? My laptop is three years old and I run 150 threads on it every time I launch the server I'm working on (dozens of times a day). No problems at all. It's in Java, in case it matters.

I believe he means that each process requires ~5 seconds of CPU/system time per update, meaning that even a single 8-core machine wouldn't keep up. It's not really related to how many threads or processes can be created, rather how many can actually be doing work in parallel.

Re: Three Months to Scale NewsBlur

#113

Post by bobdvb »

boyter wrote: post #82
MatthewPhillips wrote: post #24

Earlier quoted context omitted.

Indeed. If one were wise, you could stalk Google/Yahoo/BigCo properties that are being neglected, launch a paid competitor and reap the reward when they shut down. Pinboard and now Newsblur.

Thats actually a really good idea. I wonder what would be next? Most of Yahoo's properties seem be be neglected at the moment, but what about Google? Google News perhaps?

flikr -> openphoto

Re: Three Months to Scale NewsBlur

#114

Post by johncoltrane »

I registered progrss.net a few years ago in the hope of creating a "smart" RSS reader that could help me (and others, possibly for a fee) save time by passing my feeds through easy to set up filters. The idea, sounded good, the mockups and and the prototypes looked good, it was fast and easy to use, the UX was distinctly different from all the others… but I grew tired of RSS as a whole and of woking on progrss.net. Now I don't use RSS anymore and my project was already pretty much stalled when NewsBlur was first announced here.

The day Google announced they retired Reader was "missed opportunity day", over here.

Anyway, good luck to you.

Re: Three Months to Scale NewsBlur

#115

Post by rb2k_ »

jscn wrote: post #70
conesus wrote: post #8

And I just want to remind the technical crowd here that NewsBlur is 100% open-source: http://github.com/samuelclay I've been asked quite a few times why I open-source the code. The answer is simple. Let me use an example from Joel Spolsky. (From http://www.joelonsoftware.com/articles/fog0000000052.html ) The dominant spreadsheet, with 100% market share, is Lotus 123. You're the product manager for Microsoft Excel. As…

Just wanted to say, NewsBlur and Pinboard are both big inspirations for me. It's awesome to see people making money off of open source software.

pinboard isn't open source?

Re: Three Months to Scale NewsBlur

#116

Post by Semaphor »

tempestn wrote: post #107

A tad off topic, but I'm hoping the fastest way to get an answer: How does one move or delete multiple feeds at once in newsblur? There must be a way to do this, right? Say I import 100 feeds from an OPML file, then realize I actually want them all in a folder. At the moment the only way I can figure out how to do that is to click on each feed, click move to folder, click the folder dropdown box, click the folder I w…

Heh, wondered that myself yesterday after I realized my OPML contained old feeds for some reason. Reimporting feeds will delete your current feeds :)

Re: Three Months to Scale NewsBlur

#117

Post by Semaphor »

nicw wrote: post #98
steve19 wrote: post #43

Will you be adding a Google Reader compatible API? (So existing Google Reader apps can quickly switch to NewsBlur)

You can do this currently. My feeds were quickly ingested and everything worked great.

In case you wonder why you are getting downvoted: He is asking about a compatible API for 3rd party apps (Reeder, gReader and so on) to be able to connect to newsblur just by changing the API URL.

Re: Three Months to Scale NewsBlur

#118

Post by Semaphor »

DuskStar wrote: post #85
Maven911 wrote: post #6

Hi, So i have noticed that you are very forthcoming with data, such as providing the number of premium users on your webpage, and I hope you continue this - I really hope you become more succesful with this Google Reader debacle going on. 1) Why are you open about the number of users ? (personally I think its great) 2) How do the real-time stats work, I notice that the amount of regular users was over 20k a few days…

From another comment, those numbers appear to be the new signups in the last 24 hours - and as the free option became much harder to find recently, I would expect the number of free signups to decrease dramatically.

> and as the free option became much harder to find recently

That's only temporary though.

Re: Three Months to Scale NewsBlur

#119

Post by sergiosgc »

rssident wrote: post #71
conesus wrote: post #20

Earlier quoted context omitted.

This is something I'll be working on soon enough. There are many, many dependencies (not only for the web app, but there are three DBs that you have to have installed as prerequisites - mongo, postgres/mysql, and redis). The real problem with setting up your own instance of NewsBlur is that you'll have to do your own feed fetching. This is effectively what you're paying for when you pay for NewsBlur. Let me break it…

You should make an intelligent feed updating algorithm. If you have at least fifty previous entries from a feed it's easy to predict fairly accurately when that feed will need to be updated again. That's how the indexer for http://rssident.com works. Saves you tons of cpu cycles.

Most blog providers support pubsubhubbub(the worst protocol name ever). It allows you to avoid polling, by having the content producer notify you of feed updates.

Re: Three Months to Scale NewsBlur

#120

Post by alexfarran »

rssident wrote: post #75
lazugod wrote: post #41

Earlier quoted context omitted.

I briefly tried both. * Feedly has a simple interface which is much more like Google Reader. But it depends on a plugin (or an app) rather than being an allwhere accessible website. Their Kindle Fire app seemed to load content extremely slowly, and its interface wasn't a simple list. And it's free, so I can't be a customer and I don't know how they earn money. * NewsBlur is complicated and unresponsive. There are a l…

Try http://rssident.com I am working on making the UI more user friendly and adding OPML import/export. It's going to be supported by advertising with a subscription service for premium features and no ads.

I was attracted by the basic looking design, but I can't work out how to do anything. Why am I subscribed to a huge number of random feeds? How do I get rid of them? How do I view the content of the feed instead of opening a new window on the original site?
Post reply on HN