How to build a data mining web app? • Hacker News Board

Re: How to build a data mining web app?

#11

Post by zeratul » Fri, Dec 02, 2011, 9:06 PM UTC

rossmasters wrote: post #8

eschutte2 wrote: post #4

Earlier quoted context omitted.

Maybe think of it as two different graphs with the same x-axis, superimposed?

That's all this is really. The left Y axis applies to the sets that involve an average score, the right Y axis applies to pickup ratio only. That said, what is a pickup ratio? Some sort of measure of extra votes over time?

Pickup ratio suppose to measure your chance of getting from "newest" page to the "news" page. But the more I think about it the more I'm inclined to think it's just a very hot topic detection device.

Pickup ratio > 1 means that the "newest" page has couple very interesting stories that are also seen on the "news" page.

Pickup ratio > 2 means that there is some sort of hysteria on HN about some particular topic ...

Re: How to build a data mining web app?

#12

Post by jianxioy » Fri, Dec 02, 2011, 9:21 PM UTC

This is a really cool project. Nice work! :)

P.S. It seems like you're following me on GitHub. Makes me wonder if I know you. Email me to introduce yourself! :)

Re: How to build a data mining web app?

#13

Post by rhnet » Fri, Dec 02, 2011, 9:56 PM UTC

Thanks for linking to my Stack Overflow by Day post in the readme!

Re: How to build a data mining web app?

#14

Post by eric-hu » Fri, Dec 02, 2011, 10:25 PM UTC

zeratul wrote: post #3

mapleoin wrote: post #2

I must be stupid, but I just can't wrap my head around this three-axis-in-a-2D-space graph: http://hnpickup.appspot.com/ Is this normal?

If you click on the question mark in the top right corner you will get the definitions. More documentation is in the source code. EDIT: As I say in the "warning" section of the readme file: usually it's not a good idea to show the raw data because it will confuse the end user. And if you add more explanation you will cause clutter and cognitive overload. I show the raw data mostly for people that want to dig into the…

On the topic of confusion, I think your demo would be clearer if you renamed "News stories bottom average score" to "Front page stories bottom average score". News is just a little too close to Newest, so I had to reread the definitions of both a couple of times. Cool project otherwise!

Re: How to build a data mining web app?

#15

Post by tibbon » Fri, Dec 02, 2011, 10:59 PM UTC

The hardest thing I find about writing a scraper/miner/api consumer is the HUGE amount of irregular data that you have to check for.

I was attempting to write a crawler for a pretty big dynamic site. It worked, but the code ended up being so messy because of all the weird quirks I had to check for non-stop.

Re: How to build a data mining web app?

#16

Post by zeratul » Fri, Dec 02, 2011, 11:16 PM UTC

zeratul wrote: post #10

salimmadjd wrote: post #9

Be warned! Google App Engine Pricing has gone up by as much as 500% since Nov 7. The code is based on GAE and you just need be warned that if you spend time and build around it, be ready for some crazy billing. I'm talking from personal experience :(

Yes!!! I was hoping to have pay-per-impression advertisement that would pay the bill but apps which don't offer wordy clutter are not allowed to have ads. I was rejected by adSense, BuySellAds, and adBrite. Sometimes I think we build Internet for robots not for humans. I have ~100 visits per day. That costs me ~$0.10 per day. More details here: https://github.com/entaroadun/hnpickup/wiki/Hacker-News-Pick...

Just got an email from Kontera ads:

  Denial Reason: Insufficient Text.

What happen to saying: "picture worth thousand words"?

Re: How to build a data mining web app?

#17

Post by lurker17 » Fri, Dec 02, 2011, 11:47 PM UTC

zeratul wrote: post #3

mapleoin wrote: post #2

I must be stupid, but I just can't wrap my head around this three-axis-in-a-2D-space graph: http://hnpickup.appspot.com/ Is this normal?

If you click on the question mark in the top right corner you will get the definitions. More documentation is in the source code. EDIT: As I say in the "warning" section of the readme file: usually it's not a good idea to show the raw data because it will confuse the end user. And if you add more explanation you will cause clutter and cognitive overload. I show the raw data mostly for people that want to dig into the…

Actually, clicking anywhere on the graph will flip to the definitions, which is very bad UX.

Re: How to build a data mining web app?

#18

Post by lurker17 » Fri, Dec 02, 2011, 11:48 PM UTC

zeratul wrote: post #10

salimmadjd wrote: post #9

Be warned! Google App Engine Pricing has gone up by as much as 500% since Nov 7. The code is based on GAE and you just need be warned that if you spend time and build around it, be ready for some crazy billing. I'm talking from personal experience :(

Yes!!! I was hoping to have pay-per-impression advertisement that would pay the bill but apps which don't offer wordy clutter are not allowed to have ads. I was rejected by adSense, BuySellAds, and adBrite. Sometimes I think we build Internet for robots not for humans. I have ~100 visits per day. That costs me ~$0.10 per day. More details here: https://github.com/entaroadun/hnpickup/wiki/Hacker-News-Pick...

Put current HN headlines on the page to generate content for ads.

Re: How to build a data mining web app?

#19

Post by groby_b » Sat, Dec 03, 2011, 1:00 AM UTC

zeratul wrote: post #10

salimmadjd wrote: post #9

Be warned! Google App Engine Pricing has gone up by as much as 500% since Nov 7. The code is based on GAE and you just need be warned that if you spend time and build around it, be ready for some crazy billing. I'm talking from personal experience :(

Yes!!! I was hoping to have pay-per-impression advertisement that would pay the bill but apps which don't offer wordy clutter are not allowed to have ads. I was rejected by adSense, BuySellAds, and adBrite. Sometimes I think we build Internet for robots not for humans. I have ~100 visits per day. That costs me ~$0.10 per day. More details here: https://github.com/entaroadun/hnpickup/wiki/Hacker-News-Pick...

Looking at the details:

3.8 million data store reads for 8,000 visitors seems high. Especially for the kind of data you are serving - it's not rapidly changing.

Are you using memcache at all? A quick scan of the source doesn't show any memcache imports, but I just might've missed it. (http://code.google.com/appengine/docs/python/memcache/usingm...)

Re: How to build a data mining web app?

#20

Post by zeratul » Sat, Dec 03, 2011, 1:32 AM UTC

groby_b wrote: post #19

zeratul wrote: post #10

Earlier quoted context omitted.

Yes!!! I was hoping to have pay-per-impression advertisement that would pay the bill but apps which don't offer wordy clutter are not allowed to have ads. I was rejected by adSense, BuySellAds, and adBrite. Sometimes I think we build Internet for robots not for humans. I have ~100 visits per day. That costs me ~$0.10 per day. More details here: https://github.com/entaroadun/hnpickup/wiki/Hacker-News-Pick...

Looking at the details: 3.8 million data store reads for 8,000 visitors seems high. Especially for the kind of data you are serving - it's not rapidly changing. Are you using memcache at all? A quick scan of the source doesn't show any memcache imports, but I just might've missed it. ( http://code.google.com/appengine/docs/python/memcache/usingm... )

The content is refreshed every 15 min - both database and UI.

Probably the production version of the code will get memcache and some other features. Apart of couple suggestions from the discussion here the educational version of hnpickup will be frozen. Although, any simplifications or correction to the code or documentation are very welcome.