Hnweekly: weekly top stories from Hacker News

Post by rtw » Fri, Dec 26, 2008, 5:33 PM UTC

xtimesninety wrote: post #20

rtw wrote: post #19

Earlier quoted context omitted.

I have a screenscraping thing in place that targets an ASCII only environment (LambdaMOO mud). Manually replacing unicode stuff like this has been a pain in the butt. Luckily that was all just for fun and didn't need to be perfect. Is there a good library out there (in any language) that does good unicode --> ASCII substitutions for major languages?

I once used latin1_to_ascii (The Unicode Hammer) in python: (works great) http://code.activestate.com/recipes/251871/

Post by russell » Fri, Dec 26, 2008, 6:10 PM UTC

xtimesninety wrote: post #15

tome wrote: post #12

Doesn't seem to handle non-ASCII very well: "Does Gdel matter?"

I might have problems with unicode in PHP (I really don't know much here yet) so I opted to remove them (for now) using regex. btw I'm using http://simplehtmldom.sourceforge.net/ to scrape/parse which is great, it let's you use jquery style selectors on html.

Post by scorxn » Fri, Dec 26, 2008, 7:11 PM UTC

xtimesninety wrote: post #15

tome wrote: post #12

Doesn't seem to handle non-ASCII very well: "Does Gdel matter?"

I might have problems with unicode in PHP (I really don't know much here yet) so I opted to remove them (for now) using regex. btw I'm using http://simplehtmldom.sourceforge.net/ to scrape/parse which is great, it let's you use jquery style selectors on html.

Post by xtimesninety » Sat, Dec 27, 2008, 3:37 AM UTC

siong1987 wrote: post #26

xtimesninety wrote: post #10

Earlier quoted context omitted.

nice work too :)

How long do your crawl the Ycombinator once? Do you actually crawl every page on YC?

Post by siong1987 » Sat, Dec 27, 2008, 9:42 AM UTC

xtimesninety wrote: post #27

siong1987 wrote: post #26

Earlier quoted context omitted.

How long do your crawl the Ycombinator once? Do you actually crawl every page on YC?

I scrape only the front page once every ~4hrs (minimum) Nearlyfreespeech.net doesn't have cron yet, so I'm using onlinecronjobs.com. If you want to scrape it on demand just go to http://hnweekly.watdahel.com/index/scrape