Earlier quoted context omitted.
What is tendancy ?
trends sorry .... english is not my native speaking.
No problem, I thought maybe I don't understand some latest techno-speak ;-)
Ask HN: What info do you web scrape for?
71–80 of 115 posts
Re: Ask HN: What info do you web scrape for?
#72I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…
Regarding 2)
Why wouldn't it work for PDF's? If you're able to get the file itself, you should be able to OCR it...
Is there anything obvious that I am missing in regards to PDFs?
Re: Ask HN: What info do you web scrape for?
#73I scrape Gumtree and eBay hourly using a python script for certain things I want under a certain price. The script sends me an email with the link in it and I get on top of it sharpish. Managed to bag a lot of stuff over the last couple of years for not much money. If someone bags this up as a service I'd pay for it.
@allegory could you share the the script with us ?
Would love to but not at the minute because it has hard coded credentials for eBay API in it. It's on my list as a TODO to tidy it up. Will stuff on github and post a Show HN on it soon :)
I've got one that monitors amazon prices for sudden lows as well.
Re: Ask HN: What info do you web scrape for?
#74Re: Ask HN: What info do you web scrape for?
#75I currently scrap my lending club account to automatically trade loan notes on the secondary market. This way I can buy/sell notes that satisfy my criteria. If anyone is interested in this, I can send you the scripts.
Re: Ask HN: What info do you web scrape for?
#76I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…
Regarding 2) Why wouldn't it work for PDF's? If you're able to get the file itself, you should be able to OCR it... Is there anything obvious that I am missing in regards to PDFs?
I've worked with OCRed PDFs, the main thing that should be obvious is that OCR results range from poor to horrendous. It takes a lot of manual cleanup if a high degree of accuracy is required. Or depending on why you want the text, you can adjust expectations or add layers of software such as fuzzy search algorithms to deal with the issues.
Again depending on the application, the mixed quality of OCR isn't always a deal breaker, but it's not always as simple as it might appear.
Re: Ask HN: What info do you web scrape for?
#77I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…
Regarding 2) Why wouldn't it work for PDF's? If you're able to get the file itself, you should be able to OCR it... Is there anything obvious that I am missing in regards to PDFs?
It's not the text that's the issue, it's the structure. PDFs have nowhere near as much structure as markup. You end up having to do this for dozens of layouts and it gets hurty really fast:
http://schoolofdata.org/2013/06/18/get-started-with-scraping...
Re: Ask HN: What info do you web scrape for?
#78I'm working on a startup that has web scraping at its core. The vision is a bit larger and includes fusing data from various sources in a probabilistic way (e.g. the same people, products, or companies found on different sides with ambiguous names and information. This is based on the research I've doen at uni). However, I found that there are no web crawling frameworks out there that allow for large-scale and contin…
>I found that there are no web crawling frameworks out there that allow for large-scale and continuous crawling of changing data.
Are you distinguishing between "I found that there are no" and "I didn't find any so far"?
Which ones that came close have you rejected, and why?
Re: Ask HN: What info do you web scrape for?
#79There is a closet industry for scraping any sort of data that can move markets. Fed, crop, weather, employment,etc. Anything that is released at a certain time on a fixed calendar, you can bet that multiple parties are trying to scrape it as fast as possible. If you can scrape this data( the easy part), put it in a structured format( somewhat hard) and deliver it in under a few seconds(this is where you get paid) the…
>It's an interesting niche that hasn't been computerized yet.
That's quite an assertion. I'm certain it has been.
Re: Ask HN: What info do you web scrape for?
#80Earlier quoted context omitted.
We have a non-tech intern and import.io looks like like a great tool to get him chewing up data. I'm playing with it now. Why didn't it work out for you? Beyond the wrapped browser interface being a little funky lol. (Edit: eugh, selecting data for import is really clunky.) Ask HN: Anybody got a visual scraping service they like?
It was the data extraction and selection process I couldn't get to work. I was trying to scrape a particular search on autotrader.co.uk (I wanted more up to date results than their daily emails provide, and I wanted to filter out cars that had been written off). I don't remember all the details, but I followed the tutorial video and got to the stage where you select a single item that matches your criteria and it's s…
I found you often have to select two, then it figures it out. I assumed it was probably because of alternating odd/even row CSS classes.