The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...
There are multiple internal tools I use at work (JIRA, our ticketing system, our code review tool) that won't work because of this issue. In the meantime, I've written Tampermonkey scripts that will scrape and embedd multiple pages all hack-like, but at least I get a good CSV of the data I need. To me, the value in this tool is the user interface for creating the scrape logic. If this ran as an embeddable JS app, tha…
Portia, an open-source visual web scraper
51–60 of 70 posts
Re: Portia, an open-source visual web scraper
#52Earlier quoted context omitted.
There are multiple internal tools I use at work (JIRA, our ticketing system, our code review tool) that won't work because of this issue. In the meantime, I've written Tampermonkey scripts that will scrape and embedd multiple pages all hack-like, but at least I get a good CSV of the data I need. To me, the value in this tool is the user interface for creating the scrape logic. If this ran as an embeddable JS app, tha…
Why would scrape JIRA when they have a perfectly workable API?
Re: Portia, an open-source visual web scraper
#53Earlier quoted context omitted.
CasperJS is a higher-level wrapper for PhantomJS, so - yes, it could be done with PhantomJS solely... But you wouldn't want to, because CasperJS makes automation easier.
Are there any libraries to facilitate database-connectivity to SQL Server or MySQL from javascript? I've used CasperJS to scrape some sites, but always fall back on post-processing the scraped data with another program in order to get it into my database. I'd love to be able to do it all from one piece of code.
Re: Portia, an open-source visual web scraper
#54Re: Portia, an open-source visual web scraper
#55Earlier quoted context omitted.
Why would scrape JIRA when they have a perfectly workable API?
While their APIs are nice, they require separate permissions and having access to them isn't always a given depending on the company that runs/owns the Jira instance.
Re: Portia, an open-source visual web scraper
#56I like that there's people working to make scraping easier and friendly for everyone. Sadly (IMHO) the cases where these tools will probably fail are at the same time the same not really open on providing the data directly. Most scraper-unfriendly sites would make you request another page before to capture a cookie, set cookies on the request headers or a referer entry, or manually using regex magic to extract inform…
Re: Portia, an open-source visual web scraper
#57I like that there's people working to make scraping easier and friendly for everyone. Sadly (IMHO) the cases where these tools will probably fail are at the same time the same not really open on providing the data directly. Most scraper-unfriendly sites would make you request another page before to capture a cookie, set cookies on the request headers or a referer entry, or manually using regex magic to extract inform…
Surely there are difficulties in expecting data providers to produce their data in standard formats across industries and countries? I am naive as to how much and what data is available but that seems a stretch
There are a lot of accessible sources (though, not documented), but there are also clear examples on how one would never provide a service! Some examples [3, 4]
What I was referring, though, was in a way to avoid having to build an intermediate server scraping services that are perfectly usable (JSON, XML) just because we (all) prefer to build clients that understand one type of feed (standard).
Maybe it's not about designing a language, but just as a new way of doing things. Let's say I provide the client with the clear instructions on how to use a service (its format, and where are the fields that the client understands (in an XPath-like syntax)).
That should be enough to avoid periodically scraping good-player servers, but at the same time being able to build client apps without having to implement all the differences between feeds. Besides, it would avoid being banned for accessing too much times a service, and would give data providers insight on who is really using their data.
Let's say we want to unify the data in Feed A and Feed B. The model is about foos and bars:
Feed A:
{
"status": "ok",
"foobars": [
{
"name": "Foo",
"bar": "Baz"
}, ...
]
}
Feed B
[{"n": "foo","info": {"b": "baz"}},...]
We could provide:
{
"feeds": [
{
"name": "Feed A",
"url": "http://feed.a",
"format": "json",
"fields": {
"name": "/foobars//name",
"bar": "/foobars//bar"
}
},
{
"name": "Feed B",
"url": "http://feed.b",
"format": "json",
"fields": {
"name": "//n",
"bar": "//info/b"
}
]
}
Instead of providing a service ourselves that accesses Feed A and Feed B
every minute just because we want to ease things on the client.
Not sure if that's what you asked, though.[1]: http://citybik.es
[2]: http://github.com/eskerda/pybikes
[3]: https://github.com/eskerda/PyBikes/blob/experimental/pybikes...
[4]: https://github.com/eskerda/PyBikes/blob/experimental/pybikes...
Re: Portia, an open-source visual web scraper
#58Re: Portia, an open-source visual web scraper
#59The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...
There are multiple internal tools I use at work (JIRA, our ticketing system, our code review tool) that won't work because of this issue. In the meantime, I've written Tampermonkey scripts that will scrape and embedd multiple pages all hack-like, but at least I get a good CSV of the data I need. To me, the value in this tool is the user interface for creating the scrape logic. If this ran as an embeddable JS app, tha…
Re: Portia, an open-source visual web scraper
#60The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...
There are multiple internal tools I use at work (JIRA, our ticketing system, our code review tool) that won't work because of this issue. In the meantime, I've written Tampermonkey scripts that will scrape and embedd multiple pages all hack-like, but at least I get a good CSV of the data I need. To me, the value in this tool is the user interface for creating the scrape logic. If this ran as an embeddable JS app, tha…