Show HN: Gargl – Create an API for any website, without writing a line of code • Hacker News Board

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#11

Post by jodoglevy » Fri, Feb 21, 2014, 12:54 AM UTC

jaytaylor wrote: post #10

ChuckMcM wrote: post #3

Just as a matter of record you risk getting your IP blacklisted by using something like this without the web sites permission. Perhaps the poster child for web sites that go apeshit is CraigsList but most sites respond in one way or another. One of my favorites are the Markov generated search results that Bing sends back to robots.

You raise interesting and valid points. Notably, there is naturally a rather ubiquitous route around getting blocked: use Tor (it will work in many cases, though not all). The most intriguing thing about this Gargl thing imo is that it is a free version of for-profit SaaS website-to-api offerings such as kimonolabs[0]. I love the nobleness of taking something which is only available as a paid service and creating a f…

OP here.

I did a post comparing Gargl to Kimono actually: http://jodoglevy.com/jobloglevy/?p=146

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#12

Post by tlrobinson » Fri, Feb 21, 2014, 1:12 AM UTC

digitalboss wrote: post #4

FYI Gargl vs Kimono - mentioned at the bottom original article. http://jodoglevy.com/jobloglevy/?p=146

I'll throw my scraper creator in the ring too: http://exfiltrate.org/

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#13

Post by benwilber0 » Fri, Feb 21, 2014, 1:12 AM UTC

Your description of the problem and solution is too verbose. I need bullet points describing 1) my problems, 2) how my problems are solved by this. I'm not going to read a full-on blog post to figure out if this is relevant to me.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#14

Post by stringham » Fri, Feb 21, 2014, 1:46 AM UTC

jheriko wrote: post #5

this is a clever idea - i've had a similar idea many many years ago infact but never followed through because i strongly disagree with making it easy to e.g. abuse google or yahoo by spamming their search engine. as much as i disagree with keeping proprietary secrets i agree more that people should have freedom of choice to do that... in that regard its nice to see the big warning at the top of the page about ease of…

I wouldn't call http an api. Http is a protocol.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#15

Post by marcosdumay » Fri, Feb 21, 2014, 1:52 AM UTC

jheriko wrote: post #5

this is a clever idea - i've had a similar idea many many years ago infact but never followed through because i strongly disagree with making it easy to e.g. abuse google or yahoo by spamming their search engine. as much as i disagree with keeping proprietary secrets i agree more that people should have freedom of choice to do that... in that regard its nice to see the big warning at the top of the page about ease of…

> Here I feel almost like what this does is takes an existing 'API' (http - the internet) and refactors the interface in highly specific ways to make it easier to use...

That's what all libraries do. If we can't call that an API, we'll only be able to use the name for bare I/O operations.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#16

Post by dfgonzalez » Fri, Feb 21, 2014, 2:30 AM UTC

Love these scrapper template generators. I wonder why you chose Java instead of something like PhantomJS to run the scrapper.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#17

Post by nly » Fri, Feb 21, 2014, 3:06 AM UTC

jaytaylor wrote: post #10

ChuckMcM wrote: post #3

Just as a matter of record you risk getting your IP blacklisted by using something like this without the web sites permission. Perhaps the poster child for web sites that go apeshit is CraigsList but most sites respond in one way or another. One of my favorites are the Markov generated search results that Bing sends back to robots.

You raise interesting and valid points. Notably, there is naturally a rather ubiquitous route around getting blocked: use Tor (it will work in many cases, though not all). The most intriguing thing about this Gargl thing imo is that it is a free version of for-profit SaaS website-to-api offerings such as kimonolabs[0]. I love the nobleness of taking something which is only available as a paid service and creating a f…

> there is naturally a rather ubiquitous route around getting blocked: use Tor

How about fuck you? Seriously, don't use Tor for scraping sites. Webmasters can and will block Tor exit nodes if they feel the bad traffic outweighs the good.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#18

Post by eli » Fri, Feb 21, 2014, 3:55 AM UTC

loceng wrote: post #8

I think a better business model would be creating a service that identifies scrapers, and then blocks them. I think one might already exist, though I can't remember its name.

I don't think either is a really great idea. I think most of the people who would pay to block web scrapers are either being paranoid or are being scraped by people smart and resourceful enough to get around your filters. Any serious web scraper is going to be scripting a real browser engine, so it's going to act just like a real visitor.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#19

Post by hawkharris » Fri, Feb 21, 2014, 4:13 AM UTC

nly wrote: post #17

jaytaylor wrote: post #10

Earlier quoted context omitted.

You raise interesting and valid points. Notably, there is naturally a rather ubiquitous route around getting blocked: use Tor (it will work in many cases, though not all). The most intriguing thing about this Gargl thing imo is that it is a free version of for-profit SaaS website-to-api offerings such as kimonolabs[0]. I love the nobleness of taking something which is only available as a paid service and creating a f…

> there is naturally a rather ubiquitous route around getting blocked: use Tor How about fuck you? Seriously, don't use Tor for scraping sites. Webmasters can and will block Tor exit nodes if they feel the bad traffic outweighs the good.

You have a valid point, but the "fuck you" is unnecessary. I hope that HN will stay place where people can have civil debates, drawing on evidence instead of emotions.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#20

Post by sirclueless » Fri, Feb 21, 2014, 4:27 AM UTC

hawkharris wrote: post #19

nly wrote: post #17

Earlier quoted context omitted.

> there is naturally a rather ubiquitous route around getting blocked: use Tor How about fuck you? Seriously, don't use Tor for scraping sites. Webmasters can and will block Tor exit nodes if they feel the bad traffic outweighs the good.

You have a valid point, but the "fuck you" is unnecessary. I hope that HN will stay place where people can have civil debates, drawing on evidence instead of emotions.

It sounds like a very purposeful and directed "fuck you" to me. He's not being rude or crass, he's making the concise and vehement point that if you are a bad actor on Tor, you are harming everyone and he will hate you for it.

Enlightening debates don't come about when everyone mutes their politically incorrect emotions and speaks in platitudes, they come about when people respect each other and can speak simple truths as they would to their peers. Evidence is always useful, but rational argument is equally important.