Live data from Hacker News

Show HN: Gargl – Create an API for any website, without writing a line of code

jodoglevy.com

41–44 of 44 posts

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#41
post #37

Earlier quoted context omitted.

There are ways to detect scrapers and other bots if you really want to and services that do so.

Do you know the names of any such services off the top of your head? Thanks

If you want to learn about the concept and armsrace, here is a paper with plenty of resources (this is in a game context, though not website, although there is the most advanced detection here): http://iseclab.org/papers/botdetection-article.pdf

Here is an open source system demo'd at BlackHat Europe 2011 (that checks it is a proper browser (with DOM/Javascript/etc), also good against DDoS. https://github.com/yuri-gushin/Roboo

Project Honeypot (scans inbound ips) good against spambots: https://www.projecthoneypot.org/

Here are some commercial solutions: CloudFlare's ScrapeShield -https://www.cloudflare.com/apps/scrapeshield

Distil Networks - http://www.distilnetworks.com/

Scrape Sentry - http://www.scrapesentry.com/

Fireblade - http://www.fireblade.com/

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#42
I do this kind of stuff with wget, sed, awk so far, but it's nice to see some more thought-out alternatives.

What I like most about your competition though is the JS interface that gets used for one good last thing (before being properly scraped and de-AD- and de-java-fied): clicking on the content you want, and deselecting content you don't want: subtly, with your mouse you lead a pattern-matching algorithm doing the annoying work.

Honestly the simplicity of this interface is even more breathtaking to me than gargl :P But it's even more limited, as after clicking twice it thinks that it has understood the pattern already, although that might not be the case.

I'd suggest to integrate the idea, but to make the learning process more clever, make it possible to select more things, even though the engine thinks there can't be any more similar things. Give that AI more things to learn from. We want more identifiers than just counts and HTML elements: "2nd subelement of ".

There's good stuff you can do with statistics, too. Some data exists only once, some exists only 3 times, some always exists over 10 times. That's valuable info. Some data has many words of whitespace seperated text - oh a paragraph!

tldr We need something that generates good semantics out of normal web sites automatically, so that users can use a simple Web UI mangled into the target web site to choose the right pattern.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#43
post #37

Earlier quoted context omitted.

Do you know the names of any such services off the top of your head? Thanks

If you want to learn about the concept and armsrace, here is a paper with plenty of resources (this is in a game context, though not website, although there is the most advanced detection here): http://iseclab.org/papers/botdetection-article.pdf Here is an open source system demo'd at BlackHat Europe 2011 (that checks it is a proper browser (with DOM/Javascript/etc), also good against DDoS. https://github.com/yuri-gu…

Great. Thank you.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#44
post #5

this is a clever idea - i've had a similar idea many many years ago infact but never followed through because i strongly disagree with making it easy to e.g. abuse google or yahoo by spamming their search engine. as much as i disagree with keeping proprietary secrets i agree more that people should have freedom of choice to do that... in that regard its nice to see the big warning at the top of the page about ease of…

I wouldn't call http an api. Http is a protocol.

any interface that is designed to be, and can be used by, software is technically an api...

the thing i was trying to stab at was the recent popularity of 'API' as a term and the way it is applied...

Post reply on HN