Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Gargl – Create an API for any website, without writing a line of code

jodoglevy.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#31

Post by brey » 

  if you can see or submit data using the website, it means the website does 
  have some kind of API ... For example, if you do a search in Yahoo, you can
  see the search page sent to Yahoo’s servers has the following url ...

  https://search.yahoo.com/search?p=search+term
no. no no no. this is not an API. this is about as far from an application programming INTERFACE as it can get. this means an agreed format, where there's a contract (social or otherwise) to provide stability to APPLICATION clients. there's no contract here other than 'a human types something into the box, presses some buttons and some results appears on the website'.

/search?p=search+term is an implementation detail hidden from the humans the site is built for. they can, and most likely will, change this at any time. the HTML returned (and being scraped) is an implementation detail. today, HTML. tomorrow, AJAX. next week? who knows, maybe Flash.

fine, it's a scraper builder. but don't call what it's using an API, and don't imply it's anything more than a fragile house of cards built on the shaky foundation of 'if you get noticed you're going to get banned or a cease and desist'.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#32

Post by matznerd »

platz wrote: post #25
dfgonzalez wrote: post #16

Love these scrapper template generators. I wonder why you chose Java instead of something like PhantomJS to run the scrapper.

um, the whole thing is language agnostic, no?

The APIs generated are agnostic, but the tool to create them is java based.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#34

Post by walden42 »

tlrobinson wrote: post #28
walden42 wrote: post #22

Earlier quoted context omitted.

Nice, I like it! Pretty easy to use. Only problem is that I couldn't download any file =) When I try downloading a file, it just shows a green box saying it downloaded.

Thanks. Which browser are you using?

Firefox 26 on Linux Mint. It works fine in Chrome. There is no scrollbar on the page you're viewing, either.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#35

Post by eli »

matznerd wrote: post #30
eli wrote: post #18

Earlier quoted context omitted.

I don't think either is a really great idea. I think most of the people who would pay to block web scrapers are either being paranoid or are being scraped by people smart and resourceful enough to get around your filters. Any serious web scraper is going to be scripting a real browser engine, so it's going to act just like a real visitor.

There are ways to detect scrapers and other bots if you really want to and services that do so.

Adding a captcha to every page, maybe? There are services that will charge you money for this, but that doesn't mean it works.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#36

Post by matznerd »

eli wrote: post #35
matznerd wrote: post #30

Earlier quoted context omitted.

There are ways to detect scrapers and other bots if you really want to and services that do so.

Adding a captcha to every page, maybe? There are services that will charge you money for this, but that doesn't mean it works.

Captchas don't do anything to stop bots, they just add a small additional cost(~$1.40 per 1000 solved). I am talking about monitoring things that 90% of bots generally do not take precautions against, like tracking mouse movements and other things I won't mention here, that distinguish them from humans.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#37

Post by loceng »

matznerd wrote: post #30
eli wrote: post #18

Earlier quoted context omitted.

I don't think either is a really great idea. I think most of the people who would pay to block web scrapers are either being paranoid or are being scraped by people smart and resourceful enough to get around your filters. Any serious web scraper is going to be scripting a real browser engine, so it's going to act just like a real visitor.

There are ways to detect scrapers and other bots if you really want to and services that do so.

Do you know the names of any such services off the top of your head? Thanks

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#38

Post by platz »

matznerd wrote: post #32
platz wrote: post #25

Earlier quoted context omitted.

um, the whole thing is language agnostic, no?

The APIs generated are agnostic, but the tool to create them is java based.

Well the reference generator is in java, but take a look at the github repo, there is nothing preventing adding an additional generator in the /generators directory.

All you really need to do is output the template.

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#39

Post by eli »

matznerd wrote: post #36
eli wrote: post #35

Earlier quoted context omitted.

Adding a captcha to every page, maybe? There are services that will charge you money for this, but that doesn't mean it works.

Captchas don't do anything to stop bots, they just add a small additional cost(~$1.40 per 1000 solved). I am talking about monitoring things that 90% of bots generally do not take precautions against, like tracking mouse movements and other things I won't mention here, that distinguish them from humans.

I don't believe you. At best you can obfuscate and confuse scrapers. You can't stop them from reading a public web page. (And I shudder to think what these solutions must do to accessibility -- hope you don't have any blind readers.)

Re: Show HN: Gargl – Create an API for any website, without writing a line of code

#40

Post by matznerd »

eli wrote: post #39
matznerd wrote: post #36

Earlier quoted context omitted.

Captchas don't do anything to stop bots, they just add a small additional cost(~$1.40 per 1000 solved). I am talking about monitoring things that 90% of bots generally do not take precautions against, like tracking mouse movements and other things I won't mention here, that distinguish them from humans.

I don't believe you. At best you can obfuscate and confuse scrapers. You can't stop them from reading a public web page. (And I shudder to think what these solutions must do to accessibility -- hope you don't have any blind readers.)

Oh, I wasn't saying you can completely stop them from reading a page or individual pages. But there is activity, than can be detected as irregular. Here is a true example I know about someone who wanted to scrape their competitor's client listings. The competitor had a map with points of their customers with random user IDs and no where was the entire dataset visible. The person just built a scraper/bot, to hit every single possible ID of over 10,000 numbers. They hit a ton of empty pages, and that company should have recognized an IP incrementally crawling their data, especially empty pages...This activity should have been recognized and resulted in an IP ban.
Post reply on HN