Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

wrapapi.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#41

Post by webninja »

wrapapi wrote: post #39
webninja wrote: post #35

This tool is really well thought-out and useful! I made a working API in less than 1 hr. This tool has a much better design & implementation than Kimono and easier than using Python 3 + Beautiful Soup 4 which is how I made my previous web scrapers. This tool also works for POSTing to web forms.

Thanks webninja! The POSTing part is one of the biggest things we were trying to get right while not making it any harder to use than Kimono. Is there anything that was confusing when trying to learn it for the first time? If so, we're still trying to make it easier =)

I think it's pretty close to perfect. I made a sample WrapAPI for http://etfdailynews.com/etf/{{symbol}}/ where the symbol is a 3-4 character ETF ticker symbol like VYM. It's one of the only websites I've found that provides the full breakdown of an ETF's contents. Initially I wasn't sure where to provide sample input such as a handful of ticker 3 character strings since /{{symbol}}/ isn't a GET or a POST value. So under the JSON and Table column I had multiple symbols separated by commas and it took me a little while before I realized that you were only supposed to supply one test value and where I was supposed to supply that test value. But it "clicked" shortly after.

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#42

Post by OhSoHumble »

I've always felt that I service like this would be great for Code for America projects. A big problem I have with creating technology applications for civic goods is that the government is terrible at providing open data. Even when they do open source civic data, they do a terrible job of it.

An example of this is California drought data: automatically grabbing data on the drought is incredibly difficult because it involves scraping HTML tables. I tried to build an API that presents drought data so volunteers would have an easier time building out data visualizations. I ended up just getting exhausted doing all the scraping work.

I then moved onto a new project: building a free-to-use Padmapper for affordable housing. The data for income restricted apartment units are driven by a government contracted vendor. A city county will declare income stabilization policies and legally enforce them against landowners and then the landowners would send over their list of units to the vendor.

This would be great except the vendor does the bare minimum. Padmapper looks amazing but, really, it's only applicable for the upper middle class due to explosive housing costs in the Bay Area. So, in order to provide a more modern website and mobile application for the community, I started to scrape the vendor's website. It was terrible. I kept getting throttled. So I gave up.

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#44

Post by wolco »

cookiecaper wrote: post #38
iambpentameter wrote: post #20

Does this violate any current US law?

The software itself probably wouldn't, but the use of it for anything anyone cares about probably would. The CFAA, etc., make unwanted scraping illegal and this has been tested repeatedly in court. The company that runs this software as a service needs to be very careful. 3Taps was similar and got destroyed for relaying data scraped from Craigslist. Contacting the server after its operator has expressed its wish for…

Another court stated that copying data into a ram buffer for under 1.2 seconds was allowed. Depending on how they structure this it might be legally allowed.

https://books.google.ca/books?id=a-yu2-JUQNAC&pg=PT249&lpg=P...

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#45

Post by dhruvkar »

OhSoHumble wrote: post #42

I've always felt that I service like this would be great for Code for America projects. A big problem I have with creating technology applications for civic goods is that the government is terrible at providing open data. Even when they do open source civic data, they do a terrible job of it. An example of this is California drought data: automatically grabbing data on the drought is incredibly difficult because it i…

What was the vendor's website?

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#46

Post by wrapapi »

OhSoHumble wrote: post #42

I've always felt that I service like this would be great for Code for America projects. A big problem I have with creating technology applications for civic goods is that the government is terrible at providing open data. Even when they do open source civic data, they do a terrible job of it. An example of this is California drought data: automatically grabbing data on the drought is incredibly difficult because it i…

@OhSoHumble: this actually came to mind for us too in terms of building better experiences on top of common government services. I'd love to chat and see how I can help, so shoot me an email at peter@wrapapi.com

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#47

Post by startupdiscuss »

The problem with opaque pricing is this: people don't want to start experimenting with something if it could be infinitely bad. i.e. if they can imagine the worst.

In the back of their heads, some people imagine the service is going to be huge, and then they worry that all the profits will be paid out to wrapapi.

Better to have a high headline number and then offer discounts for certain uses (non-profit, open source, students, etc). People are optimistic about how much money they might make so a high headline future price for when you graduate from the free tier is not necessarily bad.

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#49

Post by superasn »

This is reallly neat. But I think you guys are losing a lot of potential customers by not having a clear cut pricing on you site.

I've used similar services like parsehub.com in the past and if they didn't have a pricing page I would have never tried it. Just my 2 cents.

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#50

Post by OhSoHumble »

dhruvkar wrote: post #45
OhSoHumble wrote: post #42

I've always felt that I service like this would be great for Code for America projects. A big problem I have with creating technology applications for civic goods is that the government is terrible at providing open data. Even when they do open source civic data, they do a terrible job of it. An example of this is California drought data: automatically grabbing data on the drought is incredibly difficult because it i…

What was the vendor's website?

socialserve.com
Post reply on HN