Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website
51–60 of 62 posts
Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website
#52Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website
#53This tool is really well thought-out and useful! I made a working API in less than 1 hr. This tool has a much better design & implementation than Kimono and easier than using Python 3 + Beautiful Soup 4 which is how I made my previous web scrapers. This tool also works for POSTing to web forms.
I often see one or more commenters write what seems like an excessively positive thought dump on Show HNs. It just doesn't seem like the natural conversational tone everyone uses, but I can't quite put my finger on it.
Has anyone else noticed it? Is there a term for this sort of writing style?
Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website
#54"Price: Contact us" Why?
Likely because they have custom pricing based loosely on how much business value they create for the customer. E.g. if a philatelist wanted to scrape stamp catalogs, and if an industry-specific analytics platform wanted to scrape a directory of prospects - you'd want two different prices. Otherwise, you'd either 1) leave stamp enthusiasts out in the rain, or 2) leave a whole lot of meat on the bone w/r/t enterprise p…
When I was last working inside an organization and reviewing vendors for a product, it really left a bad taste in my mouth when they had "Ask for Pricing." I get it, my consulting work is basically Ask for Pricing, I understand the business strategy. But it's such a headache to sit through bullshit product demos for multiple vendors over a few weeks just to hear that their pricing structure is way out of line.
There is this idea that a lot of companies have, where they're more "professional" or conversion-optimized by removing public pricing and putting everyone through a sales funnel. But that concept only works if 1) you have a great product and 2) you have a great sales team, capable of making my time to failure in the conversion process fast and painless. Every company thinks they have this, but they almost never do. I really don't think you want to optimize your business for keeping stamp enthusiasts happy.
Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website
#55Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website
#56Is this happening on your site? If not, would appreciate some tips about coding it and how to handle exception cases where the wizard can't keep in sync or user click on unintended page elements.
Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website
#57Earlier quoted context omitted.
The software itself probably wouldn't, but the use of it for anything anyone cares about probably would. The CFAA, etc., make unwanted scraping illegal and this has been tested repeatedly in court. The company that runs this software as a service needs to be very careful. 3Taps was similar and got destroyed for relaying data scraped from Craigslist. Contacting the server after its operator has expressed its wish for…
Another court stated that copying data into a ram buffer for under 1.2 seconds was allowed. Depending on how they structure this it might be legally allowed. https://books.google.ca/books?id=a-yu2-JUQNAC&pg=PT249&lpg=P...
Thanks for that! Like I said, I'm not a lawyer and I'm sure there are other gaps in my case knowledge. It's certainly positive to see the Second Circuit recognizing that there is some need to consider the transient nature of RAM copies before ruling them infringing.
The ruling suggests that MAI v. Peak did not address the transitory argument merely because it was not raised by the litigants, and that the precedent set there (which wouldn't have necessarily been binding anyway) is therefore not abrogated by ruling that some RAM copies are transient enough to fail to qualify.
Importantly, the durations listed here describe the runtime of the content, not the amount of time the data is held in the RAM. It is said that the system would buffer 0.1 seconds (100ms) of content at one point and 1.2 seconds of content at another point.
The Court does not seem to establish "1.2 seconds" as a general benchmark for RAM transience, but rather it suggests that transience should be considered on a case-by-case basis, per the language of the statute.
However, the general rule of thumb is that if a copy exists long enough to derive any value from it, it is non-transient. Guidance from the Copyright Office [0] reads:
>[...] we believe that Congress intended the copyright owner’s exclusive right to extend to all reproductions from which economic value can be derived. The economic value derived from a reproduction lies in the ability to copy, perceive or communicate it. Unless a reproduction manifests itself so fleetingly that it cannot be copied, perceived or communicated, the making of that copy should fall within the scope of the copyright owner’s exclusive rights. The dividing line, then, can be drawn between reproductions that exist for a sufficient period of time to be capable of being "perceived, reproduced, or otherwise communicated" and those that do not. As a practical matter, as discussed above, this would cover the temporary copies that are made in RAM in the course of using works on computers and computer networks.
and scrapers have been held liable for copyright infringement via RAM copies on multiple occasions. Ticketmaster v. RMG states:
>[...] copies of ticketmaster.com webpages automatically stored on a viewer's computer are “copies” within the meaning of the Copyright Act.
despite the fact that they likely would've been held for a much shorter time than either 100ms or 1.2 seconds.
Notably, this was before the case referenced above, but it's typical of later cases, and it succinctly demonstrates that courts are likely to find RAM copies of an entire work (the web page) more likely to be of non-transitory nature than snippets of ~ 1/1500th of an entire work, regardless of how long they're stored in RAM.
[0; PDF] https://www.copyright.gov/reports/studies/dmca/sec-104-repor...
Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website
#58Suggestion: Please be upfront about pricing. I've been bitten with similar tool dying because they were too shy to ask for money.
Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website
#59By the way your Onboarding step-by-step wizard[1] is really awesome. I've used similar scripts on my sites before but they keep breaking because the users often click on some div or button (or due to mobile phones) not intended (they're only learning) and then wizard can't sync to the next step and the whole thing breaks :/ Is this happening on your site? If not, would appreciate some tips about coding it and how to…
The most helpful part is that you can pass a callback which will trigger before/during/after each step, which can let you ensure that the state of the page matches what you're expecting. In our case, we use it to make sure that you're switched to the right tab, etc. Take a look! I highly recommend it.
Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website
#60Suggestion: Please be upfront about pricing. I've been bitten with similar tool dying because they were too shy to ask for money.
They have a link to the pricing page[1] on the navbar [1] https://wrapapi.com/v2#/pricing