Craigslist does not allow scraping. Beg for help and for support for breaking their rules elsewhere. This is Hacker News, not "help me retain a business model after I broke rules" central.
You could alternatively hire people to spam them with nice requests to reconsider.
Bye Bye Craigslist
211–220 of 312 posts
Re: Bye Bye Craigslist
#212Earlier quoted context omitted.
You're right, I was hoping I could go on, but I wasn't stealing anyone's content, I was posting links pasted geographically located with stat summaries. Look at a listing on PadMapper - there's no listing text, and when you click through, it goes to the original listing.
My apologies. I may have used some improper wording there. But if I may... you must admit that you have taken "something" from Craigslist. Whatever you want to call it... be it geographical location points, links-to-listings, content, data, images, etc... it is inevitably some form of information that was originally curated by Craigslist through their 10+ years of existence online and is thus available to be used (an…
Thanks for the explanation! I've talked with lawyers about this in the past week, and facts, totally rewritten summaries of fact, and collections of facts aren't copyrightable in the US, so it's not really an issue of their content, as far as I understand. They do have a right not to be crawled by me, though, especially if it causes a burden on them. I wrote that crawler specifically to be very light on their servers including making tradeoffs to usability (it never recrawls listings), but that's still their decision to make. If there are ways to get the data that don't involve touching their servers, then the TOU no longer applies, if I understand the law, but it would obviously make them angry, and so I'm most likely not going to do that.
Re: Bye Bye Craigslist
#213Craigslist should buy you, not block you. I'm basically a Luddite and yours is the one service I always recommend.
Maybe they should block you then buy you. Probably would help them in the negotiation...
But in all seriousness, PadMapper was a great service and seemed to do it the "right" way (providing deep links back into Craigslist's site/content), if there is a "right way" to unofficially gather data without use of a public API.
Re: Bye Bye Craigslist
#214Re: Bye Bye Craigslist
#215Why not buy a license and go hard on mobile? Then use that to get more direct listings, ideally with ACTUAL STRUCTURED DATA unlike the shit that is craigslist (i.e. closer to MLS, or even better, closer to the proprietary systems used inside large real estate investment companies), and then re-release a web version using only those listings. Maybe focus on SFBA only, since the real estate situation is acutely broken…
Maybe, if you want to chat more about this, email me at eric@padmapper.com
Re: Bye Bye Craigslist
#216You already got the traction. Offer something complete now. It's already better.
Re: Bye Bye Craigslist
#217Earlier quoted context omitted.
Sorry, I didn't mean it that way - if you know what PadMapper is and see the URL, I think it makes sense.
With the OP here, I saw the thread title and silently went "thank god" to myself immediately knowing what this meant, having used PadMapper very extensively in the last year after moving to Austin. I knew it meant a good portion of ads being gone, but I was going to get much more qualitative results in finding a condo. I really hope PadMapper can make up the volume but continue to have good results for people searchi…
Haha you could always filter out Craigslist with the sources filter (though it's a bit hidden)
Re: Bye Bye Craigslist
#218Can you open source padmapper so we can run it ourselves? That way, no matter what Craigslist does, people would still be able to use the great service.
Hm, that's kind of an interesting idea. I'd be pretty embarrassed at all the hacks I've added over the years, but I'll definitely keep that in mind.
Re: Bye Bye Craigslist
#219Earlier quoted context omitted.
My apologies. I may have used some improper wording there. But if I may... you must admit that you have taken "something" from Craigslist. Whatever you want to call it... be it geographical location points, links-to-listings, content, data, images, etc... it is inevitably some form of information that was originally curated by Craigslist through their 10+ years of existence online and is thus available to be used (an…
Thanks for the explanation! I've talked with lawyers about this in the past week, and facts, totally rewritten summaries of fact, and collections of facts aren't copyrightable in the US, so it's not really an issue of their content, as far as I understand. They do have a right not to be crawled by me, though, especially if it causes a burden on them. I wrote that crawler specifically to be very light on their servers…
What about copyright traps?
Say Craigslist created a fake listing or two for say a 2 bedroom apartment in the middle of an industrial complex? (ie, an apartment that doesn't exist, but was placed there by Craiglist to serve as evidence against your copying).
If the Craigslist lawyers find that fake listing on PadMapper, could they do anything to charge you with regards to a copyright violation?
I'm not trying to drag this on. I'm only trying to help.
Here is a smoking gun article on Rand McNally's copyright traps (or lack thereof): http://www.straightdope.com/columns/read/1058/do-maps-have-c...
Re: Bye Bye Craigslist
#220Earlier quoted context omitted.
Does that mean other people should steal the content and profit from it? Craigslist is unwilling to let Padmapper buy a license to use their content. This isn't as black and white as "stealing content".
Content not available for licensing is a justification for infringement? That doesn't ring true at all for me - exclusive content is often much more valuable to a business than content that's licensed. I doubt anyone would think that goldman sachs coverage being distributed by morgan stanley would fit under some "no harm, no foul" ethos even if Morgan significantly improved on the layout of the pages.
I don't agree with the infringement, I just think that boiling it down to "stealing" is too reductionist. The reality here is a bit more complicated.
Seriously, I think I am the only person who doesn't think it is OK to torrent Game of Thrones. :)