Similar story as https://www.padmapper.com/. It's sad that craigslist isn't willing to either update their application or allow someone to make it more useful for their customers.
Craigslist Casualty: The Story of ApartmentsNearMetro.com
11–20 of 26 posts
Re: Craigslist Casualty: The Story of ApartmentsNearMetro.com
#12FYI: this is unreadable on my WP8 device. The screen width is set smaller than the text width so I have so scroll left and right for every line.
Thanks for letting me know, I implemented a quick fix for mobile devices. Unfortunately I don't have a WP8 to test it on, but it shouldn't require scrolling left and right anymore to read the text.
Re: Craigslist Casualty: The Story of ApartmentsNearMetro.com
#13I had that exact same pain point in 2011. It's too bad the Craigslist refuses to innovate or let others innovate on their platform.
Yeah! It almost seems like a source of pride for them... What were you working on that got squashed by Craigslist?
I was simply trying to locate an apartment near Metro. I toyed with the idea of scraping their site but remembered this Wired story from 2009 [0] so I just gave up on using them for my search.
0: http://www.wired.com/entertainment/theweb/magazine/17-09/ff_...
Re: Craigslist Casualty: The Story of ApartmentsNearMetro.com
#14> Had around 1,500 unique visitors with an average time spent on the site of about 7 and a half minutes.
With no disrespect to the OP, that's not a ton of users, even for a single day...so I'm kind of surprised Craigslist noticed enough to block them. There must be tons of unapproved scraper scripts checking CL on a daily basis.
Re: Craigslist Casualty: The Story of ApartmentsNearMetro.com
#15So there's Walk Score. Apartments near (Washington) metro example — http://walk.sc/YPUq1a
This doesn't really solve the problem of Craigslist hoarding all their data to themselves (as pointed out numerous times already), but there are enough other sources to make it usable.
Re: Craigslist Casualty: The Story of ApartmentsNearMetro.com
#16FYI: this is unreadable on my WP8 device. The screen width is set smaller than the text width so I have so scroll left and right for every line.
Thanks for letting me know, I implemented a quick fix for mobile devices. Unfortunately I don't have a WP8 to test it on, but it shouldn't require scrolling left and right anymore to read the text.
Thank for fixing!
Re: Craigslist Casualty: The Story of ApartmentsNearMetro.com
#17It's the inevitable fate for anything that uses CL for data. CL is too afraid of competition, yet refuses to innovate. It's a shame.
In defense of CL (which I have never before offered), I noticed some UI improvements recently: pic view, grid view, and map view.
These are the first noticeable innovations on CL in many years. Might they be the result of the on-going parade of sites improving upon CL w/ their own data, not to mention the criticisms on HN and elsewhere?
Whether they 'hate' competition is debatable. Clearly they view their data as THEIR property. They occasionally let things like PadMapper go while putting the ban hammer down on other things, very arbitrarily.
The Wired story on them about 5-6 years back suggested they are just very skeptical of third parties and human nature in general. It left me with the impression that Craig is just deeply selfish and authoritarian.
Re: Craigslist Casualty: The Story of ApartmentsNearMetro.com
#18I don't get why people say things like, "It's too bad the Craigslist won't ... let other people innovate on their platform." It's just childish in my view. Here's another child's toys - and, granted, he has a LOT of toys[1] - and he's chosen to share them to every child in the world regardless of their background, yet you're upset because he's not sharing them in such a way that would only allow people to play with the toys on another playground. "He" doesn't want that - he wants to play, play, play with everyone. So he decides to say "No!" to anyone who wants to take his toys off of his playground.
If you don't like the way "he" plays with his toys, then tough cookies - "he" doesn't care and neither does anyone else except those who want to also complain about him not sharing in the way they want him to. You'll see this damn near every time someone posts a "Help! My API provider just changed the rules!" type of post. There are two types of responses: "I hate that - they should be more open", or "It's ridiculous to even think that you can build a business based off of leaching from someone else and that that other entity would be okay with that forever."
[1]Unlike children (who typically are given their toys), these toys are the result of "his" work.
Re: Craigslist Casualty: The Story of ApartmentsNearMetro.com
#19> Had around 1,500 unique visitors with an average time spent on the site of about 7 and a half minutes. With no disrespect to the OP, that's not a ton of users, even for a single day...so I'm kind of surprised Craigslist noticed enough to block them. There must be tons of unapproved scraper scripts checking CL on a daily basis.
That's a very good point. 1,500 unique visitors is tiny. Hell, I will probably get more visitors to this single blog post. Forgoing everything about CL blocking me and such, I never did enough to promote the site. I started the site in 2011 and had little experience with promoting content. As a newb developer, I was leery of promoting something I created. What if it broke? What if people didn't like it? When in reality, I learned over time, that most people frankly don't care. And that's not a bad thing.
The past few years I have been learning over and over: ship it, get feedback, improve, repeat.
So, getting back to how they blocked me. If I understand correctly, craigslist blocked any HTTP requests from the Google App Engine User Agent. HTTP Requests from Google App Engine always append 'AppEngine-Google' to the end of the User Agent (http://code.google.com/p/googleappengine/issues/detail?id=34...).
You can recreate this block by doing this in python (you need the amazing python requests library, so install that): import requests
r = requests.get('http://craigslist.com) print r.status_code
This should print 200 and you're good to go.
Then, try it with the 'AppEngine-Google' in the user agent: r = requests.get('http://craigslist.com, headers={'User-Agent': 'AppEngine-Google'})
and r.status_code will return 404
bingo bango, you've been blocked. I could have just moved off of Google App Engine, etc. but I would probably be faced with a similar fate as Padmapper if I ever got any kind of sizable user base.
I created this gist: https://gist.github.com/wlindner/4752122 since the python code formatting is wonky
Re: Craigslist Casualty: The Story of ApartmentsNearMetro.com
#20It's the inevitable fate for anything that uses CL for data. CL is too afraid of competition, yet refuses to innovate. It's a shame.
In defense of CL (which I have never before offered), I noticed some UI improvements recently: pic view, grid view, and map view. These are the first noticeable innovations on CL in many years. Might they be the result of the on-going parade of sites improving upon CL w/ their own data, not to mention the criticisms on HN and elsewhere? Whether they 'hate' competition is debatable. Clearly they view their data as THE…
I've seen the UI improvements, but haven't needed to use them so I don't know how good they are. If those work: Awesome! In a way, that's a victory. If CL incorporates the best things about 3rd party sites into their own site, I think that's a big win for consumers. If CL refuses to allow 3rd parties to exist because they are skeptical of human nature. Talk about the ultimate irony: a site that lives off of people's willingness to trust strangers being unwilling to trust strangers themselves.