Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Craigwatch Story

craigwatch.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: The Craigwatch Story

#21

Post by yonran »

stickfigure wrote: post #6

I used Craigwatch. It was a useful tool - when a search pops, I get an email. I used to use Google Reader for this but when Reader died so did my RSS reading habit, so now email is the only part of my daily workflow that I poll frequently. The real mystery to me is why Craigslist shut the service down, because (protocols aside) it wasn't fundamentally different from any other way of consuming craigslist. Craigslist's…

By my reading, prior to Dec 2013, the old terms[1] permitted using personal RSS readers but forbade aggregators. However, the new terms are much more restrictive; they prohibit any use of the craigslist website outside of a web browser. Thus, I think that it is now against the terms to use an RSS reader to read the craigslist RSS feeds! (By the way, it is also against the craigslist terms to write a crawler such as google.com or archive.org even if it respects robots.txt!)

I happened to review these terms because I too was sent a letter from Brian Hennessy representing craigslist last Friday regarding a little Chrome extension I wrote a couple years ago[2].

[1] https://web.archive.org/web/20131128233421/http://www.craigs...

[2] https://github.com/yonran/craigslist-shortcuts

Re: The Craigwatch Story

#22

Post by dougmccune »

pessimizer wrote: post #20
jstanek wrote: post #17

Earlier quoted context omitted.

Craigslist has a really bad history of abusing its users. For instance, a few years back they changed the TOS so that CL becomes the copyright holder of all ads placed on the site.

So trying to fend off padmapper (whether one agrees with that or not) has suddenly become "a really bad history of abusing its users"? I guess scrapers are users.

I think the objection is that the initial knee-jerk reaction to try to fend off scrapers/competitors had a (potentially unintended) consequence of being pretty anti-user by claiming exclusive ownership over the user's content. Craigslist doesn't have to play nice with other services, or aid their competitors, but trying to fight competitors by overreaching and harming your customers isn't OK, which of course, is why they then backtracked on that bad policy after enough public anger.

Re: The Craigwatch Story

#23

Post by wiseleo »

kazinator wrote: post #13
wiseleo wrote: post #7

Earlier quoted context omitted.

Useful free stuff is typically gone in minutes. Only junk remains posted. Its RSS feeds have a refresh period of 1 hour, which is simply too long to have the opportunity to grab hot items. As Craigslist has very shallow category listings in its "free" category, where you can't separate furniture from clothes as an example, the site forces users to rely on manually refreshing pages, which change very often. I have bet…

Wiseleo, is that really true about the refresh rate? RSS readers have a configurable rate. Often there is a one hour default in order to be reasonably nice to the server. Basically, a Craigslist RSS item is just a URL with the search parameters embedded. Maybe I'm wrong, but I suspect that whenever you fetch this URL, the server executes the search and produces the results as RSS XML items, so the fetch rate is contr…

Gotta love being downvoted for an informative post... :)

If you look at the XML returned for RSS feeds, you will notice:

2014-07-16T14:25:18-07:00 1 hourly

That is the configuration setting for RSS readers to not update more than once per hour.

See spec: http://web.resource.org/rss/1.0/modules/syndication/

Craigslist has aggressive blocking for excessive GET requests. The RSS feed contains only the first 250 characters of text description of the ad. Thus, you will see that something got added but you will not see its details. More importantly, attributes are not available as part of the RSS feed.

That means that you were interested in specific colors of a car, you would need to define a separate RSS feed for red, yellow, and so on.

It is hard to test whether the RSS search results are additionally throttled, but you will likely get blocked while testing. :)

While legitimately shopping, I got blocked multiple times for becoming more efficient.

RSS is not hard to read. Here are some red manual transmission cars in San Francisco Bay Area http://sfbay.craigslist.org/search/cto?auto_paint=7&auto_tra...

Re: The Craigwatch Story

#24

Post by stickfigure »

yonran wrote: post #21
stickfigure wrote: post #6

I used Craigwatch. It was a useful tool - when a search pops, I get an email. I used to use Google Reader for this but when Reader died so did my RSS reading habit, so now email is the only part of my daily workflow that I poll frequently. The real mystery to me is why Craigslist shut the service down, because (protocols aside) it wasn't fundamentally different from any other way of consuming craigslist. Craigslist's…

By my reading, prior to Dec 2013, the old terms[1] permitted using personal RSS readers but forbade aggregators. However, the new terms are much more restrictive; they prohibit any use of the craigslist website outside of a web browser. Thus, I think that it is now against the terms to use an RSS reader to read the craigslist RSS feeds! (By the way, it is also against the craigslist terms to write a crawler such as g…

How can Craigslist accuse you of violating their terms of use? The act of providing a chrome extension does not use their service. Maybe if this was some sort of copyright-circumvention software covered by the DMCA I could understand it. But this sounds like an obvious legal bluff, no?

IANAL, which is why I am asking these questions.

Re: The Craigwatch Story

#25

Post by wiseleo »

marincounty wrote: post #19
wiseleo wrote: post #7

Earlier quoted context omitted.

Useful free stuff is typically gone in minutes. Only junk remains posted. Its RSS feeds have a refresh period of 1 hour, which is simply too long to have the opportunity to grab hot items. As Craigslist has very shallow category listings in its "free" category, where you can't separate furniture from clothes as an example, the site forces users to rely on manually refreshing pages, which change very often. I have bet…

I used to give away good free stuff, but it's just not woth the hassle(going to the wrong people, people not showing up, the spam, over agressive Flaggers--you people need to get a life, etc.) While I'm here. What's with the begging to lower an already low price on a service, or item. I used to price an item realistically, then knock off 40 percent. People still want to Haggle over the price. I now just double the pr…

Yeah, I am going to work on it once I am done launching my main product. My main product is for in-person customer acquisition, so I should be able to seed this properly.

I am thinking:

* Schedule pickup (I made a scheduling product a few years ago with a nice algorithm)

* [Innovative revenue source that is the secret sauce, not involving ads or subscriptions]

* Request hold on the item (with ability to flag no shows)

* Hyper-detailed categories (I have a very interesting plan for that)

Open API, but anyone including us is free to clone what you build :)

Re: The Craigwatch Story

#26

Post by intellegacy »

it's a great story.

BTW, can anyone give me perspective on what it would take to recreate this in Python? As a learning exercise

Is it possible to do? how long would it take for someone relatively new, etc.

Re: The Craigwatch Story

#27

Post by yonran »

stickfigure wrote: post #24
yonran wrote: post #21

Earlier quoted context omitted.

By my reading, prior to Dec 2013, the old terms[1] permitted using personal RSS readers but forbade aggregators. However, the new terms are much more restrictive; they prohibit any use of the craigslist website outside of a web browser. Thus, I think that it is now against the terms to use an RSS reader to read the craigslist RSS feeds! (By the way, it is also against the craigslist terms to write a crawler such as g…

How can Craigslist accuse you of violating their terms of use? The act of providing a chrome extension does not use their service. Maybe if this was some sort of copyright-circumvention software covered by the DMCA I could understand it. But this sounds like an obvious legal bluff, no? IANAL, which is why I am asking these questions.

IANAL either, but I think it would be against craigslist terms for me to continue to use their website while “providing” my accessibility extension. So I could publish the extension, but I would have to stop using craigslist. Additionally, the Chrome Web Store developer agreement[1] prohibits publishing an extension that “knowingly violates a third party’s terms of service”, so my guess is that the lawyer would then ask Google to take it down since any user of the extension would be violating the craigslist terms and thus (after the C&D) I would be violating the Chrome Web Store terms.

It’s pretty airtight, so I took my extension down.

[1]: https://developer.chrome.com/webstore/terms

Re: The Craigwatch Story

#28

Post by bnejad »

intellegacy wrote: post #26

it's a great story. BTW, can anyone give me perspective on what it would take to recreate this in Python? As a learning exercise Is it possible to do? how long would it take for someone relatively new, etc.

With all the libraries available for parsing and scraping I suspect this could be recreated very quickly.

Re: The Craigwatch Story

#29

Post by mkaziz »

yonran wrote: post #21
stickfigure wrote: post #6

I used Craigwatch. It was a useful tool - when a search pops, I get an email. I used to use Google Reader for this but when Reader died so did my RSS reading habit, so now email is the only part of my daily workflow that I poll frequently. The real mystery to me is why Craigslist shut the service down, because (protocols aside) it wasn't fundamentally different from any other way of consuming craigslist. Craigslist's…

By my reading, prior to Dec 2013, the old terms[1] permitted using personal RSS readers but forbade aggregators. However, the new terms are much more restrictive; they prohibit any use of the craigslist website outside of a web browser. Thus, I think that it is now against the terms to use an RSS reader to read the craigslist RSS feeds! (By the way, it is also against the craigslist terms to write a crawler such as g…

I'm curious as to whether you got a response to your refusal to "comply." Did he get back to you after that?

Re: The Craigwatch Story

#30

Post by unreal37 »

melvinng wrote: post #16
melvinng wrote: post #12

Stand up to the MAN!!!

Everyone is too law abiding.. But really what laws is he breaking here?? Trademark? Copyright? Spoke to my legal counsel and I don't see either..

Terms of service are a contractual agreement between a service and its users. There's no laws being broken, but it's breach of contract.

Your legal counsel would know that I suspect.

Post reply on HN