Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finding the best ticket price – Simple web scraping with Python

danielforsyth.me

Post reply on HN
Print view View original

31–36 of 36 posts

Re: Finding the best ticket price – Simple web scraping with Python

#31

Post by dmn001 »

zo1 wrote: post #24
buro9 wrote: post #8

This reminds me of something I knocked up back in 2006. It's not a scraper, it's not Python, but here you are: http://giggr.com/?q=klaxons Searches multiple UK ticket sites and returns the artist page matching the query. Clicking a header label (i.e. Ticketweb) switches to that provider. Double-clicking the header re-searches based on the value of the search box. I use it for the 9am scramble for newly released ticke…

If you don't mind me asking. Who pays for the bandwidth cost of running "giggr"? It doesn't look like you have any ads running. Or are you monetizing on it in some other way?

Looks like it just uses browser-side js requests to get the search pages, so it would use minimal bandwidth.

Re: Finding the best ticket price – Simple web scraping with Python

#32

Post by dmn001 »

motoboi wrote: post #28

Some months ago I found https://import.io/ and it just blow my mind. I remember the pain it was to write custom scrapers every time (I used to do it with Perl, btw). They have a custom browser with a nice interface, but the biggest thing are the so called "Connectors": you instruct the system into how to query and parse results and Import.IO will give you an API endpoint for this query, now automatized. One can, say,…

Other browser based screen scrapers that are in the space are 80 legs, kiminolabs, Mozenda and OutWit Hub, I'm sure there are more. Last time I checked, import.io was a fairly lightweight browser wrapper.

I also write web scrapers using Perl and Python, recently have been gravitating towards Python as the code looks more readable. I don't use browser based scrapers because the sites I scrape are usually more complex so it is just easier to write my own code, and they lack functionality and control of the data, and there is the overhead of learning the terminology and how it works.

Re: Finding the best ticket price – Simple web scraping with Python

#33

Post by buro9 »

zo1 wrote: post #24
buro9 wrote: post #8

This reminds me of something I knocked up back in 2006. It's not a scraper, it's not Python, but here you are: http://giggr.com/?q=klaxons Searches multiple UK ticket sites and returns the artist page matching the query. Clicking a header label (i.e. Ticketweb) switches to that provider. Double-clicking the header re-searches based on the value of the search box. I use it for the 9am scramble for newly released ticke…

If you don't mind me asking. Who pays for the bandwidth cost of running "giggr"? It doesn't look like you have any ads running. Or are you monetizing on it in some other way?

It's a static web page on a Linode I use for other projects and purposes.

Even if millions of people decided to suddenly use it, the cost would be almost nothing.

I might even consider putting the free CloudFlare in front of it to ensure the cost is nothing (one static HTML file cached forever).

Heh, just looked at the source code again... it's a single request web page, not even an external CSS or JavaScript file.

You can't get cheaper really.

Re: Finding the best ticket price – Simple web scraping with Python

#34

Post by stefap2 »

jknupp wrote: post #19

A shorter, more comprehensible version: import requests from bs4 import BeautifulSoup from urlparse import urljoin URL = ' http://philadelphia.craigslist.org/search/sss?sort=date&quer... BASE = ' http://philadelphia.craigslist.org/cpg/' response = requests.get(URL) soup = BeautifulSoup(response.content) for listing in soup.find_all('p',{'class':'row'}): if listing.find('span',{'class':'price'}): price = int(listing.t…

thanks

Re: Finding the best ticket price – Simple web scraping with Python

#36

Post by dai_pole »

I had a go "just for fun" using curl, grep, sed, and tr. Probably too much regex?

    #!/bin/sh
    #
    # tickets.sh - A "no BS" ticket price scraper. Output in CSV format.
    #              Uses standard issue Unix utilities only.
    #              No soup for you!
    
    
    URL="http://philadelphia.craigslist.org"
    QUERY="firefly+tickets"
    
    RESULTS=`curl -s -m 10 "$URL/search/sss?sort=date&query=$QUERY" \
            | grep '[ \t]*$\([0-9]\{1,\}\)[^.]*>\([A-Z]\{1\}[a-z]\{2\} \{1,\}[0-9]\{1,2\}\)[^.]*]*\.html">\([^\([^.]*
\)!\1,$\2,\3,\4:!g; \
                   s!   *! !g; \
                   s!,  *!,!g' \
            | tr ':' '\n'`
    
    echo "$RESULTS"
Post reply on HN