Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finding the best ticket price – Simple web scraping with Python

danielforsyth.me

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Finding the best ticket price – Simple web scraping with Python

#21

Post by lumpypua »

bshimmin wrote: post #16
wingerlang wrote: post #14

Earlier quoted context omitted.

Seems to be their thing QHandJoob QFlatBoob

Wow. Spurred on by your discoveries, I found this: http://weboob.org/applications/qhavedate "QHaveDate is a graphical application able to interact with dating websites, and help you manage your numerous conquests." I, uh, don't really know where to start with that.

Lol, there's some ridiculous stuff on that page: "Management of the meeting places, with for each some stats for the number of contacts met, your success rate, which seduction methods have been used, if they worked or not, if you’ve slept with the contact, which sexual positions, etc."

Re: Finding the best ticket price – Simple web scraping with Python

#22

Post by danielforsyth »

nivertech wrote: post #18
danielforsyth wrote: post #17

Earlier quoted context omitted.

Ah yes thats the problem, I am using beautifulsoup4==4.3.2. Try pip install beautifulsoup4

I found the problem. I think your listing ate '_'. It should be 'soup.find_all' instead of 'soup.findall' and 'link_end' instead of 'linkend'

Good find! Fixed it, thanks!

Re: Finding the best ticket price – Simple web scraping with Python

#23

Post by kmike84 »

tst wrote: post #6

I can recommend scrapy[0] if you work on a bit bigger problem. But even then if you familiar with scrapy it's incredible fast to write a simple scraper with your data neatly exported in .json. [0]: http://scrapy.org/

Scrapy spider that is doing exactly the same::

    # It requires scrapy from github.
    # Save it to tickets.py and execute 
    # "scrapy runspider tickets.py" from the command line

    from urlparse import urljoin
    import scrapy
    
    class TicketSpider(scrapy.Spider):
        name = 'tickets'
        start_urls = ['http://philadelphia.craigslist.org/search/sss?sort=date&query=firefly%20tickets']
    
        def parse(self, response):
            for listing in response.css('p.row'):
                price_txt = listing.css('span.price').re('(\d+)')
                if not price_txt:
                    continue
                price = int(price_txt[0])
                if 100
There is no reason to prefer Scrapy for extracting information from a single webpage, but on the other hand it is not any harder than BS+pyquery+requests.

Re: Finding the best ticket price – Simple web scraping with Python

#24

Post by zo1 »

buro9 wrote: post #8

This reminds me of something I knocked up back in 2006. It's not a scraper, it's not Python, but here you are: http://giggr.com/?q=klaxons Searches multiple UK ticket sites and returns the artist page matching the query. Clicking a header label (i.e. Ticketweb) switches to that provider. Double-clicking the header re-searches based on the value of the search box. I use it for the 9am scramble for newly released ticke…

If you don't mind me asking. Who pays for the bandwidth cost of running "giggr"? It doesn't look like you have any ads running. Or are you monetizing on it in some other way?

Re: Finding the best ticket price – Simple web scraping with Python

#25

Post by gjreda »

I did this recently when trying to get tickets to a sold out Cloud Nothings show. I'd scrape Craiglist for postings every 10 minutes, and then send myself a text if any of the posts were new. I ended up getting tickets the day before the show.

Since the show was at a very small venue (capacity of maybe 500), I didn't have to worry about a constant stream of false positives. I would have needed to handle these if I were searching for tickets to a sold out show, since ticket brokers just spam Craigslist constantly with popular terms.

Re: Finding the best ticket price – Simple web scraping with Python

#26

Post by josegonzalez »

Shameless Plug: I work for an NYC-based startup - SeatGeek.com - that is basically this[1]. We used to do forecasting but found that wasn't really useful[2] or worth the time it took to maintain, so we nixed it.

- [1]: As an example, here is the Firefly event the OP was scraping. : https://seatgeek.com/firefly-music-festival-tickets

- [2]: We haven't included Craigslist because the data is much less structured and inexperienced users may have a Bad Time™. YMMV

- [3]: It was also a royal pain in the ass to maintain. I know because I had to update the underlying data provided to the model, and also modify it whenever available data changed :( . Here is a blog post on why we removed it from the product in general: http://chairnerd.seatgeek.com/removing-price-forecasts

Re: Finding the best ticket price – Simple web scraping with Python

#27

Post by kmike84 »

crdoconnor wrote: post #7
tst wrote: post #6

I can recommend scrapy[0] if you work on a bit bigger problem. But even then if you familiar with scrapy it's incredible fast to write a simple scraper with your data neatly exported in .json. [0]: http://scrapy.org/

I don't recommend scrapy. Classic example of a framework that should have been a library. It will work up until a point and then it will railroad your app and you will have a really painful time breaking out of the 'scrapy' way of doing things. Classic 'framework' problem. I prefer a combination of celery (distributed task management), mechanize (pretend web browser) and pyquery (jquery selectors for python).

I'm not sure how would you design a library for event-loop based website navigation when an event loop is explicit. Scrapy (which is a wrapper over Twisted) is already quite close to this IMHO. You can plug anything to the same event loop if needed (think twisted web services, etc).

You can parallelize synchronous mechanize/requests scripts via celery, but it is less efficient in terms of resource usage if the bottleneck is I/O; also, it has larger fixed costs per each task.

N Scrapy processes, each processing 1/N of total urls is an easy enough way to distribute load; if that is not enough then a shared queue like https://github.com/darkrho/scrapy-redis is also an option.

I think it is not "scrapy" way of doing things that causes the problems, it is an inherent complexity of concurrency; you either give up some concurrency or build your solution around it.

Re: Finding the best ticket price – Simple web scraping with Python

#28

Post by motoboi »

Some months ago I found https://import.io/ and it just blow my mind.

I remember the pain it was to write custom scrapers every time (I used to do it with Perl, btw).

They have a custom browser with a nice interface, but the biggest thing are the so called "Connectors": you instruct the system into how to query and parse results and Import.IO will give you an API endpoint for this query, now automatized.

One can, say, create a "connector" which can query Airbnb and parse results, then create another "connector" which queries booking.com. Now it is possible to use the API to make a query for Boa Vista, Roraima (my city) and get the dataset.

I am not affiliated with them in any way, just a very happy old-school scrapper.

Nice walkthrough: http://www.youtube.com/watch?v=_16O10Wx2W4

UPDATE:

Unsurprisingly, import.io was Hacker News stuff in the past: https://news.ycombinator.com/item?id=7582858

Re: Finding the best ticket price – Simple web scraping with Python

#29

Post by [deleted] »

kmike84 wrote: post #27
crdoconnor wrote: post #7

Earlier quoted context omitted.

I don't recommend scrapy. Classic example of a framework that should have been a library. It will work up until a point and then it will railroad your app and you will have a really painful time breaking out of the 'scrapy' way of doing things. Classic 'framework' problem. I prefer a combination of celery (distributed task management), mechanize (pretend web browser) and pyquery (jquery selectors for python).

I'm not sure how would you design a library for event-loop based website navigation when an event loop is explicit. Scrapy (which is a wrapper over Twisted) is already quite close to this IMHO. You can plug anything to the same event loop if needed (think twisted web services, etc). You can parallelize synchronous mechanize/requests scripts via celery, but it is less efficient in terms of resource usage if the bottle…

[deleted]

Re: Finding the best ticket price – Simple web scraping with Python

#30

Post by rakoo »

bshimmin wrote: post #16
wingerlang wrote: post #14

Earlier quoted context omitted.

Seems to be their thing QHandJoob QFlatBoob

Wow. Spurred on by your discoveries, I found this: http://weboob.org/applications/qhavedate "QHaveDate is a graphical application able to interact with dating websites, and help you manage your numerous conquests." I, uh, don't really know where to start with that.

It's funny because the previous name of qhavedate was qhavesex.

It's sad that the only thing that people can see from this project is the name.

Post reply on HN