Stop avoiding regular expressions damn it

A couple of years ago I moved to a different country, and for some reasons I needed two apartments, preferably close to each other. As you can imagine, the real estate websites are not designed for the kind of query I needed, so I wrote some code to aid me in my quest[1].

It's just shell script and text processing with awk. I download various results with all the available apartments for many real estate websites, then I scrape the data I care about (with regular expressions!) like address, rooms, price, anything really, and query the Google Maps API with all the addresses to retrieve the geographical coordinates, then I compute the distances between any two houses and sort them.

It's fantastically modular. Adding support for a new website meant just creating some regular expressions that work for that website. This was great because I was doing this on the road, as I was visiting the foreign city and found new sources of information.

Regular expressions were also great because these websites didn't have any API where I could query for the address, etc. I had to rely on what people wrote in their ads. This meant that when I wrote a regexp to match a set of results I had to inspect the failures to see new ways people described their houses and improved my matching based on that. Initially I had hoped I'd be able to parse 80% of the ads, but measurements and careful coding had allowed me to match approximately 99% of the ads!

The textual operation of this software allowed me to easily input some data manually. For example I realized that I'm also interested in having these apartments close to a subway station. No problem, just manually create the file with the subway stations in the correct, simple, textual format and the program will pick it up and use automatically.

The textual interface also helped with fancy queries, like "price between X and Y, 6 rooms total, prefer 4-2 to 3-3 if distance less than D, but 3-3 if distance greater than D, prefer Z subway line to Q, only one apartment might be from an agency rather than an individual, try to put one in K part of the city". Try to do that with an existing website.

[1] https://code.google.com/p/operation-housefinder/

    `\A@(.*)@\z`

    BenchmarkRegexp	  500000	      5181 ns/op
    BenchmarkStrings	10000000	       171 ns/op

An interface that e.g. makes me 'escape' half of my input because its designers think their special use of characters must take precedence over all user input is a bad interface.