Linus responds...
Reverse Engineering OKCupid
11–20 of 24 posts
Re: Reverse Engineering OKCupid
#12Linus responds...
I was stunned when I got the notification email to moderate him =D
Re: Reverse Engineering OKCupid
#13I do a lot of this kind of work for gmusicapi, and I still keep a Windows VM around just to use Fiddler.
Does anyone have recommendations for other tools? I came away from Burp and Charles disappointed in the past, but that was some time ago.
Re: Reverse Engineering OKCupid
#14Linus responds...
I was stunned when I got the notification email to moderate him =D
I was surprised by his recommendation. I found lxml, with it's weak xpath implementation to be a poor tool for parsing XML. How can it be a good tool for parsing non conforming HTML?
Perhaps my reply to Linus was snarky, but in my experience Beautiful Soup was easy to use for parsing HTML, where lxml for parsing XML was not. Granted the XML I had to parse used name spaces, but there was no reason for it to be difficult and there was no reason for poorly documenting which xpath features are supported.
Re: Reverse Engineering OKCupid
#15Linus responds...
He does, but sadly his comment is way off. Anyone that's done any amount of HTML scraping will use BeautifullSoup over lxml. The former being easier and more tolerant of html's nuances. The latter being brittle for anything less well formed than XHTML.
Re: Reverse Engineering OKCupid
#16Earlier quoted context omitted.
I was stunned when I got the notification email to moderate him =D
I was surprised by his recommendation. I found lxml, with it's weak xpath implementation to be a poor tool for parsing XML. How can it be a good tool for parsing non conforming HTML? Perhaps my reply to Linus was snarky, but in my experience Beautiful Soup was easy to use for parsing HTML, where lxml for parsing XML was not. Granted the XML I had to parse used name spaces, but there was no reason for it to be difficu…
No, you were right and be was wrong. Just because he made a copy of unix doesn't mean he knows how to parse HTML with python. Anyone that's done a lots of web scraping or (as in my case) crawling/indexing will see the merit of your response.
Re: Reverse Engineering OKCupid
#17Linus responds...
[deleted]
Re: Reverse Engineering OKCupid
#18I do a lot of this kind of work for gmusicapi, and I still keep a Windows VM around just to use Fiddler. Does anyone have recommendations for other tools? I came away from Burp and Charles disappointed in the past, but that was some time ago.
The most universally effective solution I've seen is Sikuli running in a VM (which is the only sane way to run it, since it hijacks your input devices). Everything else fails in some edge case. What other tool can scrape an interface that uses both HTML and Flash and is only served over HTTPS?
It is brittle, in that it can be broken by cosmetic UI changes, but the maintenance is generally trivial. Also, it's slow as all hell. But sometimes you really need that sledgehammer.
Re: Reverse Engineering OKCupid
#19This method is well known and there are ways of making this more difficult. For example, servers add a hidden request cookie which is a random number embedded in the page that the user is coming from. This forces you to actually parse the page. Then they can move it to javascript making it even more difficult.
I would just like to point out that not all sites do the following (and I'm unsure if OKC does), but watch out for tripwires as chacham15 said. Some sites that I've had the misfortune of "getting to know" use insignificant or blank inputs as a form of detecting unauthorized access. One rule I follow is to: Retrieve, Analyse and Regurgitate Everything.
By inputs do you mean CAPTCHAs or something else?
Re: Reverse Engineering OKCupid
#20Linus responds...
He does, but sadly his comment is way off. Anyone that's done any amount of HTML scraping will use BeautifullSoup over lxml. The former being easier and more tolerant of html's nuances. The latter being brittle for anything less well formed than XHTML.
I didn't want to say it but yea I thought BeautifulSoup has way more development.
I wonder if you disagree with him, he will unleash his wraith upon ye.