Viewing profile — dmn001

I find there is really no need to hide or mask the IP address when web scraping. The use of proxies or Tor to do so is completely unnecessary and maybe prohibitive e.g. try using G…

By caching, I just mean storing of data locally so you don't have to request it again under a certain timeframe. I use my own caching scripts written in Python, if you use a 3rd pa…

There is no issue with parsing and scraping in the same loop as long as there is caching in there as well. You don't want to be hitting the server repeatedly whilst you're debuggin…

On the contrary, I have found lxml suitable for all of my scraping projects where the objective is to write some XPath to parse or extract some data from some element.

That may be fine for javascript heavy websites for a site with a few pages, but for anything with more than say 1,000 pages it is much more efficient to scrape using requests with …

SVG flowcharts of many gamebooks including FF and Lone Wolf: http://outspaced.fightingfantasy.net/SVG_Flowcharts/main.htm...

Should be fixed now.

I made .svg diagrams for the Fighting Fantasy books Warlock of Firetop Mountain and Deathtrap dungeon a while back on my old blog: https://daveman.wordpress.com/2010/01/08/how-to-c…

No. Most websites don't do this.

It's extremely rare to be ip-blocked by any website just for using the Google's user agent from a non-specific range. IP's get re-used and you can switch to a new one easily, so it…

SEEKING WORK - England,UK/Remote Over 7 years experience with Python software development, cloud services, data mining, web crawling, databases. Want to extract or crawl data from …

The first part seems like a very long-winded way to say "don't use the default user agent". The captcha was unusually simple to solve, in most cases the best strategy is to avoid s…

Right, if you're just matching a substring in a list of strings then you don't need regex for that. Also, lxml is over 20x faster than BeautifulSoup, although it doesn't really mat…

SEEKING WORK - UK/Remote Over 7 years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website…

I don't think I have the other videos, IIRC he removed or made private the previous videos shortly after the later ones were uploaded and that was the impetus for downloading them …

I found some of Vova's videos: https://www.youtube.com/watch?v=_zG_QpwbV3E https://www.youtube.com/watch?v=9csBR4AcSAQ https://www.youtube.com/watch?v=XfGbWbWSr00 https://www.youtu…

It would just be much easier to use the Facebook graph api, there is an official Python module and is well documented, and would be less likely to hit rate limits or other blocks -…

SEEKING WORK - UK/Remote Over 5 years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website…

Location: UK Remote: Yes. Willing to relocate: Yes. Technologies: Python, Perl, Matlab, R, MongoDB, MySQL, Git Résumé/CV: available on request Email: dmn001 gmail

Btw, I have no association with this company, other than I like to keep track of their blog and events regularly as it ties in quite closely with my line of work. Thought I'd post …

SEEKING WORK - UK/Remote 5+ years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Want to extract or crawl data from a website suc…

> the entire content of The Pirate Bay > Not the P2P content that people download I think you're confusing different definitions of 'content' here. Also, there are many more sites …

SEEKING WORK - remote or UK 5+ years experience with Python and Perl web scraping, cloud services, databases and Linux administration. Need to extract or crawl data from a website …