Web Scraping Indeed for Key Data Science Job Skills
1–10 of 23 posts
Re: Web Scraping Indeed for Key Data Science Job Skills
#2This is very cool. I've been kicking around a similar approach to feed data into a recommendation engine: collect job listings, filter against a list of stop words, then see if ones with similar words turn out to be similar jobs.
One of the problems with this approach is (in my country at least) the heavy industry presence of recruitment firms means every job is listed up to four times: once by the employer, and once by each firm competing to find the hire.
Re: Web Scraping Indeed for Key Data Science Job Skills
#3look through Indeed's pages of job results and click on all of the job links, but only in the center of the page where all of the jobs are posted (not on the edges).
I wrote a toolkit to help solve this problem [1]. An issue with taking the approach of hard-coding the result pattern to scrape is that it can break when the page changes. E.g., the author's code has:
page_obj.find(id = 'resultsCol')
And as far as cleaning up unicode and HTML entities, I like the "he" project [2]. Within text fields these HTML parsing libraries don't do a very good job so unfortunately this extra parsing is necessary. And sometimes properly stripping all duplicate whitespace involves getting very familiar with control characters (and corrupted control characters from bad encoding) as well as left-behind html tags/entities [3].
1. https://github.com/MachinePublishers/ScreenSlicer
2. https://github.com/mathiasbynens/he
3. https://github.com/MachinePublishers/ScreenSlicer/blob/maste...
Re: Web Scraping Indeed for Key Data Science Job Skills
#4look through Indeed's pages of job results and click on all of the job links, but only in the center of the page where all of the jobs are posted (not on the edges). I wrote a toolkit to help solve this problem [1]. An issue with taking the approach of hard-coding the result pattern to scrape is that it can break when the page changes. E.g., the author's code has: page_obj.find(id = 'resultsCol') If Indeed ever chang…
Could you please elaborate on what you mean by "dynamically figure out where the results are"? Or how to go about doing it?
#Edit. I see your first link sorta answers that. And correct me if I'm wrong, but when I went there, it seemed that the library caters more towards automatic searching + paging, rather than extracting results?
Re: Web Scraping Indeed for Key Data Science Job Skills
#5re: python vs. R
I haven't done python at all, but from reading bits and piece online, it seems to me that Python is a lot more about ML than Statistics.
Also, how does one decide what is a "data scientist"? Is it only people who do Stats+ML+IT? What about a researcher in economics or biology? Or marketing. Are those included? They do a lot of Stats and increasingly a lot of ML. Not so much information technology though (less concerned with data storage and retrieval, because they have other people to do that for them). I'd venture to say that researchers like that are far bigger number than pure play data scientists and would be interesting to see the technical skills for them. I bet SPSS and SAS would look a lot more in demand.
Re: Web Scraping Indeed for Key Data Science Job Skills
#6The page is not available for me.
Re: Web Scraping Indeed for Key Data Science Job Skills
#7look through Indeed's pages of job results and click on all of the job links, but only in the center of the page where all of the jobs are posted (not on the edges). I wrote a toolkit to help solve this problem [1]. An issue with taking the approach of hard-coding the result pattern to scrape is that it can break when the page changes. E.g., the author's code has: page_obj.find(id = 'resultsCol') If Indeed ever chang…
Could you please elaborate on what you mean by "dynamically figure out where the results are"? Or how to go about doing it? #Edit. I see your first link sorta answers that. And correct me if I'm wrong, but when I went there, it seemed that the library caters more towards automatic searching + paging, rather than extracting results?
It handles extraction too, trying to find where the results are and then extracting individually the title/summary/url/date.
To elaborate on the general approach I used, it was to take each node in the web page and get stats about all of them (e.g., position on page, amount of freetext, etc) and plug those stats into a neural net.
I worked on a different project some years ago that took the approach of looking for repeating tag patterns in the page, focusing especially on structural tags (as opposed to ones that are purely for formatting).
Another possible approach might be to just plug the whole result page into something such as Boilerpipe (https://code.google.com/p/boilerpipe/) and look at the set of urls in the text block it identifies.
Re: Web Scraping Indeed for Key Data Science Job Skills
#8Perhaps offtopic: Is web scraping a desired skill in the job market? Asking because unemployed.
Re: Web Scraping Indeed for Key Data Science Job Skills
#9re: python vs. R I haven't done python at all, but from reading bits and piece online, it seems to me that Python is a lot more about ML than Statistics. Also, how does one decide what is a "data scientist"? Is it only people who do Stats+ML+IT? What about a researcher in economics or biology? Or marketing. Are those included? They do a lot of Stats and increasingly a lot of ML. Not so much information technology tho…
Just my personal biased perspective:
You are correct, there is no good definition of "Data Science". Just as any other buzzword, it is widely devoid of meaning but useful to convey a general idea to certain audiences.
Regarding R vs. Python, I personally use Python to process my data up to the point where I can hand it over to R in a convenient df-esque shape and use R for stats and plotting. Even if I do not require R for stats, I still use ggplot2 for plotting, though I'd be happy to see Seaborn or Bokeh evolve to a point where they can rival it.
Re: Web Scraping Indeed for Key Data Science Job Skills
#10re: python vs. R I haven't done python at all, but from reading bits and piece online, it seems to me that Python is a lot more about ML than Statistics. Also, how does one decide what is a "data scientist"? Is it only people who do Stats+ML+IT? What about a researcher in economics or biology? Or marketing. Are those included? They do a lot of Stats and increasingly a lot of ML. Not so much information technology tho…
You can take a look at this book Analyzing the Analyzers
The Authors surveyed data scientists, asking about their
experiences and how they viewed their own skills and careers.They answer your question what is a "data scientist"
http://www.oreilly.com/data/free/files/analyzing-the-analyze...