Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping Indeed for Key Data Science Job Skills

jessesw.com

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Web Scraping Indeed for Key Data Science Job Skills

#11

Post by blumkvist »

sososoko wrote: post #10
blumkvist wrote: post #5

re: python vs. R I haven't done python at all, but from reading bits and piece online, it seems to me that Python is a lot more about ML than Statistics. Also, how does one decide what is a "data scientist"? Is it only people who do Stats+ML+IT? What about a researcher in economics or biology? Or marketing. Are those included? They do a lot of Stats and increasingly a lot of ML. Not so much information technology tho…

You can take a look at this book Analyzing the Analyzers The Authors surveyed data scientists, asking about their experiences and how they viewed their own skills and careers.They answer your question what is a "data scientist" http://www.oreilly.com/data/free/files/analyzing-the-analyze...

That's a great paper. I wonder how I've missed it. I'm happy to to say that it partially confirms my own take on the situation. I thought of it only in 2 categories - Comp sci/math guys vs. Applied statistics guys (mostly from Humanities). Reality seems more nuanced then that (as always). Very nice read!

Re: Web Scraping Indeed for Key Data Science Job Skills

#12

Post by stewhir »

As a little side project, I built a website at: http://skill.report which instantly does this for any job title. Go try it! I'd love to hear your feedback :)

It works by sampling job ads from Indeed, then applying some information extraction/retrieval/NLP algos to extract and weight the presence of identified skills and qualities. There's some occasional glitches in the algo (I need to fix some of the disambiguation data), but it usually gives reasonable results.

I was thinking of focusing the algorithm for giving really in-depth feedback on improving your resume specifically for a job. Now if only I could finish my PhD thesis I might actually have the time to do more with it...

Re: Web Scraping Indeed for Key Data Science Job Skills

#13

Post by dheera »

I wish there was a system of web scrapers where the scraping logic is user-contributed and decentralized at the same time. Being decentralized, there would be no way for the owners of websites to stop anyone from scraping, and being user-maintained, the logic gets updated quickly whenever the original website's HTML template changes.

Re: Web Scraping Indeed for Key Data Science Job Skills

#14

Post by raus22 »

blumkvist wrote: post #5

re: python vs. R I haven't done python at all, but from reading bits and piece online, it seems to me that Python is a lot more about ML than Statistics. Also, how does one decide what is a "data scientist"? Is it only people who do Stats+ML+IT? What about a researcher in economics or biology? Or marketing. Are those included? They do a lot of Stats and increasingly a lot of ML. Not so much information technology tho…

The answer to what a data scientist does, is very different when you ask a self-proclaimed data scientist and a person that hires a "data Scientist". This creates can create disappointment on both sides.

It ranges (from the hiring side) from: "good if you know excel and you will report figures to management" to "please only apply if you have 2 Ph.Ds and 20 years of experience in C". So it is a bit wide...

the self-proclaimed "data scientists" probably goes from know a programming language/know visualising/know statistics up to "X Ph.Ds and Y years of experience where Y > 20"

Re: Web Scraping Indeed for Key Data Science Job Skills

#15

Post by iheartmemcache »

Toast_ wrote: post #8

Perhaps offtopic: Is web scraping a desired skill in the job market? Asking because unemployed.

Sort of. There are tons of people offering crawling-as-a-service, where you can do your own scraping. Then there are tools like import.io which let you point-and-click data right into your database. There's scrapy for and similar frameworks for Python. I'm sure node has more than a few npm packages out there to deal with dynamically rendered (e.g. infinite scroll) stuff out there. In short, it's a useful tool to be able to reach for in your toolbelt but not the hammer you'll use every day.

Re: Web Scraping Indeed for Key Data Science Job Skills

#16

Post by gwu78 »

dheera wrote: post #13

I wish there was a system of web scrapers where the scraping logic is user-contributed and decentralized at the same time. Being decentralized, there would be no way for the owners of websites to stop anyone from scraping, and being user-maintained, the logic gets updated quickly whenever the original website's HTML template changes.

Assuming that someone has created such a system but has not released it, and that this person asked you what you might do with her system, what would you answer?

Do you envision that users would want to run such a system, e.g., if there was a public benefit to such information sharing?

What if the implementation was a group of small programs written in C that communicated with each other, and no browser extensions or scripting languages were required?

What if the system required attachment of dedicated hardware to the user's LAN, e.g., a $25 single board computer?

Re: Web Scraping Indeed for Key Data Science Job Skills

#19

Post by jefb » 

    for script in soup_obj(["script", "style"]):
            script.extract() # Only need these two elements from the BS4 object
That comment is a bit misleading. soup.extract() will remove those tags from the tree, 'script' and 'style' are the two tags you __don't__ need.

http://www.crummy.com/software/BeautifulSoup/bs4/doc/#extrac...

Re: Web Scraping Indeed for Key Data Science Job Skills

#20

Post by artmageddon »

Toast_ wrote: post #8

Perhaps offtopic: Is web scraping a desired skill in the job market? Asking because unemployed.

My guess is that web scraping isn't something you're going to see on job postings specifically, but if you can write a web scraper with your favorite programming language, that would be something useful to discuss and maybe even land you an interview.
Post reply on HN