Web Scraping Indeed for Key Data Science Job Skills

> Assuming that someone has created such a system but has not released it, and that this person asked you what you might do with her system, what would you answer?

Build stuff that takes information and uses it in new, interesting, creative, and useful ways. Right now there is a lot of extremely useful data that is trapped inside the interfaces of websites and apps that could be used in amazing ways but unfortunately there's no easy way to get at the data.

I don't think hogging information and intellectual property will last very long as a means of creating value. We as a society need to think of better business models and better ways to define progress than this.

> Do you envision that users would want to run such a system, e.g., if there was a public benefit to such information sharing?

Sure, if they are getting something out of it too. For example, the new ways of accessing information should only be usable if they participate in running the system.

> What if the implementation was a group of small programs written in C that communicated with each other, and no browser extensions or scripting languages were required? What if the system required attachment of dedicated hardware to the user's LAN, e.g., a $25 single board computer?

All this sounds good to me. I'd want the full hardware and software stack to be open-source though if it's going to be plugged into my home network, so that there's no chance of it violating my privacy.

One hurdle will be how to enforce that users MUST contribute a piece of their bandwidth in order to be able to use the fruits of the system (e.g. you need to help others make scraping API calls before you can issue calls yourself). Napster did this for music, but as with any centralized system, it will eventually get sued and shut down.

In order to decentralize this I think a cryptographic currency similar to Bitcoin will be needed: you get points for offering bandwidth, you need to spend points in order to make calls on other peoples' bandwidth.