When I saw the column called content my face went a little bit sad.
I might be missing something, but could you explain why?
Mixnode: Turn the web into a database
31–40 of 87 posts
Re: Mixnode: Turn the web into a database
#32Their Twitter is strange:
It says
Likes: 1023
@mixnode hasn't liked any Tweets yet
Re: Mixnode: Turn the web into a database
#33I could use it for a lots of things if it could filter for HTTP header. If there would be additional plug-ins to detect e.g. 3rd party tags it would be even more powerful for testing.
It looks like there is a "headers" column, so my guess is that you can do that.
Re: Mixnode: Turn the web into a database
#34I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…
From what I understand this is not trying to solve the typical e-commerce problem of closely watching your competitors selling something, but rather trying to provide a database to people interested in content on the web.
It probably won't solve the problems you're working on, but I could imagine quite a lot of interesting text analysis cases.
Re: Mixnode: Turn the web into a database
#35I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…
I haven't tried Mixnode yet, but the way I understand it, it lets you query websites and retrieve their HTML content that you can then parse - without you having to crawl the site. Looking at their Github, they seem to utilize WARC, so they may also allow you to request the website for certain timestamps?
That being said, I find this highly interesting, if it works like that. We are working on a peer-to-peer database that lets you query a semantic database, popularized mostly by public web data, but with strong guarantees of accurate and timely data, and this could be a great way to write more robust linked-data converters.
Re: Mixnode: Turn the web into a database
#36Im confused.
Do they do the scraping crawling and we just search their database?
Do we have to do the scraping/crawling and we dump the results into a mixnode server running locally?
Re: Mixnode: Turn the web into a database
#37Reminds me of YQL, https://developer.yahoo.com/yql/ .
Which used to support a similar usage (select from url) but no longer does.
Re: Mixnode: Turn the web into a database
#38Would this work as a complement to SPARQL? Or are they completely orthogonal to each other?
Re: Mixnode: Turn the web into a database
#39This seems like a really cool idea, but I'm struggling to imagine what the use cases would look like? Anybody have any ideas?
Re: Mixnode: Turn the web into a database
#40Im confused. Do they do the scraping crawling and we just search their database? Do we have to do the scraping/crawling and we dump the results into a mixnode server running locally?
I'm slightly confused too. They say that the web is a database, but it looks like we're SQL querying their database of the web.
I'm also interested in how often they rescrape their pages, and if they have rate-limit bypassing tech (for the Amazon scrapers).
So far, I think they're calling the Web a database because you can use SQL to query their database — which makes me feel like they're missing the point.
But they've done such hard work and they look like they're really excited about it — but I just don't understand why