Interesting product. BTW: 10 times posted in HN, first time they’re trending (if you click the domain you’ll see the list). Persistence pays I guess :)
Mixnode: Turn the web into a database
41–50 of 87 posts
Re: Mixnode: Turn the web into a database
#42Earlier quoted context omitted.
It’s not even an alternative to crawling. It’s just a way to exploit the result of crawling. You have to crawl first. Very misleading title.
I don't think it's actually wrong to call it an alternative. If someone says that going to a restaurant is an alternative to cooking, you know exactly what they mean. It's not an alternative method, but it is an alternative choice. The problem is that there are two valid interpretations here, and it wasn't clear which was the right one.
As long as Mixnode took care of the cooking for us, yes, using Mixnode’s SQL cutlery is kind of an alternative to cooking.
Let’s hope that the food is varied enough and not too outdated, though.
Re: Mixnode: Turn the web into a database
#43Re: Mixnode: Turn the web into a database
#44A database can be owned. I am very uneasy with anything that can be owned by one person hypothetically where many have contributed.
Re: Mixnode: Turn the web into a database
#45Good, now you need to design an index to that database.
Re: Mixnode: Turn the web into a database
#46I wonder - if you take the top ten keywords and url info from all pages on the web - would the data fit on a micro-sd card !?
Re: Mixnode: Turn the web into a database
#47I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…
What if the product was a framework for sourcing, aggregating, and visualizing data? When the user is put in control, you don't need to trust the product to do these things for you - it simply enables you to do what you want.
I think this is where the web is headed - where common users gain the ability to perform tasks that currently only developers or technical experts can do.
Re: Mixnode: Turn the web into a database
#48Where are the querys stored?
Re: Mixnode: Turn the web into a database
#49[deleted]
Re: Mixnode: Turn the web into a database
#50This is way less cool than the title suggests. They are doing a bunch of crawling and inserting the raw html content into their big centralized database, where you can run queries on the text inside:
select
url,
string_between(content, '', '') as title
from
resources
where
content_type like 'text/html%'