Mixnode: Turn the web into a database

#1

Post by cloudblare » Sun, Oct 07, 2018, 4:27 PM UTC

Mixnode: Turn the web into a database
mixnode.com

Re: Mixnode: Turn the web into a database

#2

Post by chefqual » Sun, Oct 07, 2018, 4:41 PM UTC

This is very smart. Also a very good choice of language. Reminds me of a comment from a few days ago https://news.ycombinator.com/item?id=18144385

Re: Mixnode: Turn the web into a database

#3

Post by donkoz_ab » Sun, Oct 07, 2018, 4:47 PM UTC

chefqual wrote: post #2

This is very smart. Also a very good choice of language. Reminds me of a comment from a few days ago https://news.ycombinator.com/item?id=18144385

I think that was a smart choice too. It's good that they could see through all the hype with the more recent languages and pick SQL. interesting choice indeed. Not sure how it scales though.

Re: Mixnode: Turn the web into a database

#4

Post by pighive » Sun, Oct 07, 2018, 4:57 PM UTC

chefqual wrote: post #2

This is very smart. Also a very good choice of language. Reminds me of a comment from a few days ago https://news.ycombinator.com/item?id=18144385

I too looked at the comment and thought, yes it’s about time to get my SQL act together. I can write sql queries, and can also understand looking at them what they supposedly do, but my day job doesn’t really demand more than a simple select on two tables. Where can I go learn/explore more competitive SQL?

Re: Mixnode: Turn the web into a database

#5

Post by donkoz_ab » Sun, Oct 07, 2018, 5:03 PM UTC

pighive wrote: post #4

chefqual wrote: post #2

This is very smart. Also a very good choice of language. Reminds me of a comment from a few days ago https://news.ycombinator.com/item?id=18144385

I too looked at the comment and thought, yes it’s about time to get my SQL act together. I can write sql queries, and can also understand looking at them what they supposedly do, but my day job doesn’t really demand more than a simple select on two tables. Where can I go learn/explore more competitive SQL?

this is an amazing resource http://selectstarsql.com/

Re: Mixnode: Turn the web into a database

#6

Post by fooyc » Sun, Oct 07, 2018, 5:12 PM UTC

Is this an alternative to crawling/scraping, or a way to exploit the result of crawling/scraping ?

What they offer is not really clear from the article. It seems that they only provide a raw SQL interface over a database of crawled web pages (to be fair, they added a few HTML-related SQL functions). We don’t know where this database come from, or who is supposed to provide it.

Great to see that SQL is making a come back, though.

Re: Mixnode: Turn the web into a database

#7

Post by appdrag » Sun, Oct 07, 2018, 5:17 PM UTC

Website seems to be down ... Slashdot effect ... But i really like the idea :)

Re: Mixnode: Turn the web into a database

#8

Post by GordonS » Sun, Oct 07, 2018, 5:28 PM UTC

This does look really interesting for research and discovering conten. But I'm not sure how good a replacement it would be for more generally scraping content.

Firstly, if you are scraping you would generally only be targeting a specific list of sites, and you'd want to make sure you were getting the freshest content - which means going straight to the source.

Secondly, while plenty was shown around metadata, there wasn't much shown about extracting actual content. I had expected it to be some kind of clever, AI-hype product that extracted semantic data, but it appears to be much more rudimentary than that, effectively letting you query the DOM with SQL.

I don't mean to hate on it - this really does look interesting - I'm just not convinced there is any real value over existing (or custom) scraping tools.

Re: Mixnode: Turn the web into a database

#9

Post by adambyrtek » Sun, Oct 07, 2018, 5:33 PM UTC

pighive wrote: post #4

chefqual wrote: post #2

This is very smart. Also a very good choice of language. Reminds me of a comment from a few days ago https://news.ycombinator.com/item?id=18144385

I too looked at the comment and thought, yes it’s about time to get my SQL act together. I can write sql queries, and can also understand looking at them what they supposedly do, but my day job doesn’t really demand more than a simple select on two tables. Where can I go learn/explore more competitive SQL?

Really good introduction to some of the more "modern" features: https://modern-sql.com/

Re: Mixnode: Turn the web into a database

#10

Post by mrtksn » Sun, Oct 07, 2018, 5:35 PM UTC

This is cool. Just an idea, do you think that you can make it do select within the DOM? It would be amazing to do SELECT on the document from a human point of view.

I mean from human point of view there are no divs or spans on a page but articles, comments paragraphs pictures links and so on.

Sure it a much larger problem but Google for example seems to be able to extract categorised information from web pages.