Mixnode: Turn the web into a database • Hacker News Board

Re: Mixnode: Turn the web into a database

#61

Post by [deleted] » Sun, Oct 07, 2018, 10:06 PM UTC

dredmorbius wrote: post #59

z3t4 wrote: post #46

I wonder - if you take the top ten keywords and url info from all pages on the web - would the data fit on a micro-sd card !?

There are ~40k words in English. You don't need a full URL, but only a hash. The words could similarly be hashed, most-frequent words to smallest values. There are slightly shy 2 billion websites worldwide, 200 million are active. A 32-bit integer could index each site. A further hash for site paths. http://www.internetlivestats.com/total-number-of-websites/ There were 30 trillion unique URLs as of 2012 In August 201…

[deleted]

Re: Mixnode: Turn the web into a database

#62

Post by ma2rten » Sun, Oct 07, 2018, 10:17 PM UTC

AznHisoka wrote: post #26

I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…

I could imagine that they let you schedule a custom crawl as an added service.

Re: Mixnode: Turn the web into a database

#63

Post by NicoJuicy » Sun, Oct 07, 2018, 10:21 PM UTC

nycdatasci wrote: post #37

drgath wrote: post #20

Reminds me of YQL, https://developer.yahoo.com/yql/ .

Which used to support a similar usage (select from url) but no longer does.

Yeah, html table is deprecated :(

Re: Mixnode: Turn the web into a database

#64

Post by NicoJuicy » Sun, Oct 07, 2018, 10:21 PM UTC

bootcat wrote: post #16

Yahoo pipes had a very similar thing long back !

It was YQL i suppose

Re: Mixnode: Turn the web into a database

#65

Post by rpedela » Sun, Oct 07, 2018, 10:26 PM UTC

AznHisoka wrote: post #26

I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…

As a data engineer who needs to crawl websites sometimes, Mixnode looks interesting to me. I agree that it is hard to make scraping a product because it is so use case specific. However crawling, defined as downloading all HTML, PDF, images on a given site, is a pretty common first step and something that could be a product. Then turning that into SQL sounds pretty awesome.

Re: Mixnode: Turn the web into a database

#66

Post by keyle » Sun, Oct 07, 2018, 10:32 PM UTC

Somewhat related, I did this hacky thing in 2012. Worked a charm... No db required...

https://github.com/keyle/json-anything

Re: Mixnode: Turn the web into a database

#67

Post by adrianratnapala » Sun, Oct 07, 2018, 10:43 PM UTC

solarkraft wrote: post #34

AznHisoka wrote: post #26

I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…

From what I understand this is not trying to solve the typical e-commerce problem of closely watching your competitors selling something, but rather trying to provide a database to people interested in content on the web. It probably won't solve the problems you're working on, but I could imagine quite a lot of interesting text analysis cases.

Yes a generalisation of the way wikipedia provides a web site, and a more machine readable for of what they do.

If I get back into blogging, I would be really happy to have my posts structured and indexed in whatever standard way made them easier bots to use.

Re: Mixnode: Turn the web into a database

#68

Post by utopcell » Sun, Oct 07, 2018, 10:52 PM UTC

It seems to me that the moment you'd need to do anything interesting with a website, you'd need to crawl a lot of its pages and you'd hit robots.txt limitations very quickly.

Re: Mixnode: Turn the web into a database

#69

Post by scientits » Sun, Oct 07, 2018, 10:52 PM UTC

Can you please elaborate on the pricing model?

Re: Mixnode: Turn the web into a database

#70

Post by [deleted] » Sun, Oct 07, 2018, 11:04 PM UTC

mjhirn wrote: post #35

AznHisoka wrote: post #26

I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…

I haven't tried Mixnode yet, but the way I understand it, it lets you query websites and retrieve their HTML content that you can then parse - without you having to crawl the site. Looking at their Github, they seem to utilize WARC, so they may also allow you to request the website for certain timestamps? That being said, I find this highly interesting, if it works like that. We are working on a peer-to-peer database…

[deleted]