Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixnode: Turn the web into a database

mixnode.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: Mixnode: Turn the web into a database

#71

Post by diminoten »

AznHisoka wrote: post #26

I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…

> but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted from some obscure xpath)

What if they did? Would you buy it then? What could they possibly offer you before you'd be willing to use their product?

Re: Mixnode: Turn the web into a database

#72

Post by wolco »

ssalka wrote: post #47
AznHisoka wrote: post #26

I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…

What if the product was a framework for sourcing, aggregating, and visualizing data? When the user is put in control, you don't need to trust the product to do these things for you - it simply enables you to do what you want. I think this is where the web is headed - where common users gain the ability to perform tasks that currently only developers or technical experts can do.

It's always been the goal to empower the user but you also have the movement to simplify everything.

Re: Mixnode: Turn the web into a database

#74

Post by dang »

fooyc wrote: post #21
adrianmonk wrote: post #14

For anyone else confused by the title, this is an alternative to you doing the crawling. It's not some proposal to re-architect the web so that crawling isn't necessary. It's a data warehouse of the web as a service.

It’s not even an alternative to crawling. It’s just a way to exploit the result of crawling. You have to crawl first. Very misleading title.

Ok, we've taken that bit out of the title above.

Re: Mixnode: Turn the web into a database

#75

Post by dredmorbius »

I've conducted several analyses in which I've looked for trends or patterns across multiple websites or domains. Finding out where discussion / content covering specific topics or keywords is an example, see "Tracking the conversation":

https://www.reddit.com/r/dredmorbius/comments/3hp41w/trackin...

That consisted of querying 100 terms over about 100 sites, and scraping Google's (rather inaccurate) "results found" estimate. About 10k Google queries.

Slowing those queries to the point they don't trip Google's bot detection and request CAPTCHAs is the hard part of this -- given a single IP, the queries stretch over a week or more.

A single source to query that information directly would make these investigations far easier. I've several such projects in mind.

Re: Mixnode: Turn the web into a database

#76

Post by garyjob »

Its heartening to know that more and more folks are starting to share the vision of web as Tim Berners Lee by starting such projects. https://en.wikipedia.org/wiki/Giant_Global_Graph

The tag line of this project is similar in fashion to the following projects which I have came across.

https://www.import.io/

https://getdata.io/

https://data-miner.io/

https://www.apify.com/

I am definitely looking forward to seeing more projects like these which will be helpful in transitioning us from Web2.0 to Web3.0

I think the main hurdle we face in transitioning the web we know today to the vision behind all these projects is companies that have already aggregated huge volumes of data. (e.g. Facebook, LinkedIn, Angelist, CrunchBase, Yelp)

They are now doing their best to protect their data to secure their competitive moat. This has the effect of preventing data from being utilized in other ways than was originally intended.

I did write a post about this topic before around 3 months ago as well..

https://medium.com/@garyjob/the-web-is-a-giant-graph-databas...

Re: Mixnode: Turn the web into a database

#77

Post by sgc »

sam0x17 wrote: post #54
anonytrary wrote: post #50

This is way less cool than the title suggests. They are doing a bunch of crawling and inserting the raw html content into their big centralized database, where you can run queries on the text inside: select url, string_between(content, ' ', ' ') as title from resources where content_type like 'text/html%'

I would say this is more exciting than it looks, though. I used to do a lot of crawling in the early 2000s, and almost all of it was expressed in terms of string_between calls. XPath is more convenient, but I'd say that 85% of the time you can collapse an XPath query into a string_between-style query. It can be awkward and even inconsistent, but in practice it often works well.

Having done a fair amount of work with xslt and using regex to strip out bad data, I agree. But 85% is terrible if you are creating a database. Any regex style query on xml or pseudo-xml requires bespoke treatment and a high amount of human hours to check the results before you can be sure an edge case didn't completely destroy your model.

Re: Mixnode: Turn the web into a database

#78

Post by godot »

A few years ago I worked in a startup, and to find customers we needed to find web sites using certain technologies (e.g. wordpress and certain plugins). We used the service of an extremely similar SaaS startup for a little bit -- that basically did the exact same thing as Mixnode. That startup didn't work out and was shut down soon (and my startup didn't work out either). Wish you best of luck and hope things work out for you, maybe the tech climate and trends have evolved since a few years ago and this could work out a business now.

Re: Mixnode: Turn the web into a database

#79

Post by dragonwriter »

dredmorbius wrote: post #59
z3t4 wrote: post #46

I wonder - if you take the top ten keywords and url info from all pages on the web - would the data fit on a micro-sd card !?

There are ~40k words in English. You don't need a full URL, but only a hash. The words could similarly be hashed, most-frequent words to smallest values. There are slightly shy 2 billion websites worldwide, 200 million are active. A 32-bit integer could index each site. A further hash for site paths. http://www.internetlivestats.com/total-number-of-websites/ There were 30 trillion unique URLs as of 2012 In August 201…

> There are ~40k words in English.

The Second Edition of the OED had 171k+ full entries for words in current use, 47k+ for obsolete words, and ~ 9500 sub-entries.

And there have been a number of supplements since.

40k is low by a significant multiple.

Re: Mixnode: Turn the web into a database

#80

Post by dredmorbius »

dragonwriter wrote: post #79
dredmorbius wrote: post #59

Earlier quoted context omitted.

There are ~40k words in English. You don't need a full URL, but only a hash. The words could similarly be hashed, most-frequent words to smallest values. There are slightly shy 2 billion websites worldwide, 200 million are active. A 32-bit integer could index each site. A further hash for site paths. http://www.internetlivestats.com/total-number-of-websites/ There were 30 trillion unique URLs as of 2012 In August 201…

> There are ~40k words in English. The Second Edition of the OED had 171k+ full entries for words in current use, 47k+ for obsolete words, and ~ 9500 sub-entries. And there have been a number of supplements since. 40k is low by a significant multiple.

For key-value lookup, to rouggh magnitudes are 10^4 - 10^5 (words) vs 10^8 (active sites).

This is OOM level analysis, not higgh-precision estimation.

Though I appreciate the correction.

Post reply on HN