Ah, the latest company to replicate ql2's WebQL from the 90s.
https://www.directionsmag.com/article/2901 and still at https://www.ql2.com. Props to Ted Kubaitis who developed WebQL on the side and grew it into a very helpful company for data collection, price tracking, etc.
Mixnode: Turn the web into a database
51–60 of 87 posts
Re: Mixnode: Turn the web into a database
#52Interesting product. BTW: 10 times posted in HN, first time they’re trending (if you click the domain you’ll see the list). Persistence pays I guess :)
Almost every story is posted multiple times.
Re: Mixnode: Turn the web into a database
#53Kids these days... We could have had XHTML, xpath, and the web as a semantic DB. I wonder if the author even knows what these things are, or what happened with the vision of a semantic machine-readable web. I rarely come across engineers who even know what XML is (no, it’s not an alternative encoding format to JSON). It’d be great if CS courses and bootcamps would teach some basic web history.
The author is working with the Web, like as it is. Not an imaginary one where everyone has formatted their page in validated XHTML. This is the reality regardless of the author's age, be it 17 or 70.
Re: Mixnode: Turn the web into a database
#54This is way less cool than the title suggests. They are doing a bunch of crawling and inserting the raw html content into their big centralized database, where you can run queries on the text inside: select url, string_between(content, ' ', ' ') as title from resources where content_type like 'text/html%'
I would say this is more exciting than it looks, though. I used to do a lot of crawling in the early 2000s, and almost all of it was expressed in terms of string_between calls. XPath is more convenient, but I'd say that 85% of the time you can collapse an XPath query into a string_between-style query. It can be awkward and even inconsistent, but in practice it often works well.
Re: Mixnode: Turn the web into a database
#55This does look really interesting for research and discovering conten. But I'm not sure how good a replacement it would be for more generally scraping content. Firstly, if you are scraping you would generally only be targeting a specific list of sites, and you'd want to make sure you were getting the freshest content - which means going straight to the source. Secondly, while plenty was shown around metadata, there w…
It would be great if they allowed people to write custom views for a certain group of pages, and allowed them to be run and indexed by default. Then you could create, for example, an Amazon item page view that scrapes price and description, and reviews, and quantity, and seller and all that shit and it would be scraped and indexed for you. They could make it optional and make it default only when the view becomes pop…
And if this was centralized, everyone would benefit, since, say, amazon would only get indexed by this service, rather than thousands of individual companies with their own bots doing similar things.
Re: Mixnode: Turn the web into a database
#56Earlier quoted context omitted.
I might be missing something, but could you explain why?
Because 99% of scraping is parsing that column. At least that’s been my experience.
Scraping is literally just the successful acquisition of content.
You're getting into data parsing. Almost nobody scrapes data without processing, parsing and normalizing it, but scraping is getting the data in the first place.
Scraping isn't easy at scale, though. You have to distribute your crawlers, adhere to TOS (in theory) and avoid getting blocked. It's simple at small scale, though.
I don't know about the utility of this service, though. It handles the less interesting part of data acquisition and processing. I also agree with other comments that most scraping use cases are targeted and small in scope.
Re: Mixnode: Turn the web into a database
#57Earlier quoted context omitted.
this is an amazing resource http://selectstarsql.com/
The timing of your comment couldn't be better. Only yesterday I kinda messed up in an interview because I wasn't good at SQL. Just cursorily checked the link you posted and it is looking good. Thanks for the suggestion.
Can you remember the questions? I think that I'm relatively good with SQL, but I just realized, I have never been asked any SQL specific questions, even though most of my work has been tied to it. The questions usually revolve around specifics of the engine, not query language itself.
Re: Mixnode: Turn the web into a database
#58Interesting product. BTW: 10 times posted in HN, first time they’re trending (if you click the domain you’ll see the list). Persistence pays I guess :)
Off topic, but thanks for pointing that cool HN feature out
Re: Mixnode: Turn the web into a database
#59I wonder - if you take the top ten keywords and url info from all pages on the web - would the data fit on a micro-sd card !?
There are ~40k words in English. You don't need a full URL, but only a hash. The words could similarly be hashed, most-frequent words to smallest values.
There are slightly shy 2 billion websites worldwide, 200 million are active. A 32-bit integer could index each site. A further hash for site paths.
http://www.internetlivestats.com/total-number-of-websites/
There were 30 trillion unique URLs as of 2012
In August 2012, Amit Singhal, Senior Vice President at Google and responsible for the development of Google Search, disclosed that Google's search engine found more than 30 trillion unique URLs on the Web, crawls 20 billion sites a day, and processes 100 billion searches every month [2] (which translate to 3.3 billion searches per day and over 38,000 thousand per second).
http://www.internetlivestats.com/google-search-statistics/
There are terabyte MicroSD cards, so this looks viable.
Re: Mixnode: Turn the web into a database
#60When I saw the column called content my face went a little bit sad.
I hear you, but given they went with SQL, what choice did they have? No schema could adequately represent all the possible content in the document body.