With XHTML 2.0 and related tools like XQuery it could've been a matter of course. Hell XQuery is still a much better tool for this job than SQL, but no one cares.
I mean
> string_between(content, '', '') as title
really?
Mixnode: Turn the web into a database
11–20 of 87 posts
Re: Mixnode: Turn the web into a database
#12This does look really interesting for research and discovering conten. But I'm not sure how good a replacement it would be for more generally scraping content. Firstly, if you are scraping you would generally only be targeting a specific list of sites, and you'd want to make sure you were getting the freshest content - which means going straight to the source. Secondly, while plenty was shown around metadata, there w…
It would be great if they allowed people to write custom views for a certain group of pages, and allowed them to be run and indexed by default. Then you could create, for example, an Amazon item page view that scrapes price and description, and reviews, and quantity, and seller and all that shit and it would be scraped and indexed for you. They could make it optional and make it default only when the view becomes popular based on their own stats. How awesome and useful would that be?
Re: Mixnode: Turn the web into a database
#13With XHTML 2.0 and related tools like XQuery it could've been a matter of course. Hell XQuery is still a much better tool for this job than SQL, but no one cares. I mean > string_between(content, ' ', ' ') as title really?
I think the main point here is that you can get data from many different places without having to run crawlers. Like the etld example. tbh I too want to see better DOM handling (stringBetween is not the best function for HTML parsing lol) but the main value prop is pretty impressive.
Re: Mixnode: Turn the web into a database
#14For anyone else confused by the title, this is an alternative to you doing the crawling.
It's not some proposal to re-architect the web so that crawling isn't necessary. It's a data warehouse of the web as a service.
Re: Mixnode: Turn the web into a database
#15Kids these days... We could have had XHTML, xpath, and the web as a semantic DB. I wonder if the author even knows what these things are, or what happened with the vision of a semantic machine-readable web. I rarely come across engineers who even know what XML is (no, it’s not an alternative encoding format to JSON).
It’d be great if CS courses and bootcamps would teach some basic web history.
Re: Mixnode: Turn the web into a database
#16Yahoo pipes had a very similar thing long back !
Re: Mixnode: Turn the web into a database
#17Is this an alternative to crawling/scraping, or a way to exploit the result of crawling/scraping ? What they offer is not really clear from the article. It seems that they only provide a raw SQL interface over a database of crawled web pages (to be fair, they added a few HTML-related SQL functions). We don’t know where this database come from, or who is supposed to provide it. Great to see that SQL is making a come b…
Based on a quick google search (e.g., https://stackoverflow.com/questions/46673751/nutch-vs-heritr...), their existing product appears to be a hosted solution for crawling the web.
This new product sounds like it is just a query language that can be used on top of what you yourself have paid them to crawl. I don't believe they've actually crawled the whole web and are providing an interface to that. Their website says things like "the entire web" and "trillions of rows", but I'm guessing that's only true if you pay them a few million dollars to do that.
Re: Mixnode: Turn the web into a database
#18An interesting way to enter the search indexing space.
Re: Mixnode: Turn the web into a database
#19Earlier quoted context omitted.
I too looked at the comment and thought, yes it’s about time to get my SQL act together. I can write sql queries, and can also understand looking at them what they supposedly do, but my day job doesn’t really demand more than a simple select on two tables. Where can I go learn/explore more competitive SQL?
this is an amazing resource http://selectstarsql.com/
The timing of your comment couldn't be better.
Only yesterday I kinda messed up in an interview because I wasn't good at SQL. Just cursorily checked the link you posted and it is looking good. Thanks for the suggestion.
Re: Mixnode: Turn the web into a database
#20Reminds me of YQL, https://developer.yahoo.com/yql/.