Mixnode: Turn the web into a database • Hacker News Board

Re: Mixnode: Turn the web into a database

#81

Post by ajawee » Mon, Oct 08, 2018, 3:10 AM UTC

testplzignore wrote: post #17

fooyc wrote: post #6

Is this an alternative to crawling/scraping, or a way to exploit the result of crawling/scraping ? What they offer is not really clear from the article. It seems that they only provide a raw SQL interface over a database of crawled web pages (to be fair, they added a few HTML-related SQL functions). We don’t know where this database come from, or who is supposed to provide it. Great to see that SQL is making a come b…

Based on a quick google search (e.g., https://stackoverflow.com/questions/46673751/nutch-vs-heritr... ), their existing product appears to be a hosted solution for crawling the web. This new product sounds like it is just a query language that can be used on top of what you yourself have paid them to crawl. I don't believe they've actually crawled the whole web and are providing an interface to that. Their website sa…

I guess, there are using common crawl as a base. Not sure wether they are doing actual crawling along with it.

Re: Mixnode: Turn the web into a database

#82

Post by peteforde » Mon, Oct 08, 2018, 3:58 AM UTC

What does the creator of Mixnode expect, believe or hope that people will use this tool for?

It's a critical problem that the site doesn't explain why people would want to use it. What tool or behaviour will it replace? What are those people doing today?

"I am a paying customer; who am I and what are my problems?" How many of those customers are there, and how much are they willing to pay to solve their problems?

Is it faster/better to use Mixnode than to create my own scraper? Is it possible to purchase an enterprise instance that runs in our datacentre? Is this flexible enough to accommodate my future business rules?

How much will this cost, and who do I call if it breaks? Can I purchase an SLA comparable to what AWS offers?

Most businesses have about 100 hard questions associated with them, where if you have good answers you're probably going to do just fine. The answers are the easy part; figuring out the questions for each company is hard.

Re: Mixnode: Turn the web into a database

#83

Post by pouta » Mon, Oct 08, 2018, 4:12 AM UTC

I would love to know more about this. I extensively use scraping and crawling at my startup and it's too much one consuming

Re: Mixnode: Turn the web into a database

#84

Post by stonogo » Mon, Oct 08, 2018, 4:56 AM UTC

JustThrowMeAway wrote: post #32

Their Twitter is strange: It says Likes: 1023 But when you click on it, it says @mixnode hasn't liked any Tweets yet What happened?

It means they unliked all their likes en masse. Twitter's counts of these things are eventually consistent... with strong emphasis on eventual.

Re: Mixnode: Turn the web into a database

#85

Post by Ayesh » Mon, Oct 08, 2018, 9:08 AM UTC

AznHisoka wrote: post #26

I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…

This. I write crawler software (adapters mostly) for the same client, and I could never figure out anneasy way for my client to specify the xpath/case paths in a meaningful way and extract the data.

Every crawler task requires different paging methods, different xpath patterns, etc that it makes things more complicated to generalize it.

Re: Mixnode: Turn the web into a database

#86

Post by samat » Mon, Oct 08, 2018, 12:16 PM UTC

AznHisoka wrote: post #26

I dont want to be too harsh but I wouldnt find this useful (and my job depends a lot on crawling data) 1. When most people scrape data, they generally are interested in a very specific niche subset of the web. Sure you might have a billion row database of every article ever publisbed, but do you have all the rows of every item sold in FootLocker.com, for instance? As well as the price of each item(which is extracted…

There is a product (several of them from one company, actually) for crawling, but it's more of a tooling than a end-user product https://scrapinghub.com

Re: Mixnode: Turn the web into a database

#87

Post by rosha » Wed, Oct 10, 2018, 6:51 AM UTC

the idea of turning the web into a database colums/rows is hell of a great crazy idea, to be honest I was like wow, good luck with it.

I still believe that sometimes you might need some real anonymous way to crawl data to syndicate it, think of linkedin data, how are you going to insert the data when Linkedin blocks every single request you do? currently I use a paid service that has a crawler which I use for getting the data I need. https://proxycrawl.com/anonymous-crawler-asynchronous-scrapi...

Do you think Mixnode can help in getting to insert row data from difficult websites like Linkedin or Google?