81–90 of 102 posts
Love this, I submitted this to API List ( https://apilist.fun/api/dashblock ), been seeing more and more scraping apis become available, it seems it is a becoming a very competitive industry and this is a unique solution (at least from what I've seen)
This has been tried many times and it never seems to gain traction to become a relevant concept. Off the top of my head, I remember Kimono Labs that looked quite promising. Then it was acquired by Palantir and shut down. I also have seen many solutions that are similar (basically most scraping companies, like Diffbot which also claims to use machine learning for their extraction techniques) What's the plan here to re…
Timing is perfect because to do that, you need a robust headless browser and a smart way to locally identify the elements on the page if you don't want to maintain your scripts. That's why we use Puppeteer and TensorflowJS which didn't exists 2-3 years ago.
But sure, there are website owners who don't want an API for their website. Our plan is not to fight against them but to start with owner that are already convinced that they could benefits from automating the usage of their website. The banking sector understood that, and that's why Yodlee and Plaid are so successful today.
And if you step back, there are tons of websites that don't have the ressources to create an API (30% of the websites have been created using Wordpress) and don't know the value they could generate from it.
So yes, we'll have to overcome a lot of challenges to build this technology and make it accessible to everyone but we are convinced that the Internet will be used more and more programmatically in the future and we are just paving the way for it ;)
Hey Hugues and Max, congratulations. Can I ask some questions about how this would apply to a project of mine? I currently create a personal newspaper, printed daily in my office. It’s a reasonably large piece of software that pulls in my calendar, emails, news stories I care about, twitter feeds, weather, stock quotes, etc. I use python’s newspaper library for parsing RSS feed links to news sites, but it is, at time…
You can create an API for any website (news website included) from our Mac/Windows software and you can access this API from anywhere. It runs on our servers and you can query it from any language you want. Let us know if you need more help hello@dashblock.com
If this works for amazon.com.au with it's 20 different page layouts and page navigation systems (sometimes ajax, sometimes not) for different product types, I'll be impressed.
This has been tried many times and it never seems to gain traction to become a relevant concept. Off the top of my head, I remember Kimono Labs that looked quite promising. Then it was acquired by Palantir and shut down. I also have seen many solutions that are similar (basically most scraping companies, like Diffbot which also claims to use machine learning for their extraction techniques) What's the plan here to re…
How do you differentiate from Octoparse?
There is plenty of differences among which 1/ we don't rely on classic selectors (CSS, xPath, etc) which allows us to be resilient to website updates 2/ we offer a simple UI that automates data selection and structuration and 3/ we are available on Windows and MacOS =)
I'm not buying this, does AI process html as text lol? Surely it process it as a tree, right?
This has been tried many times and it never seems to gain traction to become a relevant concept. Off the top of my head, I remember Kimono Labs that looked quite promising. Then it was acquired by Palantir and shut down. I also have seen many solutions that are similar (basically most scraping companies, like Diffbot which also claims to use machine learning for their extraction techniques) What's the plan here to re…
In regards to your question about companies' concerns: if the data is made publicly available (i.e. web page is not behind authentication), then why should it matter how it's accessed?
For example, Google temporarily bans your IP when you hit things like Google Play urls multiple times in a few minutes. This is clearly an attempt to block anyone but a human to extract information from the Play store.
I might have missed it, but how can I see (or edit) the configuration of my configured API? It looks like all I can do is run the API or delete it.