Launch HN: Dashblock (YC S19) – Turn Any Website into an API • Hacker News Board

Re: Launch HN: Dashblock (YC S19) – Turn Any Website into an API

#71

Post by awad » Wed, Sep 18, 2019, 11:40 PM UTC

How do you differentiate from Octoparse?

Re: Launch HN: Dashblock (YC S19) – Turn Any Website into an API

#72

Post by mLuby » Thu, Sep 19, 2019, 12:09 AM UTC

gargarplex wrote: post #68

HPouillot wrote: post #54

Earlier quoted context omitted.

We did :) It works on all kind of pages. You just have to set it up on one page and it will work on all similar pages of the website. Did you have in mind to train a model to recognise careers pages across websites ?

Yeah, that would be really helpful. I want to monitor careers pages of all local companies in the Crunchbase NYC geo in order to help candidates search for local companies by keywords (eg C#). We have an API already (syncs with Algolia) to receive the jobs, with unique key on each job’s URI; and we wouldn’t want to scrape more than once per day.

Would love to use that if/when you get it working.

Re: Launch HN: Dashblock (YC S19) – Turn Any Website into an API

#73

Post by mLuby » Thu, Sep 19, 2019, 12:12 AM UTC

I like how simple it is—best of luck! (BTW I think your demo video can be shortened in the middle; after 6 selectors it's clear how that works.)

1. How hard would it be to do inputs? That is, there's a form that I have to fill out manually but I want to do so by API.

2. How well does this work for creating UX tests? The Selenium "no code" tools I've seen are terrible.

Re: Launch HN: Dashblock (YC S19) – Turn Any Website into an API

#74

Post by statictype » Thu, Sep 19, 2019, 12:28 AM UTC

tipalink wrote: post #66

whoisjuan wrote: post #59

This has been tried many times and it never seems to gain traction to become a relevant concept. Off the top of my head, I remember Kimono Labs that looked quite promising. Then it was acquired by Palantir and shut down. I also have seen many solutions that are similar (basically most scraping companies, like Diffbot which also claims to use machine learning for their extraction techniques) What's the plan here to re…

In regards to your question about companies' concerns: if the data is made publicly available (i.e. web page is not behind authentication), then why should it matter how it's accessed?

The API would bypass ads on the page?

I feel like this would have the same sort of friction that RSS had.

Which is to say, it could certainly still work.

Re: Launch HN: Dashblock (YC S19) – Turn Any Website into an API

#75

Post by MCorbani » Thu, Sep 19, 2019, 12:35 AM UTC

awad wrote: post #71

How do you differentiate from Octoparse?

There is plenty of differences among which 1/ we don't rely on classic selectors (CSS, xPath, etc) which allows us to be resilient to website updates 2/ we offer a simple UI that automates data selection and structuration and 3/ we are available on Windows and MacOS =)

Re: Launch HN: Dashblock (YC S19) – Turn Any Website into an API

#76

Post by HPouillot » Thu, Sep 19, 2019, 12:37 AM UTC

awad wrote: post #71

How do you differentiate from Octoparse?

We use machine learning to extract the content of the page, which means when the webpage changes you don't have to update your selection like Octoparse.

Re: Launch HN: Dashblock (YC S19) – Turn Any Website into an API

#77

Post by HPouillot » Thu, Sep 19, 2019, 12:41 AM UTC

mLuby wrote: post #73

I like how simple it is—best of luck! (BTW I think your demo video can be shortened in the middle; after 6 selectors it's clear how that works.) 1. How hard would it be to do inputs ? That is, there's a form that I have to fill out manually but I want to do so by API. 2. How well does this work for creating UX tests? The Selenium "no code" tools I've seen are terrible.

Thanks ! 1. It changes the user experience but the underlying model stay the same and will allow our user to record session with inputs and clicks in next releases. 2. Indeed, if you can replay a session you check the data is what you expected. What solution have you tried so far ?

Re: Launch HN: Dashblock (YC S19) – Turn Any Website into an API

#78

Post by quickthrower2 » Thu, Sep 19, 2019, 1:03 AM UTC

If this works for amazon.com.au with it's 20 different page layouts and page navigation systems (sometimes ajax, sometimes not) for different product types, I'll be impressed.

Re: Launch HN: Dashblock (YC S19) – Turn Any Website into an API

#79

Post by vessenes » Thu, Sep 19, 2019, 1:25 AM UTC

Hey Hugues and Max, congratulations.

Can I ask some questions about how this would apply to a project of mine?

I currently create a personal newspaper, printed daily in my office. It’s a reasonably large piece of software that pulls in my calendar, emails, news stories I care about, twitter feeds, weather, stock quotes, etc.

I use python’s newspaper library for parsing RSS feed links to news sites, but it is, at times lacking, so dashblock strikes me as interesting.

What I understand from the video is I could over time build out APIs with dashblock for major news sites; this would help with a few sites that are hard for newspaper.

How would I use dashblock in production - unattended, CLI Linux or Mac? Also, it looks really slow in the video, is this typical speeds? Is this something that you require be run on your cloud, or could I run it locally?

Thanks, Peter

Re: Launch HN: Dashblock (YC S19) – Turn Any Website into an API

#80

Post by surfer77 » Thu, Sep 19, 2019, 1:30 AM UTC

Love this, I submitted this to API List (https://apilist.fun/api/dashblock), been seeing more and more scraping apis become available, it seems it is a becoming a very competitive industry and this is a unique solution (at least from what I've seen)