Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Context.dev (YC S26) – API to get structured data from any website

context.dev

Post reply on HN
Print view View original

41–50 of 102 posts

Re: Launch HN: Context.dev (YC S26) – API to get structured data from any website

#44

Post by TheYahiaBakour »

dataviz1000 wrote: post #40
TheYahiaBakour wrote: post #39

Earlier quoted context omitted.

Fascinating, this is definitely a path i was thinking about for a bit but decided against doing it primarily since it changes the business fundamentally

You are smart! Another YC startup went this path, scraping targeting endpoints, and they didn't get much traction and they have since pivoted. > We maintain a caching layer and avoid hammering websites. This made me think of it. It is another way to avoid hammering websites by using extremely targeted requests for the most part bypassing HTML, DOM, and JavaScript.

100% yes, i found out about that company recently too, it's a tempting idea for sure

Re: Launch HN: Context.dev (YC S26) – API to get structured data from any website

#46

Post by twosdai »

TheYahiaBakour wrote: post #32
twosdai wrote: post #29

Are you using residential proxies? How do you handle websites that don't want to be scraped. EG if I start passing in Linkedin pages what is your expectation of the result that people would see per profile. EDIT: Congrats on the launch seriously hard work, just wanting to understand your scraping stance more. I've worked with a lot of tools on this, didn't mean for my initial comment to be adversarial.

hi, no worries at all, it came off perfectly fine yes, we use residential proxies + all requests go are js-rendered, we maintain a caching layer which is 95%+ opted into by customers it's all included in the credit price, great value compared to alternatives, the business model does rely on scale and our margin gets better the more requests we serve (esp infra cost for k8 + browser fleet) to answer your e.g., yes pub…

What major edge cases have you found so far for the browser scraping method, its really complex like dealing with Auth, or pages that use pop ups, content blockers . etc...
Post reply on HN