31–40 of 102 posts
I was using Context back when it was still Brand.dev. I found it to be a great product- one of those rare APIs that immediately made a problem I had disappear. Had it in production within an hour of signing up Agents need clean/current context from the web, and this is the best way I’ve found to give it to them. The internet is clearly moving in this direction: companies are starting to realize their sites need to be…
Are you using residential proxies? How do you handle websites that don't want to be scraped. EG if I start passing in Linkedin pages what is your expectation of the result that people would see per profile. EDIT: Congrats on the launch seriously hard work, just wanting to understand your scraping stance more. I've worked with a lot of tools on this, didn't mean for my initial comment to be adversarial.
yes, we use residential proxies + all requests go are js-rendered, we maintain a caching layer which is 95%+ opted into by customers
it's all included in the credit price, great value compared to alternatives, the business model does rely on scale and our margin gets better the more requests we serve (esp infra cost for k8 + browser fleet)
to answer your e.g., yes public linkedin pages will work fine, anything behind a login we don't really support out of the box until we can figure out a safe way to do so, since that's where red lines are drawn
we step in whenever we see our service is hitting a website more than it should, this usually means reaching out to a customer for clarity on why they are not opting into the cache, we have alot of safeguards around fraud/spam and will let someone know if their request pattern looks like they're causing harm
Earlier quoted context omitted.
Better business model would be some sort of micropayment setup - allow humans, but make scrapers pay a hundredth of a cent for access.
Is this similar? https://blog.cloudflare.com/monetization-gateway/
Are you using residential proxies? How do you handle websites that don't want to be scraped. EG if I start passing in Linkedin pages what is your expectation of the result that people would see per profile. EDIT: Congrats on the launch seriously hard work, just wanting to understand your scraping stance more. I've worked with a lot of tools on this, didn't mean for my initial comment to be adversarial.
Hey Yahia, I recently saw your startup on the YC registry and found it to be one of the most promising of the batch, so good luck with this! Do you mind if I ask how you managed to get your first users on Brand.dev?
i got my first customer after 1 month, and then just worked like hell to make him happy even though it was a very large company paying me $99/mo at the time, they're still a customer too
since then, i just locked in on every customer until they were thrilled, and once we hit 50, word of mouth started to kick in, now we're at 300+ and get a referral every other day which is awesome
i think our community appreciates effort, and when someone goes the extra mile to make sure your problems are solved, you tend to remember that. it doesn't scale super well, but you'd be surprised how far it'll take you
> Reverse-engineers any website by doing a breadth search across every transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub), listing them all, and generating a typed JSON API that bypasses almost all bot protections — including Turnstile. I didn't include the ability, but it bypassed the most advanced ChatGPT + Turnstile. Built with self-improving Claude Code agents that rewrite their own instructions until fresh agents consistently succeed.
> Once connected to a page, it intercepts every byte of network traffic — then actively drives the page to surface endpoints that only fire on interaction. It types into forms, clicks buttons, scrolls, triggers modals, paginates, submits searches, and walks through multi-step flows, watching what each action produces on the wire. Every request gets captured with its method, headers, payload shape, and response, then classified by transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub). The result is a complete map of the site's real API surface — including the hidden endpoints that only exist behind a click — turned into typed proxy routes you can curl.
Have a look at Intercept. [0] I don't have a need for it, likely it is dated and will require some more tuning, and I want to get away from scraping. Creating typed Typescript proxy API for any website might be something you find useful. > Reverse-engineers any website by doing a breadth search across every transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub), listing them all, and generating a typed JSON A…
Have a look at Intercept. [0] I don't have a need for it, likely it is dated and will require some more tuning, and I want to get away from scraping. Creating typed Typescript proxy API for any website might be something you find useful. > Reverse-engineers any website by doing a breadth search across every transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub), listing them all, and generating a typed JSON A…
Fascinating, this is definitely a path i was thinking about for a bit but decided against doing it primarily since it changes the business fundamentally
Another YC startup went this path, scraping targeting endpoints, and they didn't get much traction and they have since pivoted.
> We maintain a caching layer and avoid hammering websites.
This made me think of it. It is another way to avoid hammering websites by using extremely targeted requests for the most part bypassing HTML, DOM, and JavaScript.