61–70 of 97 posts
I tried my hand at this once from har file, build a DAG of the requests and responses and how their parameters map into each other, and then try to build a spec.
But it was very brittle. I imagine now with LLMs its much more viable. I always thought generating clients and load tests would be a cool use case.
Earlier quoted context omitted.
Definitely get that. Being hammered by scrapers is a massive PITA (especially with latest aggressive AI crawlers). We focus primarily on allowing people to automate their existing workflows. For all hosted workflows we have rate limits to prevent mass scraping/affecting server workload in any real capacity. In fact, because we don't load js/html and hit endpoints directly I would guess that we consume less server res…
Oh no I’m not worried about the resources or rate limits. If I’d make a mobile app and users simply use your automation service instead of my mobile app, I’d lose traffic/money/motivation to improve it. If they run into issues from your service now it could make my app look bad while the error isn’t with the app. See tailwind for an example tale.
Interesting product (Caido co-founder here). It is very hard to nail auth, probably the most underlooked aspect by end users. We are working on something similar for PoC reproduction of vulnerabilities. Fingerprinting is also a hard thing to match perfectly, I would be curious to know what your strategy is on that. My experience has been that unless you bundle multiple TLS lib it is almost impossible to do at 100% be…
We’re currently running a variety of stuff for TLS/HTTP2. If you download you can see the full trace of the connection. We dump the TLS connection byte for byte with the different structured subsections. With tls.peet.ws and bogdann finn’s tls-client (which we use parts of with some modifications) I would say that http3/tcp fingerprinting is probably the remaining issue. We currently don’t support http3 connections (…
It’s pretty funny to see that in their demo video given it’s a blatant violation of their ToS.
Earlier quoted context omitted.
The goal is not to scrape sites en-masse, but to allow people to automate their existing workflows and actions that they perform already via a browser. I understand the concerns around this being unethical, and it's something I spent a lot of time thinking about when I worked on automations previously. I've written a decent amount about how I don't think that sneaker bots or ticket bots are ethical. I don't support m…
> Is using puppeteer to automate a form submission a violation of ToS? If so then why is using a screen reader not? Without taking a position on the ethics of automation, surely this isn't a serious question? Things that the ToS prohibits you from doing are ToS violations, and other things aren't. For instance, from AirBnb's terms of service: "Do not use bots, crawlers, scrapers, or other automated means to access or…
Related. We have several third party web apps in use. These apps don't expose a public api, but they are all single page web apps. We wanted to connect claude code to these web apps for our limited use case. We opened chrome, navigated the entire website, the downloaded the network tab as an har file. The asked claude to analyze and document the apis as an openapi json. Worked amazing. Next step - we wrote a small py…
Congratulations on the launch. Totally unrelated, I am just curious about why you chose the name, as someone who is Ugandan and was born in raised in Kampala (which is the Capital City of Uganda BTW). Congratulations again.
It was the (generated) name of the Conductor workspace when I started the project. We were going to rename it before launch but the name stuck lol :)
Earlier quoted context omitted.
> Is using puppeteer to automate a form submission a violation of ToS? If so then why is using a screen reader not? Without taking a position on the ethics of automation, surely this isn't a serious question? Things that the ToS prohibits you from doing are ToS violations, and other things aren't. For instance, from AirBnb's terms of service: "Do not use bots, crawlers, scrapers, or other automated means to access or…
Ah yes, AirBnB, the company that famously hacked Craigslist to achieve viral growth by using a bot, crawler, scraper and definitely automated means to access and collect Craiglist's data and other content from and otherwise interact with the Craigslist platform.