161–170 of 194 posts
Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.
That’s what they are doing. This is a textbook protection racket. “Buy Cloudflare bot protection, otherwise it would be a shame if your site got scraped and ddos’d.” Who is doing the scraping and ddosing? Cloudflare.
It seems like there's a missed use case: web archiving. I don't see any mention of WARC as an output format. This could be useful to journalists and academically if they had it.
Earlier quoted context omitted.
fuck firecrawl. they copied my idea by showing interest in my product and then copied it, used their YC money to give it all out for free. fuck nick in particular. I'm still salty over this
"they copied my idea by showing interest in my product and then copied it". What exactly is revolutionary about Firecrawl or your product? Scraping APIs have been around for over a decade.
Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.
That's too funny. If true, really looking forward to the Cloudflare response here. I'm unsure how you would spin that in a way that didn't seem self-serving.
Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.
That’s what they are doing. This is a textbook protection racket. “Buy Cloudflare bot protection, otherwise it would be a shame if your site got scraped and ddos’d.” Who is doing the scraping and ddosing? Cloudflare.
It was an interesting and sometimes fun part of my career. Working on a site/application that isn't necessarily a tech site, and that I have a personal interest in was pretty great... some of the pace for sales/commercial features less so, with sales making deals requiring deep integrations on impossible timelines. You learn a lot when a self-hosted site is being kicked while it's down... The cloud migration to get a better use of flexible resources, etc.
Doesn't work for pages protected by cloudflare in my experience. What a shame, they could've produced the problem and sold the solution.
Came here to write this. I am getting much better results from Firecrawl (not affiliated with them, just a happy customer).
On one hand, their bots seem much more well behaved than others.
However, running a crawler fleet which is deceptive and evasive in its identification and don't honor REP is no way to build a business.
It seems like there's a missed use case: web archiving. I don't see any mention of WARC as an output format. This could be useful to journalists and academically if they had it.
Earlier quoted context omitted.
"they copied my idea by showing interest in my product and then copied it". What exactly is revolutionary about Firecrawl or your product? Scraping APIs have been around for over a decade.
I was the first to return markdown and use reader mode stuff to strip irrelevant stuff. Theres copying and there's talking to the founder sounding interested to have your team copy what I did in the background. One is fair game, the other is a dick head move.