Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built an SDK that scrambles HTML so scrapers get garbage

obscrd.dev

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Show HN: I built an SDK that scrambles HTML so scrapers get garbage

#33

Post by [deleted] »

larsmosr wrote: post #13
lich_king wrote: post #5

You break highlighting and copy-and-paste. If I want to share or comment on a piece of your website... I can't. I guess this can be a "feature" in some rare cases, but a major usability pain otherwise. I'm not a fan of all the documentation and marketing content for this project evidently being AI-generated because I don't know which parts of it are the things you believe and designed for, and which are just LLM verb…

Copy-paste breaking is intentional for protected content but it's opt-in per component, not whole-site. On the AI docs concern, fair point. To answer directly: I've confirmed the obfuscation defeats any scraper reading raw HTML via HTTP requests. Whether GPTBot or ClaudeBot use headless browsers internally, I honestly don't know. The README threat model lists headless browsers under "what it does NOT stop" for that r…

[deleted]

Re: Show HN: I built an SDK that scrambles HTML so scrapers get garbage

#34

Post by larsmosr »

lich_king wrote: post #5

You break highlighting and copy-and-paste. If I want to share or comment on a piece of your website... I can't. I guess this can be a "feature" in some rare cases, but a major usability pain otherwise. I'm not a fan of all the documentation and marketing content for this project evidently being AI-generated because I don't know which parts of it are the things you believe and designed for, and which are just LLM verb…

Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3;

Official OpenAI documentation: https://platform.openai.com/docs/gptbot

Re: Show HN: I built an SDK that scrambles HTML so scrapers get garbage

#36

Post by larsmosr »

ozgurozkan wrote: post #31

[flagged]

The breadcrumb approach right now is simple invisible markers, not paraphrase-resistant watermarking. You're right that semantic watermarking that survives LLM rephrasing is the harder and more interesting problem. It's on the radar but not in scope for v1.

Re: Show HN: I built an SDK that scrambles HTML so scrapers get garbage

#37

Post by yjftsjthsd-h »

larsmosr wrote: post #28
well_ackshually wrote: post #20

[flagged]

The TalkBack issue is useful feedback, thank you. I tested with NVDA and VoiceOver but not TalkBack on Android. If light mode is reading individual words instead of a continuous block that's a real bug I want to fix. On the broader point, I hear you, but I think there's a middle ground. Not all content is public knowledge. Some of it is premium, proprietary, or behind a paywall. The people publishing it should get to…

> On the broader point, I hear you, but I think there's a middle ground. Not all content is public knowledge. Some of it is premium, proprietary, or behind a paywall. The people publishing it should get to decide whether it becomes free training data.

I don't follow. Are you suggesting that someone is scraping private sites that they have to log in on in order to train AI on it?

Re: Show HN: I built an SDK that scrambles HTML so scrapers get garbage

#38

Post by dlcarrier »

Oh great, another method to make screen readers and keyboard navigation impossible.

At this point, bots are better at getting data out of web pages than people are. (And have been so for at least a few years: https://www.usenix.org/conference/usenixsecurity23/presentat...)

All we're doing now is making it easier to get data from a web scraper than to browse to the web page ourselves.

Post reply on HN