Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Someone used my open source project to phish people

andrej.sh

Post reply on HN
Print view View original

21–30 of 64 posts

Re: Someone used my open source project to phish people

#21

Post by jedberg »

MostlyStable wrote: post #19
jedberg wrote: post #15

Earlier quoted context omitted.

Have you tried putting known human writing into pangram? I have. I've gotten 100% AI with multiple samples of my own human writing. It has also given me 50% on things I know were 100% AI written (from my prompts). Pangram and everything like it is useless. The results are random on known samples.

Pangram specifically (as opposed to most other detectors) publish internal audits, and seem to welcome external audits [0]. I'm not saying that you are necessarily wrong, just that in my opinion they have earned a higher bar of criticism than random one off anecdote. [0] https://xcancel.com/JohnHolbein1/status/2059648132250570975#...

That's a fair criticism, I certainly didn't run a full benchmark. Just a few of my own pieces of writing. I also did it a few months ago, maybe it's gotten better since.

Re: Someone used my open source project to phish people

#22

Post by jedberg »

no_multitudes wrote: post #16
jedberg wrote: post #15

Earlier quoted context omitted.

Have you tried putting known human writing into pangram? I have. I've gotten 100% AI with multiple samples of my own human writing. It has also given me 50% on things I know were 100% AI written (from my prompts). Pangram and everything like it is useless. The results are random on known samples.

That's interesting! I have tried to get false positives from pangram and failed, so I trusted it a bit more than any of the others, although I generally just rely on my own intuition. I am curious what your false positive samples looked like, if you're willing to share. (I'm less interested in false negatives; I have successfully produced those myself.)

I'll try to pull them up for you, I'd have to go back and find them on my computer.

Re: Someone used my open source project to phish people

#23

Post by Barbing »

poly2it wrote: post #9
pressbuttons wrote: post #6

Earlier quoted context omitted.

Why do you think this is LLM-generated? Reads perfectly fine to me.

> What stuck with me wasn’t the scale, although 14,000 people getting a phishing email from a domain I own is bad. It was how mundane it was. > There was no exploit. No vulnerability disclosure. No CVE for me to write. The attacker filled out my signup form 942 times, made 942 workspaces, sent 942 batches of about a hundred invitations each, and stopped. They used my tool exactly as designed. The design was just bad…

The comments continue until the patterns are internalized https://news.ycombinator.com/item?id=48316049

Re: Someone used my open source project to phish people

#26

Post by titularcomment »

cobertos wrote: post #12

Is this the new norm for trying to make software projects in the wild? The 14000 sends over 3 hours ( Wondering if LLM-assisted vulnerability hunting will lead to the same gains in scale for bad actors wanting to find spammable channels in applications. The barrier to entry becomes so much greater because any small project, once found, can be wrung dry of all its trust signals by third parties

Abuse such as this wasn't uncommon before, email platforms with lax ratelimits have always been abused through their clients' unsecured infrastructure. The only difference in post-LLM world is the amount of platforms as well as clients popping up in this space with dubious code quality that may lead to more attacks as;

a) having an email-sending product typically meant you had a project with a lot of effort invested into it as well as knowledge

b) the models, tokens spent and review done differs in the world of vibecoding and there is a race to the bottom to produce, produce, produce. Quantity > quality

Re: Someone used my open source project to phish people

#27

Post by firesteelrain »

jedberg wrote: post #15
no_multitudes wrote: post #7

Earlier quoted context omitted.

The sentence construction, choice of vocabulary, and continually breathless tone are all clear indicators this was written by an llm and barely edited. I threw part of it into pangram to get a second opinion: https://www.pangram.com/history/8d6a7de3-86ac-4ce0-86c5-4f93...

Have you tried putting known human writing into pangram? I have. I've gotten 100% AI with multiple samples of my own human writing. It has also given me 50% on things I know were 100% AI written (from my prompts). Pangram and everything like it is useless. The results are random on known samples.

GP is right. This paragraph is a major tell. If you have read enough ChatGPT output that hasn’t been humanized, you start to instantly notice it:

“ The attacker hadn’t broken into anything. They’d just noticed something I hadn’t: I had a verified email-sending domain attached to open, unverified signup, and that’s a useful primitive if you don’t care what you send.”

Re: Someone used my open source project to phish people

#28

Post by choppsv1 »

pressbuttons wrote: post #6
no_multitudes wrote: post #3

[flagged]

Why do you think this is LLM-generated? Reads perfectly fine to me.

shrug it's getting really easy to spot the AI writing style. I find myself scanning for the obvious `tics' and almost always stop reading when I see them. It's too hard to focus on the content at that point for me.

Re: Someone used my open source project to phish people

#29

Post by comrade1234 »

Spammers are relentless. I had something similar happen to me 25 years ago. And once you're found out to allow any sort of information relay you end up in spamming scripts and for decades automated scripts will be trying to send/relay through you using the same api, even if you block it.

You learn to not leave anything open to spammers AT ALL, to your product's detriment because once you're labeled a spammer in this way your product is dead.

Re: Someone used my open source project to phish people

#30

Post by tpoacher »

firesteelrain wrote: post #27
jedberg wrote: post #15

Earlier quoted context omitted.

Have you tried putting known human writing into pangram? I have. I've gotten 100% AI with multiple samples of my own human writing. It has also given me 50% on things I know were 100% AI written (from my prompts). Pangram and everything like it is useless. The results are random on known samples.

GP is right. This paragraph is a major tell. If you have read enough ChatGPT output that hasn’t been humanized, you start to instantly notice it: “ The attacker hadn’t broken into anything. They’d just noticed something I hadn’t: I had a verified email-sending domain attached to open, unverified signup, and that’s a useful primitive if you don’t care what you send.”

Yes, instead he should have written: "The attacka didn broke into nuffin bruv. They jus noticing sometin I aint, yeh?"

Instant betterness.

Post reply on HN