Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Someone used my open source project to phish people

andrej.sh

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Someone used my open source project to phish people

#51

Post by eggbrain »

reconnecting wrote: post #47
eggbrain wrote: post #43

Earlier quoted context omitted.

To the end platform, what's the difference? Mitigation techniques largely remain the same, in that you make it more time / energy / money than what the end result of their abuse is worth. The platform cares about stopping the abuse -- not neccesarily correctly identifying whether the people abusing their platform are small shop "bot farms" vs organized crime.

To the platform, the difference shows up exactly in the mitigation math. The 'make it cost more than it's worth' model only works when both sides of that ratio are knowable and bounded. With bonus abuse, the reward is fixed and the math is clear, so you can reliably price the abuser out. With organized criminals, you can't actually see what the abuse is 'worth' to them. And they can escalate almost infinitely: mimick…

> [...] With organized criminals, you can't actually see what the abuse is 'worth' to them.

Even without collecting events, you can calculate what the abuse is worth to you, even if the math ends up being fuzzier.

At the small platform operator level (one guy running a platform, as this article), the cost can be as simple as "this pisses me off and I have weekends." They can burn forty hours bolting on JA4 fingerprinting and a disposable-email blocklist to stop an abuser whose dollar-EV to them was roughly zero. Looks irrational, and that's exactly the deterrent — abuse pricing assumes a rational counterpart, and a guy who'll overspend his own life-hours out of stubbornness is unpriceable.

At any scale larger than a small operator, you also do get real numbers -- you can't perfectly price reputation, but you can price traffic and ad conversions, operational costs, LTV of customers (and conversion funnel metrics) etc, all of which don't stay still while abuse increases.

> [...] That's why it's worth collecting events before acting: what the account is about, which IP network they use, whether they fake devices, whether there's any warmup prior to registration. Because that's what helps estimate whether your mitigation will actually work, and lets you respond in a balanced manner instead of under- or over-reacting.

Isn't this just a way to estimate exactly how much the 'abuse' is worth to the abusers?

Re: Someone used my open source project to phish people

#52

Post by no_multitudes »

bradley13 wrote: post #46
no_multitudes wrote: post #3

[flagged]

[flagged]

I am not sure if you are serious? It really has nothing to do with being grammatically correct. The article is chock-full of all of the most common LLM tells. The aggressive use of negative parallelism, filler adjectives like "deeply", and unvaryingly breathless tone all feel like decisions an actual thinking person would not make.

Re: Someone used my open source project to phish people

#53

Post by ShinyLeftPad »

jedberg wrote: post #15
no_multitudes wrote: post #7

Earlier quoted context omitted.

The sentence construction, choice of vocabulary, and continually breathless tone are all clear indicators this was written by an llm and barely edited. I threw part of it into pangram to get a second opinion: https://www.pangram.com/history/8d6a7de3-86ac-4ce0-86c5-4f93...

Have you tried putting known human writing into pangram? I have. I've gotten 100% AI with multiple samples of my own human writing. It has also given me 50% on things I know were 100% AI written (from my prompts). Pangram and everything like it is useless. The results are random on known samples.

It's obvious that existence of AI will hugely dissatisfy people and make them demand authenticity. Market dictates that there will be a satisfaction of that demand even if it is a smoke show.

Anyone with half a brain knows that people can write like that too. Sometimes people reinforce their point. Sometimes they even do it three times.

The problem is not in posts that fail to pass some half imaginary authenticity check. The problem is in technology that makes us want those checks.

Re: Someone used my open source project to phish people

#54

Post by Sophira »

no_multitudes wrote: post #52
bradley13 wrote: post #46

Earlier quoted context omitted.

[flagged]

I am not sure if you are serious? It really has nothing to do with being grammatically correct. The article is chock-full of all of the most common LLM tells. The aggressive use of negative parallelism, filler adjectives like "deeply", and unvaryingly breathless tone all feel like decisions an actual thinking person would not make.

On the one hand, people say that AI models (LLMs, image generators, etc) are just stealing from people and that they cannot be original.

On the other hand, people say that AI models have tells that no actual person would do.

Which is it? You can't have it both ways.

Re: Someone used my open source project to phish people

#55

Post by nottorp »

jedberg wrote: post #21
MostlyStable wrote: post #19

Earlier quoted context omitted.

Pangram specifically (as opposed to most other detectors) publish internal audits, and seem to welcome external audits [0]. I'm not saying that you are necessarily wrong, just that in my opinion they have earned a higher bar of criticism than random one off anecdote. [0] https://xcancel.com/JohnHolbein1/status/2059648132250570975#...

That's a fair criticism, I certainly didn't run a full benchmark. Just a few of my own pieces of writing. I also did it a few months ago, maybe it's gotten better since.

I wanted to test the service by having it check a few paragraphs from a known writer that lived before "AI".

I pasted the text into their form, pressed check... and only then they told me they want me to sign up first.

Good riddance.

Re: Someone used my open source project to phish people

#56

Post by [deleted] »

jedberg wrote: post #15
no_multitudes wrote: post #7

Earlier quoted context omitted.

The sentence construction, choice of vocabulary, and continually breathless tone are all clear indicators this was written by an llm and barely edited. I threw part of it into pangram to get a second opinion: https://www.pangram.com/history/8d6a7de3-86ac-4ce0-86c5-4f93...

Have you tried putting known human writing into pangram? I have. I've gotten 100% AI with multiple samples of my own human writing. It has also given me 50% on things I know were 100% AI written (from my prompts). Pangram and everything like it is useless. The results are random on known samples.

[deleted]

Re: Someone used my open source project to phish people

#57

Post by reconnecting »

eggbrain wrote: post #51
reconnecting wrote: post #47

Earlier quoted context omitted.

To the platform, the difference shows up exactly in the mitigation math. The 'make it cost more than it's worth' model only works when both sides of that ratio are knowable and bounded. With bonus abuse, the reward is fixed and the math is clear, so you can reliably price the abuser out. With organized criminals, you can't actually see what the abuse is 'worth' to them. And they can escalate almost infinitely: mimick…

> [...] With organized criminals, you can't actually see what the abuse is 'worth' to them. Even without collecting events, you can calculate what the abuse is worth to you , even if the math ends up being fuzzier. At the small platform operator level (one guy running a platform, as this article), the cost can be as simple as "this pisses me off and I have weekends." They can burn forty hours bolting on JA4 fingerpri…

[flagged]

Re: Someone used my open source project to phish people

#58

Post by dddw »

reconnecting wrote: post #39

Captcha here will only harm your real users' experience and won't protect against this kind of abuse, since it comes from real scammers, not fully automated bots. I've dealt with these and similar issues over the last 8 years, which led our team to develop a security tool 5 years ago that is now open-sourced. https://github.com/tirrenotechnologies/tirreno

Ooh nice, which I had a usecase for myself to try this out. Maybe in the future

Re: Someone used my open source project to phish people

#59

Post by noitpmeder »

Sophira wrote: post #54
no_multitudes wrote: post #52

Earlier quoted context omitted.

I am not sure if you are serious? It really has nothing to do with being grammatically correct. The article is chock-full of all of the most common LLM tells. The aggressive use of negative parallelism, filler adjectives like "deeply", and unvaryingly breathless tone all feel like decisions an actual thinking person would not make.

On the one hand, people say that AI models (LLMs, image generators, etc) are just stealing from people and that they cannot be original. On the other hand, people say that AI models have tells that no actual person would do. Which is it? You can't have it both ways.

Imo those are not nearly as mutually exclusive as you're presenting, as would be evidenced by anyone seriously reviewing the output of these systems, to the point it feels like a bad faith argument or an uninformed one

Re: Someone used my open source project to phish people

#60

Post by no_multitudes »

Sophira wrote: post #54
no_multitudes wrote: post #52

Earlier quoted context omitted.

I am not sure if you are serious? It really has nothing to do with being grammatically correct. The article is chock-full of all of the most common LLM tells. The aggressive use of negative parallelism, filler adjectives like "deeply", and unvaryingly breathless tone all feel like decisions an actual thinking person would not make.

On the one hand, people say that AI models (LLMs, image generators, etc) are just stealing from people and that they cannot be original. On the other hand, people say that AI models have tells that no actual person would do. Which is it? You can't have it both ways.

I said one of those things (sort of) but not the other. I am not sure why you are speaking as if "some people say x, and others say not x" is a gotcha. I am aware that many people say many different things.
Post reply on HN