Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Softmax forever, or why I like softmax

kyunghyuncho.me

Post reply on HN
Print view View original

71–80 of 116 posts

Re: Softmax forever, or why I like softmax

#71

Post by nomel »

alchemist1e9 wrote: post #38
jiggawatts wrote: post #31

Earlier quoted context omitted.

It's a fad associated with AI, popularised by Sam Altman especially. It's the new black turtleneck that everyone is wearing, but will swear upon their mother's life isn't because they're copying Steve Jobs.

> It's a fad associated with AI, popularised by Sam Altman especially. I know this is true but does anyone understand why they do it? It is actually cognitively disruptive when reading content because many of us are trained to simultaneously proof read while reading. So I also consider it a type of cognitive attack vector and it annoys me extremely as well.

It's not true at all. It was very common, nearly norm in all online communication until phones started auto correcting with capitalization. You could always tell who was a mac/phone user by their use of capitalization. Sam is older than when that happened. He almost certainly spent the majority of his online life typing in lowercase, as I did. Go look at any old IRC chat log, forum, etc from his era.

Re: Softmax forever, or why I like softmax

#72

Post by sodapopcan »

jppope wrote: post #46
roger_ wrote: post #27

An aside: please use proper capitalization. With this article I found myself backtracking thinking I’d missed a word, which was very annoying. Not sure what the authors intention was with that decision but please reconsider.

Language evolves. Capitalization is an artifact of a period where capitalizing the first letter made a lot of sense for the medium (parchment/paper). Modern culture is abandoning it for speed efficiency on keyboards or digital keyboards. A purist would say that we should still be using all capitals like they did in Greek/Latin which again was related to the medium. I'll likely continue using Capitalization as a prefe…

As much as I dislike it sometimes, language absolutely does evolve. Proper capitalization does not fit into this, though. It can completely change the meaning of something if it is not capitalized. It's not just at the beginning of sentences, it's proper nouns within a sentence. Unfortunately I don't have an example of this handy but it's happened to me several times in my life where I've been completely confused by this (mostly on Slack).

This is a merely showing off your personal style which, when writing a technical article, I don't care about.

Re: Softmax forever, or why I like softmax

#73

Post by meowface »

bowsamic wrote: post #49
PhilippGille wrote: post #37

Earlier quoted context omitted.

https://www.theguardian.com/society/2025/feb/18/death-of-cap...

Well I will fight this trend to the death. Thankfully I don't like to surround myself with philistines

The war is already over.

I 100% agree lowercase in longform essays is ridiculous, but I think for everything aside from essays, articles, papers, long emails, and some percentage of multi-paragraph site comments, lowercase is absolutely going to be the default online in 20 years.

Re: Softmax forever, or why I like softmax

#74

Post by meowface »

bicx wrote: post #47
jppope wrote: post #46

Earlier quoted context omitted.

Language evolves. Capitalization is an artifact of a period where capitalizing the first letter made a lot of sense for the medium (parchment/paper). Modern culture is abandoning it for speed efficiency on keyboards or digital keyboards. A purist would say that we should still be using all capitals like they did in Greek/Latin which again was related to the medium. I'll likely continue using Capitalization as a prefe…

It's slower for sure, but capitalization does impart information: beginning of sentences, proper nouns, acronyms, and such. Sure, you could re-read the sentence until you figured all that out, but you are creating unnecessary hitches in the reading process. Capitalization is an optimization for the reader, and lack of capitalization is optimization for the writer.

I think capitalization will slowly go by the wayside in most media, but one hill I'll always die on is punctuation. "i'll go" vs. "ill go"... the latter is just too crude. Many gen Z/alpha do omit it, though.

Re: Softmax forever, or why I like softmax

#75

Post by sodapopcan »

bambax wrote: post #19

OT: refusing to capitalize the first word of each sentence is an annoying posture that makes reading what you write more difficult. I tend to do it too when taking notes for myself because I'm the only reader and it saves picoseconds of typing; but I wouldn't dream on inflicting it upon others.

It shows how laid back they are, man.

/s

Re: Softmax forever, or why I like softmax

#76

Post by programjames »

AnotherGoodName wrote: post #65

For the answer of is "is softmax the only way to turn unnormalized real values into a categorial distribution" you can just use statistics. Eg. Using Bayesian stats, if i assume an even prior (pretend i have no assumptions about how biased it is), i see a coin flip heads 4 times in a row, what's the probability of it being heads? Via a long winded proof using the dirichlet distribution Bayesian stats will say "add on…

There's a rather simple proof for this "add one to the top, n to the bottom" posterior. Take a unit-perimeter circle, and divide it into n regions for each of the possible outcomes. Then lay out your k outcomes into their corresponding regions. You have n dividers and k outcomes for a total of n + k points. By rotational symmetry, the distance between any two points is equal in expectation. Thus, the expected size of any region is (1 + # outcomes in the region) / (n + k). So, if you were to take one more sample

E[sample in region i] = (1 + # outcomes in the region) / (n + k)

But, the indicator variable "sample in region i" is always either zero or one, so this must equal the probability!

Re: Softmax forever, or why I like softmax

#77

Post by omoikane »

roger_ wrote: post #27

An aside: please use proper capitalization. With this article I found myself backtracking thinking I’d missed a word, which was very annoying. Not sure what the authors intention was with that decision but please reconsider.

This looks like a personal blog post, in a blog where the author have avoided capitalization fairly consistently. The blog post was likely not meant to be a research paper, and reading it as a research paper is probably setting the wrong expectations.

If people wanted to read formal-looking formatted text, the author has linked to one in the second paragraph:

https://arxiv.org/abs/1511.07916 - Natural Language Understanding with Distributed Representation

Re: Softmax forever, or why I like softmax

#78

Post by elevatedastalt »

roger_ wrote: post #27

An aside: please use proper capitalization. With this article I found myself backtracking thinking I’d missed a word, which was very annoying. Not sure what the authors intention was with that decision but please reconsider.

Yeah, things like these make me glad that humans don't live forever. By the time you are 30 you already hate the way so many things work around you. If you argue about it you are called a philistine luddite who can't stomach change. There's no right or wrong, but it's good you don't have to deal with stuff you find annoying indefinitely. You just... die eventually.

It's a better equilibrium this way and one of the main reasons I don't care much for transhumanism.

Re: Softmax forever, or why I like softmax

#79

Post by nomel »

recursive wrote: post #63
thaumasiotes wrote: post #57

Earlier quoted context omitted.

> [I]'ll bet that the linguistics experimentalists have metrics that quantify reading speed measurements as determined by eye tracking experiments, and can verify this. You appear to be trolling for the sake of trolling, but for reference: reading speed is determined by familiarity with the style of the text. Diverging from whatever people are used to will make them slower. There is no such thing as "two spaces" in H…

> There is no such thing as "two spaces" in HTML, so good luck with that. Code point 160 followed by 32. In other words `  ` will do it.

There's: U+3000, ideographic space. It's conceptually fitting, with sentence separation being a good fit for "idea separation".

edit: well I tried to give an example, but hn seems to replace it with regular space. Here's a copy paste version: https://unicode-explorer.com/c/3000

Re: Softmax forever, or why I like softmax

#80

Post by mempko »

roger_ wrote: post #27

An aside: please use proper capitalization. With this article I found myself backtracking thinking I’d missed a word, which was very annoying. Not sure what the authors intention was with that decision but please reconsider.

At least for now, maybe this is the best way to tell if a text is written by an LLM or a person. An LLM will capitalize!
Post reply on HN