Live data from Hacker News

Softmax forever, or why I like softmax

kyunghyuncho.me

61–70 of 116 posts

Re: Softmax forever, or why I like softmax

#61
post #27

An aside: please use proper capitalization. With this article I found myself backtracking thinking I’d missed a word, which was very annoying. Not sure what the authors intention was with that decision but please reconsider.

This is the norm for Gen Z. We don’t see it because children don’t set social norms where adults are present too, but with the oldest of Gen Z about to turn 30, you and I should expect to see this more and more, and get used to it. If every kid can handle it, I think we can, too.

It doesn't change the point of your comment necessarily, but as far as TFA goes, the author was teaching a University course in 2015, so is highly unlikely to be Gen Z.

Re: Softmax forever, or why I like softmax

#62

Earlier quoted context omitted.

That would be "welcome to the world of academia". My post-doc friends won't even read a blog post prior to checking author's resume. They are very dismissive every time they notice anything they consider sloppy etc.

You seem to merge two different points. Not reading based on sloppiness is defensible. Not reading based on the author's resume less so.

When "sloppiness" is defined as "did anything on my personal list of pet peeves" (and it often is) then the defensibility of the two begin to converge.

Re: Softmax forever, or why I like softmax

#63

Earlier quoted context omitted.

an opinion, and a falsifiable hypothesis: call me old-fasahioned, but two spaces after a period will solve this problem if people insist on all-lower-case. this also helps distinguish between abbreviations such as st. martin's and the ends of sentences. i'll bet that the linguistics experimentalists have metrics that quantify reading speed measurements as determined by eye tracking experiments, and can verify this.

> [I]'ll bet that the linguistics experimentalists have metrics that quantify reading speed measurements as determined by eye tracking experiments, and can verify this. You appear to be trolling for the sake of trolling, but for reference: reading speed is determined by familiarity with the style of the text. Diverging from whatever people are used to will make them slower. There is no such thing as "two spaces" in H…

> There is no such thing as "two spaces" in HTML, so good luck with that.

Code point 160 followed by 32. In other words `  ` will do it.

Re: Softmax forever, or why I like softmax

#64
post #45

Earlier quoted context omitted.

It's a fad associated with AI, popularised by Sam Altman especially. It's the new black turtleneck that everyone is wearing, but will swear upon their mother's life isn't because they're copying Steve Jobs.

i don't want to press the shift-key everytime i need a capitalized letter on my phone and i disable auto-correct because it constantly messes with native languages etc. wasn't aware that this makes me a steve jobs copier :( EDIT: people are seriously so emotionally invested in capitalization that i get downvoted into minus, jeez.

When you consciously choose to save yourself effort in writing, at the expense of the readers who are trying to make sense of what you are saying, the people onto whom you've transferred the cognitive load are not likely to appreciate your laziness.

Re: Softmax forever, or why I like softmax

#65
For the answer of is "is softmax the only way to turn unnormalized real values into a categorial distribution" you can just use statistics.

Eg. Using Bayesian stats, if i assume an even prior (pretend i have no assumptions about how biased it is), i see a coin flip heads 4 times in a row, what's the probability of it being heads?

Via a long winded proof using the dirichlet distribution Bayesian stats will say "add one to the top and two to the bottom". Here we saw 4/4 heads. So we guess 5/6 chance of being heads (+1 to the top, +2 to the bottom) the next time or a 1/6 chance of being tails. This represents that the statistical model is assuming some bias in the coin.

That's normalized as a probability against 1 which is what we want. It works for multiple probabilities as well, you add to the bottom as many different outcomes as you have. The Dirichlet distribution allows for real numbers and you can support this too. If you feel this gives too much weight to the possibility of the coin being biased you can actually simply add more to the top and bottom which is the same as accounting for this in your prior, eg. add 100 to the top and 200 to the bottom instead.

Now this has a lot of differences with outcomes compared to softmax. It actually gives everything a non-zero chance rather than using the classic sigmoid activation function that softmax has underneath which moves things to almost absolute 0 or 1. But... other distributions like this are very helpful in many circumstances. Do you actually think the chance of tails becomes 0 if you see heads flipped 100 times in a row? Of course not.

So anyway the softmax function fits things to a particular type of distribution but you can actually fit pretty much anything to any distribution with good old statistics. Choose the right one for your use case.

Re: Softmax forever, or why I like softmax

#66
post #42

Earlier quoted context omitted.

The sibling comment to yours mentions that this is pretty common on Twitter, and I'd guess that it started as a way to making firing off tweets from a phone easier (since the extra effort to hit shift when typing on a phone keyboard is a bit higher, and the additional effort to go back and fix any typos that happen due to trying to capitalize things is also higher compared to using a traditional keyboard). Once enoug…

> since the extra effort to hit shift when typing on a phone keyboard is a bit higher, and the additional effort to go back and fix any typos that happen due to trying to capitalize things is also higher compared to using a traditional keyboard I'm a bit confused about this. Do people turn off auto capitalisation on their phones? I very rarely have to press shift on my phone

I hadn't considered that. My best guess is that it was originally an intentional decision based on consistency with nouns that people might have mid-sentence that they can't rely on autocorrect to capitalize properly.

Re: Softmax forever, or why I like softmax

#67
post #29

Earlier quoted context omitted.

I agree. I'm all for Graham's pyramid of disagreement: we should focus on the core argument, rather than superfluous things like tone, or character, or capitalisation. But this is too much for me personally. I just realised I consider the complete lack of capitalisation on a piece of public intellectual work to be obnoxious. Sorry, it's impractical, distracting and generates unnecessary cognitive load for everyone el…

It's a fad associated with AI, popularised by Sam Altman especially. It's the new black turtleneck that everyone is wearing, but will swear upon their mother's life isn't because they're copying Steve Jobs.

That is so incredibly dumb. I can get it in a tweet, but please, please, please properly capitalize in anything longer than a few words!

Re: Softmax forever, or why I like softmax

#68

Earlier quoted context omitted.

Twitter and all forms of instant messaging (SMS, WhatsApp, Discord, and the older ones like AIM/MSN/ICQ) have normalized it for many years. Sam is just one of the few large company CEOs to tweet in the style other Twitter users usually use. He's adopting the native culture rather than setting a trend. Sam still uses capitalization in all of his essays, as do most people (including young people). In essays, like this…

>Sam is just one of the few large company CEOs to tweet in the style other Twitter users usually use. Just looked at the algorithmic feed on Twitter to makes sure trends haven't shifted overnight, and zero people in that sample of hundreds of tweets used all lower case in the tweets. Not in science. Not in AI. Not in maths or politics or entertainment or media. Sam is trying to bE dIFFERENT. He isn't adopting a norm…

Most people on my feed do. It's true that professionals, executives, and public intellectuals generally don't use lowercase and that Sam is trying to normalize that, but my main point was that the medium matters. Lowercase looks ridiculous for longform essays and normal for short messages and microblog posts.

Re: Softmax forever, or why I like softmax

#69
post #45

Earlier quoted context omitted.

It's a fad associated with AI, popularised by Sam Altman especially. It's the new black turtleneck that everyone is wearing, but will swear upon their mother's life isn't because they're copying Steve Jobs.

i don't want to press the shift-key everytime i need a capitalized letter on my phone and i disable auto-correct because it constantly messes with native languages etc. wasn't aware that this makes me a steve jobs copier :( EDIT: people are seriously so emotionally invested in capitalization that i get downvoted into minus, jeez.

> EDIT: people are seriously so emotionally invested in capitalization that i get downvoted into minus, jeez.

I find it weird that you would be surprised that people care about the quality of textual communication

Re: Softmax forever, or why I like softmax

#70
post #42

Earlier quoted context omitted.

The sibling comment to yours mentions that this is pretty common on Twitter, and I'd guess that it started as a way to making firing off tweets from a phone easier (since the extra effort to hit shift when typing on a phone keyboard is a bit higher, and the additional effort to go back and fix any typos that happen due to trying to capitalize things is also higher compared to using a traditional keyboard). Once enoug…

> since the extra effort to hit shift when typing on a phone keyboard is a bit higher, and the additional effort to go back and fix any typos that happen due to trying to capitalize things is also higher compared to using a traditional keyboard I'm a bit confused about this. Do people turn off auto capitalisation on their phones? I very rarely have to press shift on my phone

I and everyone I know turns it off. On many platforms and in many cultures, capitalization often implies Solemness or even rudeness in 1-on-1 conversations, and otherwise comes across as being out of touch in other kinds of communication.
Post reply on HN