Live data from Hacker News

Why wordfreq will not be updated

github.com

391–400 of 542 posts

Re: Why wordfreq will not be updated

#391
post #220

Earlier quoted context omitted.

In this particular case, I was out of the loop regarding the clever analogy myself. I'm now a tad smarter because someone else expressed lack of understanding, and I learned from responses to this (grayed due to downvotes) comment.

The problem is that the answer was a really easy google. I didn't know what low background steel was and I just googled it.

A person asking the question here means there are now several good succinct explanations of it here.

Re: Why wordfreq will not be updated

#392
post #220
post #208

Earlier quoted context omitted.

I feel like there's a minimum "due diligence" bar to meet though before asking, otherwise it comes across as "I'm too lazy to google the reference and connect the dots myself, but can someone just go ahead and distill a nice summary for me"

In this particular case, I was out of the loop regarding the clever analogy myself. I'm now a tad smarter because someone else expressed lack of understanding, and I learned from responses to this (grayed due to downvotes) comment.

But it's right there in the header, you could just click the link and find out on the top of the webpage.

Re: Why wordfreq will not be updated

#393
post #323

Earlier quoted context omitted.

Isn't it the other way around? SEO text carefully tuned to tf-idf metrics and keyword stuffed to them empirically determined threshold Google just allows should have unnatural word frequencies. LLM content should just enhance and cement the status quo word frequencies. Outliers like the word "delve" could just be sentinels, carefully placed like trap streets on a map.

> LLM content should just enhance and cement the status quo word frequencies. TFA mentions this hasn't been the case.

Would you mind dropping the link talking about this point? (context: I'm a total outsider and have no idea what TFA is.)

Re: Why wordfreq will not be updated

#394
> It's rare to see NLP research that doesn't have a dependency on closed data controlled by OpenAI and Google, two companies that I already despise.

The dependency on closed data combined with the cost of compute to do anything interesting with LLMs has made individual contributions to NLP research extremely difficult if one is not associated with a very large tech company. It's super unfortunate, makes the subject area much less approachable, and makes the people doing research in the subject area much more homogeneous.

Re: Why wordfreq will not be updated

#395
I think the main reason for sunsetting the project is hinted at near the bottom:

> The field I know as "natural language processing" is hard to find these days. It's all being devoured by generative AI. Other techniques still exist but generative AI sucks up all the air in the room and gets all the money.

Traditional NLP has been surpassed by transformers, making this project obsolete. The rest of the post reads like rationalization and sour grapes.

Re: Why wordfreq will not be updated

#396
post #322

I'm going to call it: The Web is dead. Thanks to "AI" I spend more time now digging through searches trying to find something useful than I did back in 2005. And the sites you do find are largely garbage. As a random example: just trying to find a particular popular set of wireless earbuds takes me at least 10 minutes, when I already know the company, the company's website, other vendors that sell the company's goods…

No disagreement for the most part. I used to be able to say search for Trek bike derailleur hanger and the first result would be what I wanted. Now I have to scroll past 5 ads to buy a new bike, one that's a broken link to a third party, and if I'm really lucky, at the bottom of page 1 will be the link to that part's page. The shitification of the web is real.

R.I.P. Sheldon Brown T_T

(The Agner Fog of cycling?)

Re: Why wordfreq will not be updated

#397
post #323

Earlier quoted context omitted.

> LLM content should just enhance and cement the status quo word frequencies. TFA mentions this hasn't been the case.

Would you mind dropping the link talking about this point? (context: I'm a total outsider and have no idea what TFA is.)

TFA means "the featured article", so in this case the "Why wordfreq will not be updated" link we're talking about.

Re: Why wordfreq will not be updated

#398
post #397

Earlier quoted context omitted.

Would you mind dropping the link talking about this point? (context: I'm a total outsider and have no idea what TFA is.)

TFA means "the featured article", so in this case the "Why wordfreq will not be updated" link we're talking about.

To be pedantic, the F in TFA has the same meaning as the F in RTFM.

It’s the same origin. On Slashdot (the HN of the early 00’s) people would admonish others to RTFA. Then they started using it as a referent: TFA was the thing you were supposed to have read.

Re: Why wordfreq will not be updated

#399
post #325

Earlier quoted context omitted.

But your children's children will never learn about love because that word will have been mechanically trained out of existence.

That's pretty funny. You think love is just a word?

I leave it up to the reader to determine how serious I may be.

Re: Why wordfreq will not be updated

#400

Earlier quoted context omitted.

reading stuff like this makes me so happy. no matter how fucked up something may be there is always a way to clean right up.

glances nervously at atmospheric CO2

the easiest solution is growing dense vegetation including trees, then using that for things[0] or burying it until we have a better mitigation strategy for atmospheric carbon.

Another solution, and one that, if i weren't such a lazy, is ocean based carbon binding. You can run electricity directly through ocean water and precipitate the carbon out as calcium carbonate, which is both: useful to humans as is and after processing; and useful to the coral reefs and crustaceans/mollusks or whatever in the oceans.

If anyone wants to kick me about a million US dollars, i can make a POC on a used barge with solar panels and as much recycled material as possible, and have that just run off the coast of florida or something. I figure the total cost to get a barge is around a quarter million, all-in[1], the electronics and seawater stuff is about another $150-200 thousand, and the rest is mine for the idea and the lawyers' to get this approved and left alone to do the research.

[0] burning it for heat is fine, as the net CO2 levels will remain constant, but i mean things like houses and boardwalks and boats, furniture, and so on.

[1] could be more, now, the last time i was researching seaworthy barge costs it was between $100,000 and $200,000. I'm hoping there's someone that can donate the barge so i can make the rest more fit for purpose - redundancy, better solar, better mppt, better batteries, better materials for the electrodes (it takes platinum and titanium iirc, i haven't looked at my documents for a long while.)

Post reply on HN