11–20 of 139 posts
Earlier quoted context omitted.
I'm a huge fan of HN just for replies such as this that smash the OP's post/product with something better. It's like at least half the reason I stick around here. Thanks for the great read.
>smash with something better Not a fan of the aggressive rhetoric here...
[flagged]
> rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. How does deep learning do this? The last time I was deeply involved in machine le…
The usual approach to supervised ML is to "invent" the model (layers, their parameters) or more often copy one from known good reference, then define the cost function and feed it data. "Deep" learning just means that instead of a few layers you use a big number of them.
What you describe sounds like an automated way of tweaking the architecture, IIUC? Never done that, usually the cost of a run was too high to let an algorithm do that for me. But I'm curious if this approach is being used?
1.) Grab many GBs of text (books, etc).
2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance.
3.) For each word, store most frequent word for each $N distance. [a]
4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/distance and find most probable next set of word(s)
How close would this be to GPT 2?
[a] You could go one step further and store multiple words for each distance, ordered by frequency
I wish I had the time to try this: 1.) Grab many GBs of text (books, etc). 2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance. 3.) For each word, store most frequent word for each $N distance. [a] 4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/d…
It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear algebra topics)
And of course, for deep learning, 3b1b is great for getting some visual introduction (https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQ...).
If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…
I wish I had the time to try this: 1.) Grab many GBs of text (books, etc). 2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance. 3.) For each word, store most frequent word for each $N distance. [a] 4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/d…
I wish I had the time to try this: 1.) Grab many GBs of text (books, etc). 2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance. 3.) For each word, store most frequent word for each $N distance. [a] 4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/d…
[0]: https://infini-gram.io [1]: https://arxiv.org/abs/2410.04265v1