Earlier quoted context omitted.
Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.
I’d love to see an article delve into why that is.
Deep Learning Is Not So Mysterious or Different
51–60 of 139 posts
Re: Deep Learning Is Not So Mysterious or Different
#52I wish I had the time to try this: 1.) Grab many GBs of text (books, etc). 2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance. 3.) For each word, store most frequent word for each $N distance. [a] 4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/d…
The scaling is brutal. If you have a 20k word vocabulary and want to do 3 grams, you need a 20000^3 matrix of elements (8 trillion). Most of which is going to be empty. GPT and friends cheat by not modeling each word separately, but a large dimensional “embedding” (just a vector if you also find new vocabulary silly). The embedding represents similar words near each other in this space. The famous king-man-queen exam…
Importantly, though, LLMs do not take the embeddings as input during training; they take the tokens and learn the embeddings as part of the training.
Specifically all Transformer-based models; older models used things like word2vec or elmo, but all current LLMs train their embeddings from scratch.
Re: Deep Learning Is Not So Mysterious or Different
#53If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…
I watched the 3b1b series on neural nets years ago, and it still accounts for 95% of my understanding of AI in general.
I’m not an ML person, but still. That guy has a serious gift for explaining stuff.
His video on the uncertainty principle explained stuff to me that my entire undergrad education failed to!
Re: Deep Learning Is Not So Mysterious or Different
#54If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…
Looks nice - are there written versions?
Re: Deep Learning Is Not So Mysterious or Different
#55Correct me if I'm wrong, but an artificial neuron is just good old linear regression followed by an activation function to make it non linear. Make a network out of it and cool stuff happens.
In a sense; linear regression can be computed exactly so refers to a specific technique for producing a linear model.
Most artificial neurons are trained stochastically rather than holistically, i.e. rather than looking at the entire training set and computing the gradient to minimize the squared loss or something similar, they look at each training example and compute the local gradient and make small changes in that direction.
In addition, the "activation function" almost universally used now is the rectified linear unit, which is linear for positive input and zero for negative input. This is non-decreasing at least as a function, but the fact that it is not monotonic means that there is no additional loss accrued for overcorrecting in the negative direction.
Given this, using the term "linear regression" to describe the model of an artificial neuron is not really a useful heuristic.
Re: Deep Learning Is Not So Mysterious or Different
#56If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…
Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.
That seems utterly bizarre to me. I don't use "delve" frequently myself, but it is common enough that it doesn't jump out as an unusual word. Perhaps it is overused or used in a not-exactly-usual context that tips one off that it is LLM-generated, but by itself it signifies nothing to me.
Re: Deep Learning Is Not So Mysterious or Different
#57Re: Deep Learning Is Not So Mysterious or Different
#58If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…
and if anyone is interested in delving more deeply into the statistical concepts & results referenced in the paper of this post (e.g. VC-dimension, PAC-learning, etc), I can recommend this book: https://amzn.eu/d/7Zwe6jw
Re: Deep Learning Is Not So Mysterious or Different
#59[flagged]
Please formulate your critique instead of simply labeling it with negative words.
Sure.
"Preprint" implies prior to printing, which means that there's a reasonable expectation for this paper to be submitted, accepted, and printed in a scholarly journal.
What we have here is little more than a tongue-in-cheek submission which carries an aesthetic of "hot-take" throughout the paper. This is unbecoming of one committed to scholarly pursuits and all but guarantees rejection from journals committed to professionalism.
Furthermore, what's really interesting is how this comment section has developed. It really is the blind leading the blind here.
I will not subject myself further to the consequences of Brandolini's law except to implore the reader to consider the signal-to-noise ratio resulting from being too tolerant of posts like this.
Re: Deep Learning Is Not So Mysterious or Different
#60So where is the line that something becomes ‘AI’ and is regulated?