Earlier quoted context omitted.
>smash with something better Not a fan of the aggressive rhetoric here...
I too felt threatened
Violent disagreement is violence.
Deep Learning Is Not So Mysterious or Different
61–70 of 139 posts
Re: Deep Learning Is Not So Mysterious or Different
#62Earlier quoted context omitted.
Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.
That seems utterly bizarre to me. I don't use "delve" frequently myself, but it is common enough that it doesn't jump out as an unusual word. Perhaps it is overused or used in a not-exactly-usual context that tips one off that it is LLM-generated, but by itself it signifies nothing to me.
It is a very common word used in Nigerian style English which was a very common place they were outsourcing RLHF tasks to. A sibling comment has a link but it is also easy to google.
Re: Deep Learning Is Not So Mysterious or Different
#63Earlier quoted context omitted.
You're misunderstanding. A level of abstraction is necessary for operation of modern systems. There is no human alive who, given an intermediate step in the middle of some running learning algorithm, is able to understand and mentally model the full system at full man-made resolution, that is, down to the transistor level, on a modern CPU. Someone wishing to understand a piece of software in 2025 is forced to, at som…
It's not misunderstanding at all - but your response is certainly an attempt to obfuscate the point being made. The moment you represent anything in code, you are abstracting a real thing into it's digital representation. That digital representation if fully formed at every cycle of the digital system processing it, and the state of the system - all the way down to the transistor level may be precisely determined. To…
Okay, then, you're just using your own personal definition of "black box" instead of the one everyone else uses.
Something that's a black box is unknown to the speaker. It's not understood to be unknowable to anyone.
Re: Deep Learning Is Not So Mysterious or Different
#64I wish I had the time to try this: 1.) Grab many GBs of text (books, etc). 2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance. 3.) For each word, store most frequent word for each $N distance. [a] 4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/d…
I actually tried sth like that with the Bible back in 2021. scaling is bitch. very difficult to train these types of models.
Re: Deep Learning Is Not So Mysterious or Different
#65Earlier quoted context omitted.
The scaling is brutal. If you have a 20k word vocabulary and want to do 3 grams, you need a 20000^3 matrix of elements (8 trillion). Most of which is going to be empty. GPT and friends cheat by not modeling each word separately, but a large dimensional “embedding” (just a vector if you also find new vocabulary silly). The embedding represents similar words near each other in this space. The famous king-man-queen exam…
Importantly, though, LLMs do not take the embeddings as input during training; they take the tokens and learn the embeddings as part of the training. Specifically all Transformer-based models; older models used things like word2vec or elmo, but all current LLMs train their embeddings from scratch.
And tokens are now going down to the byte level:
https://ai.meta.com/research/publications/byte-latent-transf...
Re: Deep Learning Is Not So Mysterious or Different
#66Earlier quoted context omitted.
It's not misunderstanding at all - but your response is certainly an attempt to obfuscate the point being made. The moment you represent anything in code, you are abstracting a real thing into it's digital representation. That digital representation if fully formed at every cycle of the digital system processing it, and the state of the system - all the way down to the transistor level may be precisely determined. To…
Okay, then, you're just using your own personal definition of "black box" instead of the one everyone else uses. Something that's a black box is unknown to the speaker. It's not understood to be unknowable to anyone.
So your claim is that there are instructions, data, or both that are unable to be determined in what, is by definition, a fully deterministic machine?
Re: Deep Learning Is Not So Mysterious or Different
#67Earlier quoted context omitted.
That seems utterly bizarre to me. I don't use "delve" frequently myself, but it is common enough that it doesn't jump out as an unusual word. Perhaps it is overused or used in a not-exactly-usual context that tips one off that it is LLM-generated, but by itself it signifies nothing to me.
It is a very common word used in Nigerian style English which was a very common place they were outsourcing RLHF tasks to. A sibling comment has a link but it is also easy to google.
As a non native speaker, I didn't know the word "delve" but now I know this word. I think internet community is learning from LLM?
Re: Deep Learning Is Not So Mysterious or Different
#68DNNs do not have special generalization powers. If anything, their generalization is likely weaker than more mathematically principled techniques like the SVM.
If you try to train a DNN to solve a classical ML problem like the "Wine Quality" dataset from the UCI Machine Learning repo [0], you will get abysmal results and overfitting.
The "magic" of LLMs comes from the training paradigm. Because the optimization is word prediction, you effectively have a data sample size equal to the number of words in the corpus - an inconceivably vast number. Because you are training against a vast dataset, you can use a proportionally immense model (e.g. 400B parameters) without overfitting. This vast (but justified) model complexity is what creates the amazing abilities of GPT/etc.
What wasn't obvious 10 years ago was the principle of "reusability" - the idea that the vastly complex model you trained using the LLM paradigm would have any practical value. Why is it useful to build an immensely sophisticated word prediction machine, who cares about predicting words? The reason is that all those concepts you learned from word-prediction can be reused for related NLP tasks.
Re: Deep Learning Is Not So Mysterious or Different
#69If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…
Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.
So ChatGPT or Nigerians or me apparently... :`(
Re: Deep Learning Is Not So Mysterious or Different
#70Correct me if I'm wrong, but an artificial neuron is just good old linear regression followed by an activation function to make it non linear. Make a network out of it and cool stuff happens.
Yes. An artificial neuron, as a mathematical function f, is defined by f(x) = g(wx + b) where x is the input, w is the weight, b is the bias, and g is some non-linear activation function. Is that "good old linear regression followed by an activation function to make it non linear"? Yes, it is exactly that.