31–40 of 66 posts
Why can’t there be a discussion on machine learning without everyone on HN trying to prove how unnecessary it is. Queue the anecdotes on simpler regression based methods, over paid scientists, and how much superior some other simpler method is.
I wouldn't say that deep learning is programming. I think the key feature of programming is legibility . A program is something that is clear enough to read and understand, to be decomposed in its constituent parts, and that has understandable semantics. For example, writing an algorithm that has precise steps and procedures is programming. Putting my input into a box, shaking the box, and taking the result out is no…
For instance, imagine you have a black box that observes the horse races, Twitbook, the betting market and so on, and based on those observations executes bets for you with a bookmaker. The execution of the orders has a measurable effect on your net worth.
You might write a traditional programme which takes all of this data, and based on some ETL, statistical models and probability calculations, executes orders.
You might do some ETL, plug it all into a neural network, tune it and execute orders based on the results.
Your traditional programme is very complex, and combinations of small bugs may have large effects on the results. Your unit and integration tests may themselves be wrong. Formal testing possibly reduces the expected value of the system and is an arse to carry out for any large system. The expected value of the system itself becomes harder to reason about as the system grows, based on the operation of reading and understanding the code.
The internals of your neural network are also difficult to reason about in some ways. It is difficult to understand the workings of your neural network and specific parts' effects on the measured effects of its output. It will take time to tune it and build the most profitable model.
Both implementations of the black box may be backtested, and some sort of trust can be established over the expected value of each implementation. Both implementations allow the operations of running, and measuring the results of running. Both implementations are difficult to reason about in various ways.
We are perfectly happy to give money to people for them to do things without fully understanding their inner thoughts and the processes behind those thoughts.
Which is the golden duck?
Is linear regression programming? A hammer, nail, saw and timber can also be used to solve problems but I wouldn't call those programming in and of themselves, but they could be used to built analog computers (where cogs, cams etc. Are like lines of code or procedures). Building a neutral network to get a result is not at all like programming. There is usually not a "perfect" structure, rather there are hardware, ene…
The same can be said about complex simulations. The difference lies in knowing and being able to determine the limits of the system and the verification process.
In a simulation we can derive the accuracy of our model from parameters like numerical precision and -stability, coarseness and model used. In other words we know the function we want to model because we state it explicitly.
Neural nets can model any function and the challenge is to extract the learned function from the trained network, to examine its limits and correctness. This is what I understood the author meant by the "operational" viewpoint.
We can verify that a given network architecture combined with a given optimisation function will find a local minimum w.r.t a given set of training data. This can be verified and tested.
What's not so easy to verify and test, however, are the properties of the modelled function as well as the function itself. That's why we still have to rely on proxies like error metrics on fixed datasets or failure cases.
With a simulation on the other hand, we can easily control and predict the (quality of the-) outcome by manipulating well understood parameters (number of iterations, coarseness of the simulation, numerical precision, etc.).
I picked simulations as an example, because many other classes of program can be verified using formal methods since the desired results are usually known beforehand. Again, just another reason why the author talks about a distinction in terms of operations, not the fundamental type of programming.
I find this to be a very interesting and thought provoking idea.
Why can’t there be a discussion on machine learning without everyone on HN trying to prove how unnecessary it is. Queue the anecdotes on simpler regression based methods, over paid scientists, and how much superior some other simpler method is.
It's an expensive (hardware, time, complexity) technique that is rarely the best. Why wouldn't people discuss cheaper, faster, understandable alternatives?
Why can’t there be a discussion on machine learning without everyone on HN trying to prove how unnecessary it is. Queue the anecdotes on simpler regression based methods, over paid scientists, and how much superior some other simpler method is.
It's an expensive (hardware, time, complexity) technique that is rarely the best. Why wouldn't people discuss cheaper, faster, understandable alternatives?
I have been writing optimization solvers of many forms to solve problems in engineering for about 20 years. from "make excel do linear regression on some data" to linear least squares to some nonlinear methods, simulated annealing, bayesian methods, deep neural networks -- none of this is "a new kind of programming", it's "do a bunch of data munging, throw matrix at a function, get matrix back, interpret/plot." there…
How well does "linear least squares to some nonlinear methods, simulated annealing, bayesian methods" work when you're doing speech to text, text generation or object detection? Deep learning is different in that it's a huge leap closer to human capabilities compared to the other methods you've listed.
So, could you explain again how the concepts and foundations of deep learning differs from plain old regression techniques?
Why can’t there be a discussion on machine learning without everyone on HN trying to prove how unnecessary it is. Queue the anecdotes on simpler regression based methods, over paid scientists, and how much superior some other simpler method is.
It's an expensive (hardware, time, complexity) technique that is rarely the best. Why wouldn't people discuss cheaper, faster, understandable alternatives?
I would go a step further, and say that prompt design will become an important sector of programming. Modern language models (eg GPT-3 et al) offer the capability to take a natural language input, match it against the context of the sentence, then propose a query that is understandable to the layperson. This abstraction allows us to understand the problem better, rather than just analyzing the way the problem manifes…
I don't think that's the case at all. Mathematics developed a formalised non-natural language precisely because human language is completely unsuitable for expressing abstract concepts in a concise and unambiguous fashion.
You will find that even in non-technical fields language will quickly converge to a well-defined, coarse and highly coded subset of regular human language when efficiency and correctness are key. You can observe this in the different branches of military, medicine, and trades.
We use programming to formalise algorithms, processes, and models. Those are abstract concepts and the difficulty doesn't lie in expressing them verbally. This has been shown time and again by fruitless efforts to create localised dialects of more accessible programming languages like BASIC or Pascal.
Turns out it doesn't matter whether keywords are written in your native language or if you could write natural language-like sentences: the difficult part remained formalising the abstract concept and ideas in a meaningful, logical and sound way.
What I do think will help tremendously, however, is using system such as GPT-3 to create another level of abstraction. There are many descriptive tasks that don't need to be put into code manually. The structure and behaviour of UIs comes to mind.
Earlier quoted context omitted.
It's an expensive (hardware, time, complexity) technique that is rarely the best. Why wouldn't people discuss cheaper, faster, understandable alternatives?
Just because you aren’t employed in a job that can make use of deep learning doesn’t mean it isn’t profitable. It’s extremely profitable. I’m tired of reading about some dumb alternatives that aren’t even relevant to the topic. Deep learning works very well for a certain class of problems. Continuing on with the trope that it doesn’t work just shows you aren’t educated
Of course it is profitable. One poster showed us a 20 parameter model. He was getting paid to apply deep learning on it. Damnit.