71–80 of 106 posts
The phrase "raw numpy" strikes me as funny. I would figure that's about as abstract as you could get while still working with the math (discarding symbolic engines).
On the other hand, try implementing a convnet in numpy, especially the backprop, and you might start feeling some of its "rawness" :-)
Feynmann pretty much settles it.
Earlier quoted context omitted.
What is O(n log n)? Time complexity? What about memory requirement? Last month I decreased memory requirement from 16GB to 100MB in one implementation of a "stable&fast" algorithm. Really, using a function just because it's O(n log n) without understanding its characteristic is like a blind surgeon randomly amputating limbs because maybe it will help. But that's fine for me, I have more work then, fixing software bug…
Why do you assume the commenter doesn't understand the time and memory characteristics of quicksort?
Earlier quoted context omitted.
> finding out they were working with numpy makes it even more so (since numpy takes the pain out of the matrix operations) hmm, I did the cs231n homework and had the opposite experience. It was really easy to complete it by ignoring numpy's provided matrix methods (just write a bunch of for loops in python) but that solution was really slow. If you could use numpy's matrix methods, however, the code executed a lot fa…
I found that in the ML Class, we could complete many of the assignments by doing the calcs via "for-loops" in Octave - and as you noted, it was really slow. But I think Ng wanted us to understand what was going on under the hood in Octave when you used its in-built vector primitives, and how to think about the problems in such a way to understand how to "vectorise" them so that the solutions would be amenable to usin…
> “Why do we have to write the backward pass when frameworks in the real world, such as TensorFlow, compute them for you automatically?” How many more times do you need to see the same phenomenon under different guises before you stop asking stupid questions? "Hey teach, why do I need to learn how to multiply if I can just use a calculator?"
Isn't it somewhat reasonable question, given the relatively recent advent of TensorFlow, compared to ML curricula? The stress is on, why don't we learn TensorFlow / Caffe / etc.
Earlier quoted context omitted.
What is O(n log n)? Time complexity? What about memory requirement? Last month I decreased memory requirement from 16GB to 100MB in one implementation of a "stable&fast" algorithm. Really, using a function just because it's O(n log n) without understanding its characteristic is like a blind surgeon randomly amputating limbs because maybe it will help. But that's fine for me, I have more work then, fixing software bug…
> What is O(n log n)? Time complexity? And then worst case? Best case? Average case? Which applies to random input? Nearly-sorted input? What's the overhead for a "short sort", can/should I use this to sort small sequences in a tight-ish loop or is it only for large sequences?
I was under the illusion that Big-O was always asymptotic complexity (ie worst case) and that other notations (little-o, big-omega, big-thetha etc) were used for best/average/etc case. Perhaps I'm wrong, however.
Which applies to random input? Nearly-sorted input? What's the overhead for a "short sort", can/should I use this to sort small sequences in a tight-ish loop or is it only for large sequences?
Indeed. The details matter.
My favourite example is how a naive linear search can perform better than a non-linear search with a better (in O() terms) algorithm if, for example, the linear search can do most of its work in cache (eg small input sizes or otherwise regular access patterns (predictable for prefetch)).
The phrase "raw numpy" strikes me as funny. I would figure that's about as abstract as you could get while still working with the math (discarding symbolic engines).
Yes, after implementing a simple neural network in C (with AVX, and pthreads), "raw numpy" does sound funny! On the other hand, try implementing a convnet in numpy, especially the backprop, and you might start feeling some of its "rawness" :-)
I'm developing a model (not exactly a convnet) that uses a correlation step. Because of the above problem and its resulting pure-python loops, I may have to cythonize or use the NumPy C API for the gradient evaluation. Do you know of any examples I could check out that implement partial derivatives w.r.t. a correlation (or convolution) in "raw numpy"?
Error terms represent a sum and product of derivatives. The product of a bunch of terms will tend to get really big or really small.
The rest is detail: are the terms in some interval? Which? how many are we multiplying? how many are we summing over? do we doctor the sum after we get it?
Earlier quoted context omitted.
I think you're sweeping an important distinction under the rug. If every major language provides an O(n log n) sort function, is it still a leaky abstraction? I'd say no. You can use it without worrying much about the details. But it sounds like the situation with back-propagation is different, since the internal details of the algorithm affect whether you get a usable answer at all. A borderline case might be someth…
Speaking from experience, I've had to worry about what implementation of sort is being used in many languages, from Java to C++, even Go and Python. There are a lot of details to get right: how are elements compared? Is the sort stable? Is it efficient for small N? Is it efficient for nearly-sorted arrays? Is it efficient when almost all the elements compare equal? Is it guaranteed O(n log n) or average? If average,…
Earlier quoted context omitted.
What is O(n log n)? Time complexity? What about memory requirement? Last month I decreased memory requirement from 16GB to 100MB in one implementation of a "stable&fast" algorithm. Really, using a function just because it's O(n log n) without understanding its characteristic is like a blind surgeon randomly amputating limbs because maybe it will help. But that's fine for me, I have more work then, fixing software bug…
Why do you assume the commenter doesn't understand the time and memory characteristics of quicksort?