Live data from Hacker News

14,000x Speedup (2015)

james.hiebert.name

131–140 of 237 posts

Re: 14,000x Speedup (2015)

#131
post #77
post #8

Earlier quoted context omitted.

Correct, not all programs suffer from the Halting Problem.

I think that's kind of a malformed sentence. Programs themselves do not suffer or not suffer from the halting problem, it is only a problem when considering the space of all programs.

It's not malformed, it's just incorrect. Additionally, the halting problem is about provability. That sort of imprecision of language needs qualifiers at least.

Re: 14,000x Speedup (2015)

#132
Related: if I had a bit of open source code that I wrote that I was convinced was poorly optimized, how would I go about getting someone to take a look at it and help me improve it?

Re: 14,000x Speedup (2015)

#133

I remember many years ago I came upon a giant shell script doing some data processing (data not even that large, maybe few GBs) that took close to 25 mins (mainly due to unnecessary I/O due to tons of temp file creation) - written by "developers". I re-wrote it in Perl and it completed in less than a minute (and it wasn't even optimized because I was a non-CS newbie who was just learning Perl). I even setup a cronjob…

> It was never adopted because I was new and Perl was considered "too-risky".

What's not to love about bureaucracy?

Re: 14,000x Speedup (2015)

#134

Algorithms are important but are especially powerful in combination of knowing computer architecture and programming language intricacies. Many years ago I was asked to look at the program written in C++ that calculated Kendall-tau correlation matrix for a large amount of data. Basically Kendall Tau is a robust replacement for Pearson correlation and it had to be calculated for 0.5M^2 elements and calculation of each…

This is at the very least a clever and involved optimization. Let me tell you the story of my similar 10h to 10min fix. We had this cronjob that was supposed to run every hour, detect all changes to the customers and related table and sync with the marketing saas tool. It was written in Rails and took 10 mins at first. As we grew the time taken by this job also grew linearly. To a point where it took 10 hours and we could only sync the data twice a day. As it turns out the job was loading each user and then scanning for changes. We switched it into a somewhat complex sql query and MySQL ran it in a few mins. Add the additional processing and the whole job now took 10 mins.

Re: 14,000x Speedup (2015)

#135
Point of document implementation: please don’t use … for equations, or else the equations are simply missing in environments where MathJax doesn’t run (e.g. JS disabled, text mode browser, or JS fails to load—all up, it’s more common than you realise). Use a form of markup that will fail visible rather than fail invisible. TeX mathematics notation is far better than a void.

You could make this abuse of a bit more tolerable with something like the CSS `script[type^="math/tex"]:not([id]) { display: inline }`, but that will still be hidden to text-mode browsers or browsers where the CSS fails to load. It’s much better to just not use in this way.

Re: 14,000x Speedup (2015)

#136

Then there is the time that I wrote a program that would have taken 100 years to run and found a (perfectly useful) approximation I could calculate in 20 minutes, for a 2.6 million (x) speedup. Even counting 16 hours of programmer time that is a 55,000 (x) speedup.

My most significant optimization was showing that the thing they wanted computed was not possible. Not just "will keep running and finish soon after the universe expires", but actually impossible to compute (in the general case, which was their case, if they restricted the inputs it was feasible). Apparently they had no mathematicians evaluating the requirements. The guy who brought the problem to me had been working…

Yep, that's the heart of Computer Science.

Re: 14,000x Speedup (2015)

#137
post #126

Earlier quoted context omitted.

Yes, you must always know your language, but mastering (for example) Go is a lot easier than getting a cursory understanding of C++ (i.e., enough to write correct code for non-toy applications). You can get a cursory understanding of Go in a few hours and you can master it in 1-2 months. It would be months and years in C++, respectively. C++ is just a much larger, more complex language, so there is a lot more to know…

For a systems programming language the part of C++ that most people need to know isn’t huge. There’s a lot in c++ that only exists for back compatibility and you don’t need to use it. There’s also a lot of infrastructure so library writers can write very efficient generic code. Most people are not writing libraries and don’t need that stuff.

You got all the backwards compatibility stuff, though, if you're code base is older than today. Or if you've got co workers who learned on c++11 instead of c++19...

Also, templates, and the indecipherable errors they throw. And cpp memory handling errors are one of the largest sources of security holes.

I'm convinced that apologising for cpp is basically Stockholm syndrome. These are things we absolutely would not accept in a new language, but we must, because it's what we've got. Rust can't win fast enough, I say...

Re: 14,000x Speedup (2015)

#138

> Furthermore, there is literally no way to tell whether your program will ever actually terminate without actually executing it. This is technically not correct, is it? Or at least phrased a bit poorly. Maybe replace "your" with "any given"?

> Maybe replace "your" with "any given"? I'd be more inclined to replace it with "any non-trivial" in this case.

That would also be false. There are many non-trivial programs that can proved to halt or proved not to halt.

Re: 14,000x Speedup (2015)

#139
post #68
post #58

Earlier quoted context omitted.

This is a usual problem with C++ and why I hate it. There's a lot going on under the hoods, and you must be really knowledgeable of the language to prevent stupid things. Following some idioms you can really avoid it, but it is useless since your coworkers will fall into the language traps.

I'd argue that not knowing the language you're working with is a problem that's not exclusive to C++. There's so much more going on under the hood of more high level languages, that writing faster code is straight up black magic [1]. [1] https://stackoverflow.com/questions/28723658/python-why-is-i...

C++ has a famously complex standard though, to the point that, last I checked, there are no fully standards conforming C++ compilers. I know one member of the standards body and he is an excellent software engineer, a subject matter expert, and just wicked smart too, and he still confesses he doesn't really understand C++. There's a reason why Google doesn't actually have C++ as a supported language, but rather a curated subset thereof that's governed by their house style guide.

Re: 14,000x Speedup (2015)

#140
post #65

> In climate science we do a lot of downscaling. We take temperature and precipitation readings from a coarse scale Global Climate Model grid and map them to a fine scale local grid. Let’s say the global grid is 50x25 and the local grid is 1000x500. For each grid cell in the local grid, we want to know to which grid cell in the global grid it corresponds. Climate scientist here. Colour me a little confused. We have a…

I've upvoted this. Even the "optimized" solution here is so horrible that I almost thought the article was satire when I first read it. Maybe it is.

Even if we ignore the fact that it's grid to grid conversion, (which should just let you math the answer, without any searching) the fact that all the data is sorted means you can do the equivalent of a merge sort and only have to look at a two point for each step.

And how on earth can you take .117 seconds to line up a half a megapixel worth of grid points, even with their "fast" algorithm. They are using the R language which should be fast, right? Are they doing something expensive with it? Or perhaps there is an insane amount of data elsewhere making these lookups super expensive?

Post reply on HN