Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

14,000x Speedup (2015)

james.hiebert.name

Post reply on HN
Print view View original

221–230 of 237 posts

Re: 14,000x Speedup (2015)

#221

Post by Tomminn »

gridlockd wrote: post #32
mlang23 wrote: post #17

Most code is bad code. And if it is not algorithmically bad, there are many other errors lurking in dark corners. I am a HPC cluster admin. Many years ago, we had a (for us back then) rather large project. Several million hours of CPU time. During a support case, I happened to stumble across the source code for the project. And I was pretty surprised. It was a few hundred lines of Pascal, compiled with fpc. I knew ab…

> ...not really representative for the scientific computing community. I disagree. Source: dude, trust me.

Agree, it definitely is representative.

There is the odd project that is so computationally intense that it needs to be written near optimally, but most scientific computing is a quick-and-dirty calculation run inefficiently on a giant cluster so you can get that figure for your paper and move on.

Re: 14,000x Speedup (2015)

#222

Post by hrishios »

tetha wrote: post #168
8ytecoder wrote: post #134

Earlier quoted context omitted.

This is at the very least a clever and involved optimization. Let me tell you the story of my similar 10h to 10min fix. We had this cronjob that was supposed to run every hour, detect all changes to the customers and related table and sync with the marketing saas tool. It was written in Rails and took 10 mins at first. As we grew the time taken by this job also grew linearly. To a point where it took 10 hours and we…

This is when being the guy knowing a few database tricks is fun. We had one reporting query that ran for 13 hours or so, in a normal web request. Not very successful, but it tried its best. Running this through explain while thinking of daloks revealed some dependent subqueries. Those are the devil in mysql because they are evaluated per row in the outer query. That takes a lot of time very quickly. Eventually we wra…

+1 for wrangjangling.

Re: 14,000x Speedup (2015)

#223

Post by alanbernstein »

etimberg wrote: post #13
rocqua wrote: post #6

Earlier quoted context omitted.

Grid doesn't need to be regular

As an example, see this image of the UTM zones and look at the zones labelled in red. https://upload.wikimedia.org/wikipedia/commons/b/b7/Universa...

That drives me nuts. I don't see an explanation in the article for why there are just six irregular cells in the whole grid. It does mention that it's "in part to accommodate the southern half of the Kingdom of Norway", despite the system having been developed by the US or maybe Germany. Why Norway? Why only Norway?

Re: 14,000x Speedup (2015)

#224

Post by tumblewit »

In machine learning for trading class, a python for loop code was turned into numpy array where the code went from something like over 4 minutes to a few milliseconds. Its astounding how much you can improve on code just by a vastly better algorithm.

Note: I know numpy is C and faster than python but that was not the point. The point was using vectors vs bruteforce.

Re: 14,000x Speedup (2015)

#225

Post by necovek »

roter wrote: post #65

> In climate science we do a lot of downscaling. We take temperature and precipitation readings from a coarse scale Global Climate Model grid and map them to a fine scale local grid. Let’s say the global grid is 50x25 and the local grid is 1000x500. For each grid cell in the local grid, we want to know to which grid cell in the global grid it corresponds. Climate scientist here. Colour me a little confused. We have a…

I am not a climate specialist, but I was confused about the same thing: I would fully expect for there to be a bijective mapping between a global grid and a local grid. I mean, lat/lon is a 2d space.

My other comment was going to be that using binary search is not that much computer science. I mean, it's elementary programming knowledge, you don't need a degree to know that.

Re: 14,000x Speedup (2015)

#226

Post by ncmncm »

I don't know whose law says something like, "The amount of optimization left to do is proportional to the improvement you got on the last round".

I.e., 14000x means there's probably another factor of 10 or 100 in there. But 0.1s might be fast enough.

I get 2x on std::sort pretty easily. I am hoping to get a general speedup like that for half the algorithms in the C++ Standard Library into C++23. Maybe the other half a little later.

Caches mean nobody knows how fast anything should be, so lots of things seem fast until you see one faster, then the old one suddenly looks slow.

Re: 14,000x Speedup (2015)

#227

Post by ncmncm »

sdenton4 wrote: post #137
gumby wrote: post #126

Earlier quoted context omitted.

For a systems programming language the part of C++ that most people need to know isn’t huge. There’s a lot in c++ that only exists for back compatibility and you don’t need to use it. There’s also a lot of infrastructure so library writers can write very efficient generic code. Most people are not writing libraries and don’t need that stuff.

You got all the backwards compatibility stuff, though, if you're code base is older than today. Or if you've got co workers who learned on c++11 instead of c++19... Also, templates, and the indecipherable errors they throw. And cpp memory handling errors are one of the largest sources of security holes. I'm convinced that apologising for cpp is basically Stockholm syndrome. These are things we absolutely would not ac…

It would be great if a modern language could match C++, but none is on the horizon.

Rust is powerful enough to do a lot of what C++ is used for, but not the hardest things. There are libraries you write routinely in C++ that you can't code in, or call from, Rust, and never will. The gap will widen with time. Rust is focused on low-level safety at the expense of library power, but you get safety in C++ by coding using more powerful and safe libraries, so C++ comes out ahead in the end.

Rust is a way better language than Go or Java. It would be excellent if Rust could displace them, or even just one of them.

If its adoption rate were to increase by maybe two or three orders of magnitude, it could survive. But the Rust core team seem not interested in doing what would be needed to get that much adoption, and hype alone won't get it there.

Rust's place will be secure when more complaints are written about it than paeans.

Re: 14,000x Speedup (2015)

#228

Post by ncmncm »

andi999 wrote: post #170
formerly_proven wrote: post #110

Earlier quoted context omitted.

There's very little going on under the hood with C++, if it even has a hood.

Vtable and vtable pointer is under the hood.

Those are trivial. And have little effect on performance. And are hardly used in good programs.

In practice they mainly displace function pointers.

Re: 14,000x Speedup (2015)

#229

Post by andi999 »

ncmncm wrote: post #228
andi999 wrote: post #170

Earlier quoted context omitted.

Vtable and vtable pointer is under the hood.

Those are trivial. And have little effect on performance. And are hardly used in good programs. In practice they mainly displace function pointers.

They might destroy your cache lines though.

Re: 14,000x Speedup (2015)

#230

Post by Tomis02 »

milesvp wrote: post #122
mynegation wrote: post #104

Earlier quoted context omitted.

C++, while complicated, is actually pretty transparent about it. The difference between pass-by-value and pass-by-reference is one sigil, but at least you see it there and you have at least some idea that it is there for a reason. Compare it with lazy evaluation languages like Haskell or declarative languages like SQL where oftentimes you have to run query planner explanation to troubleshoot the performance problems.

+1 for sql. I once was looking into a slow query and had to know that ordering text field in mysql requires a filesort regardless of how much data was involved. which means touching a disk. Was easy enough to make a schema change to a varchar small enough not to also trigger a filesort. But learning that required several hours of scouring documentation and making sure the schema change wasn’t modifying the requiremen…

> Also what’s fun is that the data in the tables can change the explain plan.

1. This change tends to be in your favour, not against you.

2. Data is important, not the code. Of course the plan would change if your data changes. That is the point of the query planner, to find the most efficient way of retrieving the data, based of course on the contents of the database.

I agree that you need a good knowledge of your database in order to be effective. Just like with C++, for example. Unfortunately we're not at the point where we can achieve optimal results without understanding the fundamentals, regardless of the programming paradigm.

Post reply on HN