Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparison of C++17, Go, and Java for a next-generation sequencing tool

bmcbioinformatics.biomedcentral.com

Post reply on HN
Print view View original

111–120 of 190 posts

Re: Comparison of C++17, Go, and Java for a next-generation sequencing tool

#111

Post by beached_whale »

gpderetta wrote: post #73
ezoe wrote: post #69

After quickly glancing the code, I concluded that they wrote C++ like there is no static type. It seems they faithfully ported the very dynamic nature of their existing code to C++ without thinking. Like what is this? https://github.com/ExaScience/elprep-bench/blob/master/cpp/f... auto alns = any_cast >>>(data); So the data is sam_alignment type inside shared_ptr inside deque inside another share_ptr inside god forbi…

The use of 'any' and the ridiculous levels of boxing is bizarre. There is also absolutely no attempt at using std::move; the amount of gratuitous copies and refcount updates is staggering. Having said that, apart from the use of std::any, the code is fairly clean even though obviously is not even remotely written for performance..

A copy in function arguments often is not expressed by the compiler and often allows more optimizations.

Re: Comparison of C++17, Go, and Java for a next-generation sequencing tool

#112

Post by FpUser »

benibela wrote: post #80
ezoe wrote: post #69

After quickly glancing the code, I concluded that they wrote C++ like there is no static type. It seems they faithfully ported the very dynamic nature of their existing code to C++ without thinking. Like what is this? https://github.com/ExaScience/elprep-bench/blob/master/cpp/f... auto alns = any_cast >>>(data); So the data is sam_alignment type inside shared_ptr inside deque inside another share_ptr inside god forbi…

>the unordered_map is very slow(it's a node based hash map, not suitable for the modern hardware) It is very slow? I wrote my app in FreePascal. I need a hashmap, but FreePascal has no real standard hashmap, so I have been benchmarking Pascal hashmaps for weeks/months. Today I added std::unordered_map for comparison. It will still take a day to run the benchmark, but so far it looks that std::unordered_map is 25% fas…

Free Pascal actually does have hashmap. Check this for reference: https://wiki.freepascal.org/Data_Structures,_Containers,_Col...

Re: Comparison of C++17, Go, and Java for a next-generation sequencing tool

#114

Post by aidenn0 »

tomp wrote: post #63
aidenn0 wrote: post #21

Earlier quoted context omitted.

I guess I was wrong with this comment: https://news.ycombinator.com/item?id=22959600 1. Reference counting is a form of GC; you could implement a JVM that used reference counting (though in order to be general a small amount of additional work is needed) 2. Reference counting causes extra work every time a reference appears or disappears. Tracing GCs amortize that cost across many allocations. 2.b. This is particular…

> you could implement a JVM that used reference counting It would leak memory - reference counting cannot collect cycles (you need tracing GC for that, defeating the purpose of refcounting).

From right after what you quoted:

> (though in order to be general a small amount of additional work is needed)

Not also that there are two methods of cycle detection for a reference counted GC that are not just a backup tracing-GC

1. Trial deletion (known since at least the mid 80s)

2. Various tracing systems that exploit extra information known to reference-counted systems e.g. Levanoni/Petrank[1] which actually implemented a reference counted GC for Java.

1: https://www.cs.technion.ac.il/~erez/Papers/refcount.pdf

Re: Comparison of C++17, Go, and Java for a next-generation sequencing tool

#115

Post by aidenn0 »

ken wrote: post #51
etse wrote: post #44

Earlier quoted context omitted.

Sorry about the naive question, but if the memory management overhead is worse in Swift, is the hardware it runs on typically better? I'm assuming some of this because I've noticed Android devices tend to require more CPU/memory compared to iOS devices in the same generation.

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mo... Lattner on Swift (2016): "...while it is true that modern GC's can provide high performance, they can only do that when they are granted much more memory than the process is actually using. Generally, unless you give the GC 3-4x more memory than is needed, you’ll get thrashing and incredibly poor performance..."

Agreed. I would say 2x the heap size is table-stakes for high-performance tracing GC, and the more you give it the better performance you can get.

Re: Comparison of C++17, Go, and Java for a next-generation sequencing tool

#117

Post by Rochus »

candiodari wrote: post #87
aidenn0 wrote: post #35

Earlier quoted context omitted.

> > Common Lisp also lets you use mmap > But not without allocating dynamic memory and copying data. Sure it does. In SBCL you can force a stack allocation (though rarely does it improve performance), and very short-lived values do not leave registers in any case. > > They clearly wanted automatic memory management > Most likely because of some misconceptions. There are both good and bad reasons to want automatic mem…

> > Don't forget the data sets to be filtered, sorted an analyzed are up to 200 GB. > Which is going to be rough on any automatic memory management system You could equally say that it makes the case for actually designing memory allocation strategy (which only C++ really supports) that much more important. You always see this in Java programs for large data analysis. They pick java because of memory management and t…

Yeah, right. The people who know this can apparently be counted on one hand when I look at the advertised publication and the discussion in this forum.

Re: Comparison of C++17, Go, and Java for a next-generation sequencing tool

#118

Post by aidenn0 »

Rochus wrote: post #50
aidenn0 wrote: post #35

Earlier quoted context omitted.

> > Common Lisp also lets you use mmap > But not without allocating dynamic memory and copying data. Sure it does. In SBCL you can force a stack allocation (though rarely does it improve performance), and very short-lived values do not leave registers in any case. > > They clearly wanted automatic memory management > Most likely because of some misconceptions. There are both good and bad reasons to want automatic mem…

Common Lisp: It is permissible for an implementation to simply ignore such declarations. And you still have to copy. Ref counting: only makes sense in a few special cases. Avoiding dynamic memory management: have a look at mmap.

C++: It is permissible for an implementation to allocate every local variable on the heap.

Going back to my original point, you are suggesting a complete rearchitecture of their allocation system. That does not require switching languages to C++. If we are talking about working with 100s of GB of data, that's probably even the correct approach!

TFA does not, however, claim that they have a working set of 100s of GB of data. The data is 100s of GB at rest, but can be processed in chunks with a single pass. That, by itself, does not scream "mmap" to me. On top of that, the data is compressed at rest, so copying is inevitable.

Re: Comparison of C++17, Go, and Java for a next-generation sequencing tool

#119

Post by WatchDog »

twic wrote: post #100

> To achieve good performance, it was therefore necessary to explicitly control how often and when the garbage collector would run to avoid needless interruptions of the main program, especially during parallel phases. Why? This is a batch program, interruptions don't matter, only the end-to-end time does. > The goal of elPrep is to simultaneously keep both the runtime and the memory use low. Why? Keeping runtime low…

Yeah there are lots more knobs they could have played with in regards to Java garbage collection.

I wonder if they capped the java max heap size to what the go implementation used, how much it would have affected the runtime.

Re: Comparison of C++17, Go, and Java for a next-generation sequencing tool

#120

Post by alexhutcheson »

WesternStar wrote: post #72
ezoe wrote: post #69

After quickly glancing the code, I concluded that they wrote C++ like there is no static type. It seems they faithfully ported the very dynamic nature of their existing code to C++ without thinking. Like what is this? https://github.com/ExaScience/elprep-bench/blob/master/cpp/f... auto alns = any_cast >>>(data); So the data is sam_alignment type inside shared_ptr inside deque inside another share_ptr inside god forbi…

We should break ABI on unordered map just to stop embarrassing ourselves in public and in front of new users.

It’s unfortunately not just ABI, but also API. The standard specifies that you can get iterations to specific buckets in O(1)[1], and also specifies bucket_count(), max_bucket_count(), bucket_size() (which is specified to be O(n)), and bucket(). Those functions and their specified performance make it effectively impossible to implement a standards-compliant std::unordered_map without using separate chaining.

[1] https://en.cppreference.com/w/cpp/container/unordered_map/be...

Post reply on HN