311–320 of 376 posts
Earlier quoted context omitted.
Isn't D dead practically? https://news.ycombinator.com/item?id=21902953
And yet dicebot continues to work in D, including for me for a while. I'm hiring 25 D programmers, so I suppose it very much depends on what you mean by practically!
people will bike to work to "save the environment" but won't use C direct on metal to reduce the carbon footprint of their code. For a guy like Carmack, it may be quite frustrating working with the constraints of pytorch etc. He ll probably end up making his own pytorh frontend in C, which as a bonus people will use to deploy models.
Hypocrisy is not such a rare animal. There is however extra factor here: biking to work if possible is your personal choice. Using your favorite tools at work: often not as much
Earlier quoted context omitted.
> Are you saying this can be optimised to fit inside a single 10 core server in terms of compute loads? I'm currently employed to write software which does DNA analysis. DNA is known for being Big Data. Some applications are very compute intensive and others are very data-relation intensive. The very compute intensive applications process about 1GB of data in about 30 minutes on a 32 core Xeon 6xxx with 32GB of RAM a…
That's very interesting. Do you mind sharing some more details on what those computations are? Where can I learn more about these DNA analysis use cases?
The compute-heavy workload calculates edit distance [0] of short paired-end sequenced DNA [1] vs the human genome [2]. There is open source software to manipulate the FASTA/FASTQ [3] and SAM files [4] and run the calculations [5]. The aligned file is processed in a couple of minutes to genetic variation report [6] which is used for some of the analysis products that were purchased. One popular product will give you a haplogroup [7] which basically tells you where you are in a genetic tree.
The relationship estimator uses a different sequencing technology and basically consumes a CSV file from the sequencer's manufacturer. It uses a proprietary algorithm to calculate centimorgans [8]. That then gives relationship estimates between you and other people who've purchased the product.
[0]: https://en.wikipedia.org/wiki/Needleman%E2%80%93Wunsch_algor...
[1]: https://en.wikipedia.org/wiki/Next-generation_sequencing
[2]: https://en.wikipedia.org/wiki/Reference_genome
[3]: https://en.wikipedia.org/wiki/FASTQ_format
[4]: https://github.com/samtools/hts-specs
[5]: https://en.wikipedia.org/wiki/List_of_sequence_alignment_sof...
[6]: https://en.wikipedia.org/wiki/Variant_Call_Format
Earlier quoted context omitted.
Can you expand on this? I have some pretty massive compute loads that need to be scaled onto a cluster with 100+ workers for most computations. This is after I use a library called dask that graphically does its own mapreduce optimisation inside its modules. This is all for a relatively small 250GB raw data file that I keep in a csv (and need to convert to SQL at some point). Are you saying this can be optimised to f…
Don't know why you're being downvoted but I'll assume your question is genuine. You use a cluster when your data and compute requirements are large and parallel enough that the tax paid on network latency trumps the 10-20X speedup you get on SSD and 1000X speedup you get from just keeping data in RAM. 250 Gigs is tiny enough that you could probably much get better performance running on high memory instance in AWS or…
However, even if I could load it all into memory at once, and assuming it takes 200 gb, I'm still using a master's student access to a cluster. So I get preempted like it's nobody's business. Hence why I prefer a smaller memory footprint even if I take up cpus at variable rates through a single execution.
I did try to write my own multiprocessing code for this, but the operations are sometimes too complicated (like groupby) for me to rewrite everything from the ground up. If I'm not reliant on serial data communication between processes (like you'd need to sort a column), I can get it done pretty easily. In fact, I wrote my data cleaning code with this and cleaned up the entire file in half an hour because single chunks didn't rely on others.
However, if you have some idea of how to run these computational loads in parallel in python or any other language on single compute instances (like the size of a laptop's memory of 16 gb), I'd really love to see it. Thanks.
Earlier quoted context omitted.
Hi, I'm curious because I plan to rewrite a Rust service to Go (development velocity is too slow) Which part of your service is in Rust amd which in Go? Do you think that if it would be only in Go it could sustain such a load?
What specific ergonomics pitfalls did you experience in Rust? "Development velocity is too slow" is a bit vague; beyond the use of GC, which only really matters in specialized domains, there's not much reason to think that rewriting your service in Go would give you better 'development velocity'.
Rust is slow to compile so it breaks my deep work when programming, also it costs me a lot in CI/CD. Also the Rust type system make implementing some things really hard.
For example I wanted to implement json requests logging. It took me more than 1 day in Rust, less than 2 hour in go.
Earlier quoted context omitted.
This sounds like an amazing war story that I just want to hear more of. Is there any more? What's your c++ stack like?
Currently using Restbed ( https://github.com/Corvusoft/restbed ) as the server core, wxWidgets as a server side gui, with Boost, Curl, SQLite and Standard Lib. It's not that complex, beyond using lambdas in a few places. It has extremely high performance, and can run on an Intel Compute Stick, but I tend to use an Intel Nuc at minimum, with clients typically using whatever they have, gaining over redundancy and an ab…
Earlier quoted context omitted.
Hi, I'm curious because I plan to rewrite a Rust service to Go (development velocity is too slow) Which part of your service is in Rust amd which in Go? Do you think that if it would be only in Go it could sustain such a load?
What specific ergonomics pitfalls did you experience in Rust? "Development velocity is too slow" is a bit vague; beyond the use of GC, which only really matters in specialized domains, there's not much reason to think that rewriting your service in Go would give you better 'development velocity'.
To quote Rob Pike:
The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.
Go is aiming for the same niche that Python is. This has the accidental side effect of making it faster to write software in it than most of the ALGOL-derivatives. To quote Eric Raymond (sorry) on Python:
When you're writing working code nearly as fast as you can type and your misstep rate is near zero, it generally means you've achieved mastery of the language. But that didn't make sense, because it was still day one and I was regularly pausing to look up new language and library features!
Go and Python both allow for expression about as fast as you can type, by virtue of being designed for [children in Go's case, shell scripting in Python's case].
It's not a value judgement of Rust or anything, but Rust wasn't designed with the same goals in mind.
A site for proof. It keeps amusing me on what hardware/software Stack Overflow/Stack Exchange is running on: https://stackexchange.com/performance This is way less in HW than most people in the trade (from web devs to devops) seem to think when asked about it. SO ranks #36 in Alexa right now: https://www.alexa.com/siteinfo/stackoverflow.com
Any reason for such a low (<5%) average CPU usage? It seems like a waste of resources to me; that's assuming a "normal" CPU usage read that includes wait i/o time.
Earlier quoted context omitted.
What specific ergonomics pitfalls did you experience in Rust? "Development velocity is too slow" is a bit vague; beyond the use of GC, which only really matters in specialized domains, there's not much reason to think that rewriting your service in Go would give you better 'development velocity'.
There is, though. Go was designed to be easier to reason with than conventional ALGOL-derivatives, while Rust wasn't. To quote Rob Pike: The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use the…
This may be a problem of what's idiomatic in each language, as opposed to a matter of language design per se. After all, Rust development can be made at least as easy as, e.g. Swift, simply by adding enough uses of .clone() and RefCell. Is this suboptimal? Of course, but it will still be plenty faster than Python, and perhaps even faster than Go.
Compilation time is a separate issue which apparently OP found problematic. It's being dealt with (for non-release optimized builds) via the cranelift project, which is a Rust-specific backend much like the Go compiler, with no reliance on LLVM.
Earlier quoted context omitted.
Don't know why you're being downvoted but I'll assume your question is genuine. You use a cluster when your data and compute requirements are large and parallel enough that the tax paid on network latency trumps the 10-20X speedup you get on SSD and 1000X speedup you get from just keeping data in RAM. 250 Gigs is tiny enough that you could probably much get better performance running on high memory instance in AWS or…
I'd love to keep it in RAM if I could. The problem is, the library I'm familiar with (pandas) typically seems to take more memory than the original csv file once it loads it onto memory. I know this is due to bad data types but in certain cases, I cannot get around those. However, even if I could load it all into memory at once, and assuming it takes 200 gb, I'm still using a master's student access to a cluster. So…