31–40 of 204 posts
I'm a VC and I've seen lots of pitches lately for various tech related to new architectures, be it GPU, FPGA or RISC. I'd add cryptocurrency to list workloads with insatiable computing demands.
Massive parallel prefix and reduction operations in RAM. Start with just the MPI standard reduction ops. 1000x faster than Von Neuman bottleneck.
Sounds similar to Micron Automata Processor.
I have to say that most bottlenecks are from lazy designs, not wimpy hardware. As a thought experiment, suppose parallel processing scaled easily without worrying about Amdahl's Law and similar bottlenecks. So then we put 128 cores into our computers. Software companies would eat up every single core eventually if there's no penalty for inefficient coding. They don't pay our electric bill, we do. It's kind of like fr…
Software is a gas; it expands to fill its container - Nathan Myhrvold
Bad performance isn't really a software design problem; it's an economics problem (ditto for security). You could speed up the browser by 2x now, or reduce it's memory usage by 2x, and websites would be exactly as slow after a few months.
I'm not trying to be negative here, just stating an unfortunate systems dynamic.
Are there languages that have first-class support for representing/optimizing memory hierarchy characteristics? Optimizing C compilers, for example, may have extensions to force specific alignments: https://software.intel.com/en-us/articles/coding-for-perform... But I'm not aware of languages where e.g. declaring alignments is part of the base language. Awareness of L1, L2, L3 cache characteristics, plus NUMA nodes,…
https://scholar.google.com/scholar?cites=1705642111681573037...
I see Halide and several interesting projects. Let me know if you find anything interesting :)
It does seem to confirm that locality aware programming models are probably going to look like "big data" programming models (i.e. MapReduce and family).
Are there languages that have first-class support for representing/optimizing memory hierarchy characteristics? Optimizing C compilers, for example, may have extensions to force specific alignments: https://software.intel.com/en-us/articles/coding-for-perform... But I'm not aware of languages where e.g. declaring alignments is part of the base language. Awareness of L1, L2, L3 cache characteristics, plus NUMA nodes,…
C++ has alignas since C++11 https://en.cppreference.com/w/cpp/language/alignas and since C++17 has std::hardware_destructive_interference_size and std::hardware_constructive_interference_size https://en.cppreference.com/w/cpp/thread/hardware_destructiv... which you can use together to either make sure that 2 variables are not in the same cache line (if you are making a lock-free list for example) or to make sure at c…
It's really hard. Remember the Itanium and the Cell. You can build it, but they may not come. GPUs, though. Those have turned out to be very successful, they can be parallelized as much as you're willing to pay for, and they're good for some non-graphics tasks. Much of machine learning is a simple repetitive computation running at low precision. Special purpose hardware can do that very well. So what's the next usefu…
Well, past experiences are not that useful right now. I don't think the Itanium or the Cell were great architectures, but it's clear that their most obvious failure mode (mainstream architectures improving faster than them) isn't a showstopper anymore. For some guess on the next big thing, I would imagine that stuff that avoid the need of memory coherence have nice odds.
The main lesson I would take away from Cell is that you want to design an architecture that allows gradual performance refinement. With the Cell, it was more of a step function: good performance came from using the SPEs well, but using the SPEs well was hard. There wasn't much of an in-between.
Are there languages that have first-class support for representing/optimizing memory hierarchy characteristics? Optimizing C compilers, for example, may have extensions to force specific alignments: https://software.intel.com/en-us/articles/coding-for-perform... But I'm not aware of languages where e.g. declaring alignments is part of the base language. Awareness of L1, L2, L3 cache characteristics, plus NUMA nodes,…
I have to say that most bottlenecks are from lazy designs, not wimpy hardware. As a thought experiment, suppose parallel processing scaled easily without worrying about Amdahl's Law and similar bottlenecks. So then we put 128 cores into our computers. Software companies would eat up every single core eventually if there's no penalty for inefficient coding. They don't pay our electric bill, we do. It's kind of like fr…
Yes, a pithier way of saying this is: Software is a gas; it expands to fill its container - Nathan Myhrvold Bad performance isn't really a software design problem; it's an economics problem (ditto for security). You could speed up the browser by 2x now, or reduce it's memory usage by 2x, and websites would be exactly as slow after a few months. I'm not trying to be negative here, just stating an unfortunate systems d…
Information-theorically at least, I bet that is another 10-100x speedup opportunity right there (and next multi-billion dollar industry, perhaps)...
I have to say that most bottlenecks are from lazy designs, not wimpy hardware. As a thought experiment, suppose parallel processing scaled easily without worrying about Amdahl's Law and similar bottlenecks. So then we put 128 cores into our computers. Software companies would eat up every single core eventually if there's no penalty for inefficient coding. They don't pay our electric bill, we do. It's kind of like fr…
Yes, a pithier way of saying this is: Software is a gas; it expands to fill its container - Nathan Myhrvold Bad performance isn't really a software design problem; it's an economics problem (ditto for security). You could speed up the browser by 2x now, or reduce it's memory usage by 2x, and websites would be exactly as slow after a few months. I'm not trying to be negative here, just stating an unfortunate systems d…
Why do you say so?
As part of one of those "45 hardware startups" (in stealth mode) trying to tackle the problem, I think nevertheless there is plenty of fat to trim just on the software side still... I mean, just look at that very page: 200 MB [edit: in memory consumption] for a single static article! Information-theorically at least, I bet that is another 10-100x speedup opportunity right there (and next multi-billion dollar industry…