Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Which recent research paper blew your mind?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 174 posts

Re: Ask HN: Which recent research paper blew your mind?

#61

Post by claytonjy »

GregarianChild wrote: post #31
claytonjy wrote: post #19

Earlier quoted context omitted.

> First time for ML that is not deep learning What do you mean by this? Virtually all "classic" or "shallow" ML can be GPU-accelerated, from linear regression to SVM to GBM.

Can you point me to papers with reproducible benchmarking that achieves big speedups on those? Modern GPUs are GP -GPUs: where GP means "general purpose" : you can run any code on GPGPUs. But if you want to gain real speed-ups you will have to program in an awkward style ("data parallel"). I am not aware of GPU acceleration of the work-horses of symbolic AI, such as Prolog, or SMT solving. There has been a lot of wor…

I think we're conflating two things: shallow/classic ML is not symbolic AI. I'm not sure "ML" even encompasses anything "symbolic"; I see symbolic AI and ML as subfields with little overlap.

I'm not saying symbolic AI has been GPU accelerated in the past, but that non-deep ML has been.

Re: Ask HN: Which recent research paper blew your mind?

#62

Post by kfrzcode »

pjs_ wrote: post #56
edent wrote: post #17

"Overview of SHARD: A System for Highly Available Replicated Data" it's the first paper to introduce the concept of database sharding. It was published in 1988 by the Computer Corporation of America. It is referenced hundreds of times in many classic papers. But, here's the thing. It doesn't exist . Everyone cites Sarin, DeWitt & Rosenb[e|u]rg's paper but none have ever seen it. I've emailed dozens of academics, libr…

Going through the bibliography of other people's papers and theses, looking for papers that you better cite "for good luck", or because "you gotta cite that one" is a classic PhD student behavior (I've done it) and it's not terribly surprising that something like this can happen. In fact I'd expect it to be much more widespread...

Funny, in my high school literature class I clearly remember being chastised for having sources in my works cited; but not warranting their inclusion with an actual reference in the work.

It's kind of wild that LLMs and other models/sequences will be able to quickly suss out which papers have high levels of referential integrity.

Re: Ask HN: Which recent research paper blew your mind?

#63

Post by a-dub »

claytonjy wrote: post #19
GregarianChild wrote: post #10

Someone managed to GPU-accelerate program synthesis, a form of symbolic ML. First time for ML that is not deep learning: https://dl.acm.org/doi/10.1145/3591274 Deep learning took off precisely when the ImageNet paper dropped around 2010. Before nobody believed that backprop can be GPU-accelerated.

> First time for ML that is not deep learning What do you mean by this? Virtually all "classic" or "shallow" ML can be GPU-accelerated, from linear regression to SVM to GBM.

computing a single matrix right divide is probably faster done in place with the cpu using vector math instructions.

Re: Ask HN: Which recent research paper blew your mind?

#64

Post by GregarianChild »

d_burfoot wrote: post #25
GregarianChild wrote: post #10

Someone managed to GPU-accelerate program synthesis, a form of symbolic ML. First time for ML that is not deep learning: https://dl.acm.org/doi/10.1145/3591274 Deep learning took off precisely when the ImageNet paper dropped around 2010. Before nobody believed that backprop can be GPU-accelerated.

> ... 2010. Before nobody believed that backprop can be GPU-accelerated. When I was doing my master's in 2004-06, I talked to a guy whose MSc thesis was about running NNs with GPUs. My thought was: you're going to spend a TON of time fiddling with hacky systems code like CUDA, to get basically a minor 2x or 4x improvement in training time, for a type of ML algorithm that wasn't even that useful: in that era the SVM w…

Exactly.

It's easy to see in retrospect, but hard in prospect: the original paper [1] on GPU acceleration of NNs reports a measly 20x speedup. Assuming a bit of cherry-picking on the author's side to make get the paper published, the 'real-world speedup' will have been assumed by the readership to be less. But this triggered a virtuos cycle of continuous improvements at all levels that has been dubbed "winning the hardware lottery" [2].

[1] K.-S. Oh, K. Jung, GPU implementation of neural networks.

[2] S. Hooker, The Hardware Lottery. https://arxiv.org/abs/2009.06489

Re: Ask HN: Which recent research paper blew your mind?

#65

Post by mindcrime »

antegamisou wrote: post #54
alexb_ wrote: post #29

Earlier quoted context omitted.

This is wild if true. Surely someone has to have a copy of this. How is it even being referenced if it is non existent?

I don't know why parent comment is stirring up drama but: 1. Not available online doesn't mean the paper's existence is made up. It's a very bold claim to make for the authors that they cite work that is fabricated. From the available information, this looks like a technical report by a, probably now defunct, company back in the 80s. If this was its only form of publication, and not on some conference proceedings for…

Not available online doesn't mean the paper's existence is made up.

True, but note that the post you're referring to does say:

I've emailed dozens of academics, libraries, and archives - none of them have a copy.

So this isn't somebody just saying "I couldn't find it with Google, therefore it doesn't exist."

From the available information, this looks like a technical report by a, probably now defunct, company back in the 80s.

Yeah, I think that's the key point. An internal technical memo from a private company, from that far back, isn't likely to be easy to find. It's quite possible that it's never been digitized and put on the 'net, and it it wasn't published in a journal, it may never have been archived by any university libraries or such-like.

That said, I'd be a little surprised if a copy didn't turn up somewhere, even if it means a former employee of CCA finding a copy in a desk drawer and providing it. But who knows?

Re: Ask HN: Which recent research paper blew your mind?

#66

Post by msravi »

Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm [0]

Also known as the Viterbi algorithm. Every digital communication device in existence today most likely has an implementation of it.

Later proved optimal by Forney [1]

0. https://www.essrl.wustl.edu/~jao/itrg/viterbi.pdf

1. https://www2.isye.gatech.edu/~yxie77/ece587/viterbi_algorith...

Re: Ask HN: Which recent research paper blew your mind?

#67

Post by CamperBob2 »

edent wrote: post #17

"Overview of SHARD: A System for Highly Available Replicated Data" it's the first paper to introduce the concept of database sharding. It was published in 1988 by the Computer Corporation of America. It is referenced hundreds of times in many classic papers. But, here's the thing. It doesn't exist . Everyone cites Sarin, DeWitt & Rosenb[e|u]rg's paper but none have ever seen it. I've emailed dozens of academics, libr…

1988? As far as anyone can tell, the use of the term "shard" in the context of database replication originated with Ultima Online, which was released in 1997, and which used the term in connection with its underlying mythos (the idea of representing world instances as shards of Mondain's shattered gem).

So a documented reference to sharding that's earlier than that would be interesting to see.

(Disagree? Instead of downvoting, consider posting a citation that actually resolves to a real paper.)

Re: Ask HN: Which recent research paper blew your mind?

#68

Post by irrational »

littlestymaar wrote: post #41
HideousKojima wrote: post #27

Earlier quoted context omitted.

Happens far more than you think. It can be an innocent (sort of) mistake, where authors see the citation in a previous paper and simply copy it into their own.

> where authors see the citation in a previous paper and simply copy it into their own. Without opening the paper to even read the abstract? To me it doesn't sound like “innocent” at all, and borderline malpractice…

Funny. I read your previous comment (with the ?! ending) as sarcastic. Now I see you were serious. I would be astonished if most authors have actually read even a fraction of the sources they cite.

Re: Ask HN: Which recent research paper blew your mind?

#69

Post by rch »

d_burfoot wrote: post #25
GregarianChild wrote: post #10

Someone managed to GPU-accelerate program synthesis, a form of symbolic ML. First time for ML that is not deep learning: https://dl.acm.org/doi/10.1145/3591274 Deep learning took off precisely when the ImageNet paper dropped around 2010. Before nobody believed that backprop can be GPU-accelerated.

> ... 2010. Before nobody believed that backprop can be GPU-accelerated. When I was doing my master's in 2004-06, I talked to a guy whose MSc thesis was about running NNs with GPUs. My thought was: you're going to spend a TON of time fiddling with hacky systems code like CUDA, to get basically a minor 2x or 4x improvement in training time, for a type of ML algorithm that wasn't even that useful: in that era the SVM w…

I went to a talk on "general purpose GPU programming" at the Colorado School of Mines around 2001 that covered exactly that topic. It was very disappointing to have my interest in FPGAs for this purpose be so entirely destroyed by a quirk of graphics card design.

Hinton also addressed the contribution of hardware performance advances to practical deep neural net applications in his talks in the mid-2000s.

Re: Ask HN: Which recent research paper blew your mind?

#70

Post by numbers_guy »

w-m wrote: post #26

Integral Neural Networks (CVPR 2023 Award Candidate), a nifty way of building resizable networks. My understanding of this work: A forward pass for a (fully-connected) layer of a neural network is just a dot product of the layer input with the layer weights, followed by some activation function. Both the input and the weights are vectors of the same, fixed size. Let's imagine that the discrete values that form these…

Going just by your description this sounds like they are doing operator learning. It's actually a very old idea. The proof that started operator learning is from 1988 I believe. Mathematicians have been playing around with the idea since 2016 at least.
Post reply on HN