Memristor Breakthrough: First Single Device to Act Like a Neuron • Hacker News Board

Re: Memristor Breakthrough: First Single Device to Act Like a Neuron

#91

Post by unishark » Fri, Oct 02, 2020, 9:29 AM UTC

freeone3000 wrote: post #69

mywittyname wrote: post #53

Earlier quoted context omitted.

It's fascinating that none of the comments mention neural network applications. Seems like these people came a bit too early to the party.

We've left biomimicry in the dust a fair time ago - neurons do not work like relus after all (nor do they work like memristors), but relus are great at solving problems. None of our current NN applications would even want something that behaves like a neuron. You'd essentially have to start from scratch. It's not even clear if behaves-like-a-neuron is better or not to be worth the effort. (And that's disregarding the…

I thin it's more that we're operating in a position of extreme simplification, and don't understand most biological details well enough to apply them in our own systems. But biological motivation is quite common in neural networks. Name a popular technique (backprop, dropout, etc.) and I can probably find you a source arguing it's biological relationship. It's just at an engineering-style level of approximation that scientists may abhor.

The ReLU was originally introduced by computational neuroscientists as a more biologically accurate approximation compared to the other activation functions. E.g. https://www.nature.com/articles/35016072

I would expect similar history behind convolutional neural networks, given how the early stages of the vision system have been understood to operate.

Re: Memristor Breakthrough: First Single Device to Act Like a Neuron

#92

Post by Taniwha » Fri, Oct 02, 2020, 9:39 AM UTC

gnramires wrote: post #89

Taniwha wrote: post #52

Earlier quoted context omitted.

If you're building adders with O(n) carry times you're doing it wrong - normal carry lookahead logic is ~O(log N)

The gate count for addition is at least O(n) though. You should note the O(log n) delay growth is for a logic (ideal) model. Considering the 2D geometry of real circuits, there are models with more detail, in particular VLSI models. [1] I suspect due to routing and path lengths even the delay would turn out to be O(n) in VLSI, or at least a power of n (with a small constant). [1] Seems like a good introduction: http:…

Oh yeah definitely > O(n) - but we often trade gates (and area) for speed either by pipelining and/or more parallelism (which is what carry-lookahead logic is)

Re: Memristor Breakthrough: First Single Device to Act Like a Neuron

#93

Post by orbifold » Fri, Oct 02, 2020, 9:46 AM UTC

freeone3000 wrote: post #69

mywittyname wrote: post #53

Earlier quoted context omitted.

It's fascinating that none of the comments mention neural network applications. Seems like these people came a bit too early to the party.

We've left biomimicry in the dust a fair time ago - neurons do not work like relus after all (nor do they work like memristors), but relus are great at solving problems. None of our current NN applications would even want something that behaves like a neuron. You'd essentially have to start from scratch. It's not even clear if behaves-like-a-neuron is better or not to be worth the effort. (And that's disregarding the…

Research in this area has always been cyclical. The Neural ODE paper that won a best paper award at Neurips two years ago just implements a technique that is common knowledge in the optimal control theory community since the 60s.

I wouldn’t dismiss an approach because it isn’t popular right now. Biological neurons force you to think about constraints you normally wouldn’t and crucially have solved temporal credit assignment, memory etc. in interesting ways that already have inspired some breakthroughs.

As an example the replay memory idea was directly inspired by the rough understanding neuroscientist developed of the role of the hypocampus in learning.

Re: Memristor Breakthrough: First Single Device to Act Like a Neuron

#94

Post by Flibble21 » Fri, Oct 02, 2020, 10:59 AM UTC

visarga wrote: post #73

What neural nets need in order to improve is lots of data, and especially simulated environments. We have had the advantage of billions of years of evolution, they start the same journey from scratch. Human brain is not smart in and of itself - it learns everything from the environment, including fundamental concepts and reasoning. We are not just surrounded by nature, with all its glorious detail, but also by the hu…

Plato has entered the chat

Re: Memristor Breakthrough: First Single Device to Act Like a Neuron

#95

Post by pritovido » Fri, Oct 02, 2020, 11:26 AM UTC

mikorym wrote: post #79

mdturnerphys wrote: post #76

The news article mentions that the student on the paper has passed away. He died of suicide in his lab just a couple weeks after the paper was submitted to Nature [0]. Grad school is hard for a lot of people. Find someone to talk to if you're struggling. [0] https://padailypost.com/2019/02/16/engineering-student-found...

I've had an idea for a while, coming from a developing continent, that it would be good to have an alternative for students where they can start working when they are doing an advanced degree, but also simply a place for high school students to acclimate to some of the aspects of the work place. I know it sounds tangential, but this does resonate with my own roundabout way of going from studies -> work and the gross…

What is needed is removing the source of the problem in the first place.

You are in a foreign country, with no friends, no family, working over time, in some places extremely little money.

If you are burnout the first thing that you need to do is to stop. Make friends and lovers, travel and work outside for a year or so.

To use medication or drugs is not going to solve your problems,if you continue living such an unnatural lifestyle.

And I said that having lived that lifestyle myself, but I took my breaks from the toxic system.

Re: Memristor Breakthrough: First Single Device to Act Like a Neuron

#96

Post by ficklepickle » Fri, Oct 02, 2020, 12:48 PM UTC

adatavizguy wrote: post #60

Off topic. When something is on the tip of my tongue, I don't search for it because I have a strange belief that what my mind is trying to do to remember it is fire off neurons associated with the thought or idea, near it, to create the action potential. It is strange that I will eventually remember it within hours or the next day. When the action potential fires the neuron, dendrites are formed between neurons stren…

When I started learning programming, my French became much easier to recall. This was after ~ a decade of very infrequent French language use. Much of it has subsequently come back to me.

Sample size of 1, yadda, yadda. Make of it what you will. I think it is interesting.

Re: Memristor Breakthrough: First Single Device to Act Like a Neuron

#97

Post by speeder » Fri, Oct 02, 2020, 1:03 PM UTC

pritovido wrote: post #95

mikorym wrote: post #79

Earlier quoted context omitted.

I've had an idea for a while, coming from a developing continent, that it would be good to have an alternative for students where they can start working when they are doing an advanced degree, but also simply a place for high school students to acclimate to some of the aspects of the work place. I know it sounds tangential, but this does resonate with my own roundabout way of going from studies -> work and the gross…

What is needed is removing the source of the problem in the first place. You are in a foreign country, with no friends, no family, working over time, in some places extremely little money. If you are burnout the first thing that you need to do is to stop. Make friends and lovers, travel and work outside for a year or so. To use medication or drugs is not going to solve your problems,if you continue living such an unn…

Problem is, how?

When I went to university I ended in deep debt.

Took me years, to pay off that debt, it felt like literal slavery, since I would just take whatever job that could pay the debt that was offered (I had to decline many jobs because their pay was lower than my debt, meaning taking them I would just stay in debt forever), and then I would stay until I was fired or had to quit.

Even then, those were not "real" jobs according to our government, in my country my "real" job registry, is empty, I never "worked", because I never got hired full-time legally, I couldn't afford to, whatever people offered that the pay was good enough, I would accept, even if the employer was obviously hiring me as "freelancer" just to not pay taxes.

The times I had burned out during this, I had to keep going, travelling for a year? I dunno if I will ever afford that. Making friends? How? When? I am not in a foreign country yet I have two friends at most, and one of them moved to another country and I see him once every 5 years.

So it is not like people have a choice after they went to college, I still feel that going to college was the biggest mistake I ever did on my life, I should never have went, and never took that debt, it is paid now, but I am working with Marketing, instead of working with programming, and have little room for error, no debt, but no surplus either, if something goes wrong with the business I am screwed.

Re: Memristor Breakthrough: First Single Device to Act Like a Neuron

#98

Post by 542354234235 » Fri, Oct 02, 2020, 1:05 PM UTC

pritovido wrote: post #95

mikorym wrote: post #79

Earlier quoted context omitted.

I've had an idea for a while, coming from a developing continent, that it would be good to have an alternative for students where they can start working when they are doing an advanced degree, but also simply a place for high school students to acclimate to some of the aspects of the work place. I know it sounds tangential, but this does resonate with my own roundabout way of going from studies -> work and the gross…

What is needed is removing the source of the problem in the first place. You are in a foreign country, with no friends, no family, working over time, in some places extremely little money. If you are burnout the first thing that you need to do is to stop. Make friends and lovers, travel and work outside for a year or so. To use medication or drugs is not going to solve your problems,if you continue living such an unn…

> What is needed is removing the source of the problem in the first place...To use medication or drugs is not going to solve your problems,if you continue living such an unnatural lifestyle.

But this ignores that the source of the problem could be brain chemistry itself. Or at least that it is part of the problem. If it were situational only, everyone in the same situation would have the same outcome. But the physical structure and chemical cocktail of each individual brain plays a huge role in how those situations affect each person. The brain is an organ, and just like any other organ, there are situations that cannot be fixed with “lifestyle changes”. Medication should not just be hand waved away.

Re: Memristor Breakthrough: First Single Device to Act Like a Neuron

#99

Post by agumonkey » Fri, Oct 02, 2020, 1:10 PM UTC

jacobush wrote: post #80

mikorym wrote: post #79

Earlier quoted context omitted.

I've had an idea for a while, coming from a developing continent, that it would be good to have an alternative for students where they can start working when they are doing an advanced degree, but also simply a place for high school students to acclimate to some of the aspects of the work place. I know it sounds tangential, but this does resonate with my own roundabout way of going from studies -> work and the gross…

With depression, the "why" is never easy to suss out, even when something is stated, is that really the reason?

I have a feeling that depression relies on many pillars of your mind breaking.

Re: Memristor Breakthrough: First Single Device to Act Like a Neuron

#100

Post by 542354234235 » Fri, Oct 02, 2020, 1:11 PM UTC

adatavizguy wrote: post #60

Off topic. When something is on the tip of my tongue, I don't search for it because I have a strange belief that what my mind is trying to do to remember it is fire off neurons associated with the thought or idea, near it, to create the action potential. It is strange that I will eventually remember it within hours or the next day. When the action potential fires the neuron, dendrites are formed between neurons stren…

I seem to remember reading about how thinking about something on the tip of your tongue and failing makes it harder to remember in the future, because concentrating on trying to remember it strengthens pathways that don’t lead back to the memory. So when you try in the future, those failed paths are stronger and more likely to be used again, further reinforcing the wrong association. Letting it go and hoping it comes to you also might not work, but it doesn’t strengthen bad pathways, so is more likely to allow you to remember in the future.