Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LoRA from scratch: implementation for LLM finetuning

lightning.ai

Post reply on HN
Print view View original

81–87 of 87 posts

Re: LoRA from scratch: implementation for LLM finetuning

#81

Post by mejutoco »

r3trohack3r wrote: post #30
rsweeney21 wrote: post #23

It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."

I feel like it's the difference between something that has been engineered and something that has been discovered. I feel like most of our industry up until now has been engineered. LLMs were discovered.

I believe, from what I saw in Mathematics, this is a matter of taste. Discovered or invented are 2 perspectives. Some people prefer to think that light is reaching in previously dark corners of knowledge waiting to be discovered(discover). Others prefer to think that by force of genius they brought the thing into the world.

To me, personally, these are 2 sides of the coin, without one having more proof than the other.

Re: LoRA from scratch: implementation for LLM finetuning

#82

Post by raxxorraxor »

rsweeney21 wrote: post #23

It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."

Welcome to engineering. We don't sketch our controlled systems and forget all about systems theory. Instead we just fiddle with out controllers until the result is acceptable.

Re: LoRA from scratch: implementation for LLM finetuning

#83

Post by jncfhnb »

rsweeney21 wrote: post #23

It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."

Not strange at all. This is largely how biology operates. These things are simpler than bio and more complex than programs

Re: LoRA from scratch: implementation for LLM finetuning

#84

Post by jcuenod »

sbrother wrote: post #39
denysvitali wrote: post #29

LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym

Wait, what is the meaning other than "Low-Rank Adaptation"? It's hard to google the difference.

Trying asking an LLM :)

Re: LoRA from scratch: implementation for LLM finetuning

#85

Post by jcuenod »

pizza wrote: post #74
chenxi9649 wrote: post #24

It's still not too clear to me when we should fine tune versus RAG. In the past, I used to believe that finetuning is mostly for model behavioral change, but recently it seems that certain companies are also using fine-tuning for knowledge addition. What are the main use cases for fine tuning?

These are autoregressive models. When you have a new type of sequence where future elements are able to be predicted from previous parts of the sequence, but in a new kind of way than the models have seen before, it would make sense to finetune. Admittedly, that's a pretty vague descriptor for how to decide what to do for a given data scenario, but it might be good enough as a rough heuristic. Now, whether knowledge…

Exactly this. If you have a model that's never seen JSON and you want JSON to come out, fine-tuning probably not a bad idea. If you have a model trained on English documents and you want it to produce English documents related to your company, you don't need to fine-tune.

Re: LoRA from scratch: implementation for LLM finetuning

#87

Post by dvngnt_ »

sschueller wrote: post #31
denysvitali wrote: post #29

LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym

It's unfortunate that those two so far unrelated technologies have the same acronym.

probably better than them being similar but not exactly since context still helps
Post reply on HN