Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Fructose – LLM calls as strongly typed functions

github.com

Post reply on HN
Print view View original

71–80 of 105 posts

Re: Show HN: Fructose – LLM calls as strongly typed functions

#71

Post by lefttoreader »

edunteman wrote: post #52
itfollowsthen wrote: post #32

> not unlike other packages such as marvin This feels pretty much identical to Marvin? Like the entire API? From a genuine place of curiosity: I get that your prompts are different, but like why in the name of open source would you just not contribute to these libraries instead of starting your own from scratch?

Thanks for asking, and I'd agree. I'd give the same answer as the folks asking about instructor: we built this in a week and are sharing it early, this package API happens to have landed on what Marvin is doing, we're likely to change over time, especially leaning toward running our own models as part of it.

Wait. So why not just contribute to an existing open source project if you’re going to implement an identical API?

If you run your own models as a part of it, surely you could hook up your models as a backend to whatever abstractions you’re copying here.

Re: Show HN: Fructose – LLM calls as strongly typed functions

#72

Post by amiantos »

nextos wrote: post #21

IMHO, in the future programming may look similar to this. Write a type declaration for a function with an expressive type system, e.g. refinement types. Then use LLMs + SAT/SMT to generate provably correct code. This strikes a happy medium, where machines are assisting programmers, making them much more productive. Yet the resulting code is understandable as a human has decomposed everything into functions, and also…

> Write a type declaration for a function with an expressive type system, e.g. refinement types. Then use LLMs + SAT/SMT to generate provably correct code.

This is how I use copilot currently, so I might not be following on what part of this is 'future' facing or relevant to this Fructose project?

Not being contrarian, I thought this was an interesting point but as I thought about it more I realized, "wait, they're describing what I already do".

Re: Show HN: Fructose – LLM calls as strongly typed functions

#73

Post by jxnlco »

edunteman wrote: post #23
hedgehog wrote: post #15

How does Fructose relate or compare to Instructor ( https://github.com/jxnl/instructor )?

Currently, quite comparable and obviously Instructor is more mature and feature rich. They're going the "patch the openAI client" approach which makes code written still use openAI SDK patterns which is pretty smart. Jason seems like he knows what he's doing. We're trying to make it more of a language feature with the decorated functions. Plus exploring the hosted formatting model direction. How do you feel this comp…

i do not know what i am doing.

I just want `create(response_model=T) -> T` lol

Re: Show HN: Fructose – LLM calls as strongly typed functions

#74

Post by dragonwriter »

babyshake wrote: post #59

Does anyone else get bothered by how this seemingly results in code that won't compile? Instead of this: @ai() def describe(animals: list[str]) -> str: """ Given a list of animals, use one word that'd describe them all. """ it would seem a lot more intuitive to do this: def describe(animals: list[str]) -> str: return ai("""Given a list of animals, use one word that'd describe them all.""", animals)

Wouldn't the correct way be with the use of ellipsis, as is used in type stubs?

  @ai()
  def describe(animals: list[str]) -> str:
    """ Given a list of animals, use one word that'd describe them all. """

    ...

Re: Show HN: Fructose – LLM calls as strongly typed functions

#75

Post by dragonwriter »

edunteman wrote: post #60
babyshake wrote: post #59

Does anyone else get bothered by how this seemingly results in code that won't compile? Instead of this: @ai() def describe(animals: list[str]) -> str: """ Given a list of animals, use one word that'd describe them all. """ it would seem a lot more intuitive to do this: def describe(animals: list[str]) -> str: return ai("""Given a list of animals, use one word that'd describe them all.""", animals)

Yeah the pyright doesn't like the annotated return type not being honored by the empty stub function. I wonder if there's a way to trick it. For your suggestion, the decorator would still be required to overload the function execution with the remote call, otherwise you'd just be calling the function body, but we have considered special wrapper return types to help play better with pyright (and also give programmatic…

> Yeah the pyright doesn't like the annotated return type not being honored by the empty stub function. I wonder if there's a way to trick it.

Python has an existing convention for this (so its not a "trick"), the use of the special value Ellipsis (literal: ...)

https://mypy.readthedocs.io/en/stable/stubs.html

Re: Show HN: Fructose – LLM calls as strongly typed functions

#77

Post by ilaksh »

vineetch wrote: post #66
ilaksh wrote: post #65

I find it grating that all of these types of things say "LLMs" when in fact they literally only work with OpenAI. There are hundreds of variations of LLM models. When it works with only gpt-4-turbo or gpt-3.5-turbo, it's inaccurate to say it's a tool for LLMs in general.

So you're saying they should ensure compatibility with all LLMs on Day 0 so you can avoid a personal "grating" feeling. It's called an MVP.

They should just say it works with OpenAI or ChatGPT. It's called being honest.

Re: Show HN: Fructose – LLM calls as strongly typed functions

#78

Post by barapa »

lefttoreader wrote: post #71
edunteman wrote: post #52

Earlier quoted context omitted.

Thanks for asking, and I'd agree. I'd give the same answer as the folks asking about instructor: we built this in a week and are sharing it early, this package API happens to have landed on what Marvin is doing, we're likely to change over time, especially leaning toward running our own models as part of it.

Wait. So why not just contribute to an existing open source project if you’re going to implement an identical API? If you run your own models as a part of it, surely you could hook up your models as a backend to whatever abstractions you’re copying here.

Wait, someone made a similar comment as this elsewhere in the thread. So why don't you just upvote that?

If you have your own thoughts, surely you could just think them to yourself while upvoting.

Re: Show HN: Fructose – LLM calls as strongly typed functions

#79

Post by AgentOrange1234 »

amiantos wrote: post #72
nextos wrote: post #21

IMHO, in the future programming may look similar to this. Write a type declaration for a function with an expressive type system, e.g. refinement types. Then use LLMs + SAT/SMT to generate provably correct code. This strikes a happy medium, where machines are assisting programmers, making them much more productive. Yet the resulting code is understandable as a human has decomposed everything into functions, and also…

> Write a type declaration for a function with an expressive type system, e.g. refinement types. Then use LLMs + SAT/SMT to generate provably correct code. This is how I use copilot currently, so I might not be following on what part of this is 'future' facing or relevant to this Fructose project? Not being contrarian, I thought this was an interesting point but as I thought about it more I realized, "wait, they're d…

How do you do this? I know nothing but this sounds really interesting.

Re: Show HN: Fructose – LLM calls as strongly typed functions

#80

Post by icyfox »

Big proponent of guaranteed outputs for LLMs. I wrote a library awhile back (gpt-json) that did something similar by querying the OpenAI API. At the end of the day though while their responses are _highly likely_ to be valid JSON they're not guaranteed. There's only so much that can be done with remote calls to their model's black box.

The future here really lies in compiling down context free grammars. They let you model json, yml, csv, and other programming languages as finite state machines that can force LLM transitions. They end up being pretty magical: you can force value typing, enums, and syntax validation of multivariate payloads. For use in data pipelines they can't be beat.

I did some experiments a few weeks ago on training models to generate these formats explicitly with jsonformers/outlines. Finetuning in the right format is still important to maximize output. You can end up seeing a 7% lift if you finetune explicitly for your desired format. [^1] At inference time the CFGs will constrain your model to do what it's actually intended to.

[^1]: https://freeman.vc/notes/constraining-llm-outputs

Post reply on HN