Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Fructose – LLM calls as strongly typed functions

github.com

Post reply on HN
Print view View original

21–30 of 105 posts

Re: Show HN: Fructose – LLM calls as strongly typed functions

#21

Post by nextos »

IMHO, in the future programming may look similar to this. Write a type declaration for a function with an expressive type system, e.g. refinement types. Then use LLMs + SAT/SMT to generate provably correct code.

This strikes a happy medium, where machines are assisting programmers, making them much more productive. Yet the resulting code is understandable as a human has decomposed everything into functions, and also robust as it is formally verified.

I am working on a F# proof-of-concept system like this, there are other alternatives around implemented in Haskell and other languages with varying levels of automation. It is potentially an interesting niche for a startup.

Re: Show HN: Fructose – LLM calls as strongly typed functions

#22

Post by ramses0 »

edunteman wrote: post #6
_ea1k wrote: post #2

I love the concept, but I'd really prefer being able to use it against local llms (localai, ollama, etc).

Seems like a great feature (and honestly allows us to do smarter things for strictly structured generation). I'm curious, what's your main motivation for local llms vs hosted APIs?

Not wanting to get dinged for $20/mo? Ability to use offline for local home automation (eg: "given a verbal input request, determine the home devices in scope and their on/off state" => "given the current devices state and the verbal request, generate a list of home assistant actions to perform"), using a custom model for the above, etc.

Re: Show HN: Fructose – LLM calls as strongly typed functions

#23

Post by edunteman »

hedgehog wrote: post #15

How does Fructose relate or compare to Instructor ( https://github.com/jxnl/instructor )?

Currently, quite comparable and obviously Instructor is more mature and feature rich. They're going the "patch the openAI client" approach which makes code written still use openAI SDK patterns which is pretty smart. Jason seems like he knows what he's doing.

We're trying to make it more of a language feature with the decorated functions. Plus exploring the hosted formatting model direction.

How do you feel this compares? Do you think there's any gaps in current tools worth working on?

Re: Show HN: Fructose – LLM calls as strongly typed functions

#24

Post by Terretta »

iAkashPaul wrote: post #11

TGI just integrated Guidance in 1.4.3, that by itself can support both grammar/JSON/Pydantic & tool invocation/function calling. Langchain & Llamaindex plus Fructose really need to skip the structure adherence work & move to chunking/KG generation since that's the next pain point to tackle.

Why do you have Guidance in caps?

https://github.com/guidance-ai/guidance

or ...

https://huggingface.co/docs/text-generation-inference/concep...

or ... ?

A quick glance through these, they don't seem yet to leverage json_object on OpenAI with the word JSON in the prompt, which works wonders with the 0125 models.

Re: Show HN: Fructose – LLM calls as strongly typed functions

#25

Post by edunteman »

memothon wrote: post #17
iAkashPaul wrote: post #11

TGI just integrated Guidance in 1.4.3, that by itself can support both grammar/JSON/Pydantic & tool invocation/function calling. Langchain & Llamaindex plus Fructose really need to skip the structure adherence work & move to chunking/KG generation since that's the next pain point to tackle.

What is TGI? Is that the huggingface Text Generation Inference project on GitHub?

Yes. TGI is Huggingface's version of LLVM (some nuance, of course). LLVM also launched grammar support recently too, so we'll be looking into it.

Re: Show HN: Fructose – LLM calls as strongly typed functions

#27

Post by edunteman »

politelemon wrote: post #13

Are you planning to add other types like Claude or Llama2?

eventually, but priority goes toward finding an abstraction that feels right. We're very likely to break this package API, still v0. Sticking with openai till we have more confidence in the foundation being correct.

Re: Show HN: Fructose – LLM calls as strongly typed functions

#29

Post by edunteman »

nextos wrote: post #21

IMHO, in the future programming may look similar to this. Write a type declaration for a function with an expressive type system, e.g. refinement types. Then use LLMs + SAT/SMT to generate provably correct code. This strikes a happy medium, where machines are assisting programmers, making them much more productive. Yet the resulting code is understandable as a human has decomposed everything into functions, and also…

yeah I had a moment working with fructose where I realized "oh this is more like functional programming than I expected"

Re: Show HN: Fructose – LLM calls as strongly typed functions

#30

Post by memothon »

edunteman wrote: post #25
memothon wrote: post #17

Earlier quoted context omitted.

What is TGI? Is that the huggingface Text Generation Inference project on GitHub?

Yes. TGI is Huggingface's version of LLVM (some nuance, of course). LLVM also launched grammar support recently too, so we'll be looking into it.

All of these acronyms are so confusing. I'm assuming LLVM isn't the compiler tool, but searching "LLVM ai" doesn't give me any good results.
Post reply on HN