Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment

#21

Post by add-sub-mul-div »

Lots of grassroots interest in this from a flood of new accounts created in the last few hours.

========

baileyw6 2 hours ago [flagged] [dead] | prev | next [–] excellent work!

r0sh 3 hours ago [flagged] [dead] | prev | next [–] cracked team!

mlw14 3 hours ago [flagged] [dead] | prev | next [–] Interesting library, is it like unit testing for RAGs? Can't wait to try it out!

lncheine 2 hours ago [flagged] [dead] | prev | next [–] Interesting library, can't wait to try it out!

Linda_ll 2 hours ago [flagged] [dead] | prev | next [–] Congrats on the launch! Excited for what’s to come :)

bmountain17 3 hours ago [flagged] [dead] | prev | next [–] Great new platform to boast AI performance, can't wait to try the Python library!

jjBailey 1 hour ago [flagged] [dead] | prev | next [–] Cool library, I’ll test it out

sidkapoor39 3 hours ago [flagged] [dead] | prev | next [–] Congrats on the launch! Excited to see how this streamlines Hyperparameter optimization. Keep up the great work!

brucetry 1 hour ago [flagged] [dead] | prev | next [–] Ver interesting, similar to unit test for RAGs? Love to try it out

jjBailey 1 hour ago [flagged] [dead] | prev | next [–] Very interesting library!! Can’t wait to try it!

luxxxxx 1 hour ago [flagged] [dead] | prev | next [–] Interesting library! Is it like unit testing for RAGs? Can’t wait to try it out!

kangjl888 2 hours ago [flagged] [dead] | prev | next [–] Huge congratulations to the NomadicML team on the launch of Nomadic! The platform looks like a game-changer for optimizing AI systems, excited to see how it transforms hyperparameter search for the community.

nishsinha2345 21 minutes ago [flagged] [dead] | prev | next [–] Excited to try out this library! would this help make unit testing easier? Or be used instead of unit testing?

greysongy5 19 minutes ago [flagged] [dead] | prev | next [–] Wow, this seems like it would really help automated RAG testing. What are the top use cases today?

sidvijay10 5 minutes ago [flagged] [dead] | prev [–] We're looking for a RAG testing framework for searching UGC. So far we've just been running evals manually w/o a library. Will try out Nomadic and see if it's more convenient.

Re: Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment

#22

Post by mustafabal »

altairmn wrote: post #17

Our customers use our platform to build low-latency voice and video pipelines. They utilize RAG in voice bots to improve response accuracy. Is it possible to programmatically interface with Nomadic’s hyperparameter search through an authenticated endpoint, with the ability to generate user-specific tokens for secure access?"

Certainly!

The Nomadic SDK supports 1st-party integrations with various open & closed-source ML/LLM providers. These are done through authenticated endpoints for interfacing securely with your models. Also, as noted in the Custom Evaluation section of our docs (https://docs.nomadicml.com/features/evaluations), you can provide your custom objective_functions and detail your model access logic, which may include custom authentication & access rules. A sample of this is present in our "Basic RAG" cookbook (link: https://colab.research.google.com/drive/1rv2f-qxgoN_eVDFu6Um...).

When integrated with the upcoming Nomadic Workspace, you can obtain your Nomadic API key and sync your local Nomadic models, experiments & experiment results with our managed service. The demo of this model/experiment/experiment result visualizatio is live at https:demo.nomadicml.com, please check it out and let us know your thoughts!

Re: Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment

#23

Post by _eric_z_lin »

This looks like a really useful tool for keeping AI systems optimized, especially as models and data evolve over time. I'm curious, have you considered how Nomadic might integrate into CI/CD pipelines? It seems like it could be valuable for automatically re-tuning parameters and ensuring performance doesn't degrade with new model versions or data updates. Any plans for features that would support this kind of continuous optimization workflow?

Re: Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment

#25

Post by Jadiker »

It looks like the hallucination score is somewhat related to perplexity in the sense that it relies on specific tokens. This could cause issues because rephrasing or using slightly different terms could lead to a higher hallucination score. E.g. if the correct answer is "John Smith is the world's best baker" then "Mary Kay is the world's best baker" would have a better score (lower hallucination) than "Leading maker of baked items across all the continents: John Smith" according to your metric.

Are there any plans to make updates to this score or add in different metrics for more accurately detecting hallucinations that don't penalize rephrasing?

Re: Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment

#26

Post by varunkrishnan17 »

Jadiker wrote: post #25

It looks like the hallucination score is somewhat related to perplexity in the sense that it relies on specific tokens. This could cause issues because rephrasing or using slightly different terms could lead to a higher hallucination score. E.g. if the correct answer is "John Smith is the world's best baker" then "Mary Kay is the world's best baker" would have a better score (lower hallucination) than "Leading maker…

Thanks for the well-thought out question Jadiker!

This is a potential limitation of N-gram precision with context matching, which we were using in the RAG demo for simplicity (though even with this, I don't think it would be so extreme :-) )

We already offer two other different hallucination detection approaches which should mitigate this problem - an LLM-as-a-judge model for evaluation, and semantic similarity matching. We've also considered, for example, using metrics such as BertScore. Do you have other ideas? :-)

Re: Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment

#27

Post by varunkrishnan17 »

Jadiker wrote: post #24

Looks interesting! How does it compare to things like Optuna or RayTune or Weights and Biases?

Really appreciate it Jadiker! We're obviously in a similar space, but we think we offer some strong differentiators: (1) Functionality that is more specific to your LLM use cases (for example, being able to easily kick off a RAG Retrieval / Inferencing Experiment). (2) Ability to easily customize and visualize your results - for example, through custom evaluators, and carefully curated heatmaps - both through our SDK and our managed service.

Re: Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment

#28

Post by simbasdad »

_eric_z_lin wrote: post #23

This looks like a really useful tool for keeping AI systems optimized, especially as models and data evolve over time. I'm curious, have you considered how Nomadic might integrate into CI/CD pipelines? It seems like it could be valuable for automatically re-tuning parameters and ensuring performance doesn't degrade with new model versions or data updates. Any plans for features that would support this kind of continu…

Thank you so much. Yes, we believe CI/CD pipelines are a treasure trove of data for continuous ML system optimization (these are non-deterministic systems run repeatedly with new evaluation results at each run), where you get to learn about your own ML systems. Nomadic integrates well here to continuously collect data, that it can then use to better identify optimal HP configs on the same systems. We envision this as: every time you run CI/CD pipeline, you get more data with which you can learn about your ML system better, and Nomadic is your engine to realize this.

Re: Show HN: Nomadic – Minimize RAG Hallucinations with 1 Hyperparameter Experiment

#29

Post by elizabethhu »

Hey HN! I'm Lizzie, one of the cofounders of NomadicML - excited to get your thoughts on our demo and repo.

We started working on Nomadic because we saw people wanted to ship out powerful and reliable systems but very often didn't have a map of it:

Which embedding model works best for my RAG? What temperature to set? What threshold for similarity search?

We wanted a tool to make the decision process of answering these types of questions systematic and affordable instead of resorting to intuition or something like a single expensive grid search, then set it and forget it... give us your most honest feedback!

Post reply on HN