Viewing profile — svcrunch

For over a year, I've run the Little Dorrit benchmark. It's a vision task that converts handwritten editor marks on a printed page into structured JSON output. I hadn't updated the…

The grandparent is definitely wrong on (3). Yes, coding is a killer product, I agree with you. On (2), I agree with you for local models. BUT , there are also the open source Chine…

I generally don't waste time with patents. I think most patents in deep learning can be overturned by prior art. My current approach to IP is trade secrets. If we publish, we are c…

Thanks for your interest. The rerankers are external, GoodMem is a unified API layer that calls out to various providers. There's no model running inside the database or the GoodMe…

Hi there, thanks for writing and sharing your experiences. I'm one of the builders of GoodMem ( https://goodmem.ai/ ), which is infra to simplify end-to-end RAG/agentic memory syst…

Here's a problem that no frontier model does well on (f1 https://dorrit.pairsys.ai/ > This benchmark evaluates the ability of multimodal language models to interpret handwritten ed…

This is really cool.

Various frontier LLMs were evaluated on their ability to interpret handwritten proofreading marks in printed literary text, using a small benchmark based on Charles Dickens's "Litt…

No. But to your point, note that in 2020 neuroscientists introduced the Tolman-Eichenbaum Machine (TEM) [1], a mathematical model of the hippocampus that bears a striking resemblan…

While in Google Research, I worked with two of the authors of the "Attention is All you Need" paper, including the gentleman who chose that title. As others have pointed out, self-…

Take a look at the BEIR benchmark, which has served as one of the main drivers for development of neural IR systems since its introduction in 2020. BM25 presents a challenging cros…

> but HNSW is the best 99% of the time for both performance and latency, and is implemented in almost every modern major vector store. In my experience, HNSW indexes are very expen…

I think your comment is accurate, but regarding your last point: "However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such exa…

While transformer-based AI is very powerful, and its potential uses in the business world nearly limitless, the issue of hallucination is holding back adoption. Here, Simon Hughes …

The metrics presented in the blog post are those of our production model. When designing Boomerang, we tried to balance latency and search relevance in a manner that strikes the ri…

I believe that retrieval-augmented generation is the right path to generative AI within organizations, at least for the next few years. Trying to directly fine-tune an LLM on your …

I'm curious, is HSNW the only option? Do you support IVF-style indexes? Also, FAISS is nice because it supports a pluggable storage layer. Is this something that's easily supported…

Thank you for this! I know that Cruikshank was the original illustrator of many of Dickens's novels, but I prefer the artwork of James Mahoney. As a point of comparison, the same s…

This guide reviews 13 prominent instruction-following LLMs given constraints like commercial or non-commercial usage, and self-hosted versus API access.