Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an email-to-calendar LLM

ngacho.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: Building an email-to-calendar LLM

#11

Post by htrp »

simonw wrote: post #9
politelemon wrote: post #7

I'm looking at the AppScript site. Is this like PowerAutomate, but for G Suite products? Also I don't know why, looking at the site makes me think it's a candidate that Google is going to kill off without warning: https://www.google.com/script/start/

Apps Script is positively ancient at this point, first released in 2009: https://en.wikipedia.org/wiki/Google_Apps_Script It's one of the Google products I worry least about, mainly because there are 15+ years of existing Google Sheets documents that people have built using it at this point. I don't think even Google would lightly break THAT many of their existing (often paid) users.

> I don't think even Google would lightly break THAT many of their existing (often paid) users.

Says everyone using GCP services that get deprecated.

Re: Building an email-to-calendar LLM

#12

Post by IanCal »

denton-scratch wrote: post #5

Just use CALDAV; it's designed for making calendar entries automatically via email. I'm not hip with the fashion for putting an LLM into everything. I think it's lazy.

That's a great solution to a different problem. Unless caldav has a process for extracting dates and actions from unstructured emails? But that doesn't seem related to caldav.

Re: Building an email-to-calendar LLM

#13

Post by robertclaus »

"Setting up LLMs to output structured data is incredibly hard." resonated strongly with my experience working in similar one-off projects. I've almost always implemented some level of fuzzy-matching to validate and convert the LLM output back into my expected structured format.

I've also noticed that the LLMs are much better at writing code than structured JSON (no real surprise given the popularity of code assistants). If it makes sense in the specific situation, I now have the LLM generate code and parse it into the right structure rather than requesting structured data directly:

`generate_event("I need to do X", new Date("1-1-2025"))` seems to be more reliable to generate than `{ "description": "I need to do X", "when": "1-1-2025" }`

Re: Building an email-to-calendar LLM

#14

Post by refulgentis »

robertclaus wrote: post #13

"Setting up LLMs to output structured data is incredibly hard." resonated strongly with my experience working in similar one-off projects. I've almost always implemented some level of fuzzy-matching to validate and convert the LLM output back into my expected structured format. I've also noticed that the LLMs are much better at writing code than structured JSON (no real surprise given the popularity of code assistant…

If you're doing it locally, it's likely got llama.cpp underneath it somewhere. Ask the dev to allow specifying a JSON schema via using its grammar feature.

Re: Building an email-to-calendar LLM

#15

Post by refulgentis »

darby_eight wrote: post #6
denton-scratch wrote: post #5

Just use CALDAV; it's designed for making calendar entries automatically via email. I'm not hip with the fashion for putting an LLM into everything. I think it's lazy.

IDK about "lazy", but it's certainly an extremely expensive solution.

A 3B model runs on Android phones from 2 years ago at 6 tkns/s.

Re: Building an email-to-calendar LLM

#16

Post by el_nahual »

robertclaus wrote: post #13

"Setting up LLMs to output structured data is incredibly hard." resonated strongly with my experience working in similar one-off projects. I've almost always implemented some level of fuzzy-matching to validate and convert the LLM output back into my expected structured format. I've also noticed that the LLMs are much better at writing code than structured JSON (no real surprise given the popularity of code assistant…

As long as you can sanitize the LLM output somehow. You should never `eval` LLM code straight from the tap!

Re: Building an email-to-calendar LLM

#17

Post by jakecodes »

Hey! It's awesome to read other people's solutions to this.

I've been working on solving this for the past 2 years or so and I went through much of the same struggles in the beginning until we came up with a solution which is fairly complex, to get LLM's to output data in a way we can use.

The big problem is that 95% accuracy is not good enough for calendars. People lose confidence after 1 failed attempt. Trying to get LLM's to output JSON can have a 1 in 1000 invalid JSON problem which is unrecoverable. What I wound up doing is training models for the tasks with tremendous amounts of data. I did not use OpenAI's models as they were not right for the job. Would love feedback.

convoke.ai

Re: Building an email-to-calendar LLM

#19

Post by ljlolel »

jakecodes wrote: post #17

Hey! It's awesome to read other people's solutions to this. I've been working on solving this for the past 2 years or so and I went through much of the same struggles in the beginning until we came up with a solution which is fairly complex, to get LLM's to output data in a way we can use. The big problem is that 95% accuracy is not good enough for calendars. People lose confidence after 1 failed attempt. Trying to g…

You can just force it to output into a specified BNF grammar this is quite easy

https://www.imaurer.com/llama-cpp-grammars/

Re: Building an email-to-calendar LLM

#20

Post by darby_eight »

refulgentis wrote: post #15
darby_eight wrote: post #6

Earlier quoted context omitted.

IDK about "lazy", but it's certainly an extremely expensive solution.

A 3B model runs on Android phones from 2 years ago at 6 tkns/s.

I'm not sure what you're comparing this to or how you're making this comparison—can you enlighten us?

(Somehow I doubt whatever caldav software the above poster references takes more than a second to process multiple emails.)

Post reply on HN