141–150 of 349 posts
I initially built it using llama.cpp for offline LLM inference, but soon discovered mlc-llm and moved to using it, because the latter is way faster and flexible.
You just export and upload a WhatsApp conversation and it will learn the personality AND voice of your conversation partner. You can send/receive text or voice messages; It was pretty damn spooky to actually have a voice conversation back and forth with an AI standing in for my "friend"
****
Project 2 - I also built a YouTube summarizer for individual video called Summary Cat (https://www.summarycat.com). It is not open source for now. The stack is very similar to project 1.
****
And yes I like summarizing YouTube videos:)
I think what will be really powerful is to have a registry for plugins and agents that can be easily installed in the system. Sort of like WordPress in that way. Also similar to an open source GPT store.
https://github.com/runvnc/agenthost
I believe the are several variations of this type of idea out there.
Primarily it was a PoC to see if a document based chatbot could work without crossing trust boundaries by calling out to untrusted APIs. It only makes calls to localhost.
If you’re familiar with the novel you will be pleased to know that the chatbot ended a recent answer with, “I must go now as I have an appointment with my chamber pot and I wouldn’t want to keep it waiting.”
[1]https://github.com/FlowiseAI/Flowise
[5]https://www.gutenberg.org/ebooks/1079
Everything runs on a Mac Mini with the M2 Pro CPU/GPU and Mac OS Sonoma.
I'm building https://www.brief.news , an AI powered newsletter that condenses tens of thousands of news articles into a daily briefing of the top stories, we support 30 topics today and are adding the ability to add your own! Stack is a combination of TypeScript (Next / Node) + Python with a pretty simple deployment setup right now (GHA -> Container -> Cloud Run).
I have all of the docs with summaries on a small webserver here: https://ayylmao.info
Simple Flask site with SQLite as the database.
I hit some interesting challenges, overcoming which was a valuable set of lessons learnt:
1. GPT4 Turbo slowed down to molasses in some Azure regions recently. Microsoft is not admitting this and is telling people to use GPT3.5 instead. The lesson learned is that using a regional API exposes you to slowdowns and queuing caused by local spikes in demand, such as “back to school” or end of year exams.
2. JSON mode won’t robustly stick to higher level schemas. It’s close enough, but parsing and retries are required.
3. The 128K context in GPT4 is only for the input tokens! The output is limited to 4K.
4. Most Asian languages use as many as one token per character. Translating 1 KB of English can blow through the 4 KB token limit all too easily.
5. You can ask GPT to “continue”, but then you have to detect if you received a partial or a complete JSON response, and stitch things together yourself… and validate across message boundaries.
6. The whole process above is so slow that it hits timeouts all over the place. Microsoft didn’t bother to adjust any of their default Azure SDK timeouts for HTTP calls. You have to do this yourself. It’s easy, just figure which of the three different documented methods are still valid. (Answer: none are.)
7. You’ll need a persistent cache. Just trust me on this. I simply hashed the input and used that as a file name to store responses that passed the checks.
8. A subtitle file is about 30–100 KB so it needs many small blocks. This makes the AI lose the context. So it’s important to have several passes so it can double check and stitch things together. This is very hard with automatic parsing of outputs.
9. Last but not least: the default mode of Azure is to turn the content policy up to “puritan priest censoring books”. Movies contain swearing, violence, and sex. The delicate mind of the machine can’t handle this, and it will refuse to do as it is asked. You have to dial it down to get it to do anything. There is no “zero censorship” setting. Microsoft says that I can’t feed text to an API that I can watch on Netflix with graphic visuals.
10. The missus says that the AI-translated subtitles are “perfect”, which is a big step up from some fan translated subtitles that have many small errors. Success!
I wrote this as a C# PowerShell module because that makes it easy to integrate the utility as a part of a pipeline. E.g.: I can feed it a directory listing and it’ll translate all of the subtitles.
The performance issues meant I had to process 8x chunks in parallel. Conveniently I already had code lying around to do this in PowerShell with callbacks to the main thread to report progress, etc…