Notes on OpenAI o3-mini

Agreed that Sonnet still feels like the best all-round model. The new ones are at least on par with it for pure coding, or exceed it (r1, o1 both do IME) but don't generalize as well, especially to tasks with subjective answers. I find the latest Gemini 2.0-Flash-thinking to be closest to Sonnet on those.

Not that you'd want to have to do more steps, but how do they do if you split it up in separate parts and translate them individually, then feed it back in interleaved in parts/translated parts and ask it to keep the style but fix any errors due to it originally not having full context?

Or another approach, feed it all into context but tell it to wait and not translate, and then feed it in an additionalt time part by part asking it to translate each part and with the translation style instructions repeated.

That might work to prevent the glitches I noticed toward the end of the long texts.

In general, though, I haven’t seen any sign yet that reasoning models are better at translation than nonreasoning ones even with shorter texts. LLMs in general have reached the level where it is difficult to assess the quality of their translations with A/B comparisons or on a single linear scale, as most of the variation is within the range where reasonable people disagree about translation quality.

Im curious if you know of any tools, strategies or papers about this topic.

I’ve experienced the same thing with long-form translation. I would expect that chunking the translation (page or paragraph at a time) would fix the missing paragraph problem but potentially the loss of context would cause other problems.

This is such a common use case — would love to find resources.

There might be some papers or other guides out there, but their advice will be based on whatever tools happened to be available at the time they were written and on the particular types of translations the authors cared about. The technology is advancing so rapidly that you might be better off just experimenting with various LLMs and prompts for texts and language pairs you are interested in.

I started using LLMs for translation after GPT-4 came out in March 2023—not that long ago! At first, the biggest problem was the context window: it wasn’t possible to translate more than a couple of pages at a time. Also, prompt writing was in its infancy, and a lot of techniques that have since emerged were not yet widely known. Even now, I still do a lot of trial and error with my prompts, and I cannot say with confidence that my current prompting methods are the best.

But, for what it’s worth, here are some strategies I currently use when translating with LLMs:

- In the prompt, I explain where the source text came from, how the translation will be used, and how I want it to be translated. Below is a (fictional) example, prepared through some metaprompting experiments with Claude:

https://www.gally.net/temp/20250201sampletranslationprompt.h...

- I run the prompt and source text through several LLMs and glance at the results. If they are generally in the style I want, I start compiling my own translation based on them, choosing the sentences and paragraphs I like most from each. As I go along, I also make my own adjustments to the translation as I see fit.

- After I have finished compiling my draft based on the LLM versions, I check it paragraph by paragraph against the original Japanese (since I can read Japanese) to make sure that nothing is missing or mistranslated. I also continue polishing the English.

- When I am unable to think of a good English version for a particular sentence, I give the Japanese and English versions of the paragraph it is contained in to an LLM (usually, these days, Claude) and ask for ten suggestions for translations of the problematic sentence. Usually one or two of the suggestions work fine; if not, I ask for ten more. (Using an LLM as a sentence-level thesaurus on steroids is particularly wonderful.)

- I give the full original Japanese text and my polished version to one of the LLMs and ask it to compare them sentence by sentence and suggest corrections and improvements to the translation. (I have a separate prompt for this step.) I don’t adopt most of the LLM’s suggestions, but there are usually some that I agree would make the translation better. I update the translation accordingly. I then repeat this step with the updated translation and another LLM, starting a new chat each time. Often I cycle through ChatGPT --> Claude --> Gemini several times before I stop getting suggestions that I feel are worth adopting.

- I then put my final translation through a TTS engine—usually OpenAI’s—and listen to it read aloud. I often catch minor awkwardnesses that I would overlook if reading silently.

This particular workflow works for me because I am using LLMs to translate in the same language direction I did manually for many years. If I had to translate to or from a language I don’t know, I would add extra steps to have LLMs check and double-check the accuracy of the translation and the naturalness of the output.

I was asked recently by some academics I work with about how to use LLMs to translate documents related to their research into Japanese, a language they don’t know. It’s an interesting problem, and I am planning to spend some time thinking about it soon.

Please note that my translation process above is focused on quality, not on speed. If I needed to translate a large volume of text more quickly, I would write a program to do the translation, checking, and rechecking through API calls, accepting the fact that I would not be able to check and polish the translation manually as I do now.

If anyone here would like to brainstorm together about how to use LLMs for translation, please feel free to email me. My website, with my email address on the Contact page, is linked from my HN profile page.