Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a Git-tracked book production pipeline

djspeckhals.com

Post reply on HN
Print view View original

81–90 of 95 posts

Re: I built a Git-tracked book production pipeline

#81

Post by thangalin »

I developed https://keenwrite.com for my hard sci-fi novel. I started with OpenOffice and a spreadsheet and then realized I could combine a character sheet with a Markdown editor. The character sheet became a YAML file with interpolated strings. The editor calls out to ConTeXt for typesetting to PDF. To create an audiobook, the same character sheet identifies the characters for gemma4:31b, which excels at quotation attributions when given a cast of characters and curated list of emotions. Next, I feed the chapters coupled with JSON-formatted attributions, pronunciation guides, and voice descriptions into qwen3 (VoiceDesign, Base, and 32b) to produce an audiobook with a full cast of characters.

Here's some output (to console, not JSON for brevity here) from gemma:

    Unknown (chanting): "Free the food, free the people."
    Unknown (chanting): "Border walls trap us all."
    Chloé Angelos (focused): "Let's see,"
    Yūna Futaba (serious): "The push draws ever nearer,"
    Chloé Angelos (commanding): "Yūna, buzz the CDC,"
    Unknown (formal): "CDC Emergency Operations Centre. What's your emergency?"
    Chloé Angelos (urgent): "Pandora's brew. Populated areas. Releasing soon. Loop in Beale Air Force Base."
I haven't listed all the minor characters, yet, which is why the LLM attributed "unknown" to some quotations.

I'm in the process of containerizing the solution. If interested, email me.

Re: I built a Git-tracked book production pipeline

#82

Post by raddan »

Exoristos wrote: post #24
raddan wrote: post #16

Earlier quoted context omitted.

> Overall, as a technical writeup I enjoyed the article; however, I would caution that the author seems to approach publishing from an amateur perspective. I also worked at a publishing company (for ~6 years) in the early 2000s. While you are right that the pros have some tricks to make the process easier, the fact remains that the process is not easy at all. Unlike in academic publishing, where nothing stands betwee…

Maybe we worked at the same firm. You never know.

I worked at Bedford/St. Martin, now Macmillan Education. It would be hilarious if you worked there (or at our nemesis one floor down in the same building).

Re: I built a Git-tracked book production pipeline

#83

Post by kcartlidge »

As we're sharing pipelines:

- I write in Markdown files using VS Code and a custom syntax highlighter

- A custom C# tool stitches it together (with YAML metadata)

- It generates reports on chapters, dialogue, character presence etc

- It uses custom code to generate an EPUB (v2)

- It uses custom code to generate a DOCX (validated)

- It uses custom code to generate a PDF (print or with interactive links)

Works great and gives perfect results in seconds (beats Vellum, which I quite like). The only drawback is the yak shaving involved in my totally stand-alone solution leads to more time tinkering than writing.

Re: I built a Git-tracked book production pipeline

#84

Post by raybb »

dustin1114 wrote: post #54
raybb wrote: post #38

I've been making ebooks for a nonprofit using typst and pandoc for a few years and it works quite well. We generate a pdf ebook, a print version, and a epub. They each have little tweeks but are all defined conditionally using sys.input. It was rough at first and I've had to open around a dozen or so issues for pandoc to improve things. Now it's pretty seamless.

I saw typst in my explorations but LaTeX had a few more of the controls I was looking for in print, and I really wanted a Standard Ebooks compliant EPUB. I might revisit at some time though. Thanks for bringing it up.

What kind of controls for print? I'm pretty amature all things considered so don't use any advanced features.

Also, I doubt if pandoc produces a highly compliant epub but it is always improving so who knows.

Re: I built a Git-tracked book production pipeline

#86

Post by BrenBarn »

somat wrote: post #63
BrenBarn wrote: post #12

Earlier quoted context omitted.

The annoyances of using "soft wraps" with various kinds of tools is one of the maddening irritations of our software landscape. Inserting non-semantic newlines in content just to make things fit the screen is insane.

It is not just to fit the screen, it also fits our line orientated version control better. I don't know if this is suitable for large works(books), but for technical documentation I have my plain text source with one line per sentence, actually I go further than than and usually have one line per punctuation. The raw source reads a little hard but the version control diffs are much cleaner and editing is is easier. M…

That's just another form of warping the file to fit the tools. The tools should be built to handle files that are structured in the way that is semantically most natural.

Re: I built a Git-tracked book production pipeline

#87

Post by dustin1114 »

raybb wrote: post #84
dustin1114 wrote: post #54

Earlier quoted context omitted.

I saw typst in my explorations but LaTeX had a few more of the controls I was looking for in print, and I really wanted a Standard Ebooks compliant EPUB. I might revisit at some time though. Thanks for bringing it up.

What kind of controls for print? I'm pretty amature all things considered so don't use any advanced features. Also, I doubt if pandoc produces a highly compliant epub but it is always improving so who knows.

Last I checked, typst doesn't have baseline grid support (i.e., assures vertical lines of text across spreads are aligned, thus text doesn't bleed through recto to verso).

Re: I built a Git-tracked book production pipeline

#88

Post by dustin1114 »

voidUpdate wrote: post #73
Rp8yXmdmr wrote: post #71

Earlier quoted context omitted.

Artifact in this context is whatever is produced by build process. That is common convention in CI/CD context. And the base definition for "artifact" is very wide: anything artificial, as in not natural but made by humans.

Aren't the xhtml and TeX already artifacts though? They are produced from a script that parses the ODT

There is a bit of manual tweaking required in LaTeX that I haven't figured out how to automate yet. It's mainly related to chapter endings: if there are only 2 or 3 lines on the last page, I can subtly tighten the tracking from a paragraph on the second to last page and eliminate an almost-empty page. Also, I have PNG maps that currently need some one-off LaTeX directives to lay them out how I want. I could probably embed some properties in the ODT image, but that seems kind of janky. If I needed to regenerate from the ODT source, I use a Git three-way merge to update it (also janky, but it's expedient).

Re: I built a Git-tracked book production pipeline

#89

Post by raybb »

dustin1114 wrote: post #87
raybb wrote: post #84

Earlier quoted context omitted.

What kind of controls for print? I'm pretty amature all things considered so don't use any advanced features. Also, I doubt if pandoc produces a highly compliant epub but it is always improving so who knows.

Last I checked, typst doesn't have baseline grid support (i.e., assures vertical lines of text across spreads are aligned, thus text doesn't bleed through recto to verso).

Seems you're right and there is an open issue https://github.com/typst/typst/issues/5225

Re: I built a Git-tracked book production pipeline

#90

Post by everybodyknows »

diamondap wrote: post #47
everybodyknows wrote: post #43

Earlier quoted context omitted.

> ... markdown to HTML to PDF/X-1a processor using Python, WeasyPrint, and ghostscript. I've been converting HTML to PDF by running WeasyPrint (latest version) with options I hoped were sufficient to satisfy the X-1a rules -- can it not quite do that? Is that why you need ghostscript?

I tried make my PDFs X-1a compliant with WeasyPrint, then ran them through Adobe's PDF/X validator and they kept failing. I was in a bit of a hurry and found a way to do it with ghostscript. I would like to remove ghostscript from the mix, so when I have some time, I may try again to do it all with WeasyPrint.

WeasyPrint as a project is very much alive on Github. Filing a bug there might get your problem solved.
Post reply on HN