[deleted]
Show HN: I modeled the Voynich Manuscript with SBERT to test for structure
51–60 of 142 posts
Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure
#52TIL about the Voynich manuscript. Fascinating. Thank you.
Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure
#53The link to the write-up seems broken, can you write the correct one?
Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure
#54I see that you're looking for clusters within PCA projections -- You should look for deeper structure with hot new dimensional reduction algorithms, like PaCMAP or LocalMAP! I've been working on a project related to a sensemaking tool called Pol.is [1], but reprojecting its wiki survey data with these new algorithms instead of PCA, and it's amazing what new insight it uncovers with these new algorithms! https://patco…
I’ve had much better luck with umap than PCA and t-sne for reducing embeddings.
Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure
#55The link to the write-up seems broken, can you write the correct one?
Apologies but its not letting me edit post any longer (I'm new to HN), here's the link though: https://brig90.substack.com/p/modeling-the-voynich-manuscrip...
Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure
#56I strongly believe the manuscript is undecipherable in the sense thats it's all gibberish. I can't prove it, but at this point I think it's more likely than not to be hoax.
Statistical analyses such as this one consistently find patterns that are consistent with a proper language and would be unlikely to have emerged from someone who was just putting gibberish on the page. To get the kinds of patterns these turn up someone would have had to go a large part of the way towards building a full constructed language, which is interesting in its own right.
Even before we consider the cipher, there's a huge difference between a constructed language and a stochastic process to generate language-like text.
Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure
#57Earlier quoted context omitted.
> would be unlikely to have emerged from someone who was just putting gibberish on the page People often assert this, but I'm unsure of any evidence. If I wrote a manuscript in a pretend language, I would expect it to end up with language-like patterns, some automatically and some intentionally. Humans aren't random number generators, and they aren't stupid. Therefore, the implicit claim that a human could not create…
Creating gibberish with the statistical properties of a natural language is a very hard task if you do this hundreds of years before the discovery of said statistical properties.
I'm not sure where this claim keeps coming from. Voynichese doesn't exhibit the statistical qualities of any known natural language. In a very limited sense, yes, but on balance, no. There is too much repetition for that.
Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure
#58How expensive is a "brute force" approach to decode it? I mean, how about mapping each unknown word by a known word in a known language and improve this mapping until a 'high score' is reached?
Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure
#59what I'd expect from a handwritten book like that, if it is just a gibberish, and not a cypher of any sorts - the style, calligraphy, the words used, even letters themselves should evolve from page 1 to the last page. Pages could be reordered of course, but it still should be noticeable.
Unless author hadn't written tens of books exactly like that before, which didn't survive, of course.
I don't think it's a very novel idea, but I wonder if there's analysis for pattern like that. I haven't seen mentions of page to page consistency anywhere.
Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure
#60How expensive is a "brute force" approach to decode it? I mean, how about mapping each unknown word by a known word in a known language and improve this mapping until a 'high score' is reached?
That’s a really interesting question — and one I’ve been circling in the back of my head, honestly. I’m not a cryptographer, so I can’t speak to how feasible a brute-force approach is at scale, but the idea of mapping each Voynich “word” to a real word in another language and optimizing for coherence definitely lines up with some of the more experimental approaches people have tried.
The challenge (as I understand it) is that the vocabulary size is pretty massive — thousands of unique words — and the structure might not be 1:1 with how real language maps. Like, is a “word” in Voynich really a word? Or is it a chunk, or a stem with affixes, or something else entirely? That makes brute-forcing a direct mapping tricky.
That said… using cluster IDs instead of individual word (tokens) and scoring the outputs with something like a language model seems like a pretty compelling idea. I hadn’t thought of doing it that way. Definitely some room there for optimization or even evolutionary techniques. If nothing else, it could tell us something about how “language-like” the structure really is.
Might be worth exploring — thanks for tossing that out, hopefully someone with more awareness or knowledge in the space see's it!