Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

github.com

Post reply on HN
Print view View original

11–20 of 142 posts

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#11

Post by jszymborski »

tetris11 wrote: post #7

UMAP or TSNE would be nice, even if PCA already shows nice separation. Reference mapping each cluster to all the others would be a nice way to indicate that there's no variability left in your analysis

When I get nice separation with PCA, I personally tend to eschew UMAP, since the relative distance of all the points to one another is easier to interpret. I avoid t-SNE at all costs, because distance in those plots are pretty much meaningless.

(Before I get yelled out, this isn't prescriptive, it's a personal preference.)

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#12

Post by gchamonlive »

brig90 wrote: post #4
nine_k wrote: post #2

In short, the manuscript looks like a genuine text, not like a random bunch of characters pretending to be a text. Key Findings * Cluster 8 exhibits high frequency, low diversity, and frequent line-starts — likely a function word group * Cluster 3 has high diversity and flexible positioning — likely a root content class * Transition matrix shows strong internal structure, far from random * Cluster usage and POS patte…

Yep, that was my takeaway too — the structure feels too consistent to be random, and it echoes known linguistic patterns.

I'd be surprised if it was indeed random, but the consistency is really surprising. I say this because I imagine that anyone that would be able to produce such text is a master scribe that put countless hours writing other works, so he's supposed to be very familiar with such structure, therefore even if he was going for randomness, I would doubt he would achieve it.

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#13

Post by us-merul »

I’ve found this to be one of the most interesting hypotheses: http://voynichproject.org/

The author made an assumption that Voynichese is a Germanic language, and it looks like he was able to make some progress with it.

I’ve also come across accounts that it might be an Uralic or Finno-Ugric language. I think your approach is great, and I wonder if tweaking it for specific language families could go even further.

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#14

Post by [deleted] »

brig90 wrote: post #8
Avicebron wrote: post #6

Maybe I missed it in the README but how did you do the initial encoding for the "words"? so for example, if you have ""okeeodair" as a word, where do you map that back to original symbols?

Yep, that’s exactly right — the words like "okeeodair" come directly from the EVA transliteration files, which map the original Voynich glyphs to ASCII approximations. So I’m not working with the glyphs themselves, but rather the standardized transliterated words based on the EVA (European Voynich Alphabet) system. The transliterations I used can be found here: https://www.voynich.nu/ I didn’t re-map anything back to…

[deleted]

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#16

Post by minimaxir »

A point of note is that the text embeddings model used here is paraphrase-multilingual-MiniLM-L12-v2 (https://huggingface.co/sentence-transformers/paraphrase-mult...), which is about 4 years old. In the NLP world, that's effectively ancient, particularly as the robustness of even small embeddings models due to global LLM improvements has increased dramatically both in information representation and distinctiveness in the embedding space. Even modern text embedding models not explicitly trained for multilingual support still do extremely well on that type of data, so they may work better for the Voynich Manuscript which is a relatively unknown language.

The traditional NLP techniques of stripping suffices and POS identification may actually harm embedding quality than improvement, since that removes relevant contextual data from the global embedding.

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#17

Post by brig90 »

minimaxir wrote: post #16

A point of note is that the text embeddings model used here is paraphrase-multilingual-MiniLM-L12-v2 ( https://huggingface.co/sentence-transformers/paraphrase-mult... ), which is about 4 years old. In the NLP world, that's effectively ancient, particularly as the robustness of even small embeddings models due to global LLM improvements has increased dramatically both in information representation and distinctiveness…

Totally fair — I defaulted to paraphrase-multilingual-MiniLM-L12-v2 mostly for speed and wide compatibility, but you’re right that it’s long in the tooth by today’s standards. I’d be really curious to see how something like all-mpnet-base-v2 or even text-embedding-ada-002 would behave, especially if we keep the suffixes in and lean into full contextual embeddings rather than reducing to root forms.

Appreciate you calling that out — that’s a great push toward iteration.

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#19

Post by lolinder »

glimshe wrote: post #15

I strongly believe the manuscript is undecipherable in the sense thats it's all gibberish. I can't prove it, but at this point I think it's more likely than not to be hoax.

Statistical analyses such as this one consistently find patterns that are consistent with a proper language and would be unlikely to have emerged from someone who was just putting gibberish on the page. To get the kinds of patterns these turn up someone would have had to go a large part of the way towards building a full constructed language, which is interesting in its own right.

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#20

Post by [deleted] »

lolinder wrote: post #19
glimshe wrote: post #15

I strongly believe the manuscript is undecipherable in the sense thats it's all gibberish. I can't prove it, but at this point I think it's more likely than not to be hoax.

Statistical analyses such as this one consistently find patterns that are consistent with a proper language and would be unlikely to have emerged from someone who was just putting gibberish on the page. To get the kinds of patterns these turn up someone would have had to go a large part of the way towards building a full constructed language, which is interesting in its own right.

[deleted]
Post reply on HN