Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

github.com

Post reply on HN
Print view View original

111–120 of 142 posts

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#111

Post by khafra »

patcon wrote: post #41

I see that you're looking for clusters within PCA projections -- You should look for deeper structure with hot new dimensional reduction algorithms, like PaCMAP or LocalMAP! I've been working on a project related to a sensemaking tool called Pol.is [1], but reprojecting its wiki survey data with these new algorithms instead of PCA, and it's amazing what new insight it uncovers with these new algorithms! https://patco…

LLM model interpretability also uses Sparse Autoencoders to find concept representations (https://openai.com/index/extracting-concepts-from-gpt-4/), and, more recently, linear probes.

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#113

Post by poulpy123 »

cronopios wrote: post #106
philistine wrote: post #94

Earlier quoted context omitted.

With how undecipherable the manuscript is, my personal theory is that it's the work of a naive artist and that there's no language behind it. Just someone aping language without knowing the rules about language: https://en.wikipedia.org/wiki/Naïve_art It's not a mental issue, it's just a rare thing that happens. Voynich fits the whole bill for the work of a naive artist.

And that naïve artist somehow managed to create a work that follows Zipf's law, 4 centuries before it was discovered?

Why would it not ?

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#114

Post by poulpy123 »

PaulDavisThe1st wrote: post #90
bdbenton5255 wrote: post #84

Another great natural mystery that machine learning could tackle is earthquake prediction. Sure you could find some patterns modeling historical data.

What data would you feed it and why?

There are already works using machine learning on thebtopic

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#115

Post by DonaldFisk »

cronopios wrote: post #106
philistine wrote: post #94

Earlier quoted context omitted.

With how undecipherable the manuscript is, my personal theory is that it's the work of a naive artist and that there's no language behind it. Just someone aping language without knowing the rules about language: https://en.wikipedia.org/wiki/Naïve_art It's not a mental issue, it's just a rare thing that happens. Voynich fits the whole bill for the work of a naive artist.

And that naïve artist somehow managed to create a work that follows Zipf's law, 4 centuries before it was discovered?

Random Texts Exhibit Zipf’s-Law-Like Word Frequency Distribution: https://www.nslij-genetics.org/wp-content/uploads/2022/12/ie...

It also applies to a range of natural phenomena, e.g. lunar craters and earthquakes: https://www.cs.cornell.edu/courses/cs6241/2019sp/readings/Ne...

So the fact that word frequencies in the Voynich Manuscript follow Zipf's law doesn't prove it's written in a natural language.

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#116

Post by poulpy123 »

glimshe wrote: post #15

I strongly believe the manuscript is undecipherable in the sense thats it's all gibberish. I can't prove it, but at this point I think it's more likely than not to be hoax.

The book is obviously a hoax (either voluntary or not), the question is if the text is a cypher, a transliteration, a fake language, or just gibberish.

As far as I know it's just gibberish since it doesn't follow the statistics of the known languages or cyphers of the time.

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#117

Post by DonaldFisk »

This is very interesting. You should post a link to https://www.voynich.ninja/index.php

I'm not familiar with SBERT, or with modern statistical NLP in general, but SBERT works on sentences, and there are no obvious sentence delimiters in the Voynich Manuscript (only word and paragraph delimiters). One concern I have is "Strips common suffixes from Voynich words". Words in the Voynich Manuscript appear to be prefix + suffix, so as prefixes are quite short, you've lost roughly half the information before commencing your analysis.

You might want to verify that your method works for meaningful text in a natural language, and also for meaningless gibberish (encrypted text is somewhere in between, with simpler encryption methods closer to natural language and more complex ones to meaningless gibberish). Gordon Rugg, Torsten Timm, and myself have produced text which closely resembles the Voynich Manuscript by different methods. Mine is here: https://fmjlang.co.uk/voynich/generated-voynich-manuscript.h... and the equivalent EVA is here: https://fmjlang.co.uk/voynich/generated-voynich-manuscript.t...

Re: Show HN: I modeled the Voynich Manuscript with SBERT to test for structure

#120

Post by tough »

quantadev wrote: post #83
edoceo wrote: post #81

Earlier quoted context omitted.

Is this grey cause it talks about religion? That stuff was bigger in 1500 than 2000, from that lense as religious text seems a reasonable track to follow.

Other than war plans, religious text was pretty much the only thing in the 1500s that would have been encrypted. However war plans would be very unlikely to be disguised as a botany book, for all kinds of reasons. War plans are temporary, not something you'd dedicate that level of artistic effort and permanence to.

The art of war by Sun Tzu is pretty timeless tho
Post reply on HN