Let me nitpick a bit: these characters are not Russian, they are Cyrillic. There are some Cyrillic characters that are distinctly Russian (i.e. used only in the Russian language), but these characters can't impersonate Latin letters because they are too different from them.
Plagiarized news sites are using Cyrillic characters to avoid detection
11–20 of 33 posts
Re: Plagiarized news sites are using Cyrillic characters to avoid detection
#12In Sweden (and probably other places), a service called URKUND[0] ("deed" in Swedish) is used for automatic detection of plagiarism for school work. I have always wondered to what extent they identify stuff like this, and other potential trickery with UTF-8 or removing text layers from PDF files. 0: http://www.urkund.com/en/
Would it not be easier to just have a lookup table which maps unicode points to visually equivalent glyphs?
Re: Plagiarized news sites are using Cyrillic characters to avoid detection
#13I think at some point, a sort of visual-normalization that converts similar looking unicode to a single unique string sequence (ex: convert certain letters from Cyrillic and other language sets that are also present in Latin to just Latin) is just going to be necessary as a security precaution. Given the whole "fake news" thing over the past couple of years, I expect that the first step will be taken by one of Google…
Unicode normalization/equivalence is half of what you want and UCAPI is probably the other half.
https://en.wikipedia.org/wiki/Unicode_equivalence https://www.casaba.com/products/UCAPI/
Re: Plagiarized news sites are using Cyrillic characters to avoid detection
#14In Sweden (and probably other places), a service called URKUND[0] ("deed" in Swedish) is used for automatic detection of plagiarism for school work. I have always wondered to what extent they identify stuff like this, and other potential trickery with UTF-8 or removing text layers from PDF files. 0: http://www.urkund.com/en/
A neat trick would be to render the PDF into an image file, and then OCR that image file, and do the detection using the text file generated by the OCR process. Just like VW's defeat device, the cheater would then need to create software that outputs something different when they see they are being rendered not to display, but to an image file...
So, one needs to use a font that OCR finds hard to avoid detection. Would rubbish keming do the trick, you could abide by imposed don't requirements but change kerning/leading?
Re: Plagiarized news sites are using Cyrillic characters to avoid detection
#15Let me nitpick a bit: these characters are not Russian, they are Cyrillic. There are some Cyrillic characters that are distinctly Russian (i.e. used only in the Russian language), but these characters can't impersonate Latin letters because they are too different from them. https://en.wikipedia.org/wiki/Cyrillic_script
Yeah that bothered me as well.
Would you write: "Site uses American characters to publish fake news." ?
Re: Plagiarized news sites are using Cyrillic characters to avoid detection
#16I think at some point, a sort of visual-normalization that converts similar looking unicode to a single unique string sequence (ex: convert certain letters from Cyrillic and other language sets that are also present in Latin to just Latin) is just going to be necessary as a security precaution. Given the whole "fake news" thing over the past couple of years, I expect that the first step will be taken by one of Google…
Unicode normalization/equivalence is half of what you want and UCAPI is probably the other half. https://en.wikipedia.org/wiki/Unicode_equivalence https://www.casaba.com/products/UCAPI/
Those are interesting, especially the second one. I've read a little here and there about Unicode normalization before, but UCAPI does look like what I really wanted. However, seeing as UCAPI isn't free or even "listed price plans", I get the strong feeling that this will not see much pickup (at least until someone makes a free one).
Re: Plagiarized news sites are using Cyrillic characters to avoid detection
#17Earlier quoted context omitted.
A neat trick would be to render the PDF into an image file, and then OCR that image file, and do the detection using the text file generated by the OCR process. Just like VW's defeat device, the cheater would then need to create software that outputs something different when they see they are being rendered not to display, but to an image file...
So, one needs to use a font that OCR finds hard to avoid detection. Would rubbish keming do the trick, you could abide by imposed don't requirements but change kerning/leading?
At the point where you're doing this, it's better to reword the source. (It's what I did.)
Re: Plagiarized news sites are using Cyrillic characters to avoid detection
#18>The site is part of a growing list of fake Native American pages run out of places like Macedonia, Kosovo or Vietnam. So the headline is a little misleading... It's just that there are a growing number of websites that simply plagiarize content to get views / ad revenue. Because their titles are obfuscated to prevent detection of the plagiarism, they have to target specific niche groups to drive views. So it's not s…
Can we get the title updated?
Re: Plagiarized news sites are using Cyrillic characters to avoid detection
#19Earlier quoted context omitted.
A neat trick would be to render the PDF into an image file, and then OCR that image file, and do the detection using the text file generated by the OCR process. Just like VW's defeat device, the cheater would then need to create software that outputs something different when they see they are being rendered not to display, but to an image file...
So, one needs to use a font that OCR finds hard to avoid detection. Would rubbish keming do the trick, you could abide by imposed don't requirements but change kerning/leading?
You'd probably draw more attention to your paper if OCR fails to read it!
Re: Plagiarized news sites are using Cyrillic characters to avoid detection
#20Earlier quoted context omitted.
Unicode normalization/equivalence is half of what you want and UCAPI is probably the other half. https://en.wikipedia.org/wiki/Unicode_equivalence https://www.casaba.com/products/UCAPI/
Those are interesting, especially the second one. I've read a little here and there about Unicode normalization before, but UCAPI does look like what I really wanted. However, seeing as UCAPI isn't free or even "listed price plans", I get the strong feeling that this will not see much pickup (at least until someone makes a free one).
There are plenty of free libraries that allow you to detect and compare strings with confusables