11–20 of 80 posts
I'd also add more invalid UTF encodings and embedded null bytes, etc. The JSON format would be preferable to plain text for that though.
I can think of a few more cases that I've seen cause havoc:
- U+FEFF in the middle of a string (people are used to seeing it at the beginning of a string, because Microsoft, but elsewhere it may be more surprising)
- U+0 (it's encoded as the null byte!)
- U+1B (the codepoint for "escape")
- U+85 (Python's "codecs" module thinks this is a newline, while the "io" module and the Python 3 standard library don't)
- U+2028 and U+2029 (even weirder linebreaks that cause disagreement when used in JSON literals)
- A glyph with a million combining marks on it, but not in NFC order (do your Unicode algorithms use insertion sort?)
- The sequence U+100000 U+010000 (triggers a weird bug in Python 3.2 only)
- "Forbidden" strings that are still encodable, such as U+FFFF, U+1FFFF, and for some reason U+FDD0
People should also test what happens with isolated surrogate codepoints, such as U+D800. But these can't properly be encoded in UTF-8, so I guess don't put them in the BLNS. (If you put the fake UTF-8 for them in a file, the best thing for a program to do would be to give up on reading the file.)
If you really intend this for use in testing, I'd suggest making the injections less nasty. I could easily see a junior dev slapping this in and deleting some important stuff. I'd also add more invalid UTF encodings and embedded null bytes, etc. The JSON format would be preferable to plain text for that though.
https://code.google.com/p/fuzzdb/
Fuzz lists are to web pentesters what drain snakes are to plumbers.
Using a newline as a delimiter in that file excludes newlines from being part of the strings you are testing - but newlines are an important "naughty" character to consider. Unfortunately the same is true of basically any other common delimiter character.
Maybe base64-encoding the strings would be one way to solve for this? You could use base64-encoded values in JSON, for example.
It's not completely clear to me which encoding the blns.txt file uses. Since this project is all about weird/evil bytestrings, the encoding of the file itself is very important. Using a newline as a delimiter in that file excludes newlines from being part of the strings you are testing - but newlines are an important "naughty" character to consider. Unfortunately the same is true of basically any other common delimit…
I had it set as UTF-16 for the two-byte characters when first writing it, but that had caused issues. If there is a demand, a second list can be added.
בְּרֵאשִׁית, בָּרָא אֱלֹהִים, אֵת הַשָּׁמַיִם, וְאֵת הָאָרֶץ
Nice; sort of a programming complement to Shutterstock's _List of Dirty, Naughty, Obscene, and Otherwise Bad Words_[0]. So helpful to have a bunch of minds working on useful lists like this. Good to see that GitHub passes this test! [0] https://github.com/shutterstock/List-of-Dirty-Naughty-Obscen...
I doubt the value of this repository. The first naughty French word "allumé" can't be considered naughty, dirty, or bad, like, at all. And many others are not naughty under too many circumstances... Except very few swear words, word filtering is pretty much useless.
/dev/null; rm -rf /*; echo " That's a little aggressive for testing no?
Most of what I do involves the messy world of text, and I think this is a great resource. I wish the software I depended on tested against it. I can think of a few more cases that I've seen cause havoc: - U+FEFF in the middle of a string (people are used to seeing it at the beginning of a string, because Microsoft, but elsewhere it may be more surprising) - U+0 (it's encoded as the null byte!) - U+1B (the codepoint f…