Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faker is a Python package that generates fake data for you

joke2k.net

Post reply on HN
Print view View original

11–20 of 37 posts

Re: Faker is a Python package that generates fake data for you

#11

Post by Arnor »

sophacles wrote: post #10
kerkeslager wrote: post #9

I made a tool like this for my company in Ruby (it wasn't nearly as mature as this). The largest challenge I struggled with (and never really solved) is that ultimately, there's no way to generate data as useful as real data. The value of real data comes from the fact that it's messy. Real data is different sizes than you expect[1], collides with your sentinel values[2], and comes in with unexpected encodings[3]. And…

I'd suggest looking into fuzzers. Short version - tools designed to input messy, non-conforming data to ensure the inputs don't cause problems, that things are sanitized correctly, etc. At this point they are a mature technology, with improvements constantly being researched. They are generally thought of as security tools[1], but are very useful for basic development too. [1] The common use of fuzzers in a security…

A quick search gave me this list: http://www.infosecinstitute.com/blog/2005/12/fuzzers-ultimat... is there a notable fuzzer missing? It's a pretty long list, does anyone know which of these tools are really worth checking out?

Re: Faker is a Python package that generates fake data for you

#12

Post by davidw »

asm89 wrote: post #6

The name looked familiar and it's indeed inspired by the Faker library for PHP: https://github.com/fzaninotto/Faker Another interesting library that is build on top of Faker is Alice. It allows you to define complex fixtures in .yml: https://github.com/nelmio/alice

Aren't they all kind of inspired by the original Perl implementation? I use the Ruby one for testing.

Something kind of similar and worth thinking about is this:

http://en.wikipedia.org/wiki/QuickCheck

Re: Faker is a Python package that generates fake data for you

#14

Post by ronaldbradford »

Testing data serves multiple purposes. Boundary conditions (long fields, unicode etc) is important for invalidation testing. (i.e. Testing to break your code and functionality)

Testing at scale is important for performance and predicting bottlenecks as you grow. (i.e. Testing to break your systems capacity)

It can be difficult to generate good quality test data at scale, and data based on your specific schema.

This is how http://goodtestdata.com/ came about. It has the building blocks of core data and new sources can be built on request.

Re: Faker is a Python package that generates fake data for you

#16

Post by xzel »

davidw wrote: post #12
asm89 wrote: post #6

The name looked familiar and it's indeed inspired by the Faker library for PHP: https://github.com/fzaninotto/Faker Another interesting library that is build on top of Faker is Alice. It allows you to define complex fixtures in .yml: https://github.com/nelmio/alice

Aren't they all kind of inspired by the original Perl implementation? I use the Ruby one for testing. Something kind of similar and worth thinking about is this: http://en.wikipedia.org/wiki/QuickCheck

I was just about to comment the same thing about QuickCheck. Arbitrary data galore.

I've only done serious work with QuickCheck in Haskell but here's the python implementation I've played with: https://pypi.python.org/pypi/pytest-quickcheck/

Re: Faker is a Python package that generates fake data for you

#17

Post by tel »

I've used the Ruby version of Faker to do fuzz/property/quickcheck-style testing in Ruby. I believe this to be an incredibly important, under-recognized form of testing. Faker is not the best tool for this as you really need more sources of randomness than it provides, but it's not a bad start.

The best places to learn are from the canonical libraries, quickcheck in Haskell, Quviq in Erlang, simple-check in Clojure, and there are others.

The challenge with all of these methods is that you want some notion of referential transparency in order to make useful properties. You can at least do that in certain contexts for certain expressions in Ruby and doing so will improve code readability.

I'd love to hear from others with experience using these techniques in Ruby or Python.

Re: Faker is a Python package that generates fake data for you

#18

Post by juhanima »

Arnor wrote: post #11
sophacles wrote: post #10

Earlier quoted context omitted.

I'd suggest looking into fuzzers. Short version - tools designed to input messy, non-conforming data to ensure the inputs don't cause problems, that things are sanitized correctly, etc. At this point they are a mature technology, with improvements constantly being researched. They are generally thought of as security tools[1], but are very useful for basic development too. [1] The common use of fuzzers in a security…

A quick search gave me this list: http://www.infosecinstitute.com/blog/2005/12/fuzzers-ultimat... is there a notable fuzzer missing? It's a pretty long list, does anyone know which of these tools are really worth checking out?

That's a pretty old list. Just to name one, I would recommend taking a look of Radamsa

https://www.ee.oulu.fi/research/ouspg/Radamsa

...from the Oulu University. It's more like a framework for generating intelligent fuzzers than a shrink-wrapped product, though.

The OUSPG guys are really good at fuzzing. There is also a commercial spin-off, Codenomicon, whose tools are quite widely used.

Post reply on HN