ROOT – Data Analysis Framework

I think there are a few of objectively neat features of ROOT:

* Versioned persistency of C++ objects deriving from the TObject base class [1];

* Script-like execution of C++ and a C++ REPL based on clang [2]; and

* Dynamic bindings of the C++ classes to Python [3].

There's an accompanying, but independently developed, file access protocol for reading and writing ROOT files over a network, too [4].

On the other (subjective) hand, ROOT is regarded a pain to use by ‘analysts’, the people who use ROOT to make the results that go in to physics papers. There are already some good, old-but-still-valid critiques [5, 6], so I won't say too much, but I think a large part of the problem comes from two things:

1. ROOT tries its best to do everything that a particle physicist might want to do. This encompasses a very wide range of things, and this has lead to ROOT having a very large, often intractable codebase that cannot be modularised.

2. It has failed to keep up with contemporary coding techniques and analysis methods. Most of the PhD students I know use the Python interface to ROOT, and yet the ROOT developers are planning to drop Python support for the next major version (ROOT 7, which is expected in 2018). Those that do use C++ aren't able to use even C++11 effectively with ROOT, as its interfaces aren't compatible.

Luckily, I'm confident that analysts will move to a better way. I've been very encouraged by the astrophysics and machine learning communities in particular, who are using Python to do low- and high-level analysis on large datasets, as we do in particle physics, and are producing fantastic results. Tools like pandas, matplotlib, and scikit-learn are an absolute joy to use in comparison with ROOT, and the communities within the Python ecosystem are wonderful: they foster very open code development, and value readable, well-documented, fast code.

I don't need ROOT to get any better, because I think the future is already here.

[1]: https://root.cern.ch/root/html534/guides/users-guide/InputOu...

[2]: https://root.cern.ch/cint-prompt

[3]: https://root.cern.ch/pyroot

[4]: http://xrootd.org

[5]: http://www.insectnation.org/articles/problems-with-root.html

[6]: http://www.insectnation.org/articles/root-wishlist.html

* HEP stores about 0.5 exabytes of data in ROOT format, that's almost exclusively serialized objects that do not know anything about TObject.

* XRootD is not really specific for ROOT files. A better example would maybe be our JavaScript de-serialization library, https://root.cern.ch/js/

* No way will the python binding be dropped. I wonder where you got that rumor from. About one third of our users is using it.

* HEP is limited by CPU resources, which is part of the reason why HEP decided to use a close-to-bare-metal language for the number crunching part.

* We just made the use of python and R multivariate analysis tools with ROOT data more straightforward.

* We have people from genomics etc coming to ask for help, because they cannot find a system that scales as well as ROOT does.

And then we have a different perception of the direction out there. I see that Hadoop was nice but slow, Spark is nice but slow, so now things are moving to C++, see e.g. ScyllaDB. There is no reason for us to move away from it, but every reason to make it more usable.

And yes, I agree that this is an issue. But many physicists do not.

Nice to see it on HN.

Look, ROOT is a very complex framework for data gathering and analysis build by physics and it shows every step of the way. The bugs are everywhere and it does really weird things like setting global variables when you analyze some piece of data for instance, changing your results for all subsequent analysis (this particular bug cost me about 2 weeks).

And in the end, there isn't really any point in using ROOT.

- Data gathering can be done with a simple CSV (binary if you wish), a more advanced SQL database, or in the realm of research with the venerable HDF5 format.

- Data analysis in C++ or any compiled language, just doesn't make much sense. You can use Python or R. The libraries to read and treat data are optimized and will make the process much less error prone and probably faster in the end.

Seriously, don't make the same mistakes as I did just because some older people in your lab use ROOT and you feel compelled to do it as well. There are much better tools for the job and I regret not searching for them before wasting about 6 months of my PhD thesis trying to integrate ROOT in my research workflow.