Apache NiFi

Fundamentally NiFi is a "dataflow engine", a system that can be used to automate data transfer from different and varying types of sources and sinks. It has a fairly usable UI that enables a "dataflow manager" (end user) to perform transformation, routing and delivery of data using a "drag-n-drop" configuration approach.

Getting data into or out of your application/system, or performing simple schema transformations, is a common (maybe tedious) task that most developers face. NiFi helps connect the dots, so to speak, and decouples the receipt/delivery of data away from your application. NiFi comes with a set of "batteries included" connectors for almost every transport protocol you would generally need. And it's modular so you can create your own processing components as well.

NiFi is fundamentally modeled after what's called "Flow-Based Programming"[1], which is a style of programming that facilitates composition of black-box processing units. It can run at an enterprise or IoT level, depending on where that decomposition best fits into your architecture.

[1] https://en.wikipedia.org/wiki/Flow-based_programming

(disclaimer: I'm affiliated with the NiFi project)

1. Move data from A to B

2. Move data from A to B, but perform some intermediate processing in-between (ETL)

3. Grab data from A, write it to [INSERT SYSTEM HERE] (Mongo, Kafka, many others)

4. Periodically run an arbitrary script or program (like cron but you can schedule in seconds)

Those are just some use cases that people tend to start off doing. It's important to note...the above doesn't sound very impressive until you think about the following features:

1. Build these pipeline completely from a UI. You can leverage custom code, but the number of processors that come out of the box is astounding. If you can think of it, it most likely already has something built-in.

2. By default safety nets. Data is backed by a high performance WAL that allows for recovering in the case of a failure.

3. Ability to pause and resume specific areas of a pipeline at any time. For example, you may have a processor that's receiving data and perhaps the database you write the data to is offline (or has moved). Data is automatically buffered on disk so when the DB comes back online, you can backfill the data that you've been consuming. Further, none of the data was dropped in this process.

4. Tunable and verifiable back pressuring out of the box (_very_ hard to get right when writing custom code)

5. Easy to prioritize certain events over others (e.g. higher priority, latency sensitive data)

Lastly, when you start getting comfortable using NiFi, you will realize how general purpose the engine is. Data is represented as a "FlowFile". A "FlowFile" is essentially just a Array. This means you can operate on virtually anything. Further, you can operate on large data since its backed by a WAL and NiFi facilitates the ability to stream data from disk when you want to process it. As long as you don't read the entire thing into memory (of course).

Easiest way to get started is to just download the binaries and run it. Then go to the http://localhost:8080/nifi and just mess around. There's plenty of tutorials online (Blogs, YouTube, etc). Once you get comfortable using NiFi, people will be blown away with how fast you can get something up and running. Things that would normally take days or weeks to get into production, you can routinely do it in hours or minutes.

Hope this helps!

edit: You can download from https://nifi.apache.org/download.html. Just unzip/untar the package and run "./bin/nifi.sh run"

And maybe it's just me but everything I've seen being called ETL handled data in batches, in which case you really want support for scheduling, error handling and retries, which Nifi does not really have (there is a 'retry' function somewhere, but I found it confusing and it only seems to work for a single data item, not a good idea when you've got thousands of them). I much prefer Airflow for those scenarios.

However if you just need something to send messages then you're better off using a tool that does just that, you don't need the overhead of a system that can connect arbitrary applications that talk in incompatible protocols you just need a single protocol that allows your applications to send each other data.

In their docs Nifi calls itself a dataflow tool and calls dataflow a necessary evil. It's the band-aid you need when you've got a mismatch between the way data is generated and the way it is consumed. It would be insane to deliberately create such a mismatch just to use Nifi.