If you're considering Apache NiFi, you should also look at Apache Airflow and Uber Cadence to decide what model would work best for you.
They do have totally different use cases, so it should be fairly quick to decide which one is for you.
Apache NiFi
121–130 of 140 posts
Re: Apache NiFi
#122Our devs hated it due to ease of accidental flow fsck ups. And good luck scaling it or trying to put it behind an LB!
Re: Apache NiFi
#123Phew! Happy to have read the comments here. They say a lot. I will go with Apache Airflow for all my workflow needs from now on. I wasn't entirely sure if this was the best bet, but after seeing all of this I am now.
I know a massive installation [0] which is about to be open sourced, where Apache NIFI is used in the middle of the stack as a key component. No dismissal of the capabilities this package offers intended.
[0] https://sikkerhetsfestivalen.no/bidrag2019/138
slides [slide #32]: https://static1.squarespace.com/static/5c2f61585b409bfa28a47...
Re: Apache NiFi
#124Still uses http on its website?
Re: Apache NiFi
#125Incredible piece of software. I've used it in production at my last two jobs. You can build almost anything in NiFi once you get into the mindset of how it works. A good way to get started with NiFi is to use it as a highly available quartz-cron scheduler. For example, running "some process" every 5 seconds. Disclaimer: I'm an Apache NiFi committer. An article you might find interesting about it's ability to scale. h…
This page is classic Apache project in that I read it and have no idea what it does. Can you high level explain what this thing is really for?
Thanks for your honesty, I thought I was the only one.
Re: Apache NiFi
#126Earlier quoted context omitted.
Nifi is designed to handle the problems that crop up when two systems can't talk directly to each other. It puts a buffer between them to allow one process to keep sending data when the other isn't quite fast enough, it can do some basic transformation when the data isn't quite in the right format, etc. However if you just need something to send messages then you're better off using a tool that does just that, you do…
ok that makes sense. but in that case wouldn't be easier to write your own adapters for each data source?
Well no, Nifi has a lot of adapters that you can use out of the box, making them yourself isn't easier.
Re: Apache NiFi
#127Re: Apache NiFi
#128Earlier quoted context omitted.
> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly exp…
it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"
How do you do what has become basic best practices in a gui?
Everything under source control, defined as code.
* This gives you the ability to trace who did what
* Good code-commit hygiene with prs
* The ability to revert changes
* Ability to have an exact replication and promote across environments
Pragmatically test changes, unit test your flows
* Tie all the above up with a deployment pipeline
NiFi is being used in the critical parts of business data pipelines and all the devops rules for reliable development and deployment go out the window.
If NiFi is used by one person for a minor role the GUI is fine. But if it’s the core data pipeline which is a critical service used by multiple teams it’s not the smartest way to do things.
Re: Apache NiFi
#129I like the way how it buffers messages. You basically stop a process and it will continue when it left off. It is easy to create a distributed cluster. It has hundreds of different connectors for external sources. On the other hand is very bulky. Making it to work with https was horrible. You cannot just put it behind a reverse proxy.
Re: Apache NiFi
#130We used NiFi...one of the worst experiences. It installs like an appliance and feels like you are grappling with a legacy tool weighed down by a classic view on architecture and maintenance. We had built a data pipeline and it was for very high-scale data. The theory of it was very much like a TIBCO type approach around data-pipelines. Sadly the reality was also like a TIBCO type approach around data-pipelines. One p…
Nifi is a very powerful tool, but also a very specific one, and a self described 'necessary evil'. It does one heck of a job at getting data from A to B though.
That was exactly how folks talked about TIBCO. I didn't find it very powerful. I felt it very cumbersome. When you need to think about Day 2 is where everything completely collapsed.
We were able to pass data around in incredible lightweight ways leveraging Spring sometimes even just leveraging RestRepositories and transforming the object to our data representation by hand, it was never more than 100 lines of code for the entire thing. You could spit one out in an hour...the time was really in composing them and ensuring the architecture reflected the world and was still sensible/manageable.
We ultimately faced issues with running microservices and the licensing cost of that. Our enterprise was sadly too big, they didn't realize they needed to price their internal infrastructure competitive to legacy vendors.
You could get a WAS box for 50k and cram so much on that server until it was bursting...price didn't change. On the other hand each microservice brought a cost which added up.
The economics didn't make sense and it was a new political battle to fight with someone who had zero understanding of marketing what they've built. It just wasn't worth it. Lambdas would have been an option or something more ephemeral/serverless...but the options just weren't there for us at the time.
Enter NiFi and this "new data pipeline" and the circus began.