Apache NiFi
nifi.apache.org
Apache NiFi
1–10 of 140 posts
Re: Apache NiFi
#2Brought to you by the NSA ;-)
Re: Apache NiFi
#3Can someone explain what this is? I can't find anything on the website that explains it
Re: Apache NiFi
#4We have a team using this at work. They had built a process and needed it to be put on a VM and run periodically. They said the requirements were a dual core machine and 8gb of ram. The “binary” was like 1.8gb. I’m sure it included a jre and a full nifi runtime, but god damn that is ridiculous. Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and able to run with 512mb of ram.
Re: Apache NiFi
#5Can someone explain what this is? I can't find anything on the website that explains it
Been that way for at least 5 years https://news.ycombinator.com/item?id=10190846
There's a few articles in the old comments that explains it's use case a little
Re: Apache NiFi
#6Our team found this adapter to integrate ML with NiFI pretty handy: https://dev.to/tspannhw/easy-deep-learning-in-apache-nifi-wi...
Re: Apache NiFi
#7We used NiFi...one of the worst experiences.
It installs like an appliance and feels like you are grappling with a legacy tool weighed down by a classic view on architecture and maintenance.
We had built a data pipeline and it was for very high-scale data. The theory of it was very much like a TIBCO type approach around data-pipelines.
Sadly the reality was also like a TIBCO type approach around data-pipelines.
One persons experience and opinion and I am super jaded by it due to some vendor cramming it down one of our directors throats who subsequently crammed it down ours when we warned how it would turn out. It ended up being a very leaky and obtuse abstraction that didn't belong in our data-pipeline when you planned how it was maintained longer-term.
I ultimately left that company. It had to do with as much of their leadership and tooling dictation as anything else, NiFi was one of many pains. I am sure there are places that are using NiFi who will never outgrow the tool so take it with a grain of salt.
Said company ultimately struggled for the very reasons those of us who left were predicting (the tooling pipeline was a mess and was thrashing on trying to get it right, constantly breaking by forcing this solution, along with others, into the flow. Lots of finger-pointing).
Sucks to have that: "I told you so..." moment when you never wanted that outcome for them....I just couldn't be a part of their spiral anymore.
Re: Apache NiFi
#8For the love of god, don't use NiFi to trigger an Airflow DAG.
Re: Apache NiFi
#9We have a team using this at work. They had built a process and needed it to be put on a VM and run periodically. They said the requirements were a dual core machine and 8gb of ram. The “binary” was like 1.8gb. I’m sure it included a jre and a full nifi runtime, but god damn that is ridiculous. Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and…
Well, when one considers "no payload/data" flows through this system and it is merely managing metadata & workflow status, I wonder if it is web-scale enough :P your team will now ask for a cluster of these monstrosities.
Re: Apache NiFi
#10Can someone explain what this is? I can't find anything on the website that explains it
Think TIBCO data pipelines.
You create sources of data and can have non-technical (or maybe non-developer) roles wire together these data pipelines with transformations, aggregations etc.