For someone not involved in the web stack, reading through that page tells me nothing. Can someone tell me what use cases this is meant to solve?
Apache NiFi
91–100 of 140 posts
Re: Apache NiFi
#92For someone not involved in the web stack, reading through that page tells me nothing. Can someone tell me what use cases this is meant to solve?
I tried to give an answer here. Does this help?
Re: Apache NiFi
#93I've used it a fair bit, though not for a couple of years. Few points, some of which may be out of date: * I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. * It persisted its config to an XML file, including the positions of boxes on the UI. Trying to keep this config in source control with mu…
Its indeed drag and drop but as soon as you want to do something that's slightly a little more complex than the regular processors in place, you need to use regex filters, branching, AVRO converters, and non-tech users will be lost very quickly.
I see it very useful to automate certain operations (watch S3 storage, take action as soon as object comes in and store in into a DB), as for such use cases it's pretty much drag and drop.
Re: Apache NiFi
#94We used NiFi...one of the worst experiences. It installs like an appliance and feels like you are grappling with a legacy tool weighed down by a classic view on architecture and maintenance. We had built a data pipeline and it was for very high-scale data. The theory of it was very much like a TIBCO type approach around data-pipelines. Sadly the reality was also like a TIBCO type approach around data-pipelines. One p…
Can you elaborate by what you mean on a TIBCO like approach? I haven't used their tools, but would like to know more about the issues you ran into. What were examples of the leaky abstraction>?
TIBCO is garbage. They had a halo for a long time from Rendezvous/EMS. But their money maker was this integration suite called BusinessWorks. It was this horrifyingly complex application that forced you into these ruts so that it could compile Java code. I kid you not, the developer environment for complex code was notepad.exe.
They spent a bunch of money on M&A and eventually had to go private and buy out the founder.
Re: Apache NiFi
#95Earlier quoted context omitted.
could you elaborate on the difference between what op is asking for and what nifi/airflow does? to me the use case of moving data through a couple of different services could be solved by a message bus OR nifi.
Nifi is designed to handle the problems that crop up when two systems can't talk directly to each other. It puts a buffer between them to allow one process to keep sending data when the other isn't quite fast enough, it can do some basic transformation when the data isn't quite in the right format, etc. However if you just need something to send messages then you're better off using a tool that does just that, you do…
ok that makes sense. but in that case wouldn't be easier to write your own adapters for each data source?
Re: Apache NiFi
#96Having used NiFi in production, my biggest issue with it is handling source control and multiple environments. As the "IDE" is effectively also the runtime, the lines between "local", "stage", and "prod" are easy to blur.
They have a built-in source control product called "NiFi Registry", which can even be backed by git. The workflow for promoting flows between environments feels clunky though, especially as so much environment-specific configuration is required once your number of components gets high enough.
Moving our Java, Ruby or Go code between environments or handling versioning and releases was a piece of cake, in comparison.
Re: Apache NiFi
#97Earlier quoted context omitted.
> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly exp…
it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"
There's a general level of elitism in the programming world. GUI tools are seen by some programmers as a "worse" tool, but without any real evidence, and it's likely due to the fact that GUI tools enable non-programmers to reach some level of output that used to be the sole domain of programmers.
It's a similar train of thought that leads to a lot of game developers to look down on games made with GUI tools like gamemaker or such.
Re: Apache NiFi
#98We only use NiFi on the edge of our data lake. It's very good at bulk loading, pulling log files/sensor data from hundreds of systems into our systems.
However, it does not handle small records well, and deploying custom processors is a pain, so don't use it to replace your stream processing framework.
Re: Apache NiFi
#99Earlier quoted context omitted.
it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"
I don't think it's about whether a graphical tool is performant. Many programming languages aren't performant either. It's more that graphical tools don't generally serve their stated purpose of making it possible for non-programmers to get things done because it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax. Additionally, graphical tools tend to hav…
> it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax
Totally. This is where declarative and intention-oriented systems shine. Take something like SQL, where, in the majority of cases, the end-user needs to know next to nothing about algorithmic complexity and can still achieve excellent performance and correct results.
It'd be neat to see a system that was actually designed toward problem decomposition. What's the state of the art around this kind of stuff?
Re: Apache NiFi
#100For those who were around during the mid-2000s, is this basically another revival of SOA (Service Oriented Architecture)?
I watched one of the explanation videos and it brought back memories.
My dislike of the phase back then, which I hope they've addressed now, is that while everything looked find and dandy while designing things on a UI, when something broke it was a whole heaps of generated XML no one could read.