Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache NiFi

nifi.apache.org

Post reply on HN
Print view View original

31–40 of 140 posts

Re: Apache NiFi

#31

Post by zokier »

corndoge wrote: post #3

Can someone explain what this is? I can't find anything on the website that explains it

Disclaimer: I haven't worked with NiFi specifically, but with another similar product

As far as I know, it at least partially fulfills the role of "enterprise application integration" pattern; the idea is sort of proto-servicemesh where you have bunch of weird enterprise applications all around and you need to get them talking to each other, so you plop in a fancy EAI middleware thingy in the middle which then will talk to each of the applications individually, converting and transforming data from one format and protocol to another. But its not just a dumb hub, in addition to doing arbitrarily complex transformations, it can also make "routing" decisions based on the data itself. And this pattern being product of its time, instead of defining the network in code in some nice DSL everything is just "configured" and lots of things is achieved through clicketyclicking through GUI. I suspect this is at least partially due "configuring a turn-key product" sounding less scary to managers than "developing a system on framework".

If you look at the docs page of NiFi[1] and scroll quickly through the long list in the left sidebar with all the "processors", you already can get an idea what sort of things it does.

While this sort of solutions can be very useful in some situations, I wouldn't necessarily start designing a greenfield architecture around NiFi. But if you end up running it, then you might find that piece by piece it will accumulate all sorts of bits and bops of things because it's "convenient" to throw it in there, while the logic flows become more and more eldritch in nature.

[1] http://nifi.apache.org/docs.html

Re: Apache NiFi

#32

Post by contravariant »

_57jb wrote: post #7

We used NiFi...one of the worst experiences. It installs like an appliance and feels like you are grappling with a legacy tool weighed down by a classic view on architecture and maintenance. We had built a data pipeline and it was for very high-scale data. The theory of it was very much like a TIBCO type approach around data-pipelines. Sadly the reality was also like a TIBCO type approach around data-pipelines. One p…

Nifi is a very powerful tool, but also a very specific one, and a self described 'necessary evil'. It does one heck of a job at getting data from A to B though.

Re: Apache NiFi

#33

Post by joshz404 »

benjaminwootton wrote: post #15

I have been working on a new product which competes with NiFi, providing streaming data transformations. Think, if order value > 100 and the customer has ordered 3 times in the last hour and the product will be in stock tomorrow. Kafka streams, Flink and Dataflow are super powerful and I think there is room for a GUI tool. Would be great to hear experiences of NiFi in this domain or discuss the space with any experie…

Have you heard of BPEL? :D

Re: Apache NiFi

#36

Post by zokier »

cs02rm0 wrote: post #25

I've used it a fair bit, though not for a couple of years. Few points, some of which may be out of date: * I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. * It persisted its config to an XML file, including the positions of boxes on the UI. Trying to keep this config in source control with mu…

> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi.

That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly expert system architects or developers even if they manage to click through the UI. And many developers probably prefer just defining stuff in code instead of having fancy UIs. So it sort of falls between these to categories.

Of course there is huge spectrum of skill in people, and there are probably plenty of "semi-technical" persons to whom this is perfect match, especially if supported by some more techy people.

Re: Apache NiFi

#37

Post by closeparen »

cs02rm0 wrote: post #25

I've used it a fair bit, though not for a couple of years. Few points, some of which may be out of date: * I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. * It persisted its config to an XML file, including the positions of boxes on the UI. Trying to keep this config in source control with mu…

I have not found that argument persuasive to the managers who believe that coding is inherently wasteful if what you’re trying to do is technically possible in a workflow builder GUI.

Re: Apache NiFi

#38

Post by zokier »

joshz404 wrote: post #33
benjaminwootton wrote: post #15

I have been working on a new product which competes with NiFi, providing streaming data transformations. Think, if order value > 100 and the customer has ordered 3 times in the last hour and the product will be in stock tomorrow. Kafka streams, Flink and Dataflow are super powerful and I think there is room for a GUI tool. Would be great to hear experiences of NiFi in this domain or discuss the space with any experie…

Have you heard of BPEL? :D

From wikipedia:

> the open BPMS movement led by JBoss and Intalio Inc., IBM and Microsoft decided to combine these languages into a new language, BPEL4WS. In April 2003, BEA Systems, IBM, Microsoft, SAP, and Siebel Systems submitted BPEL4WS 1.1 to OASIS for standardization via the Web Services BPEL Technical Committee

if that doesn't scare you, then oh boy...

Just look at the list of organizations mentioned there, I don't know if you could make it more enterprisy.

Re: Apache NiFi

#39

Post by marcinzm »

closeparen wrote: post #37
cs02rm0 wrote: post #25

I've used it a fair bit, though not for a couple of years. Few points, some of which may be out of date: * I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. * It persisted its config to an XML file, including the positions of boxes on the UI. Trying to keep this config in source control with mu…

I have not found that argument persuasive to the managers who believe that coding is inherently wasteful if what you’re trying to do is technically possible in a workflow builder GUI.

That's true but, personally, I just avoid working for managers like that.

Re: Apache NiFi

#40

Post by jszymborski »

contravariant wrote: post #29
jszymborski wrote: post #16

So, the comments here have mostly ranged between neutral to negative regarding their experience. I have a problem where I want to stream data to an ML layer and then stream that to a web app (e.g. Laravel or Django). Reading the docs here, this seems like this would solve this problem, but was wondering if people had alternatives given that people seem to think poorly of this application.

Nifi is fantastically good at one thing, which is dataflow. Where you've got data coming in at point A, but you need it at point B, and for some reason can't convince either A or B to connect directly. It's not a message bus, nor is it a data processing framework, nor a scheduler, nor an ETL tool. If you try to use it for one of those you're in for a bad time. What you're describing sounds like you might need a messa…

Thanks so much for that, very helpful!
Post reply on HN