Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache NiFi

nifi.apache.org

Post reply on HN
Print view View original

71–80 of 140 posts

Re: Apache NiFi

#71

Post by taftster »

_57jb wrote: post #7

We used NiFi...one of the worst experiences. It installs like an appliance and feels like you are grappling with a legacy tool weighed down by a classic view on architecture and maintenance. We had built a data pipeline and it was for very high-scale data. The theory of it was very much like a TIBCO type approach around data-pipelines. Sadly the reality was also like a TIBCO type approach around data-pipelines. One p…

> It installs like an appliance and feels like you are grappling with a legacy tool weighed down by a classic view on architecture and maintenance.

This is actually a fair and well-articulated point of view. NiFi is currently an "appliance" like you said. Worse, it's a Pet and not a Cattle.

I believe there is active work in the community to address some of that pain. For example, there was a recent addition to NiFi called "stateless NiFi" which enables NiFi to better run in Kubernetes and other "cloud" architectures.

It's not there yet, it's still what would be described as a "fat" application. But I believe that eventually NiFi will evolve to more like a command-and-control tool for the cloud and less like something you have to install directly to your hardware. We hopefully see the day where "NiFi-as-a-Service" exists, which would really be an improvement over the current model.

Re: Apache NiFi

#72

Post by mistrial9 »

zokier wrote: post #36
cs02rm0 wrote: post #25

I've used it a fair bit, though not for a couple of years. Few points, some of which may be out of date: * I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. * It persisted its config to an XML file, including the positions of boxes on the UI. Trying to keep this config in source control with mu…

> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly exp…

it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ?

I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"

Re: Apache NiFi

#74

Post by aquaticsunset »

aasasd wrote: post #60

So Apache has at least a handful of software packages that do about the same thing, but with different interfaces and connectivity?

This has sorta been my experience with a lot of Apache projects recently. The differences between them are becoming quite nuanced.

I'm trying to piece together the main reason why someone would pick this over Camel, or vice-versa. I know they're different - but not night and day.

Re: Apache NiFi

#75

Post by brutus1213 »

edmundsauto wrote: post #50
_57jb wrote: post #7

We used NiFi...one of the worst experiences. It installs like an appliance and feels like you are grappling with a legacy tool weighed down by a classic view on architecture and maintenance. We had built a data pipeline and it was for very high-scale data. The theory of it was very much like a TIBCO type approach around data-pipelines. Sadly the reality was also like a TIBCO type approach around data-pipelines. One p…

Can you elaborate by what you mean on a TIBCO like approach? I haven't used their tools, but would like to know more about the issues you ran into. What were examples of the leaky abstraction>?

I'd like to second this request. I have encountered event buses and ETL in a number of places over my career - I don't understand what the heck TIBCO does beyond something simple like RabbitMQ/ZeroMQ. How is this different from Pub-Sub (and its variants). Any pointers to books or blog posts would be really appreciated.

Re: Apache NiFi

#76

Post by ckdarby »

monstrado wrote: post #64
nightowl_games wrote: post #59

Earlier quoted context omitted.

This page is classic Apache project in that I read it and have no idea what it does. Can you high level explain what this thing is really for?

I fine that explaining NiFi tends to be difficult, especially if you're an advanced user. This is because NiFi is extremely general purpose. I've used it for many different types of projects. That being said, here's a few use cases that are relatively easy to get started. 1. Move data from A to B 2. Move data from A to B, but perform some intermediate processing in-between (ETL) 3. Grab data from A, write it to [INSE…

Isn't most of this just Kubernetes running PrestoDB/PrestoSQL?

Re: Apache NiFi

#77

Post by sixhobbits »

I have never heard of this before, and I'm sad that profit-driven, marketing speak has taken over even non-profit product pages.

> An easy to use, powerful, and reliable system.

This is the title. That's the most important sentence, and it's absolutely meaningless.

It's bad enough that everything has to "sell" - just describe to me what your product does and I'll decide if I need or not. Don't try to convince me.

If you have to sell, do it by differentiating yourself from your competitors. No one is calling themselves "Difficult to use, weak, and unreliable", so saying the opposite is not differentiation.

When did we accept that marketing-speak was default communication. Can't we have some landing pages that are essays? Or even a few paragraphs instead of trying-to-be-catchy bullet point phrases in large font?

Re: Apache NiFi

#78

Post by josephmosby »

NiFi is a fantastic tool for a certain set of organizational constraints.

* It doesn't need much in the way of dependencies to run. If you can get Java onto a machine, you can probably get NiFi to run on that machine. - That is HUGE if you are operating in an environment where getting any new dependencies installed on a machine is an operational nightmare.

* It doesn't require a lot of overhead. Specifically, no database.

* You can write components for it that don't require a whole lot of tweaking for small changes to the incoming data. So, if I have a machine processing a JSON file that looks like XXYX and another machine processing a nearly identical JSON file that looks like XYXX, the tweaks can be made pretty easily.

So, if you're looking for a lightweight, low overhead, easily configurable tool that may be running in an environment where you've got to run lots of little instances that are mostly similar but not quite, NiFi is great.

If you are running a centralized data pipeline where you have a dedicated team of data engineers to keep the data flowing, there are better options out there.

Re: Apache NiFi

#79

Post by cameronh90 »

mistrial9 wrote: post #72
zokier wrote: post #36

Earlier quoted context omitted.

> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly exp…

it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"

I don't think it's about whether a graphical tool is performant. Many programming languages aren't performant either.

It's more that graphical tools don't generally serve their stated purpose of making it possible for non-programmers to get things done because it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax.

Additionally, graphical tools tend to have a bunch of problems inherent to them, such as being unable to write comments and harder to store in git and work on collaboratively and only having one single editor, which is usually much buggier than a text editor and compiler.

So: graphical "languages" don't make most things much easier, and make other things much harder.

Re: Apache NiFi

#80

Post by _JamesA_ »

taftster wrote: post #62
nightowl_games wrote: post #59

Earlier quoted context omitted.

This page is classic Apache project in that I read it and have no idea what it does. Can you high level explain what this thing is really for?

Agreed. So here's an attempt to describe NiFi at a high level. Fundamentally NiFi is a "dataflow engine", a system that can be used to automate data transfer from different and varying types of sources and sinks. It has a fairly usable UI that enables a "dataflow manager" (end user) to perform transformation, routing and delivery of data using a "drag-n-drop" configuration approach. Getting data into or out of your a…

Is this a layer on Apache Camel [1] or something completely different?

[1]: https://camel.apache.org/

Post reply on HN