Live data from Hacker News

Apache NiFi

nifi.apache.org

131–140 of 140 posts

Re: Apache NiFi

#131
post #18

If you're considering Apache NiFi, you should also look at Apache Airflow and Uber Cadence to decide what model would work best for you.

They do have totally different use cases, so it should be fairly quick to decide which one is for you.

Can you explain what the different use cases are? In particular, what is Airflow good at that Nifi is not?

I have used Nifi a little bit and Airflow not at all. Reading the home pages of the two products, it's hard for me to know when it would be more appropriate to use Airflow than Nifi.

They both schedule jobs and move data according to control flow topologies that you build in a GUI, right?

Re: Apache NiFi

#132
post #7

We used NiFi...one of the worst experiences. It installs like an appliance and feels like you are grappling with a legacy tool weighed down by a classic view on architecture and maintenance. We had built a data pipeline and it was for very high-scale data. The theory of it was very much like a TIBCO type approach around data-pipelines. Sadly the reality was also like a TIBCO type approach around data-pipelines. One p…

> It installs like an appliance and feels like you are grappling with a legacy tool weighed down by a classic view on architecture and maintenance. This is actually a fair and well-articulated point of view. NiFi is currently an "appliance" like you said. Worse, it's a Pet and not a Cattle. I believe there is active work in the community to address some of that pain. For example, there was a recent addition to NiFi c…

Would be a huge step.

It feels like the answer will end up being something totally different. The reality is that enterprises do well with appliances.

Selling them cattle is hard because the maintenance piece expects a certain level of hygiene, proactivity, discipline.

An appliance sits there and when the thing breaks, you call in someone to fix the box. That relationship between a customer and vendor surprisingly makes for a good selling environment/symbiosis.

It's the Cathedral and the Bazaar in another spectrum...

Re: Apache NiFi

#133

Earlier quoted context omitted.

Can you elaborate by what you mean on a TIBCO like approach? I haven't used their tools, but would like to know more about the issues you ran into. What were examples of the leaky abstraction>?

I'd like to second this request. I have encountered event buses and ETL in a number of places over my career - I don't understand what the heck TIBCO does beyond something simple like RabbitMQ/ZeroMQ. How is this different from Pub-Sub (and its variants). Any pointers to books or blog posts would be really appreciated.

TIBCO is very much providing queuing/caching to shuttle data from one point to another.

The goal is even more-so to be the interconnect for all systems across a varied enterprise at a higher level. It's all pub-sub underneath the hood. Think cheap butts in seats doing the same work for a "negligible hit on performance".

In the same way you can plug random devices into outlets around the house all served by some powerplant you don't know (or even need to care about), TIBCO attempts to provide that same interface.

Data does need some restructuring, whether these are aggregations, transformations, etc. So they provide steps in the process where you can perform these operations through a drag and drop UI.

There is an input defined and an output defined in XML that you don't have to code, but is managed and can be seen. The engine beneath provides the lower layers of routing, bytecode, implementation letting you just drag blocks around on a screen "connecting things".

The goal is very pure: I have many people in my organization that know how data flows, not all of them are developers. How can I enable them to connect my organization without everyone needing to be a developer.

In theory and in practice are always the interesting observations. What I had seen happen (as was mentioned somewhere else) is that very strong developers became weak by relying on this tool (or merely left for adequately challenging work). When the world moved on to something else, so much had changed it was almost a career change to get back into development.

They went from understanding Java 3/4, JEE to Java 11, Spring, DI Frameworks....I saw a lot give up or move over to product management roles. This only made the tension between on-premise infrastructure teams and public cloud teams more divisive and toxic. I don't think it's anything uncommon in other areas, just feel like we've reached a full revolution in this particular space (and not the first revolution either).

Re: Apache NiFi

#134

Earlier quoted context omitted.

They do have totally different use cases, so it should be fairly quick to decide which one is for you.

Can you explain what the different use cases are? In particular, what is Airflow good at that Nifi is not? I have used Nifi a little bit and Airflow not at all. Reading the home pages of the two products, it's hard for me to know when it would be more appropriate to use Airflow than Nifi. They both schedule jobs and move data according to control flow topologies that you build in a GUI, right?

From watching the presentation on Youtube (https://youtu.be/sQCgtCoZyFQ), it seems NiFi is geared towards acquisition of data, gluing/batching/massaging the flow between systems and providing the necessary interfaces to downstream systems;

- Would love to see the ability to develop custom NiFi processors in Go/Rust/Elixir etc.

- XML is a big pain in the rear.

- Being container-aware is big win. Stateless is even better.

I see a good opportunity there for users like me to explorer NiFi's capability in the future.

> They both schedule jobs and move data according to control flow topologies that you build in a GUI, right?

Airflow on the other hand is designed to run scheduled jobs (whether it be batched or otherwise). The 'job' can really be anything - build / data processing pipelines, system configuration management pipelines and so on. In Airflow parlance, one can create connected DAG's as pipelines that massage the data in a way you intend it to.

They both share some commonalities but I do gravitate towards their use cases being subtly different and an important one highlighted above.

Re: Apache NiFi

#135
post #26
post #11

Earlier quoted context omitted.

TIBCO = ?

TIBCO is an "Enterprise Message Bus." Think of RabbitMQ or 0MQ but supported by a giant enterprise grade vendor. My last experience with TIBCO was them telling us the product didn't support anything but running on bare metal because "virtualization does not guarantee the order of writes to disk" (their words not mine)

True.. if your last interaction with TIBCO was c. 2005. Back then consistency issues were a real challenge with VMs, “ESB” was a real software category with transactional messaging as a pattern (I make it a practice to not judge customer preference on how they use our tools)

Today you will find even our traditional brokered messaging products like EMS lifted and shifted on AWS/Azure infra.

OTOH, if you are building new apps - you‘re likely to deploy TIBCO’s integration or stream apps / pipelines using Flogo/Pulsar/Kafka on k8s-lambda-gcr or on a fully managed service on cloud.tibco.com.

(Source / Disclaimer: I do product stuff over at TIBCO Software. Happy to chat if there are further qns: rkozhikktibcocom)

Re: Apache NiFi

#137

Earlier quoted context omitted.

it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"

I don't think it's about whether a graphical tool is performant. Many programming languages aren't performant either. It's more that graphical tools don't generally serve their stated purpose of making it possible for non-programmers to get things done because it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax. Additionally, graphical tools tend to hav…

I think GUI tools come a cropped when they try to do too much, but can work very well in limited domains.

We have recently released a GUI tool to do data transformation/ETL ( https://www.easydatatransform.com ). It is aimed more at power Excel users, than programmers or data scientists. We have tried to overcome some of the issues mentioned here:

* It is only intended for the limited domain of data transformation, which limits the complexity.

* No need to remember any syntax/commands for the vast majority of use cases.

* The transformation template is stored in a single XML file, so it can be versioned.

* You can use regexp, Javascript and command line arguments to get extra power if you need it.

* It is written in C++ and is fast.

Obviously it can't do everything you can do with a general purpose programming language, but it hopefully do enough for many use cases and is much easier to learn.

Re: Apache NiFi

#138
post #99

Earlier quoted context omitted.

I don't think it's about whether a graphical tool is performant. Many programming languages aren't performant either. It's more that graphical tools don't generally serve their stated purpose of making it possible for non-programmers to get things done because it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax. Additionally, graphical tools tend to hav…

> it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax Totally. This is where declarative and intention-oriented systems shine. Take something like SQL, where, in the majority of cases, the end-user needs to know next to nothing about algorithmic complexity and can still achieve excellent performance and correct results. It'd be neat to see a system that…

We have very obviously worked on different types of end-user SQL. One that springs to mind had 30 or 40 tables, evolved over the years as “special cases” popped up, and by time I was asked to look at it, it was maybe too far gone. I tried. Maybe if the schema had been tended like a garden, the queries wouldn’t have been so crazy, but...

Mind, though, that this system had made a ton of money for 15 years, so in a way it was a huge win for them! They just eventually painted themselves into a corner and performance started to get worse faster than Moore’s law could save them. Last I heard, the great untangling is still in progress, and I stepped away from that project a few years ago.

They likely have still come out on top when you take the sum of revenue they made from the system, but they incurred a huge unanticipated cost and got backed into a pretty bad corner. Sales people were still selling features that didn’t exist, and the team was desperately trying to build new things while also fixing the ever degrading performance.

Re: Apache NiFi

#139
post #133

Earlier quoted context omitted.

I'd like to second this request. I have encountered event buses and ETL in a number of places over my career - I don't understand what the heck TIBCO does beyond something simple like RabbitMQ/ZeroMQ. How is this different from Pub-Sub (and its variants). Any pointers to books or blog posts would be really appreciated.

TIBCO is very much providing queuing/caching to shuttle data from one point to another. The goal is even more-so to be the interconnect for all systems across a varied enterprise at a higher level. It's all pub-sub underneath the hood. Think cheap butts in seats doing the same work for a "negligible hit on performance". In the same way you can plug random devices into outlets around the house all served by some power…

Thanks for a clear explanation without dismissing the product as garbage. (it's in that space where techies hate it, but it must provide value since it's so expensive!)

Why do non-technical people need to understand the data flow? It seems like documentation (data dictionaries) would be preferred. Or, are they useful for very non-technical people, while TIBCO data flow understanding is useful for people who are data savvy but not tech savvy?

Re: Apache NiFi

#140
post #133

Earlier quoted context omitted.

TIBCO is very much providing queuing/caching to shuttle data from one point to another. The goal is even more-so to be the interconnect for all systems across a varied enterprise at a higher level. It's all pub-sub underneath the hood. Think cheap butts in seats doing the same work for a "negligible hit on performance". In the same way you can plug random devices into outlets around the house all served by some power…

Thanks for a clear explanation without dismissing the product as garbage. (it's in that space where techies hate it, but it must provide value since it's so expensive!) Why do non-technical people need to understand the data flow? It seems like documentation (data dictionaries) would be preferred. Or, are they useful for very non-technical people, while TIBCO data flow understanding is useful for people who are data…

It is a butts in seats equation.

If you can have less expensive operators driving and mapping the world and place all the smarts in the pipes, you can drive down opex and divert cash to capex for competitive advantage.

Linux and much of the streaming software world is smart people, dumb pipes.

If you invert that you have more automation, predictability, control at lower cost. The risk is a lot of eggs in one basket and when the market turns, if the company you are buying from mismanages tech, if they can't keep pace with change...you go along for their ride. Every company big and small falls into this technical debt. I have maby opinions on why as I am sure many do.

There is a lot baked into that comment but the constant tug-of-war every CIO is trying to wrap their head around....how do we do more with less and gain an advantage.

Post reply on HN