Apache NiFi • Hacker News Board

Re: Apache NiFi

#11

Post by chipsa » Mon, May 11, 2020, 6:19 PM UTC

_57jb wrote: post #10

corndoge wrote: post #3

Can someone explain what this is? I can't find anything on the website that explains it

Think TIBCO data pipelines. You create sources of data and can have non-technical (or maybe non-developer) roles wire together these data pipelines with transformations, aggregations etc.

TIBCO = ?

Re: Apache NiFi

#12

Post by RandyRanderson » Mon, May 11, 2020, 6:20 PM UTC

mberning wrote: post #4

We have a team using this at work. They had built a process and needed it to be put on a VM and run periodically. They said the requirements were a dual core machine and 8gb of ram. The “binary” was like 1.8gb. I’m sure it included a jre and a full nifi runtime, but god damn that is ridiculous. Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and…

Interesting - can you expand on this: "Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and able to run with 512mb of ram"?

Re: Apache NiFi

#13

Post by gregw134 » Mon, May 11, 2020, 6:21 PM UTC

corndoge wrote: post #3

Can someone explain what this is? I can't find anything on the website that explains it

Sure, I've worked with it. It let's you visually build data pipelines. It's extremely useful for getting work done quickly--you just drag and drop prebuilt connectors to things like elasticsearch, s3, or twitter and you have a data pipeline, including automatic backoff and the ability to inspect the data at each step. It's visual so it's easy to tell what's going on. Biggest downside is it's not automatically distributed. You can set it up to be distributed, but you have to do the plumbing yourself on the nifi graph by dropping nodes for routing tasks between nifi servers. Overall, perfect tool for quickly building a pipeline that can be easily shown to the business and in which you can visually see all data and errors at each step.

Re: Apache NiFi

#14

Post by monstrado » Mon, May 11, 2020, 6:21 PM UTC

Incredible piece of software. I've used it in production at my last two jobs. You can build almost anything in NiFi once you get into the mindset of how it works.

A good way to get started with NiFi is to use it as a highly available quartz-cron scheduler. For example, running "some process" every 5 seconds.

Disclaimer: I'm an Apache NiFi committer.

An article you might find interesting about it's ability to scale.

https://blog.cloudera.com/benchmarking-nifi-performance-and-...

Disclaimer v2: I used to work at Cloudera

Re: Apache NiFi

#15

Post by benjaminwootton » Mon, May 11, 2020, 6:23 PM UTC

I have been working on a new product which competes with NiFi, providing streaming data transformations.

Think, if order value > 100 and the customer has ordered 3 times in the last hour and the product will be in stock tomorrow.

Kafka streams, Flink and Dataflow are super powerful and I think there is room for a GUI tool.

Would be great to hear experiences of NiFi in this domain or discuss the space with any experienced users. Will add contact details in my profile.

Re: Apache NiFi

#16

Post by jszymborski » Mon, May 11, 2020, 6:24 PM UTC

So, the comments here have mostly ranged between neutral to negative regarding their experience.

I have a problem where I want to stream data to an ML layer and then stream that to a web app (e.g. Laravel or Django).

Reading the docs here, this seems like this would solve this problem, but was wondering if people had alternatives given that people seem to think poorly of this application.

Re: Apache NiFi

#17

Post by tomrod » Mon, May 11, 2020, 6:32 PM UTC

Do I understand what this is: general purposes SSIS-type data integration, pipeline, and workflow tool?

If so, how does it compare to SSIS, dbt, and other projects (please name!)?

Otherwise, what is an analogous toolset?

Re: Apache NiFi

#18

Post by yawz » Mon, May 11, 2020, 6:34 PM UTC

If you're considering Apache NiFi, you should also look at Apache Airflow and Uber Cadence to decide what model would work best for you.

Re: Apache NiFi

#19

Post by MasterIdiot » Mon, May 11, 2020, 6:38 PM UTC

mberning wrote: post #4

We have a team using this at work. They had built a process and needed it to be put on a VM and run periodically. They said the requirements were a dual core machine and 8gb of ram. The “binary” was like 1.8gb. I’m sure it included a jre and a full nifi runtime, but god damn that is ridiculous. Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and…

NiFi is intended as a serious workhorse, we run it on a cluster with 256GM of RAM and over 20 cores for each server (on-prem), the 2GB of disk storage is negligible in any sort of scale (Just like the dual core/8gb machine)

Re: Apache NiFi

#20

Post by endlessmike89 » Mon, May 11, 2020, 6:39 PM UTC

Nifi is a good (not great) tool, mostly because of all of the functionality you get out of the box. It comes with almost any kind of connector you would need for moving data. There's a pretty steep learning curve, but once you push through that, creating a new data flow from scratch is quick and easy. It sucks that other people in this thread have had bad experiences with Nifi, and I can't say that I haven't. However, it has generally been a positive addition to my team's stack.