Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache NiFi

nifi.apache.org

Post reply on HN
Print view View original

81–90 of 140 posts

Re: Apache NiFi

#81

Post by taywrobel »

sixhobbits wrote: post #77

I have never heard of this before, and I'm sad that profit-driven, marketing speak has taken over even non-profit product pages. > An easy to use, powerful, and reliable system. This is the title. That's the most important sentence, and it's absolutely meaningless. It's bad enough that everything has to "sell" - just describe to me what your product does and I'll decide if I need or not. Don't try to convince me. If…

> This is the title. That's the most important sentence, and it's absolutely meaningless.

Well, yeah, it's meaningless if you cut off the second half...

> ...to process and distribute data.

That's what it does. The adjectives before it aren't the meat of the sentence.

Re: Apache NiFi

#82

Post by monstrado »

_JamesA_ wrote: post #80
taftster wrote: post #62

Earlier quoted context omitted.

Agreed. So here's an attempt to describe NiFi at a high level. Fundamentally NiFi is a "dataflow engine", a system that can be used to automate data transfer from different and varying types of sources and sinks. It has a fairly usable UI that enables a "dataflow manager" (end user) to perform transformation, routing and delivery of data using a "drag-n-drop" configuration approach. Getting data into or out of your a…

Is this a layer on Apache Camel [1] or something completely different? [1]: https://camel.apache.org/

It was built from scratch at the NSA and open sourced a few years ago.

Re: Apache NiFi

#83

Post by monstrado »

ckdarby wrote: post #76
monstrado wrote: post #64

Earlier quoted context omitted.

I fine that explaining NiFi tends to be difficult, especially if you're an advanced user. This is because NiFi is extremely general purpose. I've used it for many different types of projects. That being said, here's a few use cases that are relatively easy to get started. 1. Move data from A to B 2. Move data from A to B, but perform some intermediate processing in-between (ETL) 3. Grab data from A, write it to [INSE…

Isn't most of this just Kubernetes running PrestoDB/PrestoSQL?

Not sure what you mean. PrestoDB is an analytics data store. The only correlation between NiFi and Presto would be if you wrote data to Presto using NiFi.

Re: Apache NiFi

#84

Post by hestefisk »

Does NiFi still use XML output for its source code? It makes it very hard to put under source control. Overall it’s a nice tool and very fast.

Re: Apache NiFi

#85

Post by hestefisk »

gregw134 wrote: post #13
corndoge wrote: post #3

Can someone explain what this is? I can't find anything on the website that explains it

Sure, I've worked with it. It let's you visually build data pipelines. It's extremely useful for getting work done quickly--you just drag and drop prebuilt connectors to things like elasticsearch, s3, or twitter and you have a data pipeline, including automatic backoff and the ability to inspect the data at each step. It's visual so it's easy to tell what's going on. Biggest downside is it's not automatically distrib…

Why would one consider distributing it in the first place? IMHO that would require distributed transactions, which would make it hugely complex.

Re: Apache NiFi

#86

Post by advaita »

tspann wrote: post #34

No more XML. Check out NiFi 1.11.4, it does everything you need for easy ingest. If you are reading some files putting them into Kafka or S3 or a database or MongoDB or Hbase or Hive or Impala or Oracle or Kudu or ..., it's genius. https://www.datainmotion.dev/

As far as I see, your link has nothing to do with 1.11.4 release. May be you wanted to link to a specific page?

Re: Apache NiFi

#88

Post by JimmyRuska »

mistrial9 wrote: post #72
zokier wrote: post #36

Earlier quoted context omitted.

> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly exp…

it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"

I don't think it's a prejudice. I've seen pentaho introduced as a way to enable business analysts to do work otherwise expensive developers would have to do. It simply didn't work, business analysts were not able to use it, devs had to take over and when they did they did not want to use pentaho. It's an easy trap for upper management to fall into thinking the selling point of these tools is the ability to allow the non tech staff to do the programming.

Re: Apache NiFi

#89

Post by skywhopper »

mistrial9 wrote: post #72
zokier wrote: post #36

Earlier quoted context omitted.

> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly exp…

it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"

That seems to be the thrust of the comment you’re replying to: the graph is essentially a programming language, with all the same complexity but multiplied because the same level of complexity is being managed without the same level of sophistication of tools like version control and even basic code re-use constructs typical in text-based languages.

Re: Apache NiFi

#90

Post by tspann »

advaita wrote: post #86
tspann wrote: post #34

No more XML. Check out NiFi 1.11.4, it does everything you need for easy ingest. If you are reading some files putting them into Kafka or S3 or a database or MongoDB or Hbase or Hive or Impala or Oracle or Kudu or ..., it's genius. https://www.datainmotion.dev/

As far as I see, your link has nothing to do with 1.11.4 release. May be you wanted to link to a specific page?

https://www.datainmotion.dev/2019/11/exploring-apache-nifi-1...
Post reply on HN