Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache NiFi

nifi.apache.org

Post reply on HN
Print view View original

111–120 of 140 posts

Re: Apache NiFi

#111

Post by PeterStuer »

taftster wrote: post #62
nightowl_games wrote: post #59

Earlier quoted context omitted.

This page is classic Apache project in that I read it and have no idea what it does. Can you high level explain what this thing is really for?

Agreed. So here's an attempt to describe NiFi at a high level. Fundamentally NiFi is a "dataflow engine", a system that can be used to automate data transfer from different and varying types of sources and sinks. It has a fairly usable UI that enables a "dataflow manager" (end user) to perform transformation, routing and delivery of data using a "drag-n-drop" configuration approach. Getting data into or out of your a…

So is this comparable to something like the old Microsoft Biztalk?

Re: Apache NiFi

#112

Post by pjmlp »

RhodesianHunter wrote: post #106
chii wrote: post #97

Earlier quoted context omitted.

There's a general level of elitism in the programming world. GUI tools are seen by some programmers as a "worse" tool, but without any real evidence, and it's likely due to the fact that GUI tools enable non-programmers to reach some level of output that used to be the sole domain of programmers. It's a similar train of thought that leads to a lot of game developers to look down on games made with GUI tools like game…

Can we claim that it's elitism when it's born from experience? Programmers have seen these sorts of UIs fail to deliver countless times as described in the root of this thread. Where is the counter example?

Unreal, Unity, Substance, Octane Render, and Scratch.

Re: Apache NiFi

#113

Post by pjmlp »

chii wrote: post #97
mistrial9 wrote: post #72

Earlier quoted context omitted.

it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"

There's a general level of elitism in the programming world. GUI tools are seen by some programmers as a "worse" tool, but without any real evidence, and it's likely due to the fact that GUI tools enable non-programmers to reach some level of output that used to be the sole domain of programmers. It's a similar train of thought that leads to a lot of game developers to look down on games made with GUI tools like game…

Not all game developers, AAA game engines have lots of visual languages specially for what the game designers and artists end up using.

Re: Apache NiFi

#114

Post by lordlinus »

mberning wrote: post #4

We have a team using this at work. They had built a process and needed it to be put on a VM and run periodically. They said the requirements were a dual core machine and 8gb of ram. The “binary” was like 1.8gb. I’m sure it included a jre and a full nifi runtime, but god damn that is ridiculous. Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and…

The reason the binary is huge because it contains all the jar/nar for the connectors ( https://www.nifi.rocks/apache-nifi-processors/). If you only require a subset of them you can use the minifi version which is quite small ( 49 MB for java and 3.2 MB for c++ version)

Re: Apache NiFi

#115

Post by mindentropy »

taftster wrote: post #62
nightowl_games wrote: post #59

Earlier quoted context omitted.

This page is classic Apache project in that I read it and have no idea what it does. Can you high level explain what this thing is really for?

Agreed. So here's an attempt to describe NiFi at a high level. Fundamentally NiFi is a "dataflow engine", a system that can be used to automate data transfer from different and varying types of sources and sinks. It has a fairly usable UI that enables a "dataflow manager" (end user) to perform transformation, routing and delivery of data using a "drag-n-drop" configuration approach. Getting data into or out of your a…

Is this similar to NodeRed for IoT? I was trying to bring up something similar for IoT with CEP rules and source and sink connections on the Edge.

Re: Apache NiFi

#116

Post by drbw »

EvanAnderson wrote: post #63
dmix wrote: post #56

Earlier quoted context omitted.

Doesn't Microsoft Access have a similar interface? It's been a long time since I've used it.

Based on a quick look at NiFi, SQL Server Integration Services (SSIS) seems similar. The configuration management tooling for SSIS is pretty good and it's amenable to version control better than, it sounds like, NiFi is. SSIS still suffers from the potential "gotchas" the top-level poster mentions.

I haven't used SSIS since about SQL2008 - back then it terrible to use with version control - not only was it a huge blob of XML, it had more xml escaped and shoved into attributes of the main document! Whats more it seemed to re-allocate the GUIDs of the elements every time you opened a diagram, so there were always changes...

Sounds like it's improved a bit since then.

Re: Apache NiFi

#117

Post by atoav »

mistrial9 wrote: post #72
zokier wrote: post #36

Earlier quoted context omitted.

> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly exp…

it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ? I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"

I am working with graphical tools and node based structures for years. Think about Blenders shader nodes or modular synthesizers. These systems are good to play around, try things out, quickly change how things are connected.

Above a certain complexity they become very hard to organize. If the complexity of your problem is below this level and the user has any self control, all is good. If the complexity of your problem is higher you'd need a programmer who knoes how to handle this anyways.

Re: Apache NiFi

#118

Post by ashtonkem »

cs02rm0 wrote: post #25

I've used it a fair bit, though not for a couple of years. Few points, some of which may be out of date: * I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi. * It persisted its config to an XML file, including the positions of boxes on the UI. Trying to keep this config in source control with mu…

The idea that some sufficiently smart compiler/language/tool will replace expensive developers is a siren song that’s as old as COBOL. Most of these tools have historically failed to deliver much (if any) value. In the few cases where they genuinely made programming easier, the bar for what computers were expected to accomplish was raised to the point where specialized labor was once again required.

Re: Apache NiFi

#119

Post by zmmmmm »

monstrado wrote: post #82
_JamesA_ wrote: post #80

Earlier quoted context omitted.

Is this a layer on Apache Camel [1] or something completely different? [1]: https://camel.apache.org/

It was built from scratch at the NSA and open sourced a few years ago.

I'm curious how you would compare it to Apache Camel - when would you use Camel and when would you use this?

We use camel with DSLs to make programmatic workflows that connect data flows together. However Camel itself doesn't typically carry the data. Sometimes it SFTPs files around etc., but mostly, it is just a messaging layer.

Is that the main difference here?

Re: Apache NiFi

#120

Post by blendo »

fmakunbound wrote: post #109

Reminds me of the early 2000s when we were all into BPM, graphical or otherwise. The drawbacks are pretty obvious. I bet the engineers who built it had fun, tho.

As a Java programmer back then, watching with horror as consultants brought in their flavor of the month (MQ/Spring/Camel), I could only pad my resume, and pretend to say "never look a gift horse in the mouth".
Post reply on HN