Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My biggest mistake as an RPA developer

anupam.de

Post reply on HN
Print view View original

61–70 of 72 posts

Re: My biggest mistake as an RPA developer

#61

Post by burnte »

xorcist wrote: post #11

Please explain to me what the use of RPA is. Here's my background in the subject: I have met with team of a dozen people that spent half a year doing something that turned out to be the functional equivalent on an SQL one-liner. They could not within our meeting slot accurately describe what it is they were doing, apart from "automating tedious work" (yes, but what is that work, specifically?) and kept insisting I wa…

I was just involved in a $50k RPA project. We have an EMR that the buyer of our facilities wanted data from. The EMR vendor gave us two choices, a SQL dump of everything (which we took but would take many months to decode what tables did what, basically reverse engineer the storage process) or let them export the subset of data we needed. The first was free, the second was $60k and would take 6 months. Knowing that vendor, it would take closer to 9-12 due to how backed up they are and how bad they are at time estimation.

Enter a third party vendor I'd worked with before. They specialize in archiving data from EMRs into a standalone searchable database. $50k and two months, they're done. The EMR has an IE only UI, and a newer cross platform UI. We demonstrated how to extract the data required for the new owner, and they created an RPA workflow to "print" to PDF 5k medical records. It's the exact same output the EMR vendor would have given us, but they'd have used their own internal tools to do it, but they'd have taken 3-6 times longer to do it.

RPA is just screen scraping, basically, and sometimes it's still useful.

Re: My biggest mistake as an RPA developer

#62

Post by proctrap »

honey-badger wrote: post #50
ajcp wrote: post #23

"Most native RPA automation is GUI based. But let us take a moment so that this sinks in. GUI based automation involves instructing a bot to communicate with another program via the UI. This is analogous to forcing two native speakers to communicate via charades. GUI based automation is always a compromise because there is invariably a more efficient way to perform the same task under the hood. This brings me to my s…

Author here. I agree entirely with what you say here. But consider this. I once automated an RPA process to create service records for the customer service department. I did this by automating over a Java based UI app, with reliable selectors as you correctly point out. However, that app's UI would keep changing due to updates from the IT department, over which the customer service department had no control over. Thi…

True, there is (hopefully) always something like a DB on the other end, containing the data in a a fashion that makes it much easier to operate on (sql).

But is that really RPA at that point ? Suppose your task is to scratch some data from a website, calculate something and if some condition is true also perform some form submit.

If you do have access to the actual database you can just use that. If you also have access to the API of the system, you have to "check some box", you might as well just write a normal application that calls some SQL, perform some http-posts and maybe also provides a nice website with that data it crawled, formatted in the way you wanted to calculate the data, in, let's say excel. (Some would say you wrote a microservice.) And if all of this is possible then you're apparently good enough in programming that such a solution is the way to go.

But if you're in the typical corporate environment where the API doesn't exist / you can't access the database / you're not allowed to interact with the checkbox except for the proprietary application, then you're back to what an RPA was supposed to do.

Because otherwise we're down to one question: When is it an RPA ? When you're just using a fancy UI ? When you're not accessing any API ? When you don't script ? Or is the example above with SQL + API also an RPA ?

RPA to me rings "mouse recorder for dummies" or "ugly hack that breaks depending on how good your visual detection system is". At least when you described discovering that you can actually just use real scripts instead of UI indirection, it reads like something like that. A "not invented here syndrome" of that RPA tool, instead of a helper that handles checkbox-detection + clicking for you.

Re: My biggest mistake as an RPA developer

#63

Post by 8organicbits »

xorcist wrote: post #11

Please explain to me what the use of RPA is. Here's my background in the subject: I have met with team of a dozen people that spent half a year doing something that turned out to be the functional equivalent on an SQL one-liner. They could not within our meeting slot accurately describe what it is they were doing, apart from "automating tedious work" (yes, but what is that work, specifically?) and kept insisting I wa…

I've seen this pop up where team A owns a web app (likely legacy) and team B has heavy and increasing use of the app. Team B realizes that they spend so much time shuffling data in and out of the app and realizes, team A should build an API so we can automate process and reporting.

So a tentative plan forms. Team B meets team A and asks for the API. Team B seeks funding to code on top of the planned API. But it falls through, team A doesn't have time/budget for the project, isn't interested in the idea, or otherwise rejects the idea. This is especially true if team A has other larger, more important users.

Team A regroups and tries RPA to make their own programmatic interface on team B's app without their help. The RPA promise that non-programmers can automate their own jobs is also compelling, and is seen as a cost savings.

What I've seen happen next is that nothing really happens. End users are capable of writing RPA, but don't want to. Any automation is flakey, and may not actually reduce costs. Most folks continue getting paid to shuffle data, who'd want to automate away their own paycheck?

Also, since team B didn't want to build an API and isn't part of the effort, there's a bunch of ugly issues: no programmatic contract/interface, no docs, no versioning, no backwards compatibility, no forward notice of changes, no access to test environments. So the automation can unexpected break or begin corrupting data, it needs regular monitoring.

As a developer, that sounds terrible. But if there's no other way to automate, it's attractive.

Re: My biggest mistake as an RPA developer

#64

Post by 8organicbits »

ragebol wrote: post #29

Can someone explain to me where the robots in RPA are? It's just scripting for UIs, the way I understand it

Here's an example (not mine). This was likely built via UI, so this is generated XML.

https://github.com/rohanbaraskar/UiPath-11/tree/master/Autom...

There's not much RPA code on GitHub, which seems like a red flag to me. I suspect there's more hype than actual work happening.

Re: My biggest mistake as an RPA developer

#65

Post by honey-badger »

burnte wrote: post #61
xorcist wrote: post #11

Please explain to me what the use of RPA is. Here's my background in the subject: I have met with team of a dozen people that spent half a year doing something that turned out to be the functional equivalent on an SQL one-liner. They could not within our meeting slot accurately describe what it is they were doing, apart from "automating tedious work" (yes, but what is that work, specifically?) and kept insisting I wa…

I was just involved in a $50k RPA project. We have an EMR that the buyer of our facilities wanted data from. The EMR vendor gave us two choices, a SQL dump of everything (which we took but would take many months to decode what tables did what, basically reverse engineer the storage process) or let them export the subset of data we needed. The first was free, the second was $60k and would take 6 months. Knowing that v…

Precisely. In some narrowly defined scenarios where data migration is otherwise like untangling a giant ball of yarn, RPA serves an invaluable need. However, the manner in which it is marketed is out of proportion to the narrower scope in which it is valuable.

Re: My biggest mistake as an RPA developer

#66

Post by honey-badger »

_dain_ wrote: post #56
honey-badger wrote: post #41

Earlier quoted context omitted.

Amen! RPA is also about democritizing IT by handing (ideally) every white-collar worker the ability to write scripts that can automate pieces of their daily-work. Of course, those solutions aren't likely to be elegant and their code won't be clean, but since their scope is tiny, that isn't a problem.

>handing (ideally) every white-collar worker the ability to write scripts that can automate pieces of their daily-work. surely much of this could be done already with a tool like AutoHotKey? I mean it won't have some of the fancier stuff like screen scraping, but it gets you 80% of the way there in a lot of cases. the barrier is not the tools, it's the IT policies that prevent ordinary workers from being allowed to u…

It's also a marketing problem. You need a solution that appears sexy to the worker, the boss and a champion high enough in the organization to overrule outdated IT policy.

I think RPA exists because every organization is plagued by people management problems that developers and IT often fail to acknowledge.

Re: My biggest mistake as an RPA developer

#67

Post by honey-badger »

proctrap wrote: post #62
honey-badger wrote: post #50

Earlier quoted context omitted.

Author here. I agree entirely with what you say here. But consider this. I once automated an RPA process to create service records for the customer service department. I did this by automating over a Java based UI app, with reliable selectors as you correctly point out. However, that app's UI would keep changing due to updates from the IT department, over which the customer service department had no control over. Thi…

True, there is (hopefully) always something like a DB on the other end, containing the data in a a fashion that makes it much easier to operate on (sql). But is that really RPA at that point ? Suppose your task is to scratch some data from a website, calculate something and if some condition is true also perform some form submit. If you do have access to the actual database you can just use that. If you also have acc…

Well, the truth is that developers often underestimate how hard it is for non-developers to write scripts. The reason why they don't understand why something like RPA exists is also known as 'The curse of knowledge'.

Re: My biggest mistake as an RPA developer

#68

Post by honey-badger »

_dain_ wrote: post #54
honey-badger wrote: post #38

Earlier quoted context omitted.

Author here. In an ideal firm, where data is handled responsibly, you would not need RPA at all. In large organizations though, data tends to accumulate in monolithic islands that are only exposed to each other with legacy UI based applications. Moving this data tends to make up a large bulk of white-collar jobs in large organizations. RPA replicates the work that these workers do by automating on the UI of the legac…

>RPA replicates the work that these workers do by automating on the UI of the legacy application and replacing them. It is championed by ambitious business professionals who >1) Don't know that a more elegant solution exists in the traditional IT world >2) Don't receive enough support from the IT org to solve this business problem that is crucial to them. >Therefore, such business professionals collaborate with consu…

That is interesting! It rings true, but I never looked at it that way.

Re: My biggest mistake as an RPA developer

#69

Post by NikolaNovak »

dragonwriter wrote: post #33
NikolaNovak wrote: post #14

Earlier quoted context omitted.

Sometimes it's the unobvious, non-technical things. FWIW: In my world, we have native tools and methods as part of our ERP application, that would accomplish certain automation tasks more efficiently and effectively than RPA. But that would constitute a "change of application" (because it would be); change management and sign-off on these is so onerous; and the perceived risk and potential impact so high; that they n…

> RPA sales pitch to executives is "This is not changing your application; this is simply doing what your people are already doing in your application, faster and more repeatable with less errors". > This is a powerful risk-reduction pitch Is it? It seems to me a powerful risk creation pitch, where instead of dealing with the process problems that make properly controlled change expensive, you just exploit a loophole…

When you buy commercial off the shelf software such as any ERP, it is not "wrong internal policies / incorrect process" or "loopholes exploited" (notion I've seen in couple of comments here) but a design reality of the mode you're operating in.

The more you customize a COTS application, the harder your support gets and the more pain you get when you patch or upgrade.

Building on top of it is practically as well as procedurally safer. It is, legitimately, reduced risk and impact. You keep the solid stable core that is fully vendor supported, and you have the piece that you built and own neatly separated.

In that way you can almost think of rpa as a shell script: You don't customize the kernel every time you need something done. You leave kernel well alone (and reduce both risk, and pain of future os upgrades) and build on an easier language something you own on top of it that interacts with it.

Note I don't do rpa and my team doesn't do rpa, but I've seen it around and seen it's value. It does feel I'm talking To folks who want to reduce this to "management sucks" as opposed to any effort to try to actually understand a technology's Place in ecosystem though...

Re: My biggest mistake as an RPA developer

#70

Post by burnte »

honey-badger wrote: post #65
burnte wrote: post #61

Earlier quoted context omitted.

I was just involved in a $50k RPA project. We have an EMR that the buyer of our facilities wanted data from. The EMR vendor gave us two choices, a SQL dump of everything (which we took but would take many months to decode what tables did what, basically reverse engineer the storage process) or let them export the subset of data we needed. The first was free, the second was $60k and would take 6 months. Knowing that v…

Precisely. In some narrowly defined scenarios where data migration is otherwise like untangling a giant ball of yarn, RPA serves an invaluable need. However, the manner in which it is marketed is out of proportion to the narrower scope in which it is valuable.

I could say that about every technology. :)
Post reply on HN