Ask HN: What do you use to move (copy) data between systems?

#1

Post by willnewby » Mon, Jun 08, 2020, 4:23 PM UTC

Do you centralize your data? Do you run an iPaaS of some kind? Do you like it?

Context: I'm currently in the middle of a project where I'm hooking up an ERP system (SalesPad) to an eCommerce website (WooCommerce) and I've found I need to move many different kinds of data between them (Inventory/Orders/CustomPricing/etc), and I haven't found a good tool so I started building one.

Has anyone else run into this? How did you solve it? Did you like your solution?

Re: Ask HN: What do you use to move (copy) data between systems?

#2

Post by linsomniac » Mon, Jun 08, 2020, 4:49 PM UTC

Many of these sorts of things we will push to an S3 bucket, and have other parts of the system pull from that bucket. We also use "rsync" with SSH keys to copy some data. Some of our vendors provide data to us via FTP, similarly it could be provided via HTTP

Maybe Zapier can help?

Re: Ask HN: What do you use to move (copy) data between systems?

#3

Post by Jugurtha » Mon, Jun 08, 2020, 5:56 PM UTC

Can you configure database replication so your database handles the transfer to the target system, and then make the ecommerce site accessible for orders? The two databases will be kept in sync.

Re: Ask HN: What do you use to move (copy) data between systems?

#4

Post by toomuchtodo » Mon, Jun 08, 2020, 6:58 PM UTC

Have you checked out http://n8n.io? Sounds like you’re just pushing JSON blobs around on a bus.

Re: Ask HN: What do you use to move (copy) data between systems?

#5

Post by sitkack » Mon, Jun 08, 2020, 7:01 PM UTC

rsync, rclone

really depends on what formats the data is in, how it will be used, how long it needs to last, how long the code that reads and writes the data will last, how much out of band information vs inband information will be sent along with the data.

You talked about movement, movement is the last of your worries, formats and provenance should rank higher on your list of concerns.

My gotos are sqlite, xml and avro. Try to keep things under 2GB so you don't run into lame data transfer issues.

Re: Ask HN: What do you use to move (copy) data between systems?

#6

Post by nailer » Mon, Jun 08, 2020, 7:24 PM UTC

rsync. Because it's efficient and well understood and emphasises accuracy.

Re: Ask HN: What do you use to move (copy) data between systems?

#7

Post by alexchantavy » Mon, Jun 08, 2020, 7:45 PM UTC

To add to the rsync answers, I think robocopy in a Windows environment is great.

Re: Ask HN: What do you use to move (copy) data between systems?

#8

Post by eb0la » Mon, Jun 08, 2020, 8:53 PM UTC

Take a look at kafka ;-) Just put the file path as the key and the content as the message value.

You get TTL/size garbage collection for free: no need to get/develop a sophisticated client/server.

Re: Ask HN: What do you use to move (copy) data between systems?

#9

Post by beeve » Wed, Jun 10, 2020, 7:41 AM UTC

Lots of data integration software is available. In the open source world, Apache Nifi and Apache Camel are two examples.