Best way to build a integration solution for a SaaS platform

#1

Post by aschema_123 » Thu, Dec 08, 2022, 3:32 AM UTC

We're building integrations for our customers to extract data from legacy on-premise or cloud-hosted, but still old sources (think medical records, HR record servers, etc), centalize it, display it, etc.

These tools each have convoluted HTTP REST / SOAP APIs

How would you go about architecting a solution that allows a customer to self-serve connect to their sources and our product would then ingest and cache the data to be served to them in (sometimes) synthesized form?

We're looking at ETL tools etc. but any thoughts come to mind here?

Re: Best way to build a integration solution for a SaaS platform

#2

Post by pqyzwbq » Thu, Dec 08, 2022, 6:33 AM UTC

Maybe you can check:

  - https://github.com/turbot/steampipe (which will query the sources on demand): Use SQL to instantly query your cloud services (AWS, Azure, GCP and more). Open source CLI. No DB required.

  - https://github.com/cloudquery/cloudquery (which will retrieve data in batch, and save them into another database for query)

Re: Best way to build a integration solution for a SaaS platform

#3

Post by creidHyyphen » Thu, Dec 08, 2022, 5:32 PM UTC

Hey! My startup company (https://hyyphen.io) is building a workflow automation platform that addresses this need.

Not looking to cold sell my product - but I'd definitely be keen to chat with you and learn more about your use case(s). If there's a fit, great, and if not it'd still be valuable just to understand what you're looking to accomplish.

If you're interested, shoot me an e-mail - creid@hyyphen.io

Re: Best way to build a integration solution for a SaaS platform

#4

Post by ensemblehq » Fri, Dec 09, 2022, 8:04 AM UTC

The general principles of ETL/ELT I assume would apply: identify the source, determine the requirements (I.e. frequency, volume, formats, etc.), determine the transformations, and ingest. You’ll also need to think about how often you need to ingest and the sync patterns between your source and your target.

Other considerations include security for PII, HIPAA (given you’re dealing with medical records).