Having Salesforce, Zendesk, Hubspot and Pipedrive that share the same data in one company I would immediately start cutting costs instead of thinking about integration. Even if I had to think about it, the only correct approach for data propagation is directly from source of truth (or in "colonial" terms - master system) to dependent systems, and the only middleware in between them being data delivery (simple queue or ESB, workflow automation platform like Camunda or Zapier etc).
We've been thinking about software integration wrong for the past 50 years
11–20 of 40 posts
Re: We've been thinking about software integration wrong for the past 50 years
#12Hey Matt, it's a really neat product idea. I'd really recommend taking a niche and building it for that first. Selling a new Enterprise-wide integration tool is hard. Selling a new data orchestration concept will be harder. I recently moved out of that area, but I worked in this space for about a decade.
I can already see the rejections:
Debugging and Support: Rather than a centralised support system with levels, to support this you'd need a technical user familiar with the integration, or have the field mappings so well-documented and available a level 2 support rep could figure it out. The latter will never happen.
Brittle: This introduces a lot of risk to the organisation, as understanding conditions of failure and what the repercussions are in the event one of those services stops working becomes.. Complicated. There's a reason authN/authZ providers have opted for hub & spoke model, and that's because you need a highly specialised, often under-resourced team to look after it. Sound familiar? If you've worked in integration, it will.
Development and Resourcing: Like any proprietary product, this is another thing to hire for, and support. The uplift required for someone to understand this concept, and run with it for the use-case you've offered which is adopting it across the entire organisation requires.. Executive-level buy-in. To put it plainly, adopting this would introduce too much risk to the business in its current form.
I'd recommend taking a smaller bite with your approach. You're one person. Take a niche, and build something people want for that - people don't want technology for the sake of technology anymore. That money's dried up, and if you're trying to form a business off this concept - you'll need deep pockets to develop it.
Here's one that could be relatively successful: Take SIEM tooling, for example. For small and medium-sized businesses, operating a SIEM can be a massive undertaking. Usually the first thing an organisation wants to understand when they look to do this is: what audit logs do we have, and how can we look at audit logs for a user or employee across a range of services. Every company operating in a regulated environment, or company wanting to ensure they understand what an employee did during their last 2 weeks of offboarding would be interested in this.
If you built a product and demo purely to extract user/employee audit logs from services and expose them in this interface - I'd be lining up to try it.
Re: We've been thinking about software integration wrong for the past 50 years
#13This might be functional for a startup with an exit horizon of 2 years and nobody needs to inherit/own the mistakes of green biz-ops people who don't know what they're doing.
I've had to integrate new functionality with disasters like this and it is always a huge pain to support. Especially when the one guy maintaining the zendesk application leaves the company and we're like, "what is the purpose of this zendesk app in the django monolith doing?" and six months after we update it to support the new thing we added, we find out that django app's functionality was superceeded by xyz a year ago.
If I found out the mission critical biz ops side of the company was run like this I would not accept the offer.
Re: We've been thinking about software integration wrong for the past 50 years
#14This is a neat idea, but the unfortunate reality is that every link in an "integration chain" is almost guaranteed to be lossy. The author notes an obvious version of this (what if one node doesn't support a field that the other nodes do) but there are countless other types of loss in each link - structure (i.e. document data types), precision, limits (max field size, max number), other "exotic" datatypes (geography, binary blobs). Even mutability can cause lossiness - some of the fields in these systems are write-once, and even if creates and updates can be propagated, delete propagation is a Wild West in SaaS applications.
If you're old enough to remember what happens when you make a repeated analog copy of something (fax machines, copy machines, VHS cassettes, etc) you know what happens with this kind of data flow topology. We advise our customers to do the /exact/ opposite of this - follow as few links as possible for any piece of data and move towards hub-and-spoke, ideally with the hub as a highly configurable, low loss node - in my world, this is a data warehouse or data lake, but it might also mean an iPaaS or similar.
Re: We've been thinking about software integration wrong for the past 50 years
#15It may not be immediately obvious, but this is basically another cri de coeur to the effect of "Why can't we make the Semantic Web work?"
That's a good question. But you need to start from the fact that people have actually tried really, really hard, rather than nobody having tried at all, and it in fact hasn't worked despite rather substantial investment. When you start from there, you may obtain some useful understanding. Perhaps even useful understanding that will lead to solving some of the problem. I've got no problem with people jousting with windmills, I just think that anyone who wants to take such a task on really ought to research previous jousting attempts first if they want to succeed (as opposed to just doing it for fun).
But if you start from the blatently false presumption that you're the first person to ever dream of data that can just flow between systems, well, yeah looking out into the world you're going to be confused at what you see.
We have in fact tried the "correct" way, and it turned out not to be correct. Whereas what we're doing now, however annoying it definitely is, does in fact at least partially work.
Re: We've been thinking about software integration wrong for the past 50 years
#16Hey Matt, it's a really neat product idea. I'd really recommend taking a niche and building it for that first. Selling a new Enterprise-wide integration tool is hard. Selling a new data orchestration concept will be harder. I recently moved out of that area, but I worked in this space for about a decade. I can already see the rejections: Debugging and Support: Rather than a centralised support system with levels, to…
> it's a really neat product idea.
The one thing this idea is not is neat, though. It's the opposite of it. It entrenches the opposite of neat.
Re: We've been thinking about software integration wrong for the past 50 years
#17Having Salesforce, Zendesk, Hubspot and Pipedrive that share the same data in one company I would immediately start cutting costs instead of thinking about integration. Even if I had to think about it, the only correct approach for data propagation is directly from source of truth (or in "colonial" terms - master system) to dependent systems, and the only middleware in between them being data delivery (simple queue o…
Not really, usually you have some translation to ensure data from different systems is aligned/translated into a unified format. It’s expensive and cumbersome to do this in every dependent downstream system over and over again. Especially if upstream systems change, you want to implement that change in one central place instead of all downstream systems.
I think there’s a lot of value in the snowflake approach: put all your enterprise data in a single logical data layer, where you can easily share that data with different tenants. Transformation and unification can be handled by the data layer.
Systems get customized a lot: sap installations have a lot of company specific fields and objects, same for most other enterprise systems. Having all those customizations effect a network of connected systems is hard to manage. Better to dump all data into an enterprise wide data layer, and then let every tenant take whatever it needs. Also will provide for more flexible access to the data (sorting, pagination, filtering) and better performance.
Re: We've been thinking about software integration wrong for the past 50 years
#18It may not be immediately obvious, but this is basically another cri de coeur to the effect of "Why can't we make the Semantic Web work?" That's a good question. But you need to start from the fact that people have actually tried really, really hard, rather than nobody having tried at all, and it in fact hasn't worked despite rather substantial investment. When you start from there, you may obtain some useful underst…
The semantic web did work. Google was able to enrich their own searches and bought a large semantic web called Freebase and incrementally enriched their web searches . When they can’t show you a relative ad they want to capture your search regardless and show you relevant info so you don’t switch providers.
Re: We've been thinking about software integration wrong for the past 50 years
#19It may not be immediately obvious, but this is basically another cri de coeur to the effect of "Why can't we make the Semantic Web work?" That's a good question. But you need to start from the fact that people have actually tried really, really hard, rather than nobody having tried at all, and it in fact hasn't worked despite rather substantial investment. When you start from there, you may obtain some useful underst…
The semantic web did work. Google was able to enrich their own searches and bought a large semantic web called Freebase and incrementally enriched their web searches . When they can’t show you a relative ad they want to capture your search regardless and show you relevant info so you don’t switch providers.
If the "semantic web" worked, they wouldn't have had to buy a startup. You don't need a startup to read kindly-provided, accurate, well-categorized RDF tuples out of a web page.
One of my favorite ironies of the "semantic web" is almost every story cited as a success story by its advocates is in fact a failure story, in that it involves somebody having to do something that wouldn't have been necessary if the semantic web actually existed.
By the way, LLMs completely finish killing off the semantic web. The semantic web will never take the form of the entire world providing nice public RDF snippets. You "just" feed the real-world goo into an LLM (or its sequels) and extract what you need. It turns out that it was literally easier to solve the language comprehension problem than to implement the semantic web. (Even if you don't consider the language comprehension problem solved yet, we are well on track to having it solved before the semantic web is implemented.)
Re: We've been thinking about software integration wrong for the past 50 years
#20Having Salesforce, Zendesk, Hubspot and Pipedrive that share the same data in one company I would immediately start cutting costs instead of thinking about integration. Even if I had to think about it, the only correct approach for data propagation is directly from source of truth (or in "colonial" terms - master system) to dependent systems, and the only middleware in between them being data delivery (simple queue o…
Not really, usually you have some translation to ensure data from different systems is aligned/translated into a unified format. It’s expensive and cumbersome to do this in every dependent downstream system over and over again. Especially if upstream systems change, you want to implement that change in one central place instead of all downstream systems. I think there’s a lot of value in the snowflake approach: put a…
I fully agree with you and what you said is basically the same as my comment. You probably misunderstood it. What I meant is that logical data layer (I call it data delivery, which of course includes translation) should not serve any other function, i.e. be some system with business logic other than translation.