A novel approach to entity resolution using serverless technology
1–10 of 31 posts
Re: A novel approach to entity resolution using serverless technology
#2Hi, I’m one of the (prospective) co-founders of TiloDB, a serverless “entity-resolution” technology.
We built TiloDB as the tech team at a European consumer credit bureau when we were faced with the technical challenge of how to assemble hundreds of millions of data sets about tens of millions of people in a way that is scalable and allows fast searching, without breaking the bank.
We tried various technologies, such as graph databases, but none of them could give us satisfactory performance.
So we turned to the opportunities of serverless technology (AWS specifically) to build a new type of entity resolution technology.
In this article we write about the technology breakthroughs that led to TiloDB, and there is also an interactive demo where you can submit data, see it linked, and see other people submitting data in real-time.
We want to spin the tech out into a new company, release it as OSS, and so are keen to hear about potential use cases you might have.
Re: A novel approach to entity resolution using serverless technology
#3This blog post from the CTO of VMware and Springsource, gives a pretty good summary of the entity-resolution field: https://blog.acolyer.org/2020/12/14/entity-resolution/
Re: A novel approach to entity resolution using serverless technology
#4This blog are very useful and amazing thanks for sharing.
Re: A novel approach to entity resolution using serverless technology
#5from experience with similar product (where we had similarly sounding way of entity resolution based on rule based fuzzy indexes and fuzzy matching, and it was working for tens of millions of entities on regular, though beefy, RDBMS more than a decade ago) - the issue isn't that much technological, it is that each customer/client has custom everything when it comes to ER, and thus scaling that business is extremely hard (that specific business collapsed primarily for that reason)
Re: A novel approach to entity resolution using serverless technology
#6from experience with similar product (where we had similarly sounding way of entity resolution based on rule based fuzzy indexes and fuzzy matching, and it was working for tens of millions of entities on regular, though beefy, RDBMS more than a decade ago) - the issue isn't that much technological, it is that each customer/client has custom everything when it comes to ER, and thus scaling that business is extremely h…
the real technical challenge is the "transitive hop" problem that we describe. The matching of the data is not so complicated - that can be done with any technology - but searching with Data A, and getting result Z - that was the tricky bit that took us years to solve and was only possible thanks to serverless tech.
Re: A novel approach to entity resolution using serverless technology
#7Hi, I’m one of the (prospective) co-founders of TiloDB, a serverless “entity-resolution” technology. We built TiloDB as the tech team at a European consumer credit bureau when we were faced with the technical challenge of how to assemble hundreds of millions of data sets about tens of millions of people in a way that is scalable and allows fast searching, without breaking the bank. We tried various technologies, such…
Very cool project.
Setting up a business based on a new DB tech that has one user, though, is tricky.
Playing devil's advocate, how do you plan to make money? Who are the users, why do they turn to TiloDB, how do they learn about it, how do they adopt it, how do they be convinced to pay you something for it? Etc
Re: A novel approach to entity resolution using serverless technology
#8from experience with similar product (where we had similarly sounding way of entity resolution based on rule based fuzzy indexes and fuzzy matching, and it was working for tens of millions of entities on regular, though beefy, RDBMS more than a decade ago) - the issue isn't that much technological, it is that each customer/client has custom everything when it comes to ER, and thus scaling that business is extremely h…
I would really love to hear more about your experience regarding the client customizations. So far the two things I can see are domain model customization and rule customization. Obviously with the rule customizations being the more challenging one.
Re: A novel approach to entity resolution using serverless technology
#9Hi, I’m one of the (prospective) co-founders of TiloDB, a serverless “entity-resolution” technology. We built TiloDB as the tech team at a European consumer credit bureau when we were faced with the technical challenge of how to assemble hundreds of millions of data sets about tens of millions of people in a way that is scalable and allows fast searching, without breaking the bank. We tried various technologies, such…
Very cool project. Setting up a business based on a new DB tech that has one user, though, is tricky. Playing devil's advocate, how do you plan to make money? Who are the users, why do they turn to TiloDB, how do they learn about it, how do they adopt it, how do they be convinced to pay you something for it? Etc
Thanks for your question. You are right - it is not an easy business to start. Investors are more used to open source projects that are already released and have community adoption that they can measure. We are kinda the opposite - enterprise ready software that wants to go open source.
So we want to make the software open source, but restrict a few modules that would be necessary for enterprise customers, such as security and auditing features.
We have quite a few companies lined up that want to do proof of concept trials with us. So far they are mostly big fintech companies that use it for anti-fraud, also AML/KYC companies that need to match and search lots of data from different sources in real time. Also very large companies that need to solve their "data silo" problem.
Adoption - hopefully they start with the OSS version, play with then want to upgrade to the enterprise version.
One area we have less experience is with which type of OSS licence to use.
Re: A novel approach to entity resolution using serverless technology
#10from experience with similar product (where we had similarly sounding way of entity resolution based on rule based fuzzy indexes and fuzzy matching, and it was working for tens of millions of entities on regular, though beefy, RDBMS more than a decade ago) - the issue isn't that much technological, it is that each customer/client has custom everything when it comes to ER, and thus scaling that business is extremely h…
the real technical challenge is the "transitive hop" problem that we describe. The matching of the data is not so complicated - that can be done with any technology - but searching with Data A, and getting result Z - that was the tricky bit that took us years to solve and was only possible thanks to serverless tech.
in our case we didn't have explicit "transitive hop". Instead in this example it would be just one entity with a bunch of addresses (with related dates if configured so), and multiple known names, etc. attached. Granted in order to get to that state the data loading process would include a massive batch matching which would be performed on the array of worker nodes - the number could be configured for given run, so there is kind of rough technological similarity to your serverless.