Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A novel approach to entity resolution using serverless technology

tilodb.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: A novel approach to entity resolution using serverless technology

#21

Post by lmeyerov »

skafoi wrote: post #19
lmeyerov wrote: post #18

ER for identity graphs is a great use case! We see teams do this a lot and with not-great tools. (Ex: users/IPs in splunk/elastic, which are better for simpler matches.) For one Graphistry project, we run a single node neo4j with 0.5b nodes/edges, so something in the description isn't adding up for me here wrt perf. Maybe an open benchmark would help? I do agree indexing matters, as that was night/day for our use cas…

Regarding the performance on neo4j: the challenge for an honest and fair test towards this would be about how to properly compare a server-based solution vs. a serverless solution. TiloDB automatically scales up and down without any further interaction due to using Lambdas for all calculations. So would you compare it with a relatively small neo4j instance or with a large cluster? I honestly don't know. When we start…

Separate benchmark per claim and core use case :) A scale-to-zero + autoscaling graph db could be both broadly relevant and differentiated, so I'd be curious there + table stakes for regular queries.

RE:extremes, we see graph DBs OK for small time series (ex: 2 nodes with a bunch of event multiedges), but not full blown time series... where we'd use a tsdb. Some vendors demo this, but always felt like wrong tool.

The many-hop case is interesting! We don't see 1K-hops typically, and I get nervous even at 10-20 on graph DBs we've used. I can imagine in logistics or sciences that happening more, or maybe even some rdf systems. Partition keys start mattering fast, whether a kvdb or a mpp, but I don't have an intuition here. Probably easier to differentiate on, but too niche?

Re: A novel approach to entity resolution using serverless technology

#22

Post by skafoi »

lmeyerov wrote: post #21
skafoi wrote: post #19

Earlier quoted context omitted.

Regarding the performance on neo4j: the challenge for an honest and fair test towards this would be about how to properly compare a server-based solution vs. a serverless solution. TiloDB automatically scales up and down without any further interaction due to using Lambdas for all calculations. So would you compare it with a relatively small neo4j instance or with a large cluster? I honestly don't know. When we start…

Separate benchmark per claim and core use case :) A scale-to-zero + autoscaling graph db could be both broadly relevant and differentiated, so I'd be curious there + table stakes for regular queries. RE:extremes, we see graph DBs OK for small time series (ex: 2 nodes with a bunch of event multiedges), but not full blown time series... where we'd use a tsdb. Some vendors demo this, but always felt like wrong tool. The…

Thanks for your input.

1k hops is also not something we see on a regular basis in our old business, which is much about people moving houses and transactional data from payment service providers. Ppl with money issues seem to move a lot more often and also fraud cases often have a lot of hops.

Re: A novel approach to entity resolution using serverless technology

#24

Post by kall »

Major_Grooves wrote: post #9
willvarfar wrote: post #7

Earlier quoted context omitted.

Very cool project. Setting up a business based on a new DB tech that has one user, though, is tricky. Playing devil's advocate, how do you plan to make money? Who are the users, why do they turn to TiloDB, how do they learn about it, how do they adopt it, how do they be convinced to pay you something for it? Etc

Thanks for your question. You are right - it is not an easy business to start. Investors are more used to open source projects that are already released and have community adoption that they can measure. We are kinda the opposite - enterprise ready software that wants to go open source. So we want to make the software open source, but restrict a few modules that would be necessary for enterprise customers, such as se…

Just a quick suggestion: if you restrict the security module, do so in a way that someone can still run the OSS version in a basic secure way. If there are lots of insecure instances of your db out there, or someone else steps in and provides a solution, that doesn‘t reflect well on the project. This wasn‘t great about elasticsearch and they changed it later.

Re: A novel approach to entity resolution using serverless technology

#26

Post by boulos »

Major_Grooves wrote: post #9
willvarfar wrote: post #7

Earlier quoted context omitted.

Very cool project. Setting up a business based on a new DB tech that has one user, though, is tricky. Playing devil's advocate, how do you plan to make money? Who are the users, why do they turn to TiloDB, how do they learn about it, how do they adopt it, how do they be convinced to pay you something for it? Etc

Thanks for your question. You are right - it is not an easy business to start. Investors are more used to open source projects that are already released and have community adoption that they can measure. We are kinda the opposite - enterprise ready software that wants to go open source. So we want to make the software open source, but restrict a few modules that would be necessary for enterprise customers, such as se…

Disclosure: I used to work on Google Cloud.

I see a lot of similarities between Kafka and Confluent. You're looking to spin out a tool that worked well for you, and offer it commercially.

I'd suggest planning more around operating TiloDB as a managed service. You happen to just need lambda, s3, and dynamo today, but the "capturable" value for many customers will be if you also manage it all for them (especially upgrades). You can still offer the open core and let folks run their own, but it sounds like a lot of the goodness comes from the way you run it.

Having said that, licensing is currently fraught in this space. Each major "database" vendor (Elastic, Redis Labs, Confluent) is basically trying to find a way to figure out how to avoid AWS (and other clouds) from just taking their code and operating it as a service.

People have very strong opinions on this topic, ranging from "open-source isn't a business plan" to "AWS is violating the spirit of the OSS community" and many more. My personal advice would be to assess more clearly why you want to be open source (you mentioned community and applications you couldn't imagine) and whether you think open source better achieves those goals than say a free tier or distributing a core binary / container image for free.

What, more specifically, do you want to get out of being open source? Contributions to the core? Contributions to the operational part? More users and feedback?

Re: A novel approach to entity resolution using serverless technology

#27

Post by skafoi »

kall wrote: post #24
Major_Grooves wrote: post #9

Earlier quoted context omitted.

Thanks for your question. You are right - it is not an easy business to start. Investors are more used to open source projects that are already released and have community adoption that they can measure. We are kinda the opposite - enterprise ready software that wants to go open source. So we want to make the software open source, but restrict a few modules that would be necessary for enterprise customers, such as se…

Just a quick suggestion: if you restrict the security module, do so in a way that someone can still run the OSS version in a basic secure way. If there are lots of insecure instances of your db out there, or someone else steps in and provides a solution, that doesn‘t reflect well on the project. This wasn‘t great about elasticsearch and they changed it later.

The idea for that is, that typical enterprise features like authorization for certain records or even attributes are not publicly available. Also e.g. encryption of the data in S3 and other parts may be an enterprise only feature. Other things, like API authorization, preventing public access to S3 and therelike must be included in the OSS version for the same reasons you mentioned.

Re: A novel approach to entity resolution using serverless technology

#28

Post by Major_Grooves »

boulos wrote: post #26
Major_Grooves wrote: post #9

Earlier quoted context omitted.

Thanks for your question. You are right - it is not an easy business to start. Investors are more used to open source projects that are already released and have community adoption that they can measure. We are kinda the opposite - enterprise ready software that wants to go open source. So we want to make the software open source, but restrict a few modules that would be necessary for enterprise customers, such as se…

Disclosure: I used to work on Google Cloud. I see a lot of similarities between Kafka and Confluent. You're looking to spin out a tool that worked well for you, and offer it commercially. I'd suggest planning more around operating TiloDB as a managed service. You happen to just need lambda, s3, and dynamo today, but the "capturable" value for many customers will be if you also manage it all for them (especially upgra…

you make a very good point. In fact, we just came out of a call with a very knowledgable investor who wants us to go down exactly that route. It could very well be that the managed service route is the way to go. Thanks for the comment. If you'd like to connect directly, I think my email address is in my profile.

re OSS strategy - it's more users and feedback that I firstly think of. For instance, people keep telling us that there could be a really great use case in crypto compliance/auditing - tracking related wallet addresses etc. We don't really know enough about blockchain to validate that, but I think an OSS community could.

Re: A novel approach to entity resolution using serverless technology

#30

Post by smackeyacky »

Major_Grooves wrote: post #28
boulos wrote: post #26

Earlier quoted context omitted.

Disclosure: I used to work on Google Cloud. I see a lot of similarities between Kafka and Confluent. You're looking to spin out a tool that worked well for you, and offer it commercially. I'd suggest planning more around operating TiloDB as a managed service. You happen to just need lambda, s3, and dynamo today, but the "capturable" value for many customers will be if you also manage it all for them (especially upgra…

you make a very good point. In fact, we just came out of a call with a very knowledgable investor who wants us to go down exactly that route. It could very well be that the managed service route is the way to go. Thanks for the comment. If you'd like to connect directly, I think my email address is in my profile. re OSS strategy - it's more users and feedback that I firstly think of. For instance, people keep telling…

I think what you might want to do is try the closed source / SAAS route first either by locking in that 2nd customer or taking some investment. If it doesn't work, then start down the open source route.

It seems to me that all the knowledge / experience of your staff is your real asset here, not necessarily the code. While that, in theory, should mean open-sourcing the code won't matter for your business, in practice it means you will be seeding competitors unnecessarily.

Turning your staff into highly chargeable consultants could be a more sustainable business model than trying to herd the cats of the internet into trying to improve your product offering, when most of those guys won't have the experience of your existing team. By offering the code out as open source you are giving a bunch of people a leg up and cutting short your time as the only game in town, which puts extra pressure on sales and might not work out.

Post reply on HN