Grats on the launch! Would be very interested to see this working with GCP Cloud Run.
Thanks! It's on the roadmap :)
Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
41–50 of 56 posts
Re: Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
#42Congrats on the launch. I'll never forget at a previous company the CEO came to the engineering team one day to triumphantly announce "I've just signed X for $Y/m, if we give them a dedicated instance!", to which we all gasped in horror, knowing that a dedicated instance would cost at least 5x. If that sort of understanding can be better communicated throughout a company, particularly to sales, then that's great news…
Haha, we've heard many a story like this one over the months!
An advisor suggested we offer a sort of "pricing heuristics" deliverable to sales as part of our product (e.g. "if a customer needs feature foo, add $bar to the price"), and your story makes me think he's onto something.
Re: Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
#43Congrats on launching!
How does it differ from Spot.io?
Re: Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
#44how are you ingesting from kinesis to clickhouse. are you using some custome sink connector or processes on ec2 or lambda?
We actually use Kafka rather than Kinesis, although they're very similar. For writing to ClickHouse from Kafka, we use the ClickHouse Kafka sink connector: https://github.com/ClickHouse/clickhouse-kafka-connect .
we are actually trying something similar but possible kinesis + clickhouse or kafka + clickhouse. Currently kinesis seems easier to deal with but not a good intergration or sink connector available to process records at scale for kinesis to put into clickhouse. Were you ever felt into similar problems where you had to process records at huge scale to be able to insert into clickhouse without much delay.
One more thing is kinesis can have duplicates while kafka is exactly once delivery.
Re: Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
#45That our industry does basically no cost accounting is basically insane.
Your product is something that most managers don’t even realize they are missing.
I’m kind of obsessed with activity-based costing as applied to software firms. Hit me up at my username at gmail if you want to nerd out about managerial accounting and/or if you might need some marketing material! I could write for days on the subject!
Re: Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
#46Earlier quoted context omitted.
We actually use Kafka rather than Kinesis, although they're very similar. For writing to ClickHouse from Kafka, we use the ClickHouse Kafka sink connector: https://github.com/ClickHouse/clickhouse-kafka-connect .
we are actually trying something similar but possible kinesis + clickhouse or kafka + clickhouse. Currently kinesis seems easier to deal with but not a good intergration or sink connector available to process records at scale for kinesis to put into clickhouse. Were you ever felt into similar problems where you had to process records at huge scale to be able to insert into clickhouse without much delay. One more thin…
I'm not familiar with Kinesis's sink APIs, but yes I'd imagine you'll have to write your own connector from scratch.
To answer your question, though, no: in the Kafka connector, the frequency of inserts into ClickHouse is configurable relatively independent of the batch size, so you don't need massive scale for real-time CH inserts. To save you a couple hours, here's an example config for the connector:
# Snippet from connect-distributed.properties
# Max bytes per batch: 1 GB
fetch.max.bytes=1000000000
consumer.fetch.max.bytes=1000000000
max.partition.fetch.bytes=1000000000
consumer.max.partition.fetch.bytes=1000000000
# Max age per batch: 2 seconds
fetch.max.wait.ms=2000
consumer.fetch.max.wait.ms=2000
# Max records per batch: 1 million
max.poll.records=1000000
consumer.max.poll.records=1000000
# Min bytes per batch: 500 MB
fetch.min.bytes=500000000
consumer.fetch.min.bytes=500000000
If you're still deciding, I'd recommend Kafka over Kinesis because (1) it's open source so more options, e.g. self host or Confluent or AWS MSK and (2) it has a much bigger community, meaning better support, more StackOverflow answers, a plug-and-play CH Kafka connector, etc.
Re: Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
#47That our industry does basically no cost accounting is basically insane. Your product is something that most managers don’t even realize they are missing. I’m kind of obsessed with activity-based costing as applied to software firms. Hit me up at my username at gmail if you want to nerd out about managerial accounting and/or if you might need some marketing material! I could write for days on the subject!
Awesome!! We'll send you an email shortly :)
Re: Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
#48Congrats on launching! How does it differ from Spot.io?
Thank you! Spot.io falls broadly into the same category as Vantage/Ternary/others, so the same answer as here applies: https://news.ycombinator.com/item?id=39183504.
In a sentence, these tools display the same cost data available within AWS - where max granularity is per-database or per-EC2 instance - whereas, Dashdive can accurately attribute portions of usage on the same DB or instance to different features/customers.
Re: Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
#49How do you aim to attribute cost for multi-tenant services like databases? For example, if I have a database costing $100/m, and 4 equal sized customers (by data volume, compute, requests), you could say that each is costing $25/m. However if I'm over provisioned by 50%, then an incremental user is only actually $12.50. To understand this difference requires understanding resource utilisation, which particularly for…
TL;DR - We treat the per-customer usage data as the ground truth, and the per-customer cost can vary based on parameters chosen by the Dashdive user and their preferred mental model. At a minimum, every customer is assigned the cost resulting from the usage that is directly attributable to them. This "directly attributable" figure is obtained by "integrating" vCPU-seconds and RAM-seconds on a per-DB-query basis, per-…
Thank you for the very detailed response, it all sounds excellent and it's great to see you've been thinking these things through. I don't really know what my expectations were, but this seems like a good combination of flexible, with opinions based in real-world experience. You're right that in my second example it's unlikely to occur at scale, the bigger you get the more granular you can be in general, so I doubt that'll be an issue.
Re: Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
#50Very cool! Feature request: I have really struggled with turning the thing costing me money off in AWS. If, with the right master credentials, I could consistently and easily do that somehow, that'd be a 10x feature. If you made that use-case free, you'd get tons of installations from people who desperately need this in the top of your sales funnel. edit: This used to say "in your app" and that wasn't quite what I wa…
We solved that exact problem with our open source tool Resoto. Specifically our "Defrag" module, which cleans up unused and expired resources: https://resoto.com/defrag https://github.com/someengineering/resoto The magic behind the clean up is Resoto's inventory graph - the graph captures the clean up steps for each individual AWS resource. One of Resoto's users, D2iQ (now part of Nutanix), reduced their monthly clou…
This is the first time I'm hearing about your company but I am already a huge fan of the tools you're building. I've had a need for most of them and next time I am met with those needs, I will use your tools. Please keep going!