The data rules worth $40k a day • Hacker News Board

Re: The data rules worth $40k a day

#41

Post by WesleyJohnson » Thu, Jul 21, 2022, 6:44 PM UTC

johnthescott wrote: post #17

kiernanmcgowan wrote: post #7

Materialized Views are damn near magic for solving issues involving slow queries on for tools that don't need real time results (eg daily reporting). They essentially act as a cache of a query at a given point in time that you're able to refresh whenever you want. https://www.postgresql.org/docs/current/rules-materializedvi...

relational databases do best with facts. a proper materialized view is just another fact. in my experience materialized views are critical for most large databases.

One of the many reasons I wish we hadn't chosen MySQL. Our internal dashboard has loads of "reports" that are basically tabular displays of queried data with optional filtering and sorting on every column, plus pagination. It's near impossible to make any of them performant due to not being able to optimize for a specific use case; because of the myriad of combinations of filters and sorts different users might apply. We've tried to explain this many times over, even bringing in a DBA consultant who said the same thing, but we're still met with "why is this so slow?" on a regular basis.

Materialized views wouldn't be a silver bullet, but it would certainly help by allowing us to "cache" all of the joins and pre-processing into indexable views.

Re: The data rules worth $40k a day

#42

Post by didgetmaster » Thu, Jul 21, 2022, 6:44 PM UTC

mcphage wrote: post #12

royjacobs wrote: post #9

The example shown has 14GB of data, which is absolutely tiny, yet it can still rack up $40k in costs per day if used incorrectly (i.e. less than optimal, but not egregiously wrong)? Why would you use this?

Plus, on what planet would you be refreshing that query every second of the day?

Would that be the same planet where you can refresh a query every second that takes 2.4s to complete? How does that work?

Re: The data rules worth $40k a day

#43

Post by xhrpost » Thu, Jul 21, 2022, 6:57 PM UTC

ericd wrote: post #8

h4kor wrote: post #5

Earlier quoted context omitted.

I always wonder, who looks at their pricing model and decides "Paying multiple dollars to query on a few GB of data sounds reasonable"?

Seriously? A $10k dedicated server would have a payback period measured in minutes.

I don't think it's always that easy. I work on some large databases (1TB+) where even queries utilizing indexed columns still take prohibitively long to run. Sure we could look into partitioning (and we likely might) but then more work is progressively falling on engineering to keep the system running for the standards of analysts but not necessarily the product.

BigQuery in these cases has been very useful. Everything I throw at it returns in seconds. I can also do more text comparison queries that on a standard RDBMS would have required a full text index (which would probably be huge). But with BigQuery even those queries take seconds and I don't have to take production resources to view it.

Re: The data rules worth $40k a day

#44

Post by femto113 » Thu, Jul 21, 2022, 7:05 PM UTC

This headline is egregiously misleading clickbait, since it is based on an absurd and self contradictory hypothetical:

"If my product had 10,000 daily active users generating 100 events per day, I’d easily hit that size in less than a year ... If I built an application to visualize this result and refreshed it every second, I would spend $40,340.16/day on Tinybird’s Pro-plan"

There is no plausible business sense in refreshing the display of a year's worth of data every second, and even scaling back to a likely still-unreasonably-frequent refresh rate of once per hour you're down to just $11/day.

Re: The data rules worth $40k a day

#45

Post by krnlpnc » Thu, Jul 21, 2022, 7:05 PM UTC

I'm sorry but if your egress costs are 40k/day I think it's time to consider leasing your own pipe. $1.2 million per month will get you one hell of an internet connection.

Re: The data rules worth $40k a day

#46

Post by silisili » Thu, Jul 21, 2022, 7:06 PM UTC

steveBK123 wrote: post #28

cosmolev wrote: post #4

Earlier quoted context omitted.

Wow, I've previously had exactly same feelings about couple of people from whom I heard "data lakes".

Only thing worse is "data lakehouse"

I couldn't help but laugh the first time I saw that.

Should we go ahead and file copyright on Data Ocean and Data Beachhouse?

Re: The data rules worth $40k a day

#47

Post by [deleted] » Thu, Jul 21, 2022, 7:09 PM UTC

datalopers wrote: post #3

If you want to stop out of control spend, have your analysts learn SQL. Avoid database systems that charge per query (like tinybird) and make damn certain your people know SQL if you do. Ignore and preferably fire people who mention things like “data lakes”.

[deleted]

Re: The data rules worth $40k a day

#48

Post by altdataseller » Thu, Jul 21, 2022, 7:12 PM UTC

oefrha wrote: post #31

royjacobs wrote: post #9

The example shown has 14GB of data, which is absolutely tiny, yet it can still rack up $40k in costs per day if used incorrectly (i.e. less than optimal, but not egregiously wrong)? Why would you use this?

Scanned 6.67GB and processed for 2.41s, cost $40340.16 / 86400 = $0.4669. More than the hourly cost of AWS on-demand a1.x4large with 16 vCPU and 32GB of RAM. Can’t think of a worse advertisement for their product.

Seriously, when i read this, i was like, there is no way i will ever use this example product they mentioned in TinyBird.

After reading it, i was curious what the writer’s relationship was with TinyBird. Was he/she a recent user?

Then i put my palm over my face..

Re: The data rules worth $40k a day

#49

Post by teddyh » Thu, Jul 21, 2022, 7:12 PM UTC

cosmolev wrote: post #4

datalopers wrote: post #3

If you want to stop out of control spend, have your analysts learn SQL. Avoid database systems that charge per query (like tinybird) and make damn certain your people know SQL if you do. Ignore and preferably fire people who mention things like “data lakes”.

Wow, I've previously had exactly same feelings about couple of people from whom I heard "data lakes".

“We used to dream about having our data in a warehouse! We had to have our data in a lake!”

Re: The data rules worth $40k a day

#50

Post by didgetmaster » Thu, Jul 21, 2022, 7:14 PM UTC

Mythical creature vs 'Real-World Sighting'?

I try to never underestimate the potential for someone to do something really stupid and I'm sure there are some egregious examples out there where a DB was set up and run such that outrageous charges resulted; but has anyone seen a situation in real life anywhere close to this kind of example?

Even if there was a instance where a poorly designed and implemented data set caused a $40K charge for a single day; I wonder how long it would take for the bean counters to notice and take action?