The example shown has 14GB of data, which is absolutely tiny, yet it can still rack up $40k in costs per day if used incorrectly (i.e. less than optimal, but not egregiously wrong)? Why would you use this?
No post body was provided.
The data rules worth $40k a day
11–20 of 72 posts
Re: The data rules worth $40k a day
#12The example shown has 14GB of data, which is absolutely tiny, yet it can still rack up $40k in costs per day if used incorrectly (i.e. less than optimal, but not egregiously wrong)? Why would you use this?
Plus, on what planet would you be refreshing that query every second of the day?
Re: The data rules worth $40k a day
#13Isn't this all extremely elementary and common knowledge for someone designing these workflows? Next blog post about how to not waste money by just not doing something extremely poorly optimized.
Re: The data rules worth $40k a day
#14Materialized Views are damn near magic for solving issues involving slow queries on for tools that don't need real time results (eg daily reporting). They essentially act as a cache of a query at a given point in time that you're able to refresh whenever you want. https://www.postgresql.org/docs/current/rules-materializedvi...
> It’s only significant weakness now is in Materialized views, with their lack of incremental refresh.
> That work towards incrementally updated views is happening and progressing. For now, it's a separate extension, though: https://github.com/sraoss/pg_ivm.
Re: The data rules worth $40k a day
#15Earlier quoted context omitted.
I always wonder, who looks at their pricing model and decides "Paying multiple dollars to query on a few GB of data sounds reasonable"?
Seriously? A $10k dedicated server would have a payback period measured in minutes.
Importantly it depends on how often these things are run and who has to maintain a server.
Re: The data rules worth $40k a day
#16Isn't this all extremely elementary and common knowledge for someone designing these workflows? Next blog post about how to not waste money by just not doing something extremely poorly optimized.
Don't use floating point for monetary calculations.
Re: The data rules worth $40k a day
#17Materialized Views are damn near magic for solving issues involving slow queries on for tools that don't need real time results (eg daily reporting). They essentially act as a cache of a query at a given point in time that you're able to refresh whenever you want. https://www.postgresql.org/docs/current/rules-materializedvi...
relational databases do best with facts. a proper materialized view is just another fact.
in my experience materialized views are critical for most large databases.
Re: The data rules worth $40k a day
#18at $40k/day SAVINGS you could build your own dc, easily.
Re: The data rules worth $40k a day
#19The example shown has 14GB of data, which is absolutely tiny, yet it can still rack up $40k in costs per day if used incorrectly (i.e. less than optimal, but not egregiously wrong)? Why would you use this?
Plus, on what planet would you be refreshing that query every second of the day?
The one where some dev gets the query in a loop unintentionally and you discover your bankrupt the next day.
Re: The data rules worth $40k a day
#20I'm really curious how people use these types of platforms in practice. It seems really easy to screw yourself over when you're "expirementing". The advice seems really obvious, but I'm sure I'd absent mindedly do a super naive query at some point and potentially cost my employer a lot of money.
I worked on a system to capture the production test data in a semiconductor company. We had trillions of rows and terabytes of data. While we were figuring stuff out, I'm sure I ran queries that scanned the entire dataset accidentally. I imagine one of those queries would have cost at least 1k to run. Our entire setup cost less than 10k a month to run on AWS regardless of how many queries we ran. I can't imagine spending 40k on 14gb of data regardless of what you were doing.