Lost me at refreshing every second. Probably only needs updating a couple of times per day. This is why you drill into requirements rather than blindly accepting them without question.
The data rules worth $40k a day
51–60 of 72 posts
Re: The data rules worth $40k a day
#52So I read the full article and it's interesting. Maybe the title is a little misleading, but when you actually read it I think they make it pretty clear that it's an extreme example designed to prove a simple point: bad data practice has a cost. For somebody who isn't super data savvy but wants to get into the space, I actually found it helpful.
Re: The data rules worth $40k a day
#53Earlier quoted context omitted.
I always wonder, who looks at their pricing model and decides "Paying multiple dollars to query on a few GB of data sounds reasonable"?
Seriously? A $10k dedicated server would have a payback period measured in minutes.
The example in the article is 40k/day for querying on 14 GB of data, every second. You can do that on a ~100/month machine or a somewhat modern laptop.
Re: The data rules worth $40k a day
#54Earlier quoted context omitted.
relational databases do best with facts. a proper materialized view is just another fact. in my experience materialized views are critical for most large databases.
One of the many reasons I wish we hadn't chosen MySQL. Our internal dashboard has loads of "reports" that are basically tabular displays of queried data with optional filtering and sorting on every column, plus pagination. It's near impossible to make any of them performant due to not being able to optimize for a specific use case; because of the myriad of combinations of filters and sorts different users might apply…
You should schedule a "create or replace table" statement every N hours/days/etc. It will be effectively just the same as a scheduled materialized view refresh.
Re: The data rules worth $40k a day
#55Earlier quoted context omitted.
Seriously? A $10k dedicated server would have a payback period measured in minutes.
The example in the article is 40k/day for querying on 14 GB of data, every second. You can do that on a ~100/month machine or a somewhat modern laptop.
Our entire, hundreds of millions of dollars business with hundreds of millions of orders of historical data can be "explored" by a data scientist on an average macbook.
If your data scientists can't do data science, then that's pretty bad
Re: The data rules worth $40k a day
#56Earlier quoted context omitted.
The data lake problem is specifically due to microservices or SOA. People love to separate customers from orders until they realize that you want to do complex filtering on customers and join it to their orders. Then everyone says "oh crap" when they realize they've created a problem without a great solution.
Yes. Fundamentally there is almost no data that is not relational in some aspect. Any given document, you can virtually instantly start picking out "ok that could be a foreign key...", and if one does not exist, it certainly will before too many sprints go by. And most usages of NoSQL are essentially equivalent to a row within a table, especially given many NoSQL solutions have bugs or performance issues with deeply…
Storing Json data into a highly structured RDBMS table can be problematic if any document contains arrays or nested documents.
I built a new general-purpose data management system that uses key-value stores that I invented to attach meta-data tags to objects. These key-value stores can also be used to create relational tables.
Because each table is basically a columnar store, I can map multiple values to each row key to create a 3D table. It seems ideal for importing Json data where any item in a document can be an array of values. I am trying to figure out how useful this system might be to the average DBA or NoSql user.
See a quick demo at https://www.youtube.com/watch?v=1b5--ibFhWo
Re: The data rules worth $40k a day
#57[deleted]
Re: The data rules worth $40k a day
#58Re: The data rules worth $40k a day
#59The example shown has 14GB of data, which is absolutely tiny, yet it can still rack up $40k in costs per day if used incorrectly (i.e. less than optimal, but not egregiously wrong)? Why would you use this?
Quoted post unavailable.
We should start a business selling data intelligence to solve this problem.
Maybe call it... business intelligence?
Re: The data rules worth $40k a day
#60If you want to stop out of control spend, have your analysts learn SQL. Avoid database systems that charge per query (like tinybird) and make damn certain your people know SQL if you do. Ignore and preferably fire people who mention things like “data lakes”.
I always wonder, who looks at their pricing model and decides "Paying multiple dollars to query on a few GB of data sounds reasonable"?
No post body was provided.