401–410 of 444 posts
Earlier quoted context omitted.
Didn't know that Posthog is based on CH these days. Interesting!
Check the list of companies using ClickHouse: https://clickhouse.com/docs/en/introduction/adopters/
I'm curious how you have this set up. Is it currently a manual process or you use social monitoring tools to help you find mentions of ClickHouse in the wild?
Earlier quoted context omitted.
I find a lot of organizations don't have the discipline to harness whatever power their data may have. Sure collect everything, but god forbid you have any sort of data governance, or spend a single resource minute of time manually tagging or organizing or validating it. Then they try to make shitty ML models or products out of it, but don't care if the models actually work or not, just that they have AI now. Then a…
Palantir, you have to have Palantir. Oh, and a bunch of data scientists with zero domain knowledge for whatever data they are analyzing, preferably with PhDs in maths, but some ML background will do. And agile, because of course all those Palantir dashboards can only be developed using agile. Once all is said and done, zero insight was created but a whole lot of consultants, contractors and project managers have been…
Most of my analyses provide very little value because they are sort of common sense to people with domain knowledge. When I ask people what could be more useful, one of two things usually happen: 1) it's impossible due to data and/or infrastructure limitations, 2) what they ask turns out to be nonsensical in further analysis (like asking for average of something that follows a very fat tailed distribution with a few observations dominating the phenomenon. Of course it's usually impossible to explain this to people).
The more I think about this, the more I think that in truly data powered companies, both the decision making and data analysis have to be carried out by more or less the same people. The organizational hierarchies have to be much flatter. Essentially the employees will have to be some kind of "secret agents" who have both the skills and the mandate to steer the company in the direction they see fit. I sort of see this already happening in the FAANG companies where, or so I hear, it's very difficult to get hired, the staff count is quite small compared to traditional companies and the senior engineers have a lot of power in the company.
Using math PhDs or Palantir or whatever as a sort of modularized black box for "insights" while giving them no real skin-in-the-game does not work.
Earlier quoted context omitted.
Palantir, you have to have Palantir. Oh, and a bunch of data scientists with zero domain knowledge for whatever data they are analyzing, preferably with PhDs in maths, but some ML background will do. And agile, because of course all those Palantir dashboards can only be developed using agile. Once all is said and done, zero insight was created but a whole lot of consultants, contractors and project managers have been…
I'm one of those PhDs with zero domain knowledge analyzing data and I share the sentiment. Most of my analyses provide very little value because they are sort of common sense to people with domain knowledge. When I ask people what could be more useful, one of two things usually happen: 1) it's impossible due to data and/or infrastructure limitations, 2) what they ask turns out to be nonsensical in further analysis (l…
My favorite example is the WW2 bomber diagram shown to illustrate survivorship bias. Sure, working from data and first principle one could identify the vulnerable spots of the bomber. Or one could asl the designers or have an engineer, heck even a contemporary field mechanic, take a look at the actual drawings of the plane. And reach the same conclusion, faster, with a ton of additional insight and improvement ideas that can actually be implemented...
"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…
I think there is a problem when someone with such proclaimed knowledge of the sector gets to this, and similar, pieces of data, and does not attribute it to pricing. Could it be queries are short because bigquery pricing for analysis, as confusing as this models are, is based on amount of data?[0]
Because the other line of reasoning is that a big chunk of that 90% of professionals being paid to do their jobs, do NOT take into account pricing of the tool and are using it for small data, instead of thinking that people are using the best tool with the lowest price, because there's plenty of options to process and analyse data right now in the cloud.
On the "business have low amount of data", that matches my experience as well. At first I thought I was simply dealing with smaller sized companies, but it's a trend of doing big data projects for data that'd fit a pendrive.
[0] https://cloud.google.com/bigquery/pricing#analysis_pricing_m...
Earlier quoted context omitted.
Check the list of companies using ClickHouse: https://clickhouse.com/docs/en/introduction/adopters/
Really neat that you scour job postings to learn useful intelligence about companies using your product. I do this too :) I'm curious how you have this set up. Is it currently a manual process or you use social monitoring tools to help you find mentions of ClickHouse in the wild?
Earlier quoted context omitted.
A large computer is radically CPU overprovisioned for most workloads.
But we aren't talking about most workloads.
And, to address your previous statement: one beefy server is actually pretty scalable. Soft threads spin up in microseconds to serve incoming requests, communication between threads is blazing fast, caching is simpler on one machine, etc. You don't even have to worry to much about scaling, the CPU just throttles itself when there is no load.
And every once in a while you just upgrade to the next gen beefy machine.
This posting was great. Highly recommended reading through. It gets really good when the author hits "Data is a Liability". > An alternate definition of Big Data is “when the cost of keeping data around is less than the cost of figuring out what to throw away.” This is exactly it. It's way too hard to go through and make decisions about what to throw away. In many respects, companies are the ultimate hoarders and can…
In a larger sense, it's a challenge to throw away stuff, just as it's difficult to trim big data. As I reach retirement, our attic, bookshelves, and cabinets must be trimmed -- and each item requires attention and a decision. Some things in the attic are obvious liabilities (what to do with a mercury barometer? A radium dial pocket watch? Old electronics?) Disposing of other stuff requires time, insight, and a sense…