Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I improve our data infrastructure?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 110 posts

Re: Ask HN: How do I improve our data infrastructure?

#91

Post by Gpetrium »

ergest wrote: post #19

They built a pipeline that complicated for 100gb? That’s insanely over-engineered! Very typical of engineers who just want to pad their resume at the expense of unsuspecting business people. I’ve worked with single server data warehouses on SQL Server that were 10x in size and served the entire company. I don’t know what your data looks like, whether it’s just transactional or a combination of transactional and raw s…

I think it is worth noting that although some will over-engineer to pad their resume, there are other valid reasons why this may have happened.

It is entirely possible that the folks hired to do the job were better specialized at creating large scale solutions. Client/supplier may have assumed that as a big corp, this segment would scale quickly and a smaller solution would have to be re-engineered at a higher cost later on.

Unfortunately there is insufficient information from stakeholders to make a clear argument.

Re: Ask HN: How do I improve our data infrastructure?

#92

Post by mremes »

remilouf wrote: post #76
mremes wrote: post #64

I'll express an advocation for using SQL as a data pipelining language. Firstly, many SQL dialects are multi-platform and provide standardization for transformations. It's a declarative language that doesn't define how computation happens but what . Where SQL is terrible to write is when one must pivot data. Each column transformation is defined separately (case whens). When the cardinality of a pivoted vector is hig…

I love SQL. But it hard to get other DS on board who think that 40 lines of Spark is better than a 10 line SQL query. The only thing that worries me with SQL is when having to write UDFs for, say, computing a Z-score. But maybe it's just because I have never done it? Do you have any good resources about this?

Don’t worry, I’m having my battles convincing my clients (both business and DS/DEs) that this is a viable paradigm. Here’s a nice-looking z-value recipe by Silota that I just googled up: http://www.silota.com/docs/recipes/sql-z-score.html

Re: Ask HN: How do I improve our data infrastructure?

#93

Post by combatentropy »

I would pick Postgres. Tables are the best format for data storage, SQL is the best language for exploring them, and Postgres is the best SQL database. With its support of myriad contraints, aggregate functions, and data types, including JSON, you should be able to build whatever view into the data that the different people want, with either database views or functions. You might have to copy data into different databases (or just different schemas) for different teams, but maybe not.

Datasets of a certain size might make Postgres fall over, but I don't think you're anywhere near that. That doesn't mean I would always stuff all my data into Postgres raw. I don't put Word documents or images in Postgres. I leave my Apache log files as Apache log files. If I want to analyze them, I usually use Bash (sed, cut, sort, etc.). But if I want to get really fancy I will import them into Postgres to run SQL queries on them, but usually just a certain segment of them (3 months, a year, etc.), only certain rows (grep first) and only certain columns (awk first).

If your data infrastructure is staid and boring, I think it's a good sign that you're doing it right.

Re: Ask HN: How do I improve our data infrastructure?

#94

Post by delusional »

IanCal wrote: post #46
ergest wrote: post #19

They built a pipeline that complicated for 100gb? That’s insanely over-engineered! Very typical of engineers who just want to pad their resume at the expense of unsuspecting business people. I’ve worked with single server data warehouses on SQL Server that were 10x in size and served the entire company. I don’t know what your data looks like, whether it’s just transactional or a combination of transactional and raw s…

I really disagree this is over engineered. This sounds like the problem is under-engineering. You suggest setting up proper infrastrucutre, rather than what they have now which sounds like a shared drive and various different processes written in whatever the person knew to make something quickly. It's currently one step up from people running notebooks locally and having no shared space for the data.

The term over-engineered has been sufficient diluted to just mean "poorly constructed" at this point.

An my opinion, if your solution is currently not working well, then it can not be over-engineered. Over engineering leads to good solutions that are too expensive, not bad solutions.

I'm interested to hear what other views on what over engineering is. At the very least to get some form of emumerarion.

Re: Ask HN: How do I improve our data infrastructure?

#95

Post by remilouf »

mremes wrote: post #92
remilouf wrote: post #76

Earlier quoted context omitted.

I love SQL. But it hard to get other DS on board who think that 40 lines of Spark is better than a 10 line SQL query. The only thing that worries me with SQL is when having to write UDFs for, say, computing a Z-score. But maybe it's just because I have never done it? Do you have any good resources about this?

Don’t worry, I’m having my battles convincing my clients (both business and DS/DEs) that this is a viable paradigm. Here’s a nice-looking z-value recipe by Silota that I just googled up: http://www.silota.com/docs/recipes/sql-z-score.html

Thanks! Do you have any tips on convincing people that SQL is a good paradigm?

Re: Ask HN: How do I improve our data infrastructure?

#96

Post by tixocloud »

> Everyone else is a consultant, and they don’t have any incentive to care about the long-term impact of architecture choices: their management only evaluates delivery (graphs, model metrics, etc.). How do I go about raising awareness?

It sounds like management treats data science as a service function as opposed to a strategic function. Do you get the sense whether they do have any incentive to grow the data science team properly as opposed to leveraging consulting resources? My gut is that there are trending to building out a data science team given you're their first data science hire but it also means that your results will largely dictate what the team will evolve to in the future.

To raise awareness, get a good understanding of the management team and their motivations. Frame the issue to something that they are most concerned with (likely business-related) and highlight the impact of not addressing the issue of poor architecture.

Present your findings, highlight the impact and go a step further to propose 3 options. Highlight why the option you're recommending is the best.

Re: Ask HN: How do I improve our data infrastructure?

#97

Post by deepstream »

remilouf wrote: post #81
deepstream wrote: post #59

> I was just hired [...] > I was thinking about building a pipeline [...] > How do I go about raising awareness? Don't. You don't get it yet. Everything there is there for a reason. Some of it may be technical reasons, some of it may be delivery reasons, some of it may be people reasons, team reasons, political reasons, etc. You just landed a sweet perm job in a field you love. Don't ruin it by becoming that cliched…

> Don't ruin it by becoming that cliched new hire that sees all their problems and knows how to make it all better. I don't, which is why I'm asking around. I'm also scheduling chats with people to understand the background and the history to see if it's worth changing anything. I will make a move if that make sense. In the meantime, I'm just gathering information to not make a stupid decision. > And you're a perm no…

you thought it wasn't necessary because you feel I was trying to be mean to you? I thought it was necessary to remind you. did you feel I wasn't being genuine? I don't think that was necessary to take it like that. but I can definitely understand how you might feel scared you're being judged for working at a big Corp after startups.

I'm not judging you, I think big Corp is a great achievement. I am genuinely congratulating, and reminding you that big Corp is what you wanted, so you can learn to play that game. is that better now?

buy in from management is always an issue. it's just they don't want to give you the impression of friction because you've just started. they're presenting you a side they think you'll like because management has decided it's important to hire and retain talent like you.

you're not asking for life advice? you're saying that because you feel I've been giving you life advice? I can understand you really take your work personally and I think that's a good thing to be passionate at what you do. I've only given you work advice specific to the situation you describe. and I'm happy with what I've said.

but that's enough about you, I took the time to read your post and make an answer, and when you make these comments, I feel like you're making it all about you, I feel you are attacking me for that and like you're not showing any gratitude. that hurts because I just wanted to be seen for contributing my help and perspective. can you when you ask for help not only consider your feelings and also consider the feelings of those offering you help? thank you.

finally, when you ask in a public place this, the answer is not just for your benefit.

hope your new job is good.

Re: Ask HN: How do I improve our data infrastructure?

#98

Post by Boxxed »

Okay, I kind of want to question the "just throw it in Postgres," attitude I've seen throughout this thread. I've dealt with similar sizes (maybe 750GB, number of rows in the small billions) and as much as I adore Postgres it just wasn't a great solution.

-- Loading would take hours, even with dropping the indexes beforehand

-- Creating the indexes afterwards would take even longer

-- Query performance was very hard to reason about. Even simple queries required altering some spillage limits.

-- Analytical queries were basically out of the question, with something like "select sum(thing) from table" taking many minutes.

Even if I could live with the above as just an initial load, there would have been a lot of churn as new data arrived that probably would have been even worse.

Re: Ask HN: How do I improve our data infrastructure?

#99

Post by eru »

solipsism wrote: post #88
eru wrote: post #65

Earlier quoted context omitted.

Oh, and even Google has gone back to a more relational design with Spanner again.

That misses the point, doesn't it. The point isn't "maybe you don't really need nosql/non-relational", it's "maybe you don't need an expensive managed storage solution built for massive scale." Spanner was indeed built for massive scale, which is reflected in the price.

Hmm, you are probably right.

Spanner does have somewhat less scale than their NoSQL offerings; and even Google says internally to go for the somewhat less scale-y spanner than them. (Because it's easier to react to needs for scale laten than it is to live without transactions and relation querying.)

Re: Ask HN: How do I improve our data infrastructure?

#100

Post by remilouf »

sologoub wrote: post #72
remilouf wrote: post #70

Earlier quoted context omitted.

I did use BigQuery in the startup I was working for before, and it worked wonders for our 12Tb of data. I think it would be a bit overkill in our situation---even though not having to manage a DB is great.

That’s the beauty of BQ - it scales well, but it works just fine in smaller use cases. It doesn’t get simpler than SQL. Another item to consider is that BQ now has ML (simpler) models built in, further reducing the complexity of your pipeline: https://cloud.google.com/bigquery/docs/bigqueryml-intro If you are not on GCP, then I’d consider AWS Athena for querying the parquet files, but you still have to structure thes…

I will consider that. How about Redshift?
Post reply on HN