Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I improve our data infrastructure?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 110 posts

Re: Ask HN: How do I improve our data infrastructure?

#81

Post by remilouf »

deepstream wrote: post #59

> I was just hired [...] > I was thinking about building a pipeline [...] > How do I go about raising awareness? Don't. You don't get it yet. Everything there is there for a reason. Some of it may be technical reasons, some of it may be delivery reasons, some of it may be people reasons, team reasons, political reasons, etc. You just landed a sweet perm job in a field you love. Don't ruin it by becoming that cliched…

> Don't ruin it by becoming that cliched new hire that sees all their problems and knows how to make it all better.

I don't, which is why I'm asking around. I'm also scheduling chats with people to understand the background and the history to see if it's worth changing anything. I will make a move if that make sense. In the meantime, I'm just gathering information to not make a stupid decision.

> And you're a perm now, in a big corp, doing data science. Relax, you got it made, right?

> So enjoy playing that game, because you are happy to be in a big corp, so the sort of benefits that can bring you is what you want, right?

I don't think this was necessary. I've only worked for startups before, and I was hired in part to see if we can do a better job with the resources we have. Buy in from management is not an issue. I am not asking for life advice.

Re: Ask HN: How do I improve our data infrastructure?

#82

Post by sixdimensional »

usgroup wrote: post #54
aeroevan wrote: post #51

Earlier quoted context omitted.

It's 100GB compressed. Parquet does a very good job of compressing most data so that's where the estimate of 10x (so 1TB) uncompressed was mentioned as a rule of thumb. Parquet also supports much better access mechanisms, like being able to deserialize a single column without having to read in entire rows. But like you mentioned, 1TB of data in a traditional database isn't that bad.

... also remembering that a traditional dB will typically not store data raw. Row compression is normal and disk compression is normal . The typical column store advantage is block compression, predicate pushdown and column order storage.

Regular databases such as SQL Server and Oracle have had columnar compression built in as an option along with the row stores for years now. I use it in SQL Server a lot and it works great.

Re: Ask HN: How do I improve our data infrastructure?

#83

Post by remilouf »

patrickg_zill wrote: post #62

For 100 gb that might grow to 1tb: 2x NVME drives, each 1tb in size. Buy pcie adapter as needed. Buy used 256gb RAM server off eBay. Install drives into server. Put all data on the nvme drives and all other files on regular drives. Everything will fit in ram after the data is touched the first time after each boot. Can't get faster than that, usually. Create user accounts for each user, set up git etc. for code stora…

I think it may come across as trolling :)

Re: Ask HN: How do I improve our data infrastructure?

#84

Post by singhrk »

Currently I am in the same boat. Few things that we have tried/figured out are -

(a) Deprecation - Get your butcher hat on. start looking at existing things, and see how many of these are used by who all. Start deprecating (or at least archiving) the offerings that no body uses (or you are not able to find a user)

(b) Simplification - Try to find the infrastructure components (compute engines, storage frameworks) that serve the same use-case, and see if you can converge into one. For example you can converge from HDF5 and S3 to just S3. Similarly from Hive and Spark to just Spark. Don't bring another infrastructure component in the mix, otherwise someone new in your place will make another HN post in future :)

(c) Documentation - Start building a place to document all the offerings that you have. Some wiki style solution or if it works for you something as simple as google docs. Or it could be some solution like Superset/Redash that is atleast bringing everything at one place

(d) Governance - Get some power users in the system, take their help in (i) identifying important datasets, (ii) adding information about existing datasets, (iii) can review a new code/dataset/production deployment

(e) start checking in Transform code/table DDL, all metadata into some git repository. This will automatically build some documentation overtime and take care of duplicate logic overtime

Re: Ask HN: How do I improve our data infrastructure?

#85

Post by iblaine »

Hiring a data engineer would be your best option. The second best option would be to outsource the infra to a company like datacoral.com or pachyderm.io.

Be aware that a single database or tool is probably not a solution. You may get more rope to hang yourself with.

Any solution is going to be a function of data engineering, systems design and project management. If you lack one of those abilities then you'll need more of the other two to make up for it.

(Feel free to reach out to me on https://www.linkedin.com/in/iblaine/ if you like. I'm not selling anything...I have been in the DE industry for a while...)

Re: Ask HN: How do I improve our data infrastructure?

#86

Post by calvinfo »

Disclaimer: I’m a cofounder of Segment [1], we build a product to help with these problems.

Given what you’ve shared here, it sounds less like your problems are related to scaling for data volume, and more related to all of the complexity that comes with a data pipeline. Instead of adding a bunch of new components, it sounds like you need just a few.

My concrete advice:

- Standardize and document the collection point for your data. Create a tracking plan which documents how data is generated. Have an API or Libraries which enforce the schema you want. If the sources of data are inconsistent, it’s going to be hard to link them together over time. - Load all of the raw (but formatted) data onto S3 into a consistent format. This can be your long term base to start building a pipeline. And the source for loading data into a warehouse. - Load that data into BigQuery (or potentially Postgres) for interactive querying of the raw data. For your dataset, the cost will be totally insignificant and results should give your analysts a way to explore your data from the consistent base. - Have a set of airflow jobs which take that raw data and create normalized views in your database. Internally we call these “Golden” reports, and they are a more approachable means of querying your data for the questions you might ask all the time. The key is that these are built off the same raw data as the interactive queries.

We use Segment to manage all of the top three bullets (collect consistently, load into S3, load into a warehouse). Then we use airflow to create the golden reports that analysts query via Mode and Tableau. As other commenters have mentioned, there are a number of tools to do this (Stitch, Glue, Dataflow), but the key is getting consistency and a shared understanding of how data flows through your system.

This is a pattern we’ve started to see hundreds of customers converge on: a single collection API that pipes to object storage that is loaded into a warehouse for interactive queries. Custom pipelines are built with spark and Hadoop on this dataset, coordinated with airflow.

[1]: https://segment.com

Re: Ask HN: How do I improve our data infrastructure?

#87

Post by patrickg_zill »

remilouf wrote: post #83
patrickg_zill wrote: post #62

For 100 gb that might grow to 1tb: 2x NVME drives, each 1tb in size. Buy pcie adapter as needed. Buy used 256gb RAM server off eBay. Install drives into server. Put all data on the nvme drives and all other files on regular drives. Everything will fit in ram after the data is touched the first time after each boot. Can't get faster than that, usually. Create user accounts for each user, set up git etc. for code stora…

I think it may come across as trolling :)

Successful presentation of this idea is left as an exercise for the reader :)

Re: Ask HN: How do I improve our data infrastructure?

#88

Post by solipsism »

eru wrote: post #65
exelius wrote: post #36

Earlier quoted context omitted.

Spot on!! Everyone has cargo-culted distributed file databases, and they’re good in specific use cases — if you have a large volume of data with a very high number of writes. Hardware and RDBMS performance have improved over the years to the point where if you’re not Google (or certain scientific applications), you probably don’t need much more than postgres. It’s completely within the bounds of feasibility of modern…

Oh, and even Google has gone back to a more relational design with Spanner again.

That misses the point, doesn't it. The point isn't "maybe you don't really need nosql/non-relational", it's "maybe you don't need an expensive managed storage solution built for massive scale."

Spanner was indeed built for massive scale, which is reflected in the price.

Re: Ask HN: How do I improve our data infrastructure?

#89

Post by remilouf »

bjornlouser wrote: post #75

"I was just hired as the first permanent data scientist..." You hope. "I was thinking about building a [another] pipeline... " You're fired. Shouldn't you first deliver some new insight based on your data analysis skills?

I was also hired because I also have an affinity (affinity, not expertise) with data engineering and am familiar with development good practices. The idea is not to spend 100% of my time doing this, more like 30%.

Business value comes first, and with a better infrastructure we could deliver a lot more value with the same head count.

Re: Ask HN: How do I improve our data infrastructure?

#90

Post by dapearce »

1) Look at 3rd party solutions (Stitch, Fivetran, Segment, etc.) to ingest your raw data into a data warehouse (strong preference for Snowflake).

2) Use dbt (https://www.getdbt.com) to clean, transform and model the raw data into analytic tables.

3) Add a BI layer (Mode, Looker, etc.) on top of the analytic tables for reporting.

Post reply on HN