Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do I improve our data infrastructure?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 110 posts

Re: Ask HN: How do I improve our data infrastructure?

#61

Post by sreekanthr »

I cannot answer your question without full understanding of what is the current usage of your data infrastructure.

Few pointers

- Who are users of the platform? If it is only used by data science team then you can rip apart the solution and work towards a more logical infrastructure where all you are doing is cleansing, normalizing and deriving features and these become your central feature repository which your team can pull and build models. You need a governance so that team is aligned on what features are present and how do they add new features to repository. At scale of 10 people it is much easier to have this all centralized, if team is scaling out then we will have to work out a de-centralization strategy.

- If you have operational reports like business reporting & investor reporting running on this infra then I would recommend keeping analytics workload separate from operational workload. They have different needs and SLA's.

One thing which worries me is you are talking about denormalization as something you are planning to do, that should have been the starting point of any HDFS/SPARK/Parquet based solution.

I can suggest tools for explorations, data quality check etc. But that requires more understanding of what your current infrastructure is solving vs what it was intended to.

Re: Ask HN: How do I improve our data infrastructure?

#62

Post by patrickg_zill »

For 100 gb that might grow to 1tb:

2x NVME drives, each 1tb in size. Buy pcie adapter as needed.

Buy used 256gb RAM server off eBay.

Install drives into server. Put all data on the nvme drives and all other files on regular drives.

Everything will fit in ram after the data is touched the first time after each boot. Can't get faster than that, usually.

Create user accounts for each user, set up git etc. for code storage and to encourage reuse.

Re: Ask HN: How do I improve our data infrastructure?

#63

Post by jon-wood »

aerovistae wrote: post #53

Just a spectator to this conversation, ordinary web developer over here, would anyone care to explain to me what form a “pipeline” takes? Is it a server endpoint? I really have no idea.

Data pipelines are typically used to translate data from whatever format the system that produces it speaks into a format that's useful for querying.

As an example you may want to take server request logs and write them to a Postgres table for querying, in which case you'd have something like this:

    Server Logs -> S3 -> Lambda which reads new logs to extract key fields -> Postgres
Once that's done you end up with a database table containing rows for things like URL, source IP, response time. You'd probably also normalise URLs so that /products/123, /products/123/ and /products?id=123 come out as the same thing for analysis.

Re: Ask HN: How do I improve our data infrastructure?

#64

Post by mremes »

I'll express an advocation for using SQL as a data pipelining language. Firstly, many SQL dialects are multi-platform and provide standardization for transformations. It's a declarative language that doesn't define how computation happens but what.

Where SQL is terrible to write is when one must pivot data. Each column transformation is defined separately (case whens). When the cardinality of a pivoted vector is high, it results in quite a verbose declaration. This problem can be mitigated for example by generating SQL programmatically with templating languages such as Jinja2. Rendering is handled nicely on platforms such as Airflow when running the rendered SQL in cloud (for example on top of Redshift or Presto cluster, BigQuery).

For writing complex transformations, UDFs and cascading subqueries are the way to go. Window functions are useful for scanning subsets of column values (useful for example in vector transformations [doing normalization, regularization etc.])

SQL is also a language with a gentle learning curve which makes it easy to learn for less software-engineering-minded people (BI people and analysts of different departments in a decentralized data science organization). It's established itself as a lingua franca for matrix transformations already for decades.

Data processing is usually done in batches of different intervals as in traditional data science nothing really needs real-time processing for single events. Then Spark shines. But I would rather make a tradeoff of using SQL and Spark side by side when handling real-time processing than losing benefits of using SQL that I listed above.

When data transformations – with some object ontology related to it other than "just maths" – are to be done real-time, then you better start thinking about building an application for that (using your favorite programming languages).

Even with Spark, around 70% of work is done in SparkSQL.

Re: Ask HN: How do I improve our data infrastructure?

#65

Post by eru »

exelius wrote: post #36
ergest wrote: post #19

They built a pipeline that complicated for 100gb? That’s insanely over-engineered! Very typical of engineers who just want to pad their resume at the expense of unsuspecting business people. I’ve worked with single server data warehouses on SQL Server that were 10x in size and served the entire company. I don’t know what your data looks like, whether it’s just transactional or a combination of transactional and raw s…

Spot on!! Everyone has cargo-culted distributed file databases, and they’re good in specific use cases — if you have a large volume of data with a very high number of writes. Hardware and RDBMS performance have improved over the years to the point where if you’re not Google (or certain scientific applications), you probably don’t need much more than postgres. It’s completely within the bounds of feasibility of modern…

Oh, and even Google has gone back to a more relational design with Spanner again.

Re: Ask HN: How do I improve our data infrastructure?

#66

Post by playing_colours »

chrismeller wrote: post #49
hobofan wrote: post #30

Earlier quoted context omitted.

A simple architecture should be able to carry this to 10x and even to 100x if you really want to push it.

And I’m not really saying otherwise, though I would somewhat disagree. I’m just saying that they weren’t necessarily (or even likely) thieving contractors who were just looking out for themselves. They built a respectable, usable, system. Honestly the contractors I see in IT are usually the far opposite end: it works well enough that they’re happy and pay my bill and by the time it doesn’t work anymore I’ll be off to…

In many cases, the cause of the problem may not be contractors.

There are lots of clients that clearly set their expectations for contractors who they see as expensive necessary evil: they want you to deliver fast and now, they do not want to hear that bubble that it will take longer to deliver a robust system.

Re: Ask HN: How do I improve our data infrastructure?

#67

Post by [deleted] »

sb8244 wrote: post #34
ergest wrote: post #19

They built a pipeline that complicated for 100gb? That’s insanely over-engineered! Very typical of engineers who just want to pad their resume at the expense of unsuspecting business people. I’ve worked with single server data warehouses on SQL Server that were 10x in size and served the entire company. I don’t know what your data looks like, whether it’s just transactional or a combination of transactional and raw s…

One thing I don't understand regarding resume padding like this (which I do think totally happens) is how do you justify it when someone asks questions about whether it was necessary? It could be very subtle too if they know their stuff and want to see if you know it. It seems like this would come back to bite in any decent interview.

[deleted]

Re: Ask HN: How do I improve our data infrastructure?

#69

Post by georgewfraser »

Postgres is not an appropriate choice for a data warehouse, because it isn’t optimized for scans. You want either Snowflake or BigQuery. They’re easy to use and they’re so fast that you’ll be able to do most of your work using very simple, un-optimized SQL queries.

Using HDFS or Parquet-in-S3 raises the complexity of your infrastructure a LOT and I’m not hearing a good reason for it. I wrote a blog post about when you should adopt a data lake, it might help you evaluate this choice:

https://fivetran.com/blog/when-to-adopt-a-data-lake

Re: Ask HN: How do I improve our data infrastructure?

#70

Post by remilouf »

havermeyer wrote: post #45
ergest wrote: post #19

They built a pipeline that complicated for 100gb? That’s insanely over-engineered! Very typical of engineers who just want to pad their resume at the expense of unsuspecting business people. I’ve worked with single server data warehouses on SQL Server that were 10x in size and served the entire company. I don’t know what your data looks like, whether it’s just transactional or a combination of transactional and raw s…

As a disclaimer, I work on the BigQuery team, but I wanted to point out that there is now support for transferring data from S3 to BigQuery: https://cloud.google.com/bigquery/docs/s3-transfer-intro

I did use BigQuery in the startup I was working for before, and it worked wonders for our 12Tb of data. I think it would be a bit overkill in our situation---even though not having to manage a DB is great.
Post reply on HN