21–30 of 110 posts
Then you can have s3 file write triggers to ensure that every new file conforms to your new schemas
Don't dump the data to Postgres. Instead, define a data model and write an API that pulls data out of the respective places. The API will be the one place your applications get data from. You don't have to build it all at once. Just code the parts you need as you develop new applications.
Yes the first step is to document where the data lives. Take the time to dig through the mess and document what you have now and where to get all the information. You want Sales - logon here, select [this] from [that] You want forecasts - email the angry VP and ask for the latest spreadsheet Once you know where things are - then you can think about rebuilding as you need.
Unfortunately you provide few details on the structure of data, so it's hard to advise anything particular.
Storing that data on S3 is probably 50% the price of storing it on EBS and you won't have the durability guarantees of S3 when you're using PostgreSQL on EBS volumes.
If you're both exploring data and building models then Spark is fine. Its APIs are no more complicated that anything else out there for these tasks.
Hive is doing nothing more than offering schema on read and shouldn't be something you're thinking much about.
PostgreSQL is row-oriented and won't be able to offer features like row-group statistics that allow queries to get minimum and maximum values for every 10-15K rows of data for the columns their interested in. This gives queries a huge speed up over needing to scan over rows rather than just the statistics for the columns their interested in.
Remember that you can have a single engineer run a single query on Spark and distribute it across several servers. This allows you to scale CPU and memory bandwidth in a way you won't be able to with PostgreSQL.
It sounds like your data isn't well organised. If you moved it around and put some consistent naming conventions in place that could help. You could also look to build an atlas of the data for newcomers to get an overall picture of what data you're storing and where it lives.
They built a pipeline that complicated for 100gb? That’s insanely over-engineered! Very typical of engineers who just want to pad their resume at the expense of unsuspecting business people. I’ve worked with single server data warehouses on SQL Server that were 10x in size and served the entire company. I don’t know what your data looks like, whether it’s just transactional or a combination of transactional and raw s…
Or they were given the same PR crap you always get from sales people that they’re just days away from tripling the number of clients and by next year they should be 10-20x the number, so they went ahead and “built it right” so they wouldn’t run into the inevitable scaling issues they were supposedly assured to hit in short order?
I don't know your ratio of HDF5 to Parquet files but remember for every GB of parquet you have it will equate to about 10 GB of space needed in CSV or PostgreSQL's internal format. So your data set is probably closer to 1 TB than 100 GB. Storing that data on S3 is probably 50% the price of storing it on EBS and you won't have the durability guarantees of S3 when you're using PostgreSQL on EBS volumes. If you're both…
Presumably the cost of a few TB on EBS is the least of your worries.
Finally, the time saving of full transactional support and constraints + sql to write etl in will drastically reduce the amount of work needed to write etl.
IMO, if RDBMS is an option for you, do it whilst your data is small enough.
I don't know your ratio of HDF5 to Parquet files but remember for every GB of parquet you have it will equate to about 10 GB of space needed in CSV or PostgreSQL's internal format. So your data set is probably closer to 1 TB than 100 GB. Storing that data on S3 is probably 50% the price of storing it on EBS and you won't have the durability guarantees of S3 when you're using PostgreSQL on EBS volumes. If you're both…
I’d contend this personally. You can employ disk, or row compression on PG if you want. Compressed disk will actually make your queries faster. You can use cstore for ORC based column storage with PG if you want. Presumably the cost of a few TB on EBS is the least of your worries. Finally, the time saving of full transactional support and constraints + sql to write etl in will drastically reduce the amount of work ne…
Where is the transactional requirement? This person is working with a copy of the real data.
ETLs only need to be written once and if he decided on a PSQL approach he'd be writing ETLs to send the data there too. He's probably going to find a number of consistency problems so trying to normalise all this data again will just result in more work that won't make his team of DS' more productive.
If he's at ~1 TB of data today, where will he be in a few years time? What's the point of putting infrastructure in place that won't last for the next 10+ years?
They built a pipeline that complicated for 100gb? That’s insanely over-engineered! Very typical of engineers who just want to pad their resume at the expense of unsuspecting business people. I’ve worked with single server data warehouses on SQL Server that were 10x in size and served the entire company. I don’t know what your data looks like, whether it’s just transactional or a combination of transactional and raw s…
> Very typical of engineers who just want to pad their resume at the expense of unsuspecting business people. Or they were given the same PR crap you always get from sales people that they’re just days away from tripling the number of clients and by next year they should be 10-20x the number, so they went ahead and “built it right” so they wouldn’t run into the inevitable scaling issues they were supposedly assured t…