Earlier quoted context omitted.
Define "big data"? You can buy normal tier 1 server vendor hardware with 6T of memory. If it fits into 1 machine or a few, it isn't "big data".
I've heard this 6TB number thrown around a bit recently, but no one can actually provide me with links to hardware I can buy - has anyone successfully setup a machine with this much ram who can post links to what they used?
New in PostgreSQL 10
41–50 of 258 posts
Re: New in PostgreSQL 10
#42Earlier quoted context omitted.
Define "big data"? You can buy normal tier 1 server vendor hardware with 6T of memory. If it fits into 1 machine or a few, it isn't "big data".
Maybe. I recently updated some of my company's internal training material, and discovered there still isn't a single (good) definition of "Big Data". Most definitions talk evaluating data on four dimensions: volume, variety, velocity and veracity. That's still a vague, but it's more helpful than looking at volume alone. Besides, if we picked machine size as _the_ definition, it would change every year. And that feels…
I've also struggled with the definition of Big Data. I typically see "size" as being the line of demarcation people throw around when thinking about Big Data, but I think it's it's actually size + latency that's the real definition.
If I have a 12TB dataset, and I need to be able to find a single record in that dataset, and return the results to a process in less than one second, I don't believe that needs to be considered "big data". The dataset is "large", but the latency / access pattern surrounding that data is low enough that you can solve that with a handful of hard drives and your data sorted on disk.
If I have a 12TB dataset and I need probably big data. To guarantee those sort of lookup times for random IO, I either need very expensive hardware, or a software based solution to distribute my data to multiple machines.
If I have 10GB of census data, and I need to run an analysis that shows what are the 100 most similar cities (non approximation) to Los Angeles in terms of demographic make up, and I need an answer in less than 1 second, that is also probably big data. The analysis is easily parallelized, but if done serially on a single machine is a "slow" process.
I realize this is a moderately unconventional way of approaching Big Data, but it's how I've started to structure some of my thinking and explanations around why it's not just size, but size + latency (which is inherently a question of what you're trying to do with the data).
Re: New in PostgreSQL 10
#43Completely off topic... but did anybody notice how they used partitioning example column names were in Spanish?
Re: New in PostgreSQL 10
#44Earlier quoted context omitted.
Having provisioned these sorts of "big data" systems in the past, it's now about how much you have today, it's how much you'll have over a growth period. The advantages of a scale-out system like cassandra, riak (RIP), memsql, big-table, cockroach, etc., are that they can grow with you from 3TB, to 9TB, to 81TB (as an example, if you're on some exponential growth curve with your data). It's not that you can't do it w…
Designing for the future is a guaranteed project failure. If one is starting a new project, hence contemplating what DB to use, starting with an ACID db is a safe bet in most cases(unless of course they are already starting with a huge amount of data). By the time the outgrow the ACID database they will have a better idea of what exactly they need and more importantly they will have the resources to make the switch.…
No it's not, say that have 2TB of logs daily, good luck designing that with a regular RDMS without sharding / clustering / re-balancing capabilities.
Re: New in PostgreSQL 10
#45Earlier quoted context omitted.
Designing for the future is a guaranteed project failure. If one is starting a new project, hence contemplating what DB to use, starting with an ACID db is a safe bet in most cases(unless of course they are already starting with a huge amount of data). By the time the outgrow the ACID database they will have a better idea of what exactly they need and more importantly they will have the resources to make the switch.…
No it's not, say that have 2TB of logs daily, good luck designing that with a regular RDMS without sharding / clustering / re-balancing capabilities.
In which case they would not be designing for the future.
Did you miss this: "(unless of course they are already starting with a huge amount of data)"?
Re: New in PostgreSQL 10
#46I noticed hash indexes are now crash proof and replicated -- this seems to make them actually usable in production. In other words, this release effectively "adds" a new index type. That seems like a much bigger deal than is being talked about, is there any reason to believe that new databases shouldn't be using hash indexes for columns that won't be supporting range queries? (In other words, pretty much all keys.) I've seen mixed reports about hash indexes being beneficial.
Re: New in PostgreSQL 10
#47How does PostgreSQL 10 compare with Cassandra for BigData requirements?
Define "big data"? You can buy normal tier 1 server vendor hardware with 6T of memory. If it fits into 1 machine or a few, it isn't "big data".
To be honest 6 TB of data isn't that much. I'd say big data starts at around one petabyte because one petabyte requires 3 servers with 45 [1] 8TB hdds. At this point your data is so large it's impossible to store it on a single physical machine.
"big data" usually doesn't require transactions though or at least no transactions across servers so you could probably get away with sharding postgresql via citus.
Re: New in PostgreSQL 10
#48Floating point timestamps - the only item which affects my work is also one of the few in this wiki page with no description. These were removed with no breaking compatibility. For anyone else using them, I just did a bit of reading [1] and it looks like they are inferior to default 8 byte timestamps. FP timestamps have microsecond precision around the year 2000, but it drops off as it moves away from there. I was us…
Yeah.. You should never use floating point when your use case wants uniform precision across the full range. That's the opposite of what floating point gives you.
Re: New in PostgreSQL 10
#49Earlier quoted context omitted.
Having provisioned these sorts of "big data" systems in the past, it's now about how much you have today, it's how much you'll have over a growth period. The advantages of a scale-out system like cassandra, riak (RIP), memsql, big-table, cockroach, etc., are that they can grow with you from 3TB, to 9TB, to 81TB (as an example, if you're on some exponential growth curve with your data). It's not that you can't do it w…
Designing for the future is a guaranteed project failure. If one is starting a new project, hence contemplating what DB to use, starting with an ACID db is a safe bet in most cases(unless of course they are already starting with a huge amount of data). By the time the outgrow the ACID database they will have a better idea of what exactly they need and more importantly they will have the resources to make the switch.…
I don't think we disagree, but I want to be cautious about saying "whatever we start with is always good enough."
I'm co-founder of a company, doing all of the technical work, and I too am using a very vanilla stack: Postgres, Elasticsearch, Redis, Memcache, Rails. However, having worked on massive scale-out projects before, I recognize where my pain points are going to be with my tech choices.
[1] I'm storing calendar data in postgres. This data is primarily key/value based, needs to be persistent, and on a per user basis potentially is tens of thousands of records. This is a medium term problem, and I'll likely move this data out of postgres and into a scale out solution sooner rather than later. I need relatively fast lookup times, but this is primarily a "high write, low read" scenario, that Cassandra is well suited for.
[2] I'm storing a bunch of text from blog posts, news articles, etc., in Postgres. I really dont' need the data to be in postgres, it's just a handy place to store it to synchronize with Algolia & Elasticsearch, where the true work happens. However, I dont' want to hold onto TBs & TBs of news articles in a postgres system. As that system scales out, it'll be unnecessarily expensive, as I don't need all the bells and whistles I'm getting from Postgres, for what is effectively a form of cold storage. This is a long term problem, and I likely won't tackle it for well over a year.
So, you're right, I'm not contemplating what DB to use, but I'm definitely aware of why I won't just leveraging the same database in perpetuity. Being focused on delivering customer value is vital, and that's my intent; however, scalability and cost structure do matter over the long term, so an understanding of where one will go is good as well.