Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HBase Deprecation at Pinterest

medium.com

Post reply on HN
Print view View original

1–10 of 59 posts

Re: HBase Deprecation at Pinterest

#2

Post by winrid »

Wow, never realized Pinterest had 6 petabytes of data. I wonder if they're including images in that. Even billions of rows is usually around 1-2tb, so makes you wonder what they're storing many billions of.

Re: HBase Deprecation at Pinterest

#3

Post by ddorian43 »

It reminds me of the often optimization story (how we got 2x+ faster by not doing very inefficient things), in this case going 6->3 replicas.

Example: TiDB at a certain time didn't write rows clustered by the primary key on disk (they had a separate index). This is very costly in distributed setups (less costly on single-node setups like PostgreSQL).

There are many such cases in many dbs. Another point lacking in most dbs is the "lsm compaction overhead" you need to do for all replicas when you're not using shared distributed storage.

This optimization can be seen on QuickWit (building/compacting inverted index is even more expensive than LSM compaction).

Re: HBase Deprecation at Pinterest

#4

Post by ddorian43 »

winrid wrote: post #2

Wow, never realized Pinterest had 6 petabytes of data. I wonder if they're including images in that. Even billions of rows is usually around 1-2tb, so makes you wonder what they're storing many billions of.

It makes no sense at their scale to store images in Hbase. It's probably just user tracking/analytics.

Re: HBase Deprecation at Pinterest

#6

Post by badpun »

winrid wrote: post #2

Wow, never realized Pinterest had 6 petabytes of data. I wonder if they're including images in that. Even billions of rows is usually around 1-2tb, so makes you wonder what they're storing many billions of.

In NoSQL world, you often store multiple duplicates of the same data, each of which is optimized for a different use case.

Re: HBase Deprecation at Pinterest

#9

Post by vkazanov »

redwood wrote: post #8

Shocking to see a company like Pinterest has built multiple in house data stores. Talking about a maintenance burden...

My company's data function, established 14 years ago, has a bunch of custom stores, as well as data-related tools. Back then data use cases were not really well-known, we had to build things from scratch a lot.

It is a very different landscape these days!

Re: HBase Deprecation at Pinterest

#10

Post by pjmlp »

As usual, with any technology adoption wave, remember this two key historical moments,

> Introduced in 2013, HBase was Pinterest’s first NoSQL datastore. Along with the rising popularity of NoSQL, HBase quickly became one of the most widely used storage backends at Pinterest.

Followed by,

> For the past few years, we have seen a seemingly steady decline in HBase usage and community activity in the industry, as many peer companies were looking for better alternatives to replace HBase in their production environments. This in turn has led to a shrinking talent pool, higher barrier to entry, and lower incentive for new engineers to become a subject matter expert of HBase.

Lets see how TiDB holds on in the same timeframe, although being boring SQL might keep it around.

Post reply on HN