Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Pegasus - A horizontally scalable, high-performance key-value store

github.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Apache Pegasus - A horizontally scalable, high-performance key-value store

#31

Post by xwowsersx »

ddorian43 wrote: post #26
yla92 wrote: post #12

Earlier quoted context omitted.

> Those are all many times slower than Redis though I'd appreciate if there any links/doc that I could look into to learn more about this?

Blog post from antirez http://oldblog.antirez.com/post/redis-persistence-demystifie...

Thanks for the link. In that post, he writes "Note: when we talk about disk controller we actually mean the caching performed by the controller or the disk itself. In environments where durability is important system administrators usually disable this layer of caching." I'm not sure I completely understand that note. Can you illuminate?

Re: Apache Pegasus - A horizontally scalable, high-performance key-value store

#32

Post by karmakaze »

xwowsersx wrote: post #31
ddorian43 wrote: post #26

Earlier quoted context omitted.

Blog post from antirez http://oldblog.antirez.com/post/redis-persistence-demystifie...

Thanks for the link. In that post, he writes "Note: when we talk about disk controller we actually mean the caching performed by the controller or the disk itself. In environments where durability is important system administrators usually disable this layer of caching." I'm not sure I completely understand that note. Can you illuminate?

That's probably in cases where the disk controller can't guarantee that accepted writes will make it to the disk in the case of power failure. I worked with bare metal servers that had redundant power supplies including to the caching/raid disk controller and disks that in the event of power loss could still guarantee that writes that were 'sync'd by software were written after power loss.

Re: Apache Pegasus - A horizontally scalable, high-performance key-value store

#33

Post by xwowsersx »

karmakaze wrote: post #32
xwowsersx wrote: post #31

Earlier quoted context omitted.

Thanks for the link. In that post, he writes "Note: when we talk about disk controller we actually mean the caching performed by the controller or the disk itself. In environments where durability is important system administrators usually disable this layer of caching." I'm not sure I completely understand that note. Can you illuminate?

That's probably in cases where the disk controller can't guarantee that accepted writes will make it to the disk in the case of power failure. I worked with bare metal servers that had redundant power supplies including to the caching/raid disk controller and disks that in the event of power loss could still guarantee that writes that were 'sync'd by software were written after power loss.

Ah got it, thanks. What a fantastic read btw, his writing is incredibly concise and clear. I have to binge some of his other writing.

Re: Apache Pegasus - A horizontally scalable, high-performance key-value store

#34

Post by l2dy »

seeekr wrote: post #18

Note that this seems to be a relatively old project, first commits from 2015. The project seems active, but most of the work seems to have been done around its inception, with some significant activity from 2020 onwards. Speculation/interpretation: So this might be a project that was used internally by some company, but perhaps not any more, and they've decided to open-source it at some point (2017-2018?) because som…

Pegasus was open-sourced by Xiaomi, and is still used internally according to https://apachecon.com/acasia2022/sessions/ai-1125.html.

Source: https://www.zhihu.com/question/66719537/answer/245270169 (in Chinese)

Re: Apache Pegasus - A horizontally scalable, high-performance key-value store

#36

Post by asjo »

studmuffin650 wrote: post #30
xani_ wrote: post #23

Earlier quoted context omitted.

Apache foundation is retirement home for projects so it checks out

I wouldn't say thats accurate. Theres many super successful and active apache projects still. To name a few: Kafka, Cassandra, Zookeeper, Spark, Tomcat, Superset, Storm, Lucene,Log4j2, Hadoop, etc. The list goes on, but I would safely say that a majority of the world's systems run on Apache projects which are for the most part actively developed

And arguably the venerable Apache HTTP Server · https://httpd.apache.org/ :-)

Re: Apache Pegasus - A horizontally scalable, high-performance key-value store

#37

Post by ramesh31 »

asjo wrote: post #36
studmuffin650 wrote: post #30

Earlier quoted context omitted.

I wouldn't say thats accurate. Theres many super successful and active apache projects still. To name a few: Kafka, Cassandra, Zookeeper, Spark, Tomcat, Superset, Storm, Lucene,Log4j2, Hadoop, etc. The list goes on, but I would safely say that a majority of the world's systems run on Apache projects which are for the most part actively developed

And arguably the venerable Apache HTTP Server · https://httpd.apache.org/ :-)

Despite all the modern alternatives, I'd bet my life that a majority (>50%) of the web is still being served by httpd.

Re: Apache Pegasus - A horizontally scalable, high-performance key-value store

#39

Post by mekster »

studmuffin650 wrote: post #30
xani_ wrote: post #23

Earlier quoted context omitted.

Apache foundation is retirement home for projects so it checks out

I wouldn't say thats accurate. Theres many super successful and active apache projects still. To name a few: Kafka, Cassandra, Zookeeper, Spark, Tomcat, Superset, Storm, Lucene,Log4j2, Hadoop, etc. The list goes on, but I would safely say that a majority of the world's systems run on Apache projects which are for the most part actively developed

Not sure about others but tools like Superset have plenty of better alternatives like Grafana and Metabase.

So when there are better alternatives, it's not too inaccurate.

Re: Apache Pegasus - A horizontally scalable, high-performance key-value store

#40

Post by michaeltlewis »

pas wrote: post #19
seeekr wrote: post #18

Note that this seems to be a relatively old project, first commits from 2015. The project seems active, but most of the work seems to have been done around its inception, with some significant activity from 2020 onwards. Speculation/interpretation: So this might be a project that was used internally by some company, but perhaps not any more, and they've decided to open-source it at some point (2017-2018?) because som…

> yet another RocksDB-based KV store Aaah, there was a super informative talk about the different databases at Facebook, most of them built on RocksDB, with different trade offs. (And I can't find the video :((((( ) Anyway, it makes sense to have yet another it if serves a different purpose. Eg. for read-heavy workloads (caches, serving user feeds, whatever), or write-heavy (monitoring, storing that sweet sweet track…

That video sounds interesting, if you do manage to find it.
Post reply on HN