Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a very fast cache service with millions of entries in Go

allegro.tech

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Writing a very fast cache service with millions of entries in Go

#71

Post by im_down_w_otp »

Xorlev wrote: post #69
im_down_w_otp wrote: post #64

Earlier quoted context omitted.

Uses Raft. Not highly-available enough unfortunately. Availability > Consistency for the use case that matters to me (basically a service message bus where all ACK'd writes will eventually be read by some consumer(s), but I don't care when or in what order, just that they all eventually get consumed, and where as long as any node in the cluster is up, it'll ACK a write).

> just that they all eventually get consumed Unless you have some sort of replication or consensus, this is still just best-effort delivery. For example, the message could be acknowledged & fsynced followed by drive death. At scale, this would happen. However, with asynchronous best-effort replication, broker death would be much less likely to lead to a loss of messages. In general, it's all about playing the odds. A…

If I tune my Kafka settings I can't get this behavior.

The closest I can get is allowing sloppy leader election, but that will definitely, definitely lose writes that have been ACK'd by a stale leader who comes back and has to become a follower. So I can get Kafka to basically accept all writes regardless of level of cluster degradation, but I can't keep it from dropping some of those writes silently onto the floor.

Re: Writing a very fast cache service with millions of entries in Go

#72

Post by lobster_johnson »

im_down_w_otp wrote: post #55
endymi0n wrote: post #38

I still don’t get why people try writing their own data store, especially in a language that's simply not very well suited to that task (and we're an almost 100% Golang shop here). Seems to be a rite of passage. The requirements are literally Public service announcement: Don't write your own data store. Repeat after me: Don't write your own data store, except if you want to experimentally find out how to build data s…

There are two datastores I would really like that don't exist. 1) An efficient, high-performance distributed persistence store for arbitrarily large CRDT's. 2) A Kafka-like highly-available distributed binary log w/ cheap topics, that doesn't require external coordination, and doesn't lose acknowledged writes (which I'll happily give up any shape of linearization guarantee for).

Riak does CRDTs (not sure about arbitrarily large), though I never understood their reasoning for requiring that CRDTs be pre-registered in a schema — it's a major drawback in an otherwise schemaless K/V store.

Re: Writing a very fast cache service with millions of entries in Go

#73

Post by jakub_h »

endymi0n wrote: post #38

I still don’t get why people try writing their own data store, especially in a language that's simply not very well suited to that task (and we're an almost 100% Golang shop here). Seems to be a rite of passage. The requirements are literally Public service announcement: Don't write your own data store. Repeat after me: Don't write your own data store, except if you want to experimentally find out how to build data s…

That sounds a little bit harsh (even if I ignore the claims of unsuitability of Go for that purpose). What if I want to write, for example, a Go port of DataDraw? Perhaps with profiling-based memory layout optimization thrown in? And maybe even caching/lazy/caching+lazy collections/maps. If I were obeying your recommendations verbatim, I'd probably still be using mainframe libraries.

Re: Writing a very fast cache service with millions of entries in Go

#74

Post by stcredzero »

mhd wrote: post #30
vog wrote: post #8

Earlier quoted context omitted.

Maybe this type of program is better suited for a language like Rust. However, while not having a GC, you have to take care of all memory issues in a way that you convince the compiler that your won't ever blow up. It would be very interesting to have such a comparison, so we could see whether it's easier to work around the GC, or easier to write bullet-proof code with manual memory management. I'd expect the Rust to…

I'm not sure how the current implementation of Go handles it, but its spiritual relatives Modula-3/Oberon handled this quite well, with a GC for most occasions and ways to bypass this with "unsafe" modules that allowed for untracked allocations/deallocations and pointer arithmetic. It's not really an either/or situation by (language) definition...

I'm puzzled as to why all mature GC doesn't have "permspace."

Re: Writing a very fast cache service with millions of entries in Go

#75

Post by im_down_w_otp »

lobster_johnson wrote: post #72
im_down_w_otp wrote: post #55

Earlier quoted context omitted.

There are two datastores I would really like that don't exist. 1) An efficient, high-performance distributed persistence store for arbitrarily large CRDT's. 2) A Kafka-like highly-available distributed binary log w/ cheap topics, that doesn't require external coordination, and doesn't lose acknowledged writes (which I'll happily give up any shape of linearization guarantee for).

Riak does CRDTs (not sure about arbitrarily large), though I never understood their reasoning for requiring that CRDTs be pre-registered in a schema — it's a major drawback in an otherwise schemaless K/V store.

But they're also required to be relatively small, because of how they're treated on disk and inside the system as opaque at the storage layer. As a result of this, and a smattering of other reasons, they're also not very fast.

Re: Writing a very fast cache service with millions of entries in Go

#76

Post by QuercusMax »

im_down_w_otp wrote: post #63
chris_va wrote: post #59

Earlier quoted context omitted.

(1) Spanner does exist, though not really available outside of Google

Yes, Spanner does exist. Spanner also does not satisfy what I said in #1.

Can you elaborate on this?

Re: Writing a very fast cache service with millions of entries in Go

#77

Post by schmichael »

stcredzero wrote: post #74
mhd wrote: post #30

Earlier quoted context omitted.

I'm not sure how the current implementation of Go handles it, but its spiritual relatives Modula-3/Oberon handled this quite well, with a GC for most occasions and ways to bypass this with "unsafe" modules that allowed for untracked allocations/deallocations and pointer arithmetic. It's not really an either/or situation by (language) definition...

I'm puzzled as to why all mature GC doesn't have "permspace."

Go's GC is non-moving and therefore cannot be generational.

http://llvm.cc/t/go-1-4-garbage-collection-plan-and-roadmap-...

Re: Writing a very fast cache service with millions of entries in Go

#78

Post by pcwalton »

schmichael wrote: post #77
stcredzero wrote: post #74

Earlier quoted context omitted.

I'm puzzled as to why all mature GC doesn't have "permspace."

Go's GC is non-moving and therefore cannot be generational. http://llvm.cc/t/go-1-4-garbage-collection-plan-and-roadmap-...

Nitpick: you don't need moving GC for generational GC. But you lose most (but not all) of the benefit of generational GC without moving GC.

Re: Writing a very fast cache service with millions of entries in Go

#79

Post by endymi0n »

maleck13 wrote: post #70
endymi0n wrote: post #38

I still don’t get why people try writing their own data store, especially in a language that's simply not very well suited to that task (and we're an almost 100% Golang shop here). Seems to be a rite of passage. The requirements are literally Public service announcement: Don't write your own data store. Repeat after me: Don't write your own data store, except if you want to experimentally find out how to build data s…

Can you explain further why golang is bad for data stores, do you mean cache in particular? Seems contrary to many of the new datastores that I see being developed: etcd[1] , cockroachdb[2], influxdb[3] just to name a couple that are written in go. [1] https://github.com/coreos/etcd [2] https://github.com/cockroachdb/cockroach [3] https://github.com/influxdata/influxdb

Go is awesome, but has very few facilities for effectively working very close to the machine and memory - you can't skip the runtime or GC. Of course, this mostly affects high throughput, high mutation rate, high memory use data stores. Eventually, you'll end up fighting the garbage collector all the time. ETCD isn't directly affected as it isn't so affected as it's not really about larger data or performance, cockroachdb is still more proof-of-concept of a pretty gigantic (though promising) architecture and Influx has honestly been an imperformant mess for us in Prod so far. It's not that it's not possible to write a decent data store in Go, but eventually, as your GC runs wild from the purposefully simple and naive stdlib implementations of maps for example, the neccessary solutions of aggressive pooling or mmapping off-heap space and writing your own allocators for it (while constantly casting something from and to *unsafe) will suck you deep down the rabbit hole and wish you had written it in C(++) or Rust in the first place.

Re: Writing a very fast cache service with millions of entries in Go

#80

Post by jasonwatkinspdx »

im_down_w_otp wrote: post #64
endymi0n wrote: post #57

Earlier quoted context omitted.

can't help with 1), but as for 2), maybe Kudu is an option for you? http://blog.rodeo.io/2016/01/24/kudu-as-a-more-flexible-kafk...

Uses Raft. Not highly-available enough unfortunately. Availability > Consistency for the use case that matters to me (basically a service message bus where all ACK'd writes will eventually be read by some consumer(s), but I don't care when or in what order, just that they all eventually get consumed, and where as long as any node in the cluster is up, it'll ACK a write).

> Uses Raft. Not highly-available enough unfortunately.

Did you test this or is it just a general view? What sort of write rate are you looking for, per topic?

Post reply on HN