Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CockroachDB 1.0

cockroachlabs.com

Post reply on HN
Print view View original

311–320 of 366 posts

Re: CockroachDB 1.0

#311

Post by newman314 »

nvarsj wrote: post #268
newman314 wrote: post #158

Earlier quoted context omitted.

Do people not run NTP on their VMs? Or are you saying that you see heavy clock skew despite having NTP in place?

Cassandra user here in AWS. Clock drift is a big problem on VMs. NTP is not aggressive enough in these environments to keep clocks relatively in sync. We regularly had several hundred milli drifts between nodes. As cassandra is extremely clock sensitive, this is a big problem. We ended up using chrony with very aggressive settings to keep things in the sub-ms range for the most part. But it's still possible to get "h…

Interesting. I wonder if anyone has documented any best practices for timekeeping in VMs.

VMware has this but it does not appear to have been updated in a while. https://kb.vmware.com/selfservice/microsites/search.do?langu...

Re: CockroachDB 1.0

#312

Post by d4l3k »

frik wrote: post #250
dis-sys wrote: post #6

I really like the fact that the CockroachDB team recently did a detailed Jepsen test with Aphyr. The follow up articles from both CockroachDB and Aphyr explaining the findings are very interesting to read. For those who might be interested - https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-j... https://jepsen.io/analyses/cockroachdb-beta-20160829

@CockroachDB dev: Has CockroachDB some health status page or REST api? (like Nginx/Apache/Redis/Memcached or a special table like MySQL) It would be helpful to monitor the CockroachDB database in production. I see there is some feature inbuild, but it only sends that data home to your server for analytics. (can be turned off) https://www.cockroachlabs.com/docs/diagnostics-reporting.htm...

CockroachDB has a rather nice admin interface that monitors the health of the cluster.

https://www.cockroachlabs.com/docs/explore-the-admin-ui.html

There's also a lot of rpc end points used for the admin UI that can be queried to get more fine grain info. However, they're primarily for internal use and might change in the future.

https://github.com/cockroachdb/cockroach/blob/master/pkg/ser...

https://github.com/cockroachdb/cockroach/blob/master/pkg/ser...

Re: CockroachDB 1.0

#313

Post by agibsonccc »

Others wrote: post #296
arjunnarayan wrote: post #122

Earlier quoted context omitted.

We have a post on why we chose Go, from a year and a half ago: https://www.cockroachlabs.com/blog/why-go-was-the-right-choi... More technically, here's a somewhat random set of thoughts on the subject: The Go GC is performant and predictable, unlike the JVM GC. We do have some very memory-allocation-conscious code patterns to minimize the performance impact of working in a garbage-collected language runtime, but in t…

Why do you think the Go GC is better than any the JVM options? From what I've seen, while the Go GC is well tuned for low latency, by picking the right JVM GC parameters you can on balance get a better throughput latency tradeoff. I'm just wondering if you have any reliable benchmarks or evidence to support what your saying? I don't use either language for work, so I think you might have better information than I.

We got around this by writing our own GC management: https://deeplearning4j.org/workspaces

We write our own GPU algorithms, Java native interface transpiler (eg: we generate JNI bindings) as well as our own memory management.

We've found the JVM to be more than suitable. Granted - we wrote our own tooling and had reasons we can't move (those customers are a neat thing most people don't think about :D)

I understand why you guys did go though. Congrats on pushing the limits of the runtime.

Re: CockroachDB 1.0

#314

Post by eblanshey »

dianasaur323 wrote: post #33
sergiotapia wrote: post #22

Is Cockroach DB intended for just "big-data" companies? Would a small project run really well with Cockroach DB? Of course a small database probably won't need a lot of the unique features, but is this aiming to replace PG/MySQL in the small/mid-size projects?

[cockroachdb here] Yes! In addition to being highly scalable, CockroachDB also comes with built-in replication. That means that even with a smaller project that hasn't scaled yet, you still get the benefit of a more resilient database. Also, CockroachDB is super easy to install and get started with!

I've come across many projects that are easy to get started with, but the main stuff to look for is in the details. Although MySQL might be easy to get into, for example, it takes time to learn the intricacies for query optimizations, and importantly, what to do when SHTF, like when a table gets corrupted.

My question is, in your opinion, what does it take to become proficient in CockroachDB sufficiently enough to be comfortable using it in a high volume, high-uptime-required environment?

Thanks.

Re: CockroachDB 1.0

#315

Post by state_machine »

frik wrote: post #250
dis-sys wrote: post #6

I really like the fact that the CockroachDB team recently did a detailed Jepsen test with Aphyr. The follow up articles from both CockroachDB and Aphyr explaining the findings are very interesting to read. For those who might be interested - https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-j... https://jepsen.io/analyses/cockroachdb-beta-20160829

@CockroachDB dev: Has CockroachDB some health status page or REST api? (like Nginx/Apache/Redis/Memcached or a special table like MySQL) It would be helpful to monitor the CockroachDB database in production. I see there is some feature inbuild, but it only sends that data home to your server for analytics. (can be turned off) https://www.cockroachlabs.com/docs/diagnostics-reporting.htm...

We're still working on integration with other monitoring systems, but the one we've tested the most and documented is prometheus: https://www.cockroachlabs.com/docs/monitor-cockroachdb-with-...

Additionally, you can get some of the same status info on the dashboard using the `node status` command (https://www.cockroachlabs.com/docs/view-node-details.html).

Re: CockroachDB 1.0

#316

Post by matt4077 »

thraway2016 wrote: post #79
anthonylebrun wrote: post #56

Since there's a little side riff about the name going on I thought I'd throw in my 2 cents. Personally I love the name. I think it does a great job of conveying the spirit of the project and provides unlimited pun opportunities. Plus it's memorable, just like a real life roach encounter. Unfortunately I'm sure some people will discriminate against your DB on the basis of name alone. That's ludicrous, but that's our s…

I see it as technical people on HN who appreciate the metaphor, versus marketing/business people who can only think of "image". It's to expected with the massive infestation of HN by suits and khakis in the last few years.

It appears to me that "marketing/business" people are simply stereotyped in this thread, because surely the complaints come mostly from "tech people".

It's the classic case of everyone saying "I think it's great buy will complain". Which ends in mindless mediocracy.

Go CockroachDB!

Re: CockroachDB 1.0

#317

Post by matt4077 »

jondubois wrote: post #121
anthonylebrun wrote: post #56

Since there's a little side riff about the name going on I thought I'd throw in my 2 cents. Personally I love the name. I think it does a great job of conveying the spirit of the project and provides unlimited pun opportunities. Plus it's memorable, just like a real life roach encounter. Unfortunately I'm sure some people will discriminate against your DB on the basis of name alone. That's ludicrous, but that's our s…

As the creator of a moderately popular open source project, I can attest that the name of the project is very important. A common problem for open source projects is that the name is not recognizable enough (e.g. too technical) or too generic (e.g. a simple English word which makes is heard to search on Google). In this case the name evokes negative emotions of fear and disgust which are not what you want to associat…

But you'd want your airline named "Virgin" and your morning-after pill named "Plan B"?

The name is, indeed, evocative. Good names don't have to universally convey "positive" emotions.

Re: CockroachDB 1.0

#319

Post by state_machine »

ralusek wrote: post #294
arjunnarayan wrote: post #83

Earlier quoted context omitted.

Not yet, but it's on our roadmap.

Just out of curiosity, do you mind elaborating a little bit on why not? It strikes me as something that would be very easy to implement in a database, is there a reason why so few databases have a mechanism to do this? If it's about maintaining an open connection in order to notify the client, that part makes sense, but at the very least the changefeed itself should be toggleable and easy to query in any DB.

One of the challenges for us in implementing something like LISTEN/NOTIFY comes from our distributed nature: since a table is likely broken up across many nodes, you somehow need to aggregate changes from all of them back into a single change feed wherever the listener is, and in such a way that it doesn't create a single point of failure.

Re: CockroachDB 1.0

#320

Post by sixdimensional »

ww520 wrote: post #278
sixdimensional wrote: post #138

How does Cockroach efficiently handle the shuffle step when data is on many nodes on the cluster and has to move to be joined? Does Cockroach need high capacity network links to function well? I always see companies making the claim of linear speedup with more nodes but surely that can't be the case if the nodes are geographically disjointed over anything less than gigabit links? Perhaps linear speedup with more node…

15 years ago I was working on a similar distributed DB product. At the time, the idea was to send the query execution plan to each node to execute any filtering criteria to trim down the candidate row set. Then compute a Bloom Filter on the joining keys on the node with the largest candidate set (using some heuristic statistics), ship the Bloom Filter to other nodes with smaller data set to greatly reduce the non-mat…

I've heard of pushdown techniques including function, predicate and aggregate pushdown in distributed relational engines before.

Another interesting idea I read about (I can't find it anywhere online) was called "join zippering". Basically you first request the cluster to solve a join by querying and streaming the key columns from a join predicate back into the cluster itself to identify which nodes have matches and then streaming the results from each node in parallel, and doing the join in the stream.

This is hard stuff but so cool too :)

Post reply on HN