41–50 of 51 posts
For god's sake, man, run a linter on this thing. The mixing of spaces and tabs alone makes the code embarrassingly difficult to read. http://imgur.com/a/XlSQD
Fix your editor and set the tabspace to the correct value ? I'm guessing 2 spaces is the right value in this case.
Earlier quoted context omitted.
Look not to sound harsh but since you're claiming the "80mm ops/sec" as a technical achievement "not possible with other databases" I found it to be fair to actually review the benchmark you linked. What you're benchmarking for the "read" is this: benchmark(function(){ gun.val(ok); }); Looking at what the gun.val call does it seems to be more or less an identitiy/noop function (that simply returns its input). It appe…
The nitty gritty details are on this readthesource.io podcast: https://youtu.be/70dn1oZQFCk (watch on 2X speed). Each process does concurrency control and then has a centralized in-memory cache for the values (like what a lot of other in-memory databases do). So when `gun.val(cb)` is called, prototype context holds its value and is able to do an immediate read - without this, JS is so slow that every function call lo…
Are you defending the fact that you benchmarked whats essentially a "function() { return 1; }", called that a "database operation" and then proceeded to claim to have achieved a level of performance "not possible with other databases"? Can you really not see how that is wrong on more than just one level?
Or are you saying it's okay because other "javascript database" vendors are also cheating in their benchmarks?
Earlier quoted context omitted.
The nitty gritty details are on this readthesource.io podcast: https://youtu.be/70dn1oZQFCk (watch on 2X speed). Each process does concurrency control and then has a centralized in-memory cache for the values (like what a lot of other in-memory databases do). So when `gun.val(cb)` is called, prototype context holds its value and is able to do an immediate read - without this, JS is so slow that every function call lo…
I don't really get what you're trying to say. Are you defending the fact that you benchmarked whats essentially a "function() { return 1; }", called that a "database operation" and then proceeded to claim to have achieved a level of performance "not possible with other databases"? Can you really not see how that is wrong on more than just one level? Or are you saying it's okay because other "javascript database" vend…
Other javascript databases and even other regular in-memory databases do similar benchmarks. We (both them and us) note at the bottom of the article that: "Take all performance testing benchmarks with a huge grain of salt." Which is why we have our PANIC tests (see my other comments on where to find them and how to run them, let me know if you need any help).
There is nothing unethical about our tests (and they are not the only tests), so /please/ call me out, but please don't misinform other readers that the op is a static function, that is misleading and damaging. Is there anything specific I can do to alleviate any concerns?
The title and the first line of the README > GUN is a realtime, distributed, offline-first, graph database engine. Doing 15M+ ops/sec in just ~12KB gzipped. But no mention of any guarantees provided by the system. What kind of transactions are supported? are writes consensus-based, best effort, eventual consistent? The "GUN Survives a Primary Fault" raises more questions than answers. The "primary" appears to be the…
- CAP Theorem tradeoffs: https://github.com/amark/gun/wiki/CAP-Theorem (We are AP, which is eventually consistent)
- My tech talk of how the CRDT works: http://gun.js.org/distributed/matters.html (compared to consensus protocols)
Clients keep a copy of data that they are interested in (not the whole data set, correct), and together they can reconstruct the entire data set. You can also run multiple server peers that backup the whole data set, or use a shard key to determine which subsets. If this isn't fault tolerant, would you mind giving me some examples of what is?
Yes, good point, to prevent data from getting trashed you need to run auth on the system, more info here: https://github.com/amark/gun/wiki/auth
We certainly aren't a storage engine, but we are a database/datastore (whatever you want to call it), or more appropriately: an open data sync protocol. I have tried to make our guarantees/tradeoffs very well laid out in the above mentioned articles/README, anything I can do to make this more clear/obvious?
millions of reads per second on mobile devices for a distributed database? just come on, why even post this?
Earlier quoted context omitted.
Yes, Andre Staltz's (of CycleJS) work finally got it working on Android and iOS with React-Native! We're planning on having an example/starter app for it soon (unfortunately, Google Chrome and iOS follow the WebSocket spec just slightly differently, so it errors on iOS currently but it is an easy fix, we just need to figure out a way for them to both work simultaneously).
That sounds great :). Any place I should keep an eye on for updates?
This poster has been working on this and spamming it all over HN for years, it's been proven to have major flaws and full of misleading claims with all those posts. millions of reads per second on mobile devices for a distributed database? just come on, why even post this?
But proven flawed? Could you send me some links/evidence on that? I'd love to do my best to reply/address any concerns.
The title and the first line of the README > GUN is a realtime, distributed, offline-first, graph database engine. Doing 15M+ ops/sec in just ~12KB gzipped. But no mention of any guarantees provided by the system. What kind of transactions are supported? are writes consensus-based, best effort, eventual consistent? The "GUN Survives a Primary Fault" raises more questions than answers. The "primary" appears to be the…
The README mentions it in the #documentation section and links to these resources: (let me know how they can be improved) - CAP Theorem tradeoffs: https://github.com/amark/gun/wiki/CAP-Theorem (We are AP, which is eventually consistent) - My tech talk of how the CRDT works: http://gun.js.org/distributed/matters.html (compared to consensus protocols) Clients keep a copy of data that they are interested in (not the who…
Maybe I'm just being silly, but that talk describes the problem. It doesn't really going into detail about the CRDT(s) you're using.
> Yes, good point, to prevent data from getting trashed you need to run auth on the system, more info here: https://github.com/amark/gun/wiki/auth
I wasn't talking about auth.
If conflict resolution is timestamp based, It feels like you could very easy end up with inconsistent data. I update field Y based upon the value of field X I see. So does someone else, but based upon a different value for field X. If I can't wrap this this in a transaction, then I could update fields based on bad/out-of-date data.
> Clients keep a copy of data that they are interested in (not the whole data set, correct), and together they can reconstruct the entire data set. You can also run multiple server peers that backup the whole data set, or use a shard key to determine which subsets. If this isn't fault tolerant, would you mind giving me some examples of what is?
I mean, I guess if you accept being able to loose data as fault tolerant, then OK. Having multiple, dedicated, full-database backups is what every other distributed database does as well.
I mean, perhaps I'm being too hash; there are many usecases, particularly with data of little value, that systems like this make easy to build. I just feel like so many people are working on datastores with loose guarantees about anything and hail them as awesome, new things.
Heck, even Wave had a more refined system where you wouldn't just randomly loose data.
You're not Google. You're not Amazon. You don't need to embrace all of this looseness and lack of guarantees.
Earlier quoted context omitted.
The README mentions it in the #documentation section and links to these resources: (let me know how they can be improved) - CAP Theorem tradeoffs: https://github.com/amark/gun/wiki/CAP-Theorem (We are AP, which is eventually consistent) - My tech talk of how the CRDT works: http://gun.js.org/distributed/matters.html (compared to consensus protocols) Clients keep a copy of data that they are interested in (not the who…
> - My tech talk of how the CRDT works: http://gun.js.org/distributed/matters.html (compared to consensus protocols) Maybe I'm just being silly, but that talk describes the problem. It doesn't really going into detail about the CRDT(s) you're using. > Yes, good point, to prevent data from getting trashed you need to run auth on the system, more info here: https://github.com/amark/gun/wiki/auth I wasn't talking about…
Yes, the first half of the talk reviews the problem, but the last half (starting with the boundary functions) explains the CRDT. If that wasn't helpful, then there is also this article on the implementation (not written by me) https://github.com/amark/gun/wiki/Conflict-Resolution-with-G....
Timestamps are bad, yes (I mention this in the talk as well), GUN uses a hybrid vector/timestamp/lexical CRDT. Lets take your analogy "updating Y when someone else sees X", a perfect example of this is realtime document collaboration (gDocs, etc.). Even with GUN, you'd not want to have the collaborative paragraph as a value on a property in the node. Each value is treated as atomic, which if two people write at the same time would cause what you are saying: them to overwrite each other. Instead, we can preserve the intent by running distributed linked list (a DAG, actually) on gun, this works quite nicely. See:
- Early working prototype/demo here: https://youtu.be/rci89p0o2wQ
- Based off the interactive data algorithms explainer article here: http://gun.js.org/explainers/school/class.html
If instead, you don't want the results to merge, but indeed be atomic (only one person or the other "wins") then even with transactions only one is going to overwrite the other. Transactions don't help unless you have a journal to rollback from - and guess what, that is possible with gun too. Or alternatively, if you just want users to be informed of the conflict such that they can decide, it is trivial to store both updates and then present them with the conflicting values to choose from, which it then saves.
Does that make sense? Do you see/have any problems with those approaches? Thanks for your input so far.
Regarding clients. Yes, absolutely, you should still run multiple dedicated full-database backups. I don't disagree with you on that point, in fact we make it easy and scalable (see our demo video of a prototype storage engine that did 100M+ records for $10/day all costs - servers, disk, and backup: https://youtu.be/x_WqBuEA7s8). The unique thing about GUN is that it is still capable of surviving complete data center outages, because the data is also backed up on the edge peers.
You can also reduce your bandwidth costs by having edge peers distribute data to their nearby peers, versus always having to pull from your data center. Aka, the Bittorrent model.
Lets be clear here, there is a big difference between Master-Slave systems and data guarantees. Databases like Cassandra have better data availability guarantees because they are HA (and gun is the same), despite not being Master-Slave.
But at the end of the day, you are right: Your not-losing-your-data is only as good as how many full replication backups you have. What I hope to have communicated to you is that gun makes it ridiculously easy to make full (and partial) replications beyond traditional databases/datastores/whatever-you-call-thems, and I hope you do think that is awesome.
Just don't use us to balance bank account data, cause we don't provide those types of guarantees, but we do provide the HA / AP / fault-tolerance ones. :)
Earlier quoted context omitted.
> - My tech talk of how the CRDT works: http://gun.js.org/distributed/matters.html (compared to consensus protocols) Maybe I'm just being silly, but that talk describes the problem. It doesn't really going into detail about the CRDT(s) you're using. > Yes, good point, to prevent data from getting trashed you need to run auth on the system, more info here: https://github.com/amark/gun/wiki/auth I wasn't talking about…
Thanks for the reply! I really appreciate the dialogue. Yes, the first half of the talk reviews the problem, but the last half (starting with the boundary functions) explains the CRDT. If that wasn't helpful, then there is also this article on the implementation (not written by me) https://github.com/amark/gun/wiki/Conflict-Resolution-with-G... . Timestamps are bad, yes (I mention this in the talk as well), GUN uses…
This is actacly my point. And GDocs is a perfect example of what _I_ mean. The Wave protocol (and operational transforms in general), or I guess they use a decedent of it now, was made for exactly these types of use-cases so that you don't loose data unexpectedly when editing values.
> Or alternatively, if you just want users to be informed of the conflict such that they can decide, it is trivial to store both updates and then present them with the conflicting values to choose from, which it then saves.
> Does that make sense? Do you see/have any problems with those approaches? Thanks for your input so far.
How is it trivial? Do you have array-value types? What's trivial about it?
And yes, I do have problems when the default conflict resolution method is to simply select a value and toss the other out. It's a bad default. You have a default that causes data loss, it _will_ come back to bite you.
This is perhaps the most egregious thing and honestly makes me have 0 trust in your system.
> Lets be clear here, there is a big difference between Master-Slave systems and data guarantees. Databases like Cassandra have better data availability guarantees because they are HA (and gun is the same), despite not being Master-Slave.
I never mentioned master-slave. You can set up most SQL databases to be async- or sync- master-master.
> You can also reduce your bandwidth costs by having edge peers distribute data to their nearby peers, versus always having to pull from your data center. Aka, the Bittorrent model.
Maybe I'm not sure what you mean by edge here. Do you mean it in the CDN sense, or in the client sense? If you're expecting your clients to be part of your DR scheme, I'm sorry, but that's moronic. You can't guarantee client availability. You can't guarantee that all available clients will even have a copy of all data. Sure, it's great that you're using the same protocol between servers, but you can't say that because of that clients can be part of DR.