This website would actually be better if they removed all the CSS and images.
LokiJS – In-memory JavaScript Datastore with Persistence
11–20 of 31 posts
Re: LokiJS – In-memory JavaScript Datastore with Persistence
#12While "Performance over everything" is obviously hyperbole, I hope they really don't mean it and actually prioritise correctness: that it apparently does actually store and retrieve data suggests that at some level, at least, they do.
There are plenty of use-cases where the risk of losing data to gain performance is perfectly acceptable.
What is important is being clear about what trade-offs they make.
As an example: Years ago I worked on a classifieds aggregation engine; we crawled hundreds of thousands of feeds, indexing tens of millions of listings.
All the queues in our processing pipeline were in-memory only: if we lost a few items (or a million) it didn't matter; worst case we'd pick them up no later than 24 hours later - in the case of more severe loss we'd simply trigger re-indexing of the feeds likely to have been "in flight" at the time. Failures would affect freshness and cause us to lose out on some updates, but as long as they were not severe enough or too frequent nobody would even notice, because the inventory was large and fleeting.
There are lots of cases like that where "performance over everything" can be a totally valid trade-off, because your data store is not the canonical source of information, doesn't need to be precise or comprehensive, and/or can be easily refreshed, for example.
This does require you to know and understand your requirements and what the datastore in question guarantees (or not), of course. If you don't, then you better stick with a "safe" option.
Re: LokiJS – In-memory JavaScript Datastore with Persistence
#13> In-Browser NoSQL db with syncing and persisting Is the syncing part as straightforward as Pouch[1]? I couldn't find anything about syncing on its docs. If a db comes with an easy offline-first and sync-when-possible model with granular permissions, it will be very interesting. Current solutions have (IMHO unacceptable) fixed dependencies on the backend or the architecture. [1]: https://pouchdb.com/
I always wondered what the use-cases for PouchDB are. Who wants to replicate their "whole" database locally?
You can filter what you are syncing or create a db per user and declare interactions with foreign data in their own instance, which will be executed according to permissions in the backend.
What I actually want is MeteorJS with neither MongoDB or the JavaScript backend, or PouchDB without Couch, or Jaydata + ASP.NET WebAPI with more open source compatibility...
Re: LokiJS – In-memory JavaScript Datastore with Persistence
#14While "Performance over everything" is obviously hyperbole, I hope they really don't mean it and actually prioritise correctness: that it apparently does actually store and retrieve data suggests that at some level, at least, they do.
MongoDB all over again.
Well, if you're building the next SnapChat, MongoDB is an entirely reasonable option and may even save you some coding time!
Re: LokiJS – In-memory JavaScript Datastore with Persistence
#15> In-Browser NoSQL db with syncing and persisting Is the syncing part as straightforward as Pouch[1]? I couldn't find anything about syncing on its docs. If a db comes with an easy offline-first and sync-when-possible model with granular permissions, it will be very interesting. Current solutions have (IMHO unacceptable) fixed dependencies on the backend or the architecture. [1]: https://pouchdb.com/
I always wondered what the use-cases for PouchDB are. Who wants to replicate their "whole" database locally?
there is a forthcoming couch plugin that does filtered replication by user
Re: LokiJS – In-memory JavaScript Datastore with Persistence
#16Earlier quoted context omitted.
MongoDB all over again.
Well, if you're building the next SnapChat, MongoDB is an entirely reasonable option and may even save you some coding time!
Simply use MongoDB with its default settings, and you're good to go with the next Snap-whatever product you want to build.
Re: LokiJS – In-memory JavaScript Datastore with Persistence
#17While "Performance over everything" is obviously hyperbole, I hope they really don't mean it and actually prioritise correctness: that it apparently does actually store and retrieve data suggests that at some level, at least, they do.
There are plenty of use-cases where the risk of losing data to gain performance is perfectly acceptable. What is important is being clear about what trade-offs they make. As an example: Years ago I worked on a classifieds aggregation engine; we crawled hundreds of thousands of feeds, indexing tens of millions of listings. All the queues in our processing pipeline were in-memory only: if we lost a few items (or a mill…
That's true, but it's got to be a trade-off and a balance of risk, because "Performance over everything" makes no sense at all: the most performant thing to do is to store no data and return no data.
And even with that trade-off, I'd argue that there's a level of correctness that should be non-negotiable. Missing data might be OK, but corrupting data on the way in is bad, and failing to process everything might be OK (eventual consistency and all that) but you should meet your published guarantees and you shouldn't corrupt your results.
So I would argue for correctness over performance, with the caveat that "correct" has a technical meaning specific to the project.
Re: LokiJS – In-memory JavaScript Datastore with Persistence
#18While "Performance over everything" is obviously hyperbole, I hope they really don't mean it and actually prioritise correctness: that it apparently does actually store and retrieve data suggests that at some level, at least, they do.
(Full disclosure, I work at a database company) Their performance is pretty impressive, but you can get over 30X their speed (30M ops/sec) by caching - see https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... .
And this can be done without compromising the storage or integrity of the data. The Observable pattern lets you update caches while you perform writes so the data stays correct, reflecting what is on disk. If caches don't exist then you pull from disk (slow) and write to the caches.
Re: LokiJS – In-memory JavaScript Datastore with Persistence
#19> In-Browser NoSQL db with syncing and persisting Is the syncing part as straightforward as Pouch[1]? I couldn't find anything about syncing on its docs. If a db comes with an easy offline-first and sync-when-possible model with granular permissions, it will be very interesting. Current solutions have (IMHO unacceptable) fixed dependencies on the backend or the architecture. [1]: https://pouchdb.com/
I too haven't seen anything on Loki's sync ability. Obviously would be a very nice feature but difficult to do right.
Pouch, Firebase, and GUN[1] (disclaimer, I work there) are examples. If you bake permissions into the system itself, you get monoliths like Meteor. But you can still achieve granular permissions by rejecting/accepting network requests (like with ExpressJS or Hapi, etc.), which gives you the balance of using whatever backend you like.
Re: LokiJS – In-memory JavaScript Datastore with Persistence
#20While "Performance over everything" is obviously hyperbole, I hope they really don't mean it and actually prioritise correctness: that it apparently does actually store and retrieve data suggests that at some level, at least, they do.
(Full disclosure, I work at a database company) Their performance is pretty impressive, but you can get over 30X their speed (30M ops/sec) by caching - see https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2... . And this can be done without compromising the storage or integrity of the data. The Observable pattern lets you update caches while you perform writes so the data stays correct, reflecting what is o…
> Caching speed is nx database speed.
Well, no shit?