311–320 of 427 posts
Earlier quoted context omitted.
Encryption at rest is still important as it closes off a few attack/loss vectors: mis-disposed hard drives, re-allocated hosts. I'm probably missing a few others.
Yeah, but it doesn't really address my concern.
How is running on AWS different than Guardian Cloud in their basement?
Earlier quoted context omitted.
Yeah, but it doesn't really address my concern.
Anyone who can control a server in any environment can potentially interact with the database powering applications running on that server. How is running on AWS different than Guardian Cloud in their basement?
Did reports of the Snowden revelations reside on the CMS?
Earlier quoted context omitted.
Yes, you can! The jsonb_array_elements function is roughly similar to Mongo’s $unwind pipeline op. It explodes a JSON array into a set of rows. From there it’s pretty simple aggregates to achieve what you’re looking for. I was evaluating Mongo a couple months back to solve roughly the same problems. Eventually discovered Postgres already had what I was looking for.
Allow me to restate this question: > Does it do that? It was supposed to be clear from the context that this meant: > Does building queries programmatically with SQLAlchemy do that? Maybe I'm misreading your comment, but you seem to just be talking about writing queries directly in SQL. If not, could you give an example/link of how to programmically build a query in SQLAlchemy that dynamically makes use of jsonb_arra…
Earlier quoted context omitted.
Anyone who can control a server in any environment can potentially interact with the database powering applications running on that server. How is running on AWS different than Guardian Cloud in their basement?
The level of control over who has physical access, of course. Did reports of the Snowden revelations reside on the CMS?
After all, the client computer that connects to the CMS is just as, or more likely to be compromised. I wouldn't be surprised if the coverage (or at least parts of it) were edited on airgapped laptops.
Earlier quoted context omitted.
You make it sound like learning SQL is like learning Assembler. It's not that hard. And ORMs exist in every language to abstract it all away. PostgreSQL looks scary because it is a swiss army knife. It has a million different features and data structures. MongoDB does only one thing.
Well, that and SQL has a somewhat undeserved reputation for being easy to learn, but also easy to screw up. Like you write a simple looking query and it turns out to have O^2 complexity and your system ends up bogged down in the database forever. In practice people who fall into complexity traps are usually asking a lot more of their database engine than any beginner. It's usually not that hard to figure out the appr…
Or you have a simple fast query with a lovely plan until the database engine decides that because you now have 75 records in the middle table instead of 74, the indexes are suddenly made of lava and now you're table-scanning the big tables and your plan looks like an eldritch horror.
[Not looking at MySQL in particular, oh no.]
Earlier quoted context omitted.
Saying "I don't know SQL so I will just use JSON" really misses the point though. SQL is easy. Data is hard. NoSQL products offer to get rid of SQL which includes an implication that SQL itself was the challenge in the first place. The problem then is that you have lost one of the best tools for working with data.
I dunno that SQL is exactly easy, though. It's one thing to say "select statements are essentially identical to Python list comprehensions", but in practice I still have to look up the Venn diagram chart every time I need to join anything, and performance optimization is still a dark art. I'd say SQL is easy in the same way that Git is easy: you can get away with using just 5% of it, but you'll still need to consult…
Mongo and Javascript don't solve that either. In fact you get additional problems by virtue of not being able to do a variety of joins. For extra points, you're going to need to go well beyond javascript with mongo if you want performance. 10gen invented this whole "aggregation framework" to sidestep the performance penalty that javascript brings to the table.
On the other side, the postgresql documentation is second to none. SQL isn't necessarily easy but the postgres documentation gives you an excellent starting point.
Earlier quoted context omitted.
The level of control over who has physical access, of course. Did reports of the Snowden revelations reside on the CMS?
Probably not, no, until they were about to be published. I imagine that the choice between "run an entire data centre ourselves, store everything there" and "use AWS, but keep high sensitivity stories on local machines" is an easy one. After all, the client computer that connects to the CMS is just as, or more likely to be compromised. I wouldn't be surprised if the coverage (or at least parts of it) were edited on a…
If those were the only two choices, you might be right. But the resources needed for the actual CMS functionality sound modest enough to run independently of the main website.
> the client computer that connects to the CMS is just as, or more likely to be compromised
That's faulty reasoning.
Earlier quoted context omitted.
You make it sound like learning SQL is like learning Assembler. It's not that hard. And ORMs exist in every language to abstract it all away. PostgreSQL looks scary because it is a swiss army knife. It has a million different features and data structures. MongoDB does only one thing.
> You make it sound like learning SQL is like learning Assembler It's not that learning SQL is hard. It's that people are inherently lazy. "Learn another thing on top of the thing it already took me a couple of years to learn? No thanks." You seem like the kind of person ready and willing to learn the right tool for the job. From my experience a few years ago on an accredit computing course that covered database admi…
And that's a big fat mistake. There are so many ways to shoot yourself in the foot with mongo such that simply knowing the language mongo uses for most of its queries while not actually knowing the particulars of how mongo uses that language… well that's just a road to a world of hurt.
For example, when I first inherited a mongo deployment I noticed the queries were painfully slow. Ah hah says me, let's index some shit. Guess what? Creating an index on a running system with that version of mongo = segfault.
After a bunch of hair pulling I got mongo up and running and got the data indexed. But the map reduce job was STILL running so slowly that we couldn't ingest data from a few tens of sensors in real time. So I made sure to set up queues locally on the sensors to buy myself some time.
Even in my little test environment with nothing else hitting the mongo server, mongod was still completely unable to run its map reduce nonsense in a performant manner. Mongo wisdom was: shard it! wait for our magical aggregation framework! Here's the thing: working at a dinky startup we can't afford to throw hardware at it especially that early in the game. Sharding the damn thing would also bring in mongo's inflexible and somewhat magical and unreliable sharding doohickey.
So I thought back to previous experience with time series data. BTDT with MySQL, you're just trading one awful lock (javascript vm) for another (auto increment). So I set up a test rig with postgres. Bam. I was able to ingest the data around 18x faster.
And that's the thing. Mongo appeals to people who are comfortable with javascript and resistant to learning domain specific knowledge. All that appealing javascript goodness comes with a gigantic cost. If you're blindly following the path of least resistance you're in for a bad time.
P.S. plv8 is a thing, and you can script postgres in javascript if you really wanted to.
Earlier quoted context omitted.
> You make it sound like learning SQL is like learning Assembler It's not that learning SQL is hard. It's that people are inherently lazy. "Learn another thing on top of the thing it already took me a couple of years to learn? No thanks." You seem like the kind of person ready and willing to learn the right tool for the job. From my experience a few years ago on an accredit computing course that covered database admi…
I think what happens (and I have this attitude too) is that "learning" SQL takes a weekend...but then you know you'll wind up having to spend a lot longer learning the patterns of the language, and the nuances of the specific dialect, and which of the integration tools will work well with your workflow and pipeline. So while "sure I'll just learn SQL" is great for a personal or school project, when you've got to get…
You have to do the exact same things with Mongo+JS (e.g. learning when to avoid the JS bits like the plague).
Earlier quoted context omitted.
>Since all our other services are running in AWS, the obvious choice was DynamoDB – Amazon’s NoSQL database offering. Unfortunately at the time Dynamo didn’t support encryption at rest. After waiting around nine months for this feature to be added, we ended up giving up and looking for something else, ultimately choosing to use Postgres on AWS RDS.
Anyone who gets control of the live server can still read a database, even if it encrypts its storage.
Disclaimer: The voice is my head does not come out of nowhere. I am building a product which addresses this: https://github.com/wallix/datapeps-sdk-js is a API/SDK solution for E2EE. Sample app integration is available at: https://github.com/wallix/notes (you can switch between master and datapeps branches to see the changes of the E2EE integration)