61–70 of 76 posts
One of the better decisions we took at my firm was to not allow direct access to any production DB to analytics visualization tools like Metabase and Redash. Always write your analytics data to a separate DB in a periodically run job. Only store aggregated anonymized data in the analytics DB you expose to internal stakeholders via tools like Metabase.
One of the better decisions we took at my firm was to not allow direct access to any production DB to analytics visualization tools like Metabase and Redash. Always write your analytics data to a separate DB in a periodically run job. Only store aggregated anonymized data in the analytics DB you expose to internal stakeholders via tools like Metabase.
This should further emphasize the need to isolate these tools and ensure they are only accessible to people who need them.
This is why I try to put everything behind NGINX with basic auth. Unfortunately not everything works well that way but in this case I suspect that this is made unexploitable by anyone without the password.
One of the better decisions we took at my firm was to not allow direct access to any production DB to analytics visualization tools like Metabase and Redash. Always write your analytics data to a separate DB in a periodically run job. Only store aggregated anonymized data in the analytics DB you expose to internal stakeholders via tools like Metabase.
What kind of tooling do you/people use for that? Or just custom scripts?
One of the better decisions we took at my firm was to not allow direct access to any production DB to analytics visualization tools like Metabase and Redash. Always write your analytics data to a separate DB in a periodically run job. Only store aggregated anonymized data in the analytics DB you expose to internal stakeholders via tools like Metabase.
Also your production database is optimized for different workloads than your analytics database. Usually production is used for fetching and updating a small number of records at a time (think updating a shopping cart), and has strict latency requirements whereas analytics involves reading a large amount of data in columns (think count group by one or two columns), and can be done in batches where the results can get…
This is why I try to put everything behind NGINX with basic auth. Unfortunately not everything works well that way but in this case I suspect that this is made unexploitable by anyone without the password.
One of the better decisions we took at my firm was to not allow direct access to any production DB to analytics visualization tools like Metabase and Redash. Always write your analytics data to a separate DB in a periodically run job. Only store aggregated anonymized data in the analytics DB you expose to internal stakeholders via tools like Metabase.
What kind of tooling do you/people use for that? Or just custom scripts?
One of the better decisions we took at my firm was to not allow direct access to any production DB to analytics visualization tools like Metabase and Redash. Always write your analytics data to a separate DB in a periodically run job. Only store aggregated anonymized data in the analytics DB you expose to internal stakeholders via tools like Metabase.
What kind of tooling do you/people use for that? Or just custom scripts?
I can't recommend any specific tools without knowing a lot about the environment, but if you're looking for terms to google: ELT (Extract, Load, Transform) and CDC (Change Data Capture) will give you a sense of the landscape.
edit: the sibling comment that mentions Airflow is a good answer for an example of an ELT workflow.
Earlier quoted context omitted.
I thought when the OP of this comment thread said locally they meant like, it isn't exposed to the Internet
"exposed" as a word does a lot of heavy lifting here. When someone is asking me casually "hey, is this server exposed to the public internet"? I take it to mean "can someone connect to it in an inbound manner from the public internet?" If the answer is no, it doesn't necessarily mean that packets don't have other ways of making their way to the server, for example, a service running locally could have a webhook mecha…
And yes, that is what I meant. curl hackmeplease.com 57 stack traces down.