Show HN: Database for analyzing US companies, visualize using Apache SuperSet

My main motivation was that I wanted to be able to drill down and filter across all the available stocks, look at the data for myself, and narrow down on the stocks I am interested based on my own sets of criteria, and make data-driven analysis for my personal investment strategies.

I used PostgreSQL as the backend database for ELT data pipelines, and used Citus Data cstore_fdw for columnar compression for the final dataset. All financial data is coming from SEC Edgar, https://www.sec.gov/developer. I used Python for downloading most of the data.

I also run the data load development locally on my home Ubuntu server that I built 5 years ago. I bought 4TB of M2 disks for best database performance, with PRIME B360M-A motherboard and Intel Chip Coffee Lake S.

I built the website simply using WordPress, and I run Apache Superset using gunicorn via Apache Webserver reverse proxy.

The registration form I had to build myself with PHP and some JavaScript, because it needed to automatically create a SuperSet user upon registration. Otherwise, I would need to input everyone manually. I used Python again for the data integration.

Please don't use the database directly as an investment tool, as its in Beta, and the data still needs to undergo heavy data quality checks, please confirm all the numbers yourself, as I provide a link for every company to the SEC filings.

Show HN: Database for analyzing US companies, visualize using Apache SuperSet
tesseractanalytics.ai

How often is the database updated? For instance on stock price data can it be considered realtime (or 15 min lag) for daily market tracking? Not familiar with that SEC api.

The data is refreshed nightly for stocks. And monthly from SEC once it is available to download. There is no guaranteed data delivery SLA that I know about from SEC, so I get it monthly once it arrives. I talk more about this in detail on the website, I will also update the data documentation tab, and in tutorials. https://tesseractanalytics.ai/finance-data-quarterly-data-do...

I looked at superset a few years ago and looked great but not popular enough to trust it. Seems the same state now?

I meant that it wasn't popular enough to use it because it might be buggy and/or wont be around in 5 years.