Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Where do you get your financial data?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 18 posts

Ask HN: Where do you get your financial data?

#1

Post by sdcoffey »

Hey Hacker News,

We're working on an investment aggregator that tracks the value of our customers' portfolios over time. We've faced a bunch of challenges in getting public stock market data (live/historical prices, splits, etc). We're currently cobbling together a dataset from a few different sources (Polygon, IEX, etc), but it's been a massive pain*

I'm wondering if this is the case for other fintech devs. Does everyone face the pain of assembling their own financial datasets? Or do we have unique needs/a bad solution?

So HN, where do you all source your financial data?

* Our main challenges:

Data quality: - Ticker symbol changes (and CUSIP/ISIN changes/challenges) - Missing or wrong values for some days - Missing or incorrect splits Speed: - Tens of API calls that would be necessary to render one screen - Historical data syncing that would take days of API calls Burden: - Enterprise sales contracts instead of self-serve - Building and maintaining your own ETL pipeline to ingest data

Re: Ask HN: Where do you get your financial data?

#2

Post by Kon-Peki »

You have to agree to serious licenses, and then you pay serious money for it.

Here's a good start to the list of vendors (HINT: click on "Data Dictionary" for things that look interesting):

https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/

For linking across different data sets and tracking companies over their entire history (check out the video starting around 6 minutes in):

https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/...

EDIT - FYI, WRDS is the financial data platform that almost all large universities use (worldwide). They handle all the management of it, and your school just has to write a few checks. Many of these data sets are licensed to all students, not just business school students. So if you are a CS student and want this kind of data for building an ML model or something like that, you should be able to get it by requesting an account on the WRDS page linked above. They might push back on you a little, in which case you'll have to go over to the business school at your Uni to get things ironed out. They have a non-techie-friendly interface, but also offer a Postgres interface so you can connect directly from Python or R or whatever with your account credentials.

Re: Ask HN: Where do you get your financial data?

#3

Post by tdubhro1 »

Everyone faces this pain. There is no single good source for all the data any nontrivial app needs. The main vendors are predatory and will change the goalposts to charge you as much as you can bear and more as you grow. In my experience this is a huge obstacle and probably a major reason there is limited innovation in anything like derived market data analytics. It’s not just you, it’s everyone.

Re: Ask HN: Where do you get your financial data?

#4

Post by sdcoffey »

Kon-Peki wrote: post #2

You have to agree to serious licenses, and then you pay serious money for it. Here's a good start to the list of vendors (HINT: click on "Data Dictionary" for things that look interesting): https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/ For linking across different data sets and tracking companies over their entire history (check out the video starting around 6 minutes in): https://wrds-www.wharton.upen…

Interesting! I'm no longer a student, so I couldn't get access, but it's cool to know that this exists.

Curious if you've used any of these WRDS data sets? Also really interested to know if you've used the postgres interface to this data, and how you liked working with it as opposed to a regular API-like interface. Thanks!

Re: Ask HN: Where do you get your financial data?

#5

Post by dvasdekis »

FWIW, I went deep with Interactive Brokers for my last Fintech. It was the only place we could source real-time currency options data and we managed to reduce our total latency from market to database to about 10ms by using a NYC datacenter, which was enough for us. They had historic ticks too, but an inbuilt ratelimiter made it a multi-month project to pull serious volumes of data.

ib_insync was the python client library, and I OS'd the market gateway I built: https://github.com/dvasdekis/ib-gateway-docker-gcp

Re: Ask HN: Where do you get your financial data?

#6

Post by sdcoffey »

dvasdekis wrote: post #5

FWIW, I went deep with Interactive Brokers for my last Fintech. It was the only place we could source real-time currency options data and we managed to reduce our total latency from market to database to about 10ms by using a NYC datacenter, which was enough for us. They had historic ticks too, but an inbuilt ratelimiter made it a multi-month project to pull serious volumes of data. ib_insync was the python client li…

Multi-month, wow. Cool to know this exists though, thanks!

Re: Ask HN: Where do you get your financial data?

#7

Post by Kon-Peki »

sdcoffey wrote: post #4
Kon-Peki wrote: post #2

You have to agree to serious licenses, and then you pay serious money for it. Here's a good start to the list of vendors (HINT: click on "Data Dictionary" for things that look interesting): https://wrds-www.wharton.upenn.edu/pages/about/data-vendors/ For linking across different data sets and tracking companies over their entire history (check out the video starting around 6 minutes in): https://wrds-www.wharton.upen…

Interesting! I'm no longer a student, so I couldn't get access, but it's cool to know that this exists. Curious if you've used any of these WRDS data sets? Also really interested to know if you've used the postgres interface to this data, and how you liked working with it as opposed to a regular API-like interface. Thanks!

I have used some of those data sets.

This book demonstrates some usage with R and the WRDS Postgres connector (but obviously you can use anything):

https://tidy-finance.org/accessing-managing-financial-data.h...

PS - some of the data sets referenced in that book are freely available.

Re: Ask HN: Where do you get your financial data?

#8

Post by melony »

Have you tried the obvious sources? Bloomberg, S&P, Nasdaq. Fintech isn't a cheap game. If you get your data from small data vendor startups, you risk getting poor fundamentals data. You don't want to waste time on debugging the calculations when you could be iterating. Some brokers like IB are also have a data vendoring side business.

If you are really tight on cash, try Intrinio. No idea about their quality but they have been around for a while.

Re: Ask HN: Where do you get your financial data?

#9

Post by sdcoffey »

melony wrote: post #8

Have you tried the obvious sources? Bloomberg, S&P, Nasdaq. Fintech isn't a cheap game. If you get your data from small data vendor startups, you risk getting poor fundamentals data. You don't want to waste time on debugging the calculations when you could be iterating. Some brokers like IB are also have a data vendoring side business. If you are really tight on cash, try Intrinio. No idea about their quality but the…

It is indeed, which is why we stopped after window-shopping Bloomberg and Nasdaq. Seems like it's hard to have cost-effective, high-quality data with and API that's amenable to bulk backfilling.

We have actually looked at Intrinio! Specifically for options data. Again the problem was that the API is not setup for bulk, historical backfills.

Post reply on HN