Monitoring 9600 banks at scale • Hacker News Board

Re: Monitoring 9600 banks at scale

#11

Post by steakknife » Thu, Jun 28, 2018, 8:15 PM UTC

Interesting writeup. This is also a major issue for us at TradeIt (we do something similar but for stock brokers and portfolio/trading) as the brokers we integrate are not always...ahem..."robust". We've found that our upstream users really appreciate that often we can tell them about brokers' service outages before the brokers even announce it (when the brokers even bother). Sometimes the brokers don't even realize their system is malfunctioning until we poke them to ask what's going on.

Our throughput numbers are much lower and and our integrations are much fewer than Plaid, so we have been able to get away with keeping a close eye on Graphite/Grafana for spikes in request failures/timeouts. Seems like eventually we will need to implement some kind of statistical monitoring and alerting.

Re: Monitoring 9600 banks at scale

#12

Post by funkymatt » Thu, Jun 28, 2018, 8:46 PM UTC

steakknife wrote: post #11

Interesting writeup. This is also a major issue for us at TradeIt (we do something similar but for stock brokers and portfolio/trading) as the brokers we integrate are not always... ahem ..."robust". We've found that our upstream users really appreciate that often we can tell them about brokers' service outages before the brokers even announce it (when the brokers even bother). Sometimes the brokers don't even realiz…

grafana has that ability built in!

Re: Monitoring 9600 banks at scale

#13

Post by throwawaymath » Thu, Jun 28, 2018, 8:48 PM UTC

lordxenu wrote: post #6

How do you get the data from banks? Are you scraping the webpage after the user logs in? Not many banks I know of have public apis.

Yes, for any bank that doesn't provide them with API access they're scraping the login pages. They even do this for banks which implement anti-scraping measures.

Re: Monitoring 9600 banks at scale

#14

Post by Terretta » Fri, Jun 29, 2018, 2:33 AM UTC

jeeyoungk wrote: post #8

divxflounder wrote: post #3

Great article! I'm definitely taking an action item to look into Prometheus. I own DevOps/Monitoring and Alerting my org and it's really cool to see how other companies skin this cat. I saw Cloudwatch in the pipeline, which is an Amazon product. I know I'm going to make a very controversial statement here, but - why Amazon? With volumes like yours, your scale will eventually hit the point where your cost skyrockets.…

One of the authors here. Thanks for enjoying the article! Re: AWS. We're not at a point where we are overburdened by the AWS spending. Many things are more efficient with AWS, as we have a fairly small engineering team. We use various different AWS products (Aurora, Kinesis, to name a few) that we are utilizing. Regarding metrics & percentiles - Yes I agree. 99th percentile is what we try to look at the most, as most…

Pretty unofficial here, but I prefer engineering channel to biz dev channel... Drop me a note, loop in whoever would be interested? I’ve been meaning to get our companies better acquainted — your fantastic write up reminded me.