Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fintech Engineering Handbook

w.pitula.me

Post reply on HN
Print view View original

221–230 of 234 posts

Re: Fintech Engineering Handbook

#221

Post by nly »

jdw64 wrote: post #53

As a programmer, what I feel when I see fintech programmers each speaking from their own different experiences and perspectives is that it makes me wonder what it really means to be good at programming. What user xlii said about not storing monetary amounts as floats is a common IEEE 754 issue. And while it's true that financial tracking should be done through immutable logs or event-based records, I don't think ever…

> However, I've heard that in HFT, they don't do that because the overhead itself is too costly.

I've worked at several HFTs and 2 had independently settled on 64bit signed fixed point with a implied 10^9 scaling factor between internal systems.

However in-process one just used 'double' for all FX conversions, scaling etc. With 15 digits of precision and careful rounding choices it's fine.

Dealing with the outside world you'd obviously just convert as cheaply as possible - and there are some crazy fast algorithms doing fast binary float -> correctly rounded decimal conversions these days

Re: Fintech Engineering Handbook

#222

Post by seff »

jdw64 wrote: post #53

As a programmer, what I feel when I see fintech programmers each speaking from their own different experiences and perspectives is that it makes me wonder what it really means to be good at programming. What user xlii said about not storing monetary amounts as floats is a common IEEE 754 issue. And while it's true that financial tracking should be done through immutable logs or event-based records, I don't think ever…

It's very true. As a merchant acquirer, everything is BigDecimal, only the auth path needs real performance, and FX occurs at a fixed rate per-day which is easy to reconcile.

It's fun reading all the different ways fintech has to adapt their systems for their specific niche.

Re: Fintech Engineering Handbook

#223

Post by CyberDildonics »

Maxatar wrote: post #166
lxgr wrote: post #151

Earlier quoted context omitted.

Usually that, but in binary, which causes lots of headaches for financial math.

For those who don't understand what they're doing, yes. But those people end up having headaches and causing trouble no matter what.

So all the people who write financial software and avoid floats don't know what they're doing? That's would end up being just about everyone. You should teach them, you know so much more than they do.

Re: Fintech Engineering Handbook

#224

Post by notpushkin »

lxgr wrote: post #66
notpushkin wrote: post #46

Earlier quoted context omitted.

It is fine as long as you don’t cross any edge cases (crypto, or more recently stuff like AI token pricing) and don’t forget to account for third party quirks (e.g. Stripe’s zero-decimal currencies: https://docs.stripe.com/currencies#zero-decimal ).

JPY not having any minor units is arguably not a “third party quirk” but just how the currency works. The same goes for various three decimal digit currencies.

I mean yeah, but it does make things more complicated. Currencies change to 0-decimal, but some systems still expect 2-decimal representation for backcompat reasons (like ISK and UGX in Stripe).

Re: Fintech Engineering Handbook

#225

Post by pcstl »

oddthink wrote: post #144
xlii wrote: post #29

I glanced, and I found this handbook shallow and - in some areas - even bad advice. E.g. If I ever see a monetary value stored in something else than integers I'm going to run away screaming (thank you Rust decimals represented as JSON floats). It's always integers unless you have a VERY good reason to do otherwise (though exported view can be in anything, even in weird bitcoded formats). FX exchange. Resolution of F…

> E.g. If I ever see a monetary value stored in something else than integers I'm going to run away screaming (thank you Rust decimals represented as JSON floats). It's always integers unless you have a VERY good reason to do otherwise (though exported view can be in anything, even in weird bitcoded formats). That really overstates the issue. Whole domains of finance run just fine on doubles. If you're doing Monte Car…

Rule of thumb here is that investments will work fine using doubles, but transfers need to run on int or fixed point.

Re: Fintech Engineering Handbook

#226

Post by belmarca »

potamic wrote: post #174
belmarca wrote: post #58

Nice. The book contains a bunch of good information that could already be found elsewhere but collecting it is quite practical. I highly suggest to read Kleppmann's Designing Data-Intensive Applications. The first edition was very good, a second one came out recently. I was CTO of a FinTech where I built the whole software stack from scratch: the lessons in the book are mostly correct. I say mostly, because as always…

Why content-addressed storage? I'm assuming this is mostly for auditability after the fact? Wouldn't it make sense to use some business identifier to be able to refer it back easily?

You will need a combination of the two. The filename should contain a compound business identifier, essentially a tuple of appropriate facts, as well as a (truncated) hash of the content.

And yes, the reason is auditability but also bare necessity.

Imagine vendor A publishes fresh files every day, but you don't exactly know when. And suppose that the vendor can (and does) republish some of them at any time, with or without notice. You need to watch those files for changes and update your own sources accordingly.

For the collection and dedup, you can construct a tuple of meta-information that will give you a good idea if the file is the same (name, time last modified, file size, etc). Then only if you're still unsure do you download the file and hash it, compare with what you already have, and make a decision. (What if the file name changed slightly and now you must re-categorize the data in the file, or what if a single row was added or removed, etc).

It's also useful when auditing (tamper-proof) or when re-running or debugging calculations that used the data in question. If a user used the data available at 9AM, which was unfortunately incomplete and led to issues, having a full data lineage really helps. You can trivially re-run the computation with the information available at that time to see what happened. Or you can mark some results as stale, etc.

There are different ways to do this and one is keeping track of your content-adressed files in a relational table and give each row a unique identifier, so that you can tell that this particular row in your AAPL OHLCV data comes from this particular file which was fetched at this particular time etc, etc. Pair that ID with a timestamp and when doing computations you can query for "the latest" AAPL data while storing the exact file ID (time is fuzzy, content hash is not). Look into temporal tables and the like.

Re: Fintech Engineering Handbook

#227

Post by notpushkin »

notpushkin wrote: post #224
lxgr wrote: post #66

Earlier quoted context omitted.

JPY not having any minor units is arguably not a “third party quirk” but just how the currency works. The same goes for various three decimal digit currencies.

I mean yeah, but it does make things more complicated. Currencies change to 0-decimal, but some systems still expect 2-decimal representation for backcompat reasons (like ISK and UGX in Stripe).

(To clarify some more, we’re in agreement here. My point was mainly that “just storing cents” seems like an easy solution and it might seem to be working well – until it’s not :-)

Re: Fintech Engineering Handbook

#228

Post by notpushkin »

lxgr wrote: post #84
jameshart wrote: post #62

Earlier quoted context omitted.

It is explicit about the fact that that number of decimal places is part of the data. The semantics for your string “10.00” are complex - is it considered equal to “10”? To “10.000”? To “10.001”? A user interacting with an API that uses such a string might make all sorts of assumptions about what it supports. A user interacting with an API that has an explicit decimal places concept is being told ‘decimals matter! Th…

> The semantics for your string “10.00” are complex - is it considered equal to “10”? Yes, but "10 USD" would be a non-canonical representation and you probably serialized incorrectly. > To “10.000”? Yes, but same caveat as above applies. > To “10.001”? Obviously not, and any system you'd ever want to use in a financial context will tell you so.

String and two-field exponent/mantissa representations are mostly the same in terms of semantics, yes. Making it two separate fields makes it less likely it would be put into `parseFloat`, but after doing some research I think strings are more popular in JSON [1, 2], so probably I’d stick to that as well.

[1]: https://msgspec.dev/supported-types#decimal

[2]: e.g. https://getlago.com/docs/api-reference/fees/fee-object#schem..., although they still use `amount_cents` for all currencies as the base rate

Re: Fintech Engineering Handbook

#229

Post by m00x »

FabHK wrote: post #206
m00x wrote: post #128

Earlier quoted context omitted.

You can't do everything you need with an integer. There are values you might want to display or calculate with that are smaller than cents. In some places you'll need things like BigDecimal, which are immune to floating point errors in most cases. It's also safe to return decimal values for displaying values.

But it's a good idea to keep those values distinct from actual money. You can't pay a fraction of a cent to anyone.

If you're calculating stock gains, taxes, etc. You'll want to do it on BigDecimal on line items, then sum it up. Cents could lose precision leading to a large difference on millions of line items.

If you're doing static payments or displaying a number, it doesn't matter if it's float amounts or integer cents since you're not doing any calculations anyway.

Stripe uses cents mostly because it's the most foolproof way that won't cause massive issues as a public API.

Re: Fintech Engineering Handbook

#230

Post by nmcost16 »

I really enjoyed this. This year I joined a fintech company, coming from a non-financial domain. This is the kind of information that I had to learn from experience, other devs, and random blogs/articles. The fact that this is a centralized location is awesome! I wish I knew about this when I was onboarding.
Post reply on HN