Earlier quoted context omitted.
Haha yeah for the past tax year I I looked at my transaction history and it was missing some that I expected to be there. Turns out, the exchange (Gemini, for crypto) uses UTC, but the IRS docs always tell you that a cutoff is relative to your own time zone. The transactions were in the late evening on New Year's Eve, central US time, but that was already the next year by UTC, so any readout of "transactions for 2022…
That is supposed to be a festive hour, not one for trading.
Fintech engineering mistakes (2022)
61–70 of 112 posts
Re: Fintech engineering mistakes (2022)
#62One more: Timezones Various platforms use: UTC, the user’s timezone as set in their dashboard, the user’s detected timezone, the timezone of the server that is generating the reports. Aggregating or reconciling data from different platforms can be a pain if the timezones aren’t clearly indicated. I’ve had bank statements that didn’t agree to CSV exports of the same data because the servers generating the two reports…
For example, an invoice due on Friday is probably actually due by close of business (5pm say) in the timezone your business operates, and if created at 11pm it would be processed the next day (or even on Monday, don’t get me started about business day calculations).
Re: Fintech engineering mistakes (2022)
#63Shit, I wish I read this prior to my interviews at Cash App.
In what way do you think it would've made a difference? Asking as someone who's applied to Block/Square/Cash App in the past.
Here's something small but I was doing a SQL interview one involved a a card_tx table with the amounts stored in integers values in cents. I immediately noticed this and chatted with the interviewer about this. My assumptions were wrong, and the floating point arithmetic was a simpler reason than what I was thinking.
Anyway you can display you have thought about relevant business problems to where you are interview engineers tend to have a strong reflex to engage the person more.
Re: Fintech engineering mistakes (2022)
#64Earlier quoted context omitted.
For example 16.10 is not exactly 16.10 in IEEE floating point. When you do enough operations, and depending on the order of operations, you can wind up several cents off. That sounds small, but can be enough to give your auditors heart-burn. COBOL does BCD arithmetic, (not really but at least conceptually), and it's penny accurate to 31 digits (as per the standard - implementations may have greater accuracy). Frankly…
It would be neat to have a fundamental "money" primitive alongside int, float, string, etc. It might not be simple to implement, though. Off the top of my head, * Can you confidently say that $100>¥10 in an offline environment? * You'd need to support different bases for splitting units to deal with eldritch values such as farthings. * Where is the line drawn? Is a Bitcoin a valid currency? A gold Krugerrand? A bundl…
Re: Fintech engineering mistakes (2022)
#65Earlier quoted context omitted.
For example 16.10 is not exactly 16.10 in IEEE floating point. When you do enough operations, and depending on the order of operations, you can wind up several cents off. That sounds small, but can be enough to give your auditors heart-burn. COBOL does BCD arithmetic, (not really but at least conceptually), and it's penny accurate to 31 digits (as per the standard - implementations may have greater accuracy). Frankly…
It would be neat to have a fundamental "money" primitive alongside int, float, string, etc. It might not be simple to implement, though. Off the top of my head, * Can you confidently say that $100>¥10 in an offline environment? * You'd need to support different bases for splitting units to deal with eldritch values such as farthings. * Where is the line drawn? Is a Bitcoin a valid currency? A gold Krugerrand? A bundl…
Yeah, one more reason for it to be a different type.
> Where is the line drawn?
It's not.
As a rule, measurement unities have an absurdly bad support from computers.
Re: Fintech engineering mistakes (2022)
#66For “using floating point data types”, it’s even worse; you often need to use strings, for example if you need to store a bank account number “01234567789” will have the leading zero stripped if you use a numeric type.
“Updating transactions” would be better phrased as “use an append-only log / evented architecture”. (Also, “use a double-entry ledger” is probably the most valuable advice I could have sent myself prior to getting into FinTech.)
“Be careful with retry” should be more strictly “use idempotent operations” and link to the canonical Stripe article on idempotency keys (https://stripe.com/blog/idempotency).
Another important one to think about is bitemporality. “Created at” vs “effective at”. Not obvious at first and you’ll have some painful migrations if you don’t build it in. Fowler has a good overview here: https://martinfowler.com/eaaDev/timeNarrative.html.
Edit to add - the advice that maybe using a NoSQL database is pretty bad IMO. I’d advise in the opposite direction - use SERIALIZABLE isolation in a SQL database. Read up on your Aphyr blog posts before trying to do anything distributed. Be paranoid about race conditions / serialization anomalies. If you eventually hit performance issues you need to think hard about what anomalies your access patterns might be subject to. (Obviously HFT won’t use serializable SQL).
Re: Fintech engineering mistakes (2022)
#67One more: Timezones Various platforms use: UTC, the user’s timezone as set in their dashboard, the user’s detected timezone, the timezone of the server that is generating the reports. Aggregating or reconciling data from different platforms can be a pain if the timezones aren’t clearly indicated. I’ve had bank statements that didn’t agree to CSV exports of the same data because the servers generating the two reports…
Furthermore, even for “date fields”, consider using a datetime. Every date implicitly exists in a timezone, and if you ignore that ambiguity you’ll get bitten later. For example, an invoice due on Friday is probably actually due by close of business (5pm say) in the timezone your business operates, and if created at 11pm it would be processed the next day (or even on Monday, don’t get me started about business day ca…
Solid advice, and must come from painful burns. I've been preaching from the same book for a few years now: a timestamp without timezone offset is worse than useless.
Or as the DB expert in previous job so eloquently put it... Timestamp with zone tells you when an event actually happened. A timestamp without zone or offset is equal to wall clock time inside a windowless room, which itself is in an unspecified location somewhere on the planet.
Re: Fintech engineering mistakes (2022)
#68> Over the last decade, fintechs using RDBMS databases to record transactions have built features onto their databases like a journal of all changes to all data and clever ways of making sure the data hasn’t been modified by storing checksums of data as transactions are added. (under “Updating transactions”) It is worth noting that many of these methods don't prevent tampering, they only make it visible when you look…
The number one thing they crave is immutability. But when that's not an option, tamper-evident comes as a close second.
Re: Fintech engineering mistakes (2022)
#69Why should anyone have a look at TigerBeetle, a bleeding edge accounting system built with a language no one understands? Since no one understands Zig, no one will be able to maintain TigerBeetle. The company is exposing itself to a great deal of risk with this decision. Adopting TigerBeetle is a colossal business mistake, not an engineering mistake.
You raise a fair question.
Coincidentally, one of the reasons we picked Zig was for how readable it was, and strikingly so, even for high level programmers who might not understand systems programming or C. Because Zig reads like TypeScript, and we were working in payment switches where the majority of programmers could read that. This particular switch, in fact, had this same business requirement, that programmers should be able to at least read the systems language.
But generally, our experience has been that people who understand C will understand how to maintain Zig [0]. Zig's toolchain is also more accessible, and across all platforms. Zig's compiler is already being used by Uber for hermetic builds.
It's also easy to learn. You can pick up Zig in a week and be comfortable in a month. Zig has a simple grammar. I love how, when we have someone join the team, we never have a discussion about how to learn Zig, as if it's a difficult language to master (like C++!). Rather, there's excitement around learning the language, even ahead of starting at work, and within a day or two they're committing.
We made this decision for TigerBeetle in July 2020, and didn't take it lightly. We had already followed Zig's progress for 2 years by that point, and many factors were considered [1][2][3]. C was the other contender, given that we had to handle memory allocation failure.
The crux of the decision, then, was whether to invest in a systems language of the last 30 years, or in a systems language of the next 30 years. A distributed database is a big investment. It made sense to invest for the future. If anything, it would have been a colossal business mistake to have picked C or C++, which would have crippled our development velocity.
Furthermore, for TigerBeetle's design goals, especially w.r.t. our adoption of NASA's Power of Ten Rules for Safety-Critical Code and thus static memory allocation, Zig made (and continues to make) the most sense.
We also liked the efficient performance culture surrounding Zig, with talented game developers like Michal Ziulek and Stephen Gutekanst, and embedded programmers like xq, Matt Knight, Jens Goldberg and others moving to it. These industries (gaming, embedded) are often a good litmus test of where systems programming is at.
More details (our thinking on Zig through the lens of safety/performance/tooling/ecosystem/hiring/marketing) here:
[0] https://kristoff.it/blog/maintain-it-with-zig/
[1] https://docs.tigerbeetle.com/FAQ#why-is-tigerbeetle-written-...
[2] Building a Distributed DB in Zig: https://www.youtube.com/watch?v=0pHHb6ONTbw
[3] TigerStyle! (How to Design Safer Systems in Less Time): https://www.youtube.com/watch?v=w3WYdYyjek4
Re: Fintech engineering mistakes (2022)
#70Earlier quoted context omitted.
I have worked at banks and fintechs for the past 30 years and honestly have never used anything but a doubles for money with no issues (and a simpler code base.) I understand the sentiment and the potential issues, but it's really kind of domain dependent.
Is storing $1.05 really that much simpler than storing something as 105? Use a long to store money, pretty easy.
Not that fix nums are wrong, but they are far from safe.