Thanks!
Hmm, it's hard to quantify it exactly, but overall there isn't that much maintenance required once it's all set up (except for the few manual bits). Let's say it could all run for months without needing any intervention. Maybe one or two data sources would stop exporting due to an expired token or some API weirdness, but the downstream services that use the data will carry on running against older data.
It's possible due to the design: some general 'architectural' decisions [0]: separating the data retrieval from data processing, avoiding databases where it's not absolutely necessary [1], 'defensive enough' error handling [2]. (but naturally I fix small things here and there when I notice just like with other chores)
I do spend quite a lot of time on it, it's sort of a hobby at this point (e.g. to add/connect new things, experiment etc.). But, lately most time was spent documenting, releasing different parts, and writing about it to share with people -- this is much more time consuming (and certainly less fun than tinkering). I'd like to reach a certain milestone and get some rest from it, switch to something else :)
> if you were to start over with a new organizational methodology, what would you do differently knowing what you know now?
Frankly, I'm not really regretting doing anything the way I did it (at least for now) -- most things make sense to me and what I implemented is a 'dependency' for more sophisticated methodologies anyway, so hopefully no effort wasted here. I wish I came up to the principles I described in the links quicker, for a while it was a 'random walk' without clear vision how to compose and connect things.
[0] https://beepb00p.xyz/exports.html
[1] https://beepb00p.xyz/unnecessary-db.html
[2] https://beepb00p.xyz/mypy-error-handling.html