Live data from Hacker News

How I build software quickly

evanhahn.com

161–170 of 215 posts

Re: How I build software quickly

#161
post #80

Earlier quoted context omitted.

This. I use celery with the same code base/docker image. Just a different entry point to start a celery worker instead of a wsgi (web) worker. Too many http requests? Add web worker instances. Background jobs piling up? Add celery workers. Clearly separate read endpoints from write/transactional endpoints and you can hit a slave postgres db or the master db depending on the http call. This creates a very robust syste…

I do the same, it's easy enough and doesn't require a ton of hosting logic. Out of interest, how do you run your migrations in production, deploy the service then run an ad-hoc job with the same container again? That was one thing I was never super happy with.

In an ideal world your code-base is always compatible with the previous migration state.

So new version can work with the previous version's DB schema.

Then, yes, simply run the migration in a transaction once the new code is deployed. Postgres has fully transactional DDL changes which helps.

Of course, it heavily depends on the actual change being made. Some changes will require downtime or must be avoided if it is too heavy on the db.

Another approach if the migration can be applied quickly is to just run the migrations as part of the deployment script. This will cause a downtime but can be short.

Easiest is just to do runmigrations in your docker image start commands, so DB is always migrated when the container starts.

tl;dr: It depends on the complexity of the migrations and your uptime requirements.

Re: How I build software quickly

#162
post #46

In recent years, I have learned how to build sufficiently robust systems fast. Here are some things I have learned: * Learn one tool well. It is often better to use a tool that you know really well than something that on the surface seems to be more appropriate for the problem. For extremely large number of real-life problems, Django hits the sweet spot. Several times I have started a project thinking that maybe Djan…

> Always choose extremely boring technology. Just use python/Django/Postgres for everything. Hell, think twice before you consider postgres. Sqlite scales further than most people would expect it to, especially for local development / spinning up isolated CI instances. And for small apps it tends to be good enough for production too.

Local Postgres is easy. If you're deploying to some cloud service, Postgres is still probably easier than SQLite, since persistent local storage is a bit more special there. I'd also be worried that SQLite doesn't provide something I need out of the box.

This isn't to complain about SQLite overall, it's perfect for local DBs like you'd see in app frontends or OS services.

Re: How I build software quickly

#163

Earlier quoted context omitted.

A lot of the conversation about "simplicity" with kubernetes ignores that there are MANY different distributions of k8s. I would argue that k3s on a vm is not too much harder to setup than a normal reverse proxy setup. And the configs are a bit easier to find on a whim when coming back after months. Obviously a full self hosted setup of k8s is pretty large task but in other forms its a competitive option in the "simp…

K8s is chock full of complexity though. Complexity that the vast majority of applications do not, and will not need

I mean a basic k3s install + a barebones deployment combined with a nodeport service is about as trivial as you can get. its not THAT much more "lines of config (60-70 max)" then an nginx conf + "git clone + other stuff" deploy script.

Re: How I build software quickly

#164

I've found that a "rough draft" is pretty hard to maintain as a "draft," when you have a typical tech manager. Instead, it becomes "final ship" code. I tend to write ship code from the start, but do so, in a manner that allows a lot of flexibility. I've learned to write "ship everywhere," even my test harnesses tend to be fairly robust, ship-Quality apps. A big part of that, is very high-Quality modules. There's alwa…

[deleted]

Re: How I build software quickly

#165

Earlier quoted context omitted.

K8s is chock full of complexity though. Complexity that the vast majority of applications do not, and will not need

I mean a basic k3s install + a barebones deployment combined with a nodeport service is about as trivial as you can get. its not THAT much more "lines of config (60-70 max)" then an nginx conf + "git clone + other stuff" deploy script.

I dunno, a sane nginx config is like 30 lines max for a basic service. You can mange it entirely with git (`git config receive.denyCurrentBranch updateInstead`) and some post-receive / push-to-checkout hooks.

Keep in mind, for K8s to work you also need to containerize your application, even if it's just static files (or I guess you can break containment and do volume mounts, or some sort of static file resource if you're serving static content). For non-static applications, you still need to configure and run it, which is just harder with k8s/docker in the mix.

Re: How I build software quickly

#166

Earlier quoted context omitted.

Last time I tried what stumped me was getting "ReadWriteMany" volumes working. My use case was pretty straightforward, a series of processing steps with the caveat that storing the actual results in a DB would be bad due to the blobs size. So, instead, the DB is used to signal downstream consumers files are ready and instead write the files in the ReadWriteMany volumes so that downstream files can simply read them. I…

Ya this kind of usecase is for sure straying into the "this is more of a pain than its worth" with regards to k8s. Esp when you can just do basic nfs mounts in nix anyways. If you are already familiar with k8s volumes and csi's its not a huge problem but if you arent its not worth learning if your goal is simple. At least in my opinion.

Thank you for taking the time to confirm this.

Yeah, that is also my opinion after that experience.

Also, it was really frustrating as the number of moving parts that have to play nice to each other is high.

Re: How I build software quickly

#167

Earlier quoted context omitted.

Last time I tried what stumped me was getting "ReadWriteMany" volumes working. My use case was pretty straightforward, a series of processing steps with the caveat that storing the actual results in a DB would be bad due to the blobs size. So, instead, the DB is used to signal downstream consumers files are ready and instead write the files in the ReadWriteMany volumes so that downstream files can simply read them. I…

Use a bucket for this. Or if you’re in a home lab use a boring old NFS server outside k8s

Thanks. The Apache beam code in the end doesn't need a bucket but the HDD attached to the VM was more than enough to store everything and then moving things out of it as the final step.

Re: How I build software quickly

#168
post #86
post #46

Earlier quoted context omitted.

> Always choose extremely boring technology. Just use python/Django/Postgres for everything. Hell, think twice before you consider postgres. Sqlite scales further than most people would expect it to, especially for local development / spinning up isolated CI instances. And for small apps it tends to be good enough for production too.

while i'm currently using sqlite, since it has only one write transaction at a time, if I understand correctly, opening a tx and doing outside requests while in the transaction could potentially block your whole application if the outside requests keep timing out... so you kinda need to watch out for that

Treat transactions like mutexes has always been the prevailing wisdom has not not? Keep them as short as possible and do not make blocking calls within one.

This would be true for any database, something read / written during a transaction would block at least that table universally until the transaction is finalised.

Re: How I build software quickly

#169
post #82

Earlier quoted context omitted.

I also like Django a lot. I can get a working project up and running trivially fast. In my day job I work with Go and while it's fine, I end up writing 10x more code for simple API endpoints and as soon as you add query parameters for filtering, pagination, etc. etc. it gets even longer. Adding a permissions model on top does similar. Of course there's a big performance difference but largely the DB queries dominate…

Yes I really wish for something like Django for a statically typed language (maybe Spring? Haven't tried it). I'm writing a CRUD CLI ( https://github.com/bbkane/enventory/ ) partly to practice doing it in Go, and while I'm mostly happy with the resulting code, theres just a whole lot of it. At least it's simple enough that I can trust LLMs to ok jobs with detailed prompts (example: https://github.com/bbkane/enventory…

https://loco.rs, it's like Ruby on Rails, but for Rust (as their website says).

Personally I find Django and Rails too magical for my taste so I just use simpler libraries like Axum, but it's nice if you want all the bells and whistles. Plus, it's much faster than either Python or Ruby.

Re: How I build software quickly

#170
My 2 cents: I prefer to emphasize quality over scope, meaning relentlessly adopt habits that reduce bugs to rare events. Don't take on extra scope unless you can deliver a working system, wait until next iteration.

Reason (my context: small team(s) but in a somewhat complex and integrated environment):

Many times, bugs in production cost the company (IT and business areas) much much more to identify/correct/resolve then making sure the stuff works in the first place.

Incorrect transactions flowing between different systems and different areas of the business and possibly out to customers and other external partners are time consuming to correct, or require costly manual labor to work around.

Post reply on HN