Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things I learned after getting users

basementcommunity.bearblog.dev

Post reply on HN
Print view View original

141–150 of 155 posts

Re: Things I learned after getting users

#141

Post by adammarples »

manv1 wrote: post #97
dahwolf wrote: post #30

"this is mostly because i relied on a SQL ORM which in short is a tool that makes writing SQL easier to pick up and faster to develop. the biggest downside is that it might execute 50 queries to your database to get a list of information, when it probably only needs 1, which will cause slowdown." I appreciate this honesty. Listen to this old man's advise: learn SQL properly. It's not that hard. Focus on it for a few…

It's hard to explain to developers why an ORM isn't really that great of a solution....but at a high level: * your database is the core of your app. Abstraction implies that you don't care about the details of your data store, which eventually will lead to two major problems: 1. When things are slow you won't be able to debug it, because you're abstracted away your database and don't understand what queries your ORM…

I think you're mixing up someone who's decided to use an ORM rather than some hodgepodge of string concatenation with someone who has no idea what a database is, is unable to set it up effectively, and is unable for some reason to debug the sql emitted by ORM queries or use their ORM at all effectively to avoid N+1 problems. Not really a fair comparison.

Re: Things I learned after getting users

#142

Post by Joel_Mckay »

hderms wrote: post #127
Joel_Mckay wrote: post #29

Yes, there are numerous automated and human-powered nuisance traffic streams. 1. CMS sites are constant maintenance, as most are an endless supply of issues. However, some have content caching to reduce the SQL workload. 2. Delayed registration with CAPTCHA and a brief explanation of why you are there. Quiet banning IP filter applied to list to boot pending users who enter emails that bonce or fail to authenticate. 3…

Hey there! Thanks for your comment. What do you specifically mean by rescale/watermark/scrub content to protect users from themselves?

This is done mainly to strip off meta/EXIF/GPS Data, and mitigate other format violations often used to cause "trouble". Resampling/transcoding also tends to corrupt steganography utilities, thrash user hardware thumbprints, and standardize the web experience by excluding unsafe formats in favor of well-tested efficient traditional codecs.

Specifics preclude the multifaceted nature of the policy.

https://www.youtube.com/watch?v=cJMwBwFj5nQ

Happy computing =)

Re: Things I learned after getting users

#143

Post by manv1 »

selcuka wrote: post #104
manv1 wrote: post #97

Earlier quoted context omitted.

It's hard to explain to developers why an ORM isn't really that great of a solution....but at a high level: * your database is the core of your app. Abstraction implies that you don't care about the details of your data store, which eventually will lead to two major problems: 1. When things are slow you won't be able to debug it, because you're abstracted away your database and don't understand what queries your ORM…

> I mean it knows what you're using to select data, so it should auto-create indexes for you, right? Curious, how does it know what you're using to select data?

Because you're using the ORM to do selects.

I'll give you an example: prisma has a "find/findMany." As part of the build process it should examine your usage and put indexes in the column that you're using in find/findMany

Indexes are an implementation detail - but to use RDBMSs successfully you need indexes. ORMs try to hide those implementation details, but without those kinds of details you get bad performance.

Prisma just fixed another problem it had with transaction isolation levels. To even understand the problem you need to understand SQL, how databases work, and how each database implements that. It's another important implementation detail that's abstracted away...until it bites you in the ass.

Prisma also finally allowed you to use database-specific types instead of just using varchar(191), which was its default for strings.

I'm picking on prisma because I happen to be using it right now. There are good things about it, like the way it does migrations and handles schema changes. But there are times its abstractions get in the way, like when you're trying to do aggregated group by (which is easier in SQL than it is in prisma). And its migrations often fail, and to understand why you really need to understand SQL.

Hibernate was worse, because it provides a transparent object-level backing store.

Re: Things I learned after getting users

#144

Post by manv1 »

watwut wrote: post #114
manv1 wrote: post #97

Earlier quoted context omitted.

It's hard to explain to developers why an ORM isn't really that great of a solution....but at a high level: * your database is the core of your app. Abstraction implies that you don't care about the details of your data store, which eventually will lead to two major problems: 1. When things are slow you won't be able to debug it, because you're abstracted away your database and don't understand what queries your ORM…

Ad2: Every single real world java project I have seen used hibernate (ORM) with flyway. I never ever seen any project in production where some underlying tech would be deleting tables autonomously. Ad1: People do debug ORMs, actually. Including queries effectivity.

Ad2 -> I've run into lots of hibernate apps. They also tend to use Spring, and the team often gets lost trying to figure out which one is the cause of the problem. It's usually both.

Hibernate and Prisma migrations will delete tables if you allow them to.

Ad1 -> I'm sure they do...but usually they call in a DBA, who basically says "this is a disaster" then recommends that they refactor everything because their database schema is fucked up because it's based on some weird object hierarchy instead of RDBMS principle.

Re: Things I learned after getting users

#145

Post by webnrrd2k »

epgui wrote: post #63
webnrrd2k wrote: post #55

Earlier quoted context omitted.

Yes, both SQL and ORMs can give you problems, but I have seen far more serious, hard to diagnose trouble from ORMs than straight SQL. Learning straight SQL is far more transferable across databases than learning an ORM. There are reasons to have ORMs, for e.g. to keep a large code base consistent. It's useful for keeping relatively simple queries consistent across an ERP system, stuff like that. However, learning SQL…

I’m not sure I understand the consistency argument… Do you have an example, even if it’s just a silly one?

The example is from a python ERP that I modified a few years ago. All the db interaction was done through an ORM and I thought that it was an OK strategy for keeping the SQL consistent and manageable.

There were other problems with the ORM approach -- mostly that it ran tens or hundreds of extra queries for relatively straightforward SQL queries. But the software was only used by a few people at a time, so the performance wasn't that big of a deal.

Re: Things I learned after getting users

#146

Post by [deleted] »

dahwolf wrote: post #138
ookblah wrote: post #99

Earlier quoted context omitted.

Honestly, I started on the ORM hate train when I first started and wrote my own queries in an effort to be "performant", but my frankenstein of writing direct queries and abstracting code to be reusable became basically a super shitty ORM that was a nightmare to maintain for a large codebase lol. I don' think there's anything wrong with it and for 90% of CRUD actions it's what you should choose. By the time you hit p…

I'm wired differently. Even if I'd run a database operation just on a local machine and it takes 100x more load than an optimized SQL query, I still care. It's craftsmanship. Sounds like you may have a case of the abstraction disease. You keep reaching for abstractions that never accomplish the actual benefits of a truly solid abstraction.

[deleted]

Re: Things I learned after getting users

#147

Post by carom »

brycelarkin wrote: post #101
ookblah wrote: post #99

Earlier quoted context omitted.

Honestly, I started on the ORM hate train when I first started and wrote my own queries in an effort to be "performant", but my frankenstein of writing direct queries and abstracting code to be reusable became basically a super shitty ORM that was a nightmare to maintain for a large codebase lol. I don' think there's anything wrong with it and for 90% of CRUD actions it's what you should choose. By the time you hit p…

I felt the same way until I found sqlx ( https://github.com/launchbadge/sqlx ). The package lets you write sql in your app with compile-time checks. Was so much easier to get up and running than learning a new ORM.

Need this for Go. Been considering writing a static checker for ages.

Re: Things I learned after getting users

#148

Post by superb-owl »

phendrenad2 wrote: post #85
superb-owl wrote: post #75

Earlier quoted context omitted.

ORMs are great for avoiding boilerplate and adding some typesafety. But once you start doing complicated things like JOINs, it's better to drop into raw SQL. > Listen to this old man's advise: learn SQL properly. It's not that hard. I couldn't agree more with the first half, or disagree more with the second :) Once you're operating at scale, it takes a lot of fine-tuning, know-how, experimentation, and reading docs t…

What are some examples of fine-tuning or trial-and-error you might need at scale?

Postgres has a ton of parameters that can greatly affect performance. I've used https://pgtune.leopard.in.ua/ to help w/ that.

In terms of schemas, where you put your indexes matters a TON. Indexes make reads faster and writes slower (to oversimplify).

Figuring out where to break out a separate table vs keep everything in one table is also a common issue.

Re: Things I learned after getting users

#149

Post by phendrenad2 »

superb-owl wrote: post #148
phendrenad2 wrote: post #85

Earlier quoted context omitted.

What are some examples of fine-tuning or trial-and-error you might need at scale?

Postgres has a ton of parameters that can greatly affect performance. I've used https://pgtune.leopard.in.ua/ to help w/ that. In terms of schemas, where you put your indexes matters a TON. Indexes make reads faster and writes slower (to oversimplify). Figuring out where to break out a separate table vs keep everything in one table is also a common issue.

True, all of these would still be an issue with ORMs.

Re: Things I learned after getting users

#150

Post by selcuka »

manv1 wrote: post #143
selcuka wrote: post #104

Earlier quoted context omitted.

> I mean it knows what you're using to select data, so it should auto-create indexes for you, right? Curious, how does it know what you're using to select data?

Because you're using the ORM to do selects. I'll give you an example: prisma has a "find/findMany." As part of the build process it should examine your usage and put indexes in the column that you're using in find/findMany Indexes are an implementation detail - but to use RDBMSs successfully you need indexes. ORMs try to hide those implementation details, but without those kinds of details you get bad performance. Pr…

> As part of the build process it should examine your usage and put indexes in the column that you're using in find/findMany

This is not as easy as it sounds, though. Reliably inspecting source code is hard, and definitely not in the scope of an ORM. It also can't catch dynamically generated queries (think user configurable filters). It might be a good idea for a third party product that analyses your code and suggest adding indexes to models.

You know who knows best about your queries? It's the RDBMS. There are tools that can add/suggest indexes by searching your database logs for slow running queries.

Post reply on HN