Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You don’t need to be “enterprise-ready” or “scalable”

gorelay.co

Post reply on HN
Print view View original

171–180 of 185 posts

Re: You don’t need to be “enterprise-ready” or “scalable”

#171

Post by mbrodersen »

wereHamster wrote: post #19

I'm working on a project where the backend devs decided to use Cassandra to manage a few hundred entities. Due to different access modes they had to denormalize the data into multiple tables and we're constantly running into consistency issues. A fucking sqlite db with a few tables and indexes would do just fine. Also, the tool has maybe a dozen DAUs. But it's web scale, so, yeah…

You are absolutely right. 95% of problems in software development are created by the software developers themselves. The fastest/smartest developers I have worked with simply make less dumb mistakes. They keep it simple. That’s all they have to do to be 10x more productive.

Re: You don’t need to be “enterprise-ready” or “scalable”

#172

Post by mbrodersen »

FpUser wrote: post #102
winrid wrote: post #92

Earlier quoted context omitted.

Sounds interesting. What's the DB? What framework/type of API are you exposing on the C++ size?

Postgresql. No framework (well there is rather low level one of my own but it is fluid). Using 4 libs for JSON, postgres, HTTP, logging. I expose basically JSON based RPC. Peers / Clients post JSON commands and receive JSON replies. Upon startup all data (except couple of giant tables) is sucked from DB into RAM into highly efficient data structures hence most reading requests are handled in microseconds. Those 2 gia…

Yep that’s the way to do it. One improvement IMHO would be to have an event log as the truth and everything else (in-memory/databases/…) derived from that event log. It is an architecture that scales from tiny to Google scale. You don’t even need something like Kafka. You can literally start with a single disk file as the event long and read it into memory at server startup. And then “scale up” by keeping the event log in a single table in a DB (SQLite/Postgres’s etc.)

Re: You don’t need to be “enterprise-ready” or “scalable”

#173

Post by lmm »

housecarpenter wrote: post #159
lmm wrote: post #125

Earlier quoted context omitted.

> So decades of PHP+MySQL and Rails+PostgreSQL were simply misguided? To the extent that transactions were the point, yes (though there may not have been many alternatvies for a datastore to use when Rails was getting started). In the case of PHP+MySQL of course MySQL transactions were disabled by default in that era (up until 2010 in fact).

I'd be interested to hear why you think this, or if you can link to something that explains this perspective.

There's a fundamental mismatch between the HTTP request-response model and the SQL "open transaction" model - you can't open a transaction, serve a form, and wait for a user to submit it and commit that in the same transaction, because what if the user walks away? But if a transaction doesn't extend to the actual user action then it's basically useless; you can't do transactional read-modify-write when the one doing the modifying is the user.

After all, what do you do when a transaction fails? Most webapps either retry or fail (if they even bother to handle that case at all), but you don't need transactions to do either of those. In an actual database application you can tell the user their transaction didn't commit and ask them to deal with it, but again you can't really do that over the HTTP request/response cycle (because what if the user submits the form and walks away? They expect their input to be saved, whatever happens on your backend).

Re: You don’t need to be “enterprise-ready” or “scalable”

#174

Post by cutler »

roflyear wrote: post #161
cutler wrote: post #154

Earlier quoted context omitted.

I'm not sure. The bar for UI has been raised significantly since Facebook launched and mobile has changed the game to such an extent that you need a presence on 3 platforms out of the door (iOS, Android & web) to even stand a chance.

Not entirely true for many apps. The app I work on does not need a working mobile site, because we don't think our consumers are going to use it on mobile. However, we do very much care about our UI - and I think react at this point helps with that. But I don't think React complicates the stack too much. It may even make things easier, tbh - I don't have to test pages on backend tech, just endpoints.

React easier compared with what - imperative PHP4 at Facebook in 2004? I'd have to disagree. React state management is much heavier than returning HTML from an embedded PHP template.

Re: You don’t need to be “enterprise-ready” or “scalable”

#175

Post by FpUser »

mbrodersen wrote: post #172
FpUser wrote: post #102

Earlier quoted context omitted.

Postgresql. No framework (well there is rather low level one of my own but it is fluid). Using 4 libs for JSON, postgres, HTTP, logging. I expose basically JSON based RPC. Peers / Clients post JSON commands and receive JSON replies. Upon startup all data (except couple of giant tables) is sucked from DB into RAM into highly efficient data structures hence most reading requests are handled in microseconds. Those 2 gia…

Yep that’s the way to do it. One improvement IMHO would be to have an event log as the truth and everything else (in-memory/databases/…) derived from that event log. It is an architecture that scales from tiny to Google scale. You don’t even need something like Kafka. You can literally start with a single disk file as the event long and read it into memory at server startup. And then “scale up” by keeping the event l…

For business it is better to have "properly" designed database with the clean structure looking at which it is easy to understand how everything is organized and works. It can be used / interrogated by other software (writes are of course prohibited).

Sure it will not do Google scale and I do not loose my sleep over it. For most "normal" businesses what I have is way more than enough.

Besides, as I've already said I treat each product individually so my architecture reflects the actual needs and changes accordingly.

Re: You don’t need to be “enterprise-ready” or “scalable”

#176

Post by winrid »

FpUser wrote: post #147
winrid wrote: post #127

Earlier quoted context omitted.

Nice! A lot of game servers are similar to this too, which you of course know. In my case I usually just "snapshot" the in memory structures to disk... BTW for others considering this approach, the DB will do similar things that FpUser is doing in the app layer - keeping hot/recent stuff in memory. But when tail latency and cost really matters this is a good technique.

>"DB will do similar things" Not really as the storage data layout and application data layout are different. If I were to use database only relying on it's caching abilities it would lead to complex queries that are definitely way less performant than getting data from in memory structures optimized for this particular application / usage patterns. On top of that some requests require some relatively complex calcula…

Sure, I definitely don't know what you're doing. Usually I would reach for preaggregation here, computing the values on write, but I don't know anything about the use case so... :)

Re: You don’t need to be “enterprise-ready” or “scalable”

#177

Post by akhmatova »

lmm wrote: post #87
jsiaajdsdaa wrote: post #24

Earlier quoted context omitted.

Why would a team decide to use nosql to manage "a few hundred" entities? Nosql databases excel at simple lookups and gets by ID, rather than trying to sort and join pretty much ad hoc.

Why would you ever use SQL? The SQL transaction model doesn't make a lot of sense for most use cases (including essentially anything that uses the internet), and trivial usability things like being able to have collection columns without having to define a separate table all add up.

That must have been what they were thinking when they made the decision to go with Cassandra.

Re: You don’t need to be “enterprise-ready” or “scalable”

#178

Post by lmm »

akhmatova wrote: post #177
lmm wrote: post #87

Earlier quoted context omitted.

Why would you ever use SQL? The SQL transaction model doesn't make a lot of sense for most use cases (including essentially anything that uses the internet), and trivial usability things like being able to have collection columns without having to define a separate table all add up.

That must have been what they were thinking when they made the decision to go with Cassandra.

I'm working on a system that's using Cassandra on a small scale (like, probably less than a million entities) now, it's great. Haven't had any consistency issues because we've designed our data flow properly (SQL will help you paper over a bad data flow for longer but that's a double-edged sword), and getting true just-works HA (master-master where a failover doesn't interrupt your writes) out of the box is a huge advantage over something like Postgres.

Re: You don’t need to be “enterprise-ready” or “scalable”

#179

Post by akhmatova »

lmm wrote: post #178
akhmatova wrote: post #177

Earlier quoted context omitted.

That must have been what they were thinking when they made the decision to go with Cassandra.

I'm working on a system that's using Cassandra on a small scale (like, probably less than a million entities) now, it's great. Haven't had any consistency issues because we've designed our data flow properly (SQL will help you paper over a bad data flow for longer but that's a double-edged sword), and getting true just-works HA (master-master where a failover doesn't interrupt your writes) out of the box is a huge ad…

Oh, I don't dispute that there are valid use cases.

But the above instance sounds like a variant of the "because, bwah, SQL is hard" line of thought that seems to have powered much of the (now historical) NoSQL boom.

Re: You don’t need to be “enterprise-ready” or “scalable”

#180

Post by davegauer »

There's this great quote at the beginning of the chapter I'm currently reading in the book _Designing Data-Intensive Applications_:

> "A complex system that works is invariably found to have evolved from a simple system that works. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work."

--John Gall, Systemantics (1975)

Post reply on HN