Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We saved $50k/year with a Go microservice coded in a hackathon

movio.co

Post reply on HN
Print view View original

251–260 of 264 posts

Re: We saved $50k/year with a Go microservice coded in a hackathon

#251

Post by carterehsmith »

LoSboccacc wrote: post #133
bufferoverflow wrote: post #113

Earlier quoted context omitted.

Can you demonstrate that Postgre is significantly faster than MySQL on average? Highly doubt so. The problem is in the way data model was architected and implemented.

apples to oranges. you need to compare PostgreSQL with other database that don't take shortcut around ACID for performances (for example DDL forcing implicit commit in transactions)

So this is true, but I find that it does not matter much.

Like, how often one needs to rollback a DDL statement? I did that like... never.

And, what is the use case? Like, you added a column to a table by accident? Well, that will not break anything, so no harm done.

That is way different from regular dml rollback which may recover 1bn records and save your life :)

Re: We saved $50k/year with a Go microservice coded in a hackathon

#252

Post by manigandham »

pisky wrote: post #171
zimpenfish wrote: post #165

Earlier quoted context omitted.

> those fools at Google, Facebook, Twitter, Pinterest, Amazon ... have dedicated hundreds of engineers and millions of dollars to nothing more than keeping MySQL up, running, and not crapping the bed every time someone looks at it funny. If you can afford that resource expenditure, by all means go nuts with MySQL. Most companies can't and would be far better served by something which doesn't need that amount of handh…

There is one thing that bugs me about all the talk of postgres' superiority: why haven't these companies switched to postgresql? Surely they weren't all too far invested in mysql before a "more knowledgeable" DBA came along saying postgresql is better.

MySQL for years was far easier to install, configure and run than PostgreSQL, especially features like replication which was much better than other options. Big companies use these databases as more like simple key/value systems rather than complex relational schemas so strong replication and operational simplicity was favored over the rich featureset of Postgres.

Eventually Postgres caught up in most things, and the delay was in some part because of implementing those features "correctly" and with more thought, but it's still a delay that hurt the uptake in the early days.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#253

Post by pandapunchpower »

EGreg wrote: post #214
pandapunchpower wrote: post #70

Earlier quoted context omitted.

The column type would have to be the type of the encrypted value. The type of the unencrypted data could not be enforced by the DB and you would have to rely on code doing the correct thing. I am however extremely wary of doing it that way. I don't know your requirements of course.

Here is the thing - encrypted stuff is just a weird encoded string. So I can’t really use columns normally. What I really need is just a huge table with two fields: “token”, “content” And the token is basically the primary key but encrypted with whatever encryption. You could even do foreign keys this way. Hmm I suddenly have an idea. What about a layer above the database that basically enforces foreign keys and join…

>What I really need is just a huge table with two fields: “token”, “content”

Sounds more like a key value store and less like a relational database. Although you can store key value data in a relational db of course, there may be a better tool for the job.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#254

Post by e12e »

paulryanrogers wrote: post #242
e12e wrote: post #240

Earlier quoted context omitted.

> will accept "0000-00-00 00:00:00" Isn't it? I thought that today()-(2018 years, 4 months and 10 days) would be approximately that date? Maybe you prefer +0000 vs just 0000? 'ISO 8601 prescribes, as a minimum, a four-digit year [YYYY] to avoid the year 2000 problem. It therefore represents years from 0000 to 9999, year 0000 being equal to 1 BC and all others AD. However, years prior to 1583 are not automatically all…

Technically true but still 0 is not yet a month or day of the month, so 0000-00-00 is still bad data

Ah, yeah. Was too focused on the year to consider the month/day!

Re: We saved $50k/year with a Go microservice coded in a hackathon

#255

Post by rypskar »

scruple wrote: post #228
rypskar wrote: post #193

Earlier quoted context omitted.

OK, the part about not using ORM if you know SQL is a bit of an exaggeration. At least when you know SQL you know when to use an ORM and when to not use it. If all you know is ORM then you will always use it, and ORM seems to lead to many developers not learning SQL

I think it depends... Who are we talking about here? Juniors, even intermediates, in my experience, haven't had enough time on the job to have learned enough to be writing raw SQL statements or query objects unless they're actively punching up on a daily basis. I am unfortunately talking from experience here. What I am saying is, I really do not want a situation on my hands where the juniors that I work with, or most…

I do also understand your points and think we agree on most. I think that if a "developer" can't write SQL I would not trust that one to set up the ORM correct either. For basic usage, sure they will get it to work and all is good. But when you want to join table or run aggregate functions the same peoples who write bad SQL could also write bad ORM code with N+1 queries. ORM has its place and optimized beautiful SQL has its place, a craftsman know which tool to use where and when to ask for help.

One of the problems, in my opinion, is that SQL isn't "cool" or hip and by many seen as not important to learn. While the new fancy Javascript based language or framework which nobody use and that will be replaced next week is much more important to learn.

Btw, get of my lawn :) /end old man rant

Re: We saved $50k/year with a Go microservice coded in a hackathon

#256

Post by TimJYoung »

83457 wrote: post #204
TimJYoung wrote: post #59

Earlier quoted context omitted.

There's no need for the extra joins, you can just do the one join and then filter everything in the WHERE clause: SELECT DISTINCT loyaltyMemberID from members as m INNER JOIN properties as p on m.id=p.user_id WHERE (p.prop='name' AND p.value = value) AND ...etc.

But how would you do exclusions with your approach?

I'm not sure what you're asking - could you give me an example of what you're envisioning that couldn't be satisfied with a combination of Boolean expressions in the WHERE clause ?

Re: We saved $50k/year with a Go microservice coded in a hackathon

#257

Post by JoachimSchipper »

EGreg wrote: post #214
pandapunchpower wrote: post #70

Earlier quoted context omitted.

The column type would have to be the type of the encrypted value. The type of the unencrypted data could not be enforced by the DB and you would have to rely on code doing the correct thing. I am however extremely wary of doing it that way. I don't know your requirements of course.

Here is the thing - encrypted stuff is just a weird encoded string. So I can’t really use columns normally. What I really need is just a huge table with two fields: “token”, “content” And the token is basically the primary key but encrypted with whatever encryption. You could even do foreign keys this way. Hmm I suddenly have an idea. What about a layer above the database that basically enforces foreign keys and join…

You'll be interested in CryptDB.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#258

Post by scruple »

rypskar wrote: post #255
scruple wrote: post #228

Earlier quoted context omitted.

I think it depends... Who are we talking about here? Juniors, even intermediates, in my experience, haven't had enough time on the job to have learned enough to be writing raw SQL statements or query objects unless they're actively punching up on a daily basis. I am unfortunately talking from experience here. What I am saying is, I really do not want a situation on my hands where the juniors that I work with, or most…

I do also understand your points and think we agree on most. I think that if a "developer" can't write SQL I would not trust that one to set up the ORM correct either. For basic usage, sure they will get it to work and all is good. But when you want to join table or run aggregate functions the same peoples who write bad SQL could also write bad ORM code with N+1 queries. ORM has its place and optimized beautiful SQL…

Yes, exactly!

> could also ORM code with N+1 queries

Oh, they absolutely do, and when we're lucky they actually catch them on their own before they get to code review. Some folks reach for tools like Bullet [0] and, that's great, but unfortunately, sometimes they treat that tooling like the Holy Gospel. They develop an over-reliance on them as if those tools exist to offload critical thinking. Drives me crazy... in my experience, it's been hard to combat this type of thing, too. The pace of "agile," the calculus between paying down technical debt and mentoring and progress, I don't really know why but I haven't had a lot of long-term luck.

> One of the problems, in my opinion, is that SQL isn't "cool" or hip and by many seen as not important to learn.

I think you're really right about that. I happen to like writing SQL quite a bit and I take a little bit of pride in that I kind of sort of actually understand a little about what is going on in the database and even then I neglect that skill. I picked up copies of both "SQL Anti-Patterns" and "SQL Performance Explained" based on recommendations from this thread and am eager to get in to them this weekend. Still lots to learn... And, I have some SQL problems that I can see coming up over the horizon today and I hope this gives me the edge I need to start grappling with them sooner rather than later.

[0]: https://github.com/flyerhzm/bullet

Re: We saved $50k/year with a Go microservice coded in a hackathon

#259

Post by astura »

carterehsmith wrote: post #251
LoSboccacc wrote: post #133

Earlier quoted context omitted.

apples to oranges. you need to compare PostgreSQL with other database that don't take shortcut around ACID for performances (for example DDL forcing implicit commit in transactions)

So this is true, but I find that it does not matter much. Like, how often one needs to rollback a DDL statement? I did that like... never. And, what is the use case? Like, you added a column to a table by accident? Well, that will not break anything, so no harm done. That is way different from regular dml rollback which may recover 1bn records and save your life :)

Seriously? You don't need to do DLL operations in day-to-day use but you'll need to when you're doing development work.

For example, you had a "color" column on a table, for a new feature youre now adding the ability to have multiple colors. You're going to create a new column, create a new table, populate that table, and drop the old column. If anything fails during that process you'd like to be able to roll back.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#260

Post by carterehsmith »

astura wrote: post #259
carterehsmith wrote: post #251

Earlier quoted context omitted.

So this is true, but I find that it does not matter much. Like, how often one needs to rollback a DDL statement? I did that like... never. And, what is the use case? Like, you added a column to a table by accident? Well, that will not break anything, so no harm done. That is way different from regular dml rollback which may recover 1bn records and save your life :)

Seriously? You don't need to do DLL operations in day-to-day use but you'll need to when you're doing development work. For example, you had a "color" column on a table, for a new feature youre now adding the ability to have multiple colors. You're going to create a new column, create a new table, populate that table, and drop the old column. If anything fails during that process you'd like to be able to roll back.

Good points, I am familiar with the process.

There is a concept or "forward-compatible change". Basically, you don't do things that will break your software.

Example, you don't add a NOT-NULL column unless you can give it a good DEFAULT value, to make it work.

Also dont' drop columns until the software is ready for it, etc.

If you have a decent ORM, it will compare your "how it needs to be" sql schema with the "how it is" schema. Then it will generate appropriate "ALTER TABLE ...." "CREATE INDEX " etc statements. Note that this is automated and you never need to type SQL statements to achieve that.

All together in the last XXX years, I did not really need to do a rollback on a dml statement.

Post reply on HN