Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We saved $50k/year with a Go microservice coded in a hackathon

movio.co

Post reply on HN
Print view View original

211–220 of 264 posts

Re: We saved $50k/year with a Go microservice coded in a hackathon

#211

Post by collyw »

bitexploder wrote: post #79
nstart wrote: post #69

Earlier quoted context omitted.

Oh gosh this pattern. The first time I encountered it was in my first job where we used Magento. Super flexible. Also super slow. Does anyone have any advice how to make a db design like this work faster? Generally I thought when data is arranged like this it might be a prime candidate for document based storage. But I'm no dba so I have no idea if that would be correct.

This just isn't that hard. They don't have that much data. It is really late for me, but, put it all in memory and figure it out. These just aren't hard problems. DBAs have been solving performance issues for decades with a clever index on the right column for 30+ years. Sorry if this is get off my lawn-ish, but I have been on too many projects where I made a DB index and solved a major bottleneck. Too many new devel…

I am not a DBA, but I could see this crap when the NoSQL hype took off.

"Relational databases don't scale"

Well they worked fine for decades before Mongo's marketing claimed so.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#212

Post by benmmurphy »

orf wrote: post #8

I don't quite get this. How fast was running this query: Select loyaltyMemberID from table WHERE gender = x AND (age = y OR censor = z) Why the random complexity with individual unions and a group? Of course that's going to be dog slow. Sure, the filters can be arbitrary but with an ORM it's really really simple to build them up from your app code. The Django ORM with Q objects is particularly great at this. Obviousl…

age/gender are probably simple but i'm guessing censor is probably derived from a transaction table. if they are letting users select arbitrary time ranges to filter the transactions then you can't store a precomputed censor value for each user. but seeing that they are talking about caching maybe a lot of stuff can be precomputed.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#213

Post by collyw »

orf wrote: post #174
rypskar wrote: post #173

Earlier quoted context omitted.

I blame ORMs', if you don't understand SQL and how databases work you should not be allowed to use an ORM. If you know how databases work you, in many cases, will not use an ORM except for the most simple CRUD operations.

I know how databases work and I use a (good) ORM almost exclusively for app work (SQL for exploration usually though). The benefits are huge.

I use both. I know the Django ORM and I know where its limits are. For getting data in it saves a load of time. When the query gets complex it starts adding time, or making things impossible (multiple join conditions weren't possible unless its been update in the most recent version).

Re: We saved $50k/year with a Go microservice coded in a hackathon

#214

Post by EGreg »

pandapunchpower wrote: post #70
EGreg wrote: post #52

Earlier quoted context omitted.

Question, though... If your columns have types, how can you encrypt them using custom keys for each one? Is it possible? I want the keys to reside on the client and search for encrypted data. Basically single row lookups at a time.

The column type would have to be the type of the encrypted value. The type of the unencrypted data could not be enforced by the DB and you would have to rely on code doing the correct thing. I am however extremely wary of doing it that way. I don't know your requirements of course.

Here is the thing - encrypted stuff is just a weird encoded string. So I can’t really use columns normally.

What I really need is just a huge table with two fields: “token”, “content”

And the token is basically the primary key but encrypted with whatever encryption.

You could even do foreign keys this way.

Hmm I suddenly have an idea. What about a layer above the database that basically enforces foreign keys and joins in this way to support end to end encryption? The content would reference ENCRYPTED foreign keys. Only clients would decrypt stuff.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#215

Post by dboreham »

attaboyjon wrote: post #78

I've come to the conclusion that the problem in tech is that all the people doing the work are in their early twenties and have no idea what they are doing. Once they get some experience they are quickly promoted to the CTO position. Rinse and repeat. What we have here is a classic dbms problem and no one at Movio seems to know how to deal with that. Instead of migrating from Mysql to something serious (Postgres) the…

Agree. I'm not sure per se this is an age thing: the field is just so big and apparently (but perhaps less so in reality) in a constant state of tech churn, that it is hard to anchor to consistent proven techniques and practices.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#216

Post by dboreham »

bufferoverflow wrote: post #113
attaboyjon wrote: post #78

I've come to the conclusion that the problem in tech is that all the people doing the work are in their early twenties and have no idea what they are doing. Once they get some experience they are quickly promoted to the CTO position. Rinse and repeat. What we have here is a classic dbms problem and no one at Movio seems to know how to deal with that. Instead of migrating from Mysql to something serious (Postgres) the…

Can you demonstrate that Postgre is significantly faster than MySQL on average? Highly doubt so. The problem is in the way data model was architected and implemented.

Postgresql is different not because it is fast but because it works. And keeps working.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#217

Post by amichal »

z3t4 wrote: post #139

mySQL is very slow when it comes to joins, groups, etc. You are always better off with simple select's. If you are using for example PHP the only viable solution is to have the db crunch it. But when using Go, NodeJS et al. you can pull the data out as a stream/array-like and apply filter/map/reduce and the logic would probably be easier to manage, rather then generating a complex SQL query. And would also allow you…

i have a 10+year old project on mysql doing 6+ way joins, groups, etc against 100million row tables in sub second time. It all depends on the indexing, disk layout and size of intermediate products etc. We dont know why they did "UNION ALL" with huge intermediate products (it seems like the query could be a single index/table scan on its face) but that is likely the slow down "EXPLAIN SELECT" would tell us

Re: We saved $50k/year with a Go microservice coded in a hackathon

#219

Post by fbr »

ksec wrote: post #116
attaboyjon wrote: post #78

I've come to the conclusion that the problem in tech is that all the people doing the work are in their early twenties and have no idea what they are doing. Once they get some experience they are quickly promoted to the CTO position. Rinse and repeat. What we have here is a classic dbms problem and no one at Movio seems to know how to deal with that. Instead of migrating from Mysql to something serious (Postgres) the…

If I ever be a CEO of the company / Startup, that one criteria I made is either I decide on all the technologies we use, or there is no CTO so i make those decision. And that criteria of technologies could be summed into one sentence. Use something boring . No Hyped programming languages / DB / tools allowed. Of course some would argue you would be doing it wrong even if it was using old tech / programming / tools. W…

I agree with the main idea: working with hyped technologies is not a solution and you can build most of the things out there with boring technology.

But then ... you have to find, attract and hire good developers. That's already difficult, adding an extra layer of 'boring technology' will make this task even more challenging.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#220

Post by cleong »

wlll wrote: post #196
pisky wrote: post #171

Earlier quoted context omitted.

There is one thing that bugs me about all the talk of postgres' superiority: why haven't these companies switched to postgresql? Surely they weren't all too far invested in mysql before a "more knowledgeable" DBA came along saying postgresql is better.

Because MySQL is the PHP of databases. Much like PHP is designed to just carry on truckin' whatever problem it encounters ( https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/ ), MySQL was too. It's easy to slam some data into it and get it back without having to think too hard about what you're actually storing ( https://grimoire.ca/mysql/choose-something-else ). It's ease of use made it pervasive, much like…

PHP was built for the web and has been successful at that job. It is easy to use because the core developers have made some good design choices for the task at hand. For example no threads, stateless requests, core functionality focused on outputting HTML, etc.

MySql and PHP are good. They do the job they were designed for in a cost effective way and of course that means there will be trade offs.

Post reply on HN