Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We saved $50k/year with a Go microservice coded in a hackathon

movio.co

Post reply on HN
Print view View original

131–140 of 264 posts

Re: We saved $50k/year with a Go microservice coded in a hackathon

#131

Post by jjeaff »

candiodari wrote: post #98
ajcodez wrote: post #92

Earlier quoted context omitted.

It makes sense if each client gets to add their own fields.

Using the builtin type for that purpose is going to work way better. This depends on the DB you're using but is generally referred to as a "JSON" field (why ? Because they're a response to MongoDB, which calls is that). Oracle and SQL server have very similar things. In Mysql, it is JSON data type [1], in Postgres JSON/JSONB [2]. Creating indexes across them is doable, through a workaround (involving what is generall…

At least in MySQL, json field types are rather new. MySQL 5.7 is not yet an option with AWS Aurora or Google cloud SQL even.

And I don't think you will necessarily get better performance with json fields vs an EAV model. Yes, you can index json fields by creating virtual views, but that requires that you know the field ahead of time. With an EAV model, you can have your values table indexed and then join.

But I am excited to start using the json field types. In many cases, it will really simplify things over the traditional EAV stuff.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#132

Post by zaarn »

icebraining wrote: post #94
matte_black wrote: post #77

Earlier quoted context omitted.

If you are using Postgres, the JSONB datatype will let you do exactly this while still using the full power of SQL. Simply create a column where you keep a JSON object full of random user properties, if flexibility is what you want. You can even index properties.

The question is whether it can be stored like this while allowing for fast queries. For example, unless it changed recently, Postgres doesn't calculate statistics to help the query planner on jsonb fields.

JSONB columns should behave just like any datatype with a GIN index in the recent releases, to my knowledge.

Still, a JSON column will arguably be faster than a pure KV table since you can more efficiently query it, especially any non-JSON columns.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#133

Post by LoSboccacc »

bufferoverflow wrote: post #113
attaboyjon wrote: post #78

I've come to the conclusion that the problem in tech is that all the people doing the work are in their early twenties and have no idea what they are doing. Once they get some experience they are quickly promoted to the CTO position. Rinse and repeat. What we have here is a classic dbms problem and no one at Movio seems to know how to deal with that. Instead of migrating from Mysql to something serious (Postgres) the…

Can you demonstrate that Postgre is significantly faster than MySQL on average? Highly doubt so. The problem is in the way data model was architected and implemented.

apples to oranges. you need to compare PostgreSQL with other database that don't take shortcut around ACID for performances (for example DDL forcing implicit commit in transactions)

Re: We saved $50k/year with a Go microservice coded in a hackathon

#134

Post by zaarn »

bobwaycott wrote: post #41
hippich wrote: post #14

What else it shows - how expensive AWS hardware vs hosting own hardware. I guess you have to consider how often you have to scale, but hetzner offers dedicated servers with 64Gb and NVMe drives starting from 54 euros per month - https://www.hetzner.com/dedicated-rootserver?country=us - compare that to $580 per month these guys were paying for i3.2xlarge instance..

I’ve heard mention of Hetzner no less than a dozen times in the last couple days. What’s their deal? I’m not quite sure I grok this server auction thing they do, or how they’re so cheap.

They offer bare metal outside the auction, which I recommend unless you look for the absolute cheapest.

The Server Auction is basically servers they previously used and were freed up (because a customer didn't need it anymore, for example).

It's somewhat similar to what OVH is doing with KimSufi or SYS.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#135

Post by LoSboccacc »

mathgladiator wrote: post #51
Matheus28 wrote: post #24

Earlier quoted context omitted.

If you aren't implementing your own tabular database on top of your existing tabular database, you aren't 2018 enough :)

I'm changing the game by code generating a tabular database on top of an eventualy consistent store. 2020 here we come!!!

oh boy do I have news for you https://www.cockroachlabs.com/

Re: We saved $50k/year with a Go microservice coded in a hackathon

#136

Post by olalonde »

chiefalchemist wrote: post #63
jehlakj wrote: post #61

Earlier quoted context omitted.

You can thank full stack developers for that

Them, or the manager that hired them? If no one up the chain brings on a DBA what are they supposed to do? I hear ya. But this is as much a symptom of naive (and budget stretched) leadership as it is of the hands on deck.

Your solution costs 150K$/year while theirs took a weekend and saves 50K$/year...

Re: We saved $50k/year with a Go microservice coded in a hackathon

#137

Post by jjeaff »

philliphaydon wrote: post #49
deizel wrote: post #38

Earlier quoted context omitted.

Can be tricky to use EAV data models with traditional ORMs.. this type of functionality can often be slow or require plugins, if implemented at all: https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...

Wondering if they shoved the data in PostgreSQL with JSONB how well it would perform over EAV.

I think that jsonb may not be as performant as EAV. You don't need joins or unions, but if you are dealing with dynamic fields, you need to know the fields ahead of time and set indexes for them in jsonb. For eav you just have to index your values table.

Re: We saved $50k/year with a Go microservice coded in a hackathon

#138

Post by reificator »

MrMorden wrote: post #56
cookiecaper wrote: post #19

Earlier quoted context omitted.

Yeah, sadly, this is not too much of an exaggeration. I've worked on teams that insisted they needed DynamoDB, because, well, Dynamo is for "Big Data", and they certainly wouldn't work somewhere that had "Small Data"! Replace the buzzwords/products as applicable; you could actually probably just scramble them and it'd work just as well, since someone out there thinks "RabbitMQ means Web Scale", etc. SQL databases are…

Every fool knows that MongoDB is web scale. (Not as web scale as /dev/null, but MongoDB has a better logo.)

I disagree. /dev/null's logo is way cooler.

https://www.nasa.gov/sites/default/files/cygx1_ill.jpg

Re: We saved $50k/year with a Go microservice coded in a hackathon

#139

Post by z3t4 »

mySQL is very slow when it comes to joins, groups, etc. You are always better off with simple select's. If you are using for example PHP the only viable solution is to have the db crunch it. But when using Go, NodeJS et al. you can pull the data out as a stream/array-like and apply filter/map/reduce and the logic would probably be easier to manage, rather then generating a complex SQL query. And would also allow you to stream the result to the client, instead of having the user wait for it all before they see anything. A lot of money could probably be saved by having the data on the client side, in for example web db, and only use the servers for backups and syncing the data between clients.
Post reply on HN