Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we build microservices

blog.yourkarma.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: How we build microservices

#31

Post by amstrad464 »

marceldegraaf wrote: post #17
iain_hecker wrote: post #15

Earlier quoted context omitted.

Yup we are. There might be multiple applications interested in the event and they each get their own queue.

Have you looked into a "real" messaging queue for this, like RabbitMQ or another AMQP based system for this? Or are you bound to SNS/SQS for a particular reason?

"Real messaging" aside, this is an interesting question. It'd be great to hear from iain_hecker if they considered using RabbitMQ (or similar) and if yes, why SQS/SNS were chosen.

Also, I'd be interested to read more about how you handle authentication across web/mobile. Thank you for the blog post and for taking the time to answer questions here. :-)

Re: How we build microservices

#32

Post by futurepaul »

amstrad464 wrote: post #31
marceldegraaf wrote: post #17

Earlier quoted context omitted.

Have you looked into a "real" messaging queue for this, like RabbitMQ or another AMQP based system for this? Or are you bound to SNS/SQS for a particular reason?

"Real messaging" aside, this is an interesting question. It'd be great to hear from iain_hecker if they considered using RabbitMQ (or similar) and if yes, why SQS/SNS were chosen. Also, I'd be interested to read more about how you handle authentication across web/mobile. Thank you for the blog post and for taking the time to answer questions here. :-)

We have considered stuff like RabbitMQ. They would be excellent solutions too and we are working on a part of our architecture with MQTT/protobuf (more on that later, I'm sure). We went with SNS/SQS because we don't have to build a cluster with that. Building clusters is hard and Amazon takes care of that. Since we were already using a bunch of AWS products, this was a nice fit and we're quite happy with it.

Authentication is a good idea for an article too. Thanks :)

Re: How we build microservices

#33

Post by futurepaul »

pdx wrote: post #24
iain_hecker wrote: post #22

Earlier quoted context omitted.

Each app gets their own queue. So there is a queue for the shipment app and for the mailer app. When an event is published to SNS, it gets added to both queues.

I see. So you never add a listener, without also modifying the emitter to send to an additional queue.

We don't change the emitter, the listener changes the SNS config. The emitter publishes to SQS under a certain topic. Listeners that interested in that topic subscribe to it by changing the SNS config, basically saying "give me a copy of those events too and put it onto my own queue". SNS copies the messages to every queue.

Re: How we build microservices

#34

Post by cwt137 »

I've been thinking a lot about SOA. Articles like this and Living Social's SOA blog posts series are a big help. But, I feel something is still missing. For example, no one talks about security of the various services. Are there any good books someone can recommend on SOA?

Re: How we build microservices

#35

Post by marceldegraaf »

noselasd wrote: post #28
marceldegraaf wrote: post #17

Earlier quoted context omitted.

Have you looked into a "real" messaging queue for this, like RabbitMQ or another AMQP based system for this? Or are you bound to SNS/SQS for a particular reason?

Why isn't SNS/SQS a "real" messaging queue ?

Sorry, my question was a bit vague. Generally SQS works with polling, and once a message has been pulled from a queue it is not available to other viewers any longer. This makes it hard to create a system where you have an unknown number of clients that may or may not be interested in certain queues, or even in messages of certain types.

RabbitMQ (and other AMQP based queueing systems) have built-in solutions for this that SQS doesn't offer. Karma solved it by sending the same message to a number of different queues, but this makes scaling to more clients harder, and reduces flexibility in subscribing clients to certain message types.

Hope this makes it clearer :-).

Re: How we build microservices

#36

Post by iain_hecker »

cwt137 wrote: post #34

I've been thinking a lot about SOA. Articles like this and Living Social's SOA blog posts series are a big help. But, I feel something is still missing. For example, no one talks about security of the various services. Are there any good books someone can recommend on SOA?

We have all services running inside a VPN (see one of our older posts: https://blog.yourkarma.com/building-private-clouds-with-amaz...) and we also use HTTP Basic token auth that are configured upon deployment. Every app gets its own token, so we can trace which app does what.

Re: How we build microservices

#37

Post by iain_hecker »

marceldegraaf wrote: post #35
noselasd wrote: post #28

Earlier quoted context omitted.

Why isn't SNS/SQS a "real" messaging queue ?

Sorry, my question was a bit vague. Generally SQS works with polling, and once a message has been pulled from a queue it is not available to other viewers any longer. This makes it hard to create a system where you have an unknown number of clients that may or may not be interested in certain queues, or even in messages of certain types. RabbitMQ (and other AMQP based queueing systems) have built-in solutions for thi…

That's where SNS comes in. We publish to SNS which will fan out to multiple SQS queues. When a new service is built, it says to SNS "subscribe my queue to these topics". The publisher of the event doesn't know about the subscribers.

Re: How we build microservices

#38

Post by cwt137 »

iain_hecker wrote: post #36
cwt137 wrote: post #34

I've been thinking a lot about SOA. Articles like this and Living Social's SOA blog posts series are a big help. But, I feel something is still missing. For example, no one talks about security of the various services. Are there any good books someone can recommend on SOA?

We have all services running inside a VPN (see one of our older posts: https://blog.yourkarma.com/building-private-clouds-with-amaz... ) and we also use HTTP Basic token auth that are configured upon deployment. Every app gets its own token, so we can trace which app does what.

I'd figured the services were on a private network (or at least the app is listening on an interface on a private network), but it is good someone confirmed it. Thanks for your insight on security.

Re: How we build microservices

#39

Post by marceldegraaf »

iain_hecker wrote: post #37
marceldegraaf wrote: post #35

Earlier quoted context omitted.

Sorry, my question was a bit vague. Generally SQS works with polling, and once a message has been pulled from a queue it is not available to other viewers any longer. This makes it hard to create a system where you have an unknown number of clients that may or may not be interested in certain queues, or even in messages of certain types. RabbitMQ (and other AMQP based queueing systems) have built-in solutions for thi…

That's where SNS comes in. We publish to SNS which will fan out to multiple SQS queues. When a new service is built, it says to SNS "subscribe my queue to these topics". The publisher of the event doesn't know about the subscribers.

Ah, nice, I didn't know the integration of SNS and SQS worked that way. Cool!

Re: How we build microservices

#40

Post by edwinnathaniel »

AdrianRossouw wrote: post #12
xetorthio wrote: post #11

Earlier quoted context omitted.

In my experience even getting all the pieces installed in a testing environment is hard. Specially when you need to make sure that they are all wired correctly, with the correct version, with all their dependencies (maybe different kind of databases, queues, caching layers) and on top of that if services are written in different languages, that adds another level of complexity. Eventually when the system grows big en…

this is where docker and the ecosystem around it starts filling in the pieces. also, it makes failures hurt less, and rollbacks much easier.

Docker answers half of the issues: deployment.

The other halves: distributed logging (how do you trace a 'transaction' that is being executed against multimachines/multiplaths?)

What about rolling releases? what about versioning?

What about service discovery (still immature field, look at how many products/tools out there trying to be THE service discovery choice).

Post reply on HN