Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Another reason why Docker containers may be slow

hackernoon.com

Post reply on HN
Print view View original

41–50 of 72 posts

Re: Another reason why Docker containers may be slow

#41

Post by asdbffg »

aaronchall wrote: post #38
travbrack wrote: post #12

Earlier quoted context omitted.

Illumos/SmartOS is FOSS

I remember being told Java is FOSS too... now tell that to Google...

Java is FOSS.

Oracle used excuse, that it's current OpenJDK license (GPLv2) is incompatible with license, used by Google's runtime (Apache 2). If Google re-licensed it's Java implementation under GPL, some of arguments, used by Oracle lawyers (code reuse and patent (?) violations), would have been void, and arguing about reuse of APIs would have been a lot harder.

Of course, this does not really matter, because the whole lawsuit is just excuse for power games between corporations. Oracle's goal wasn't about Java licensing, it was about gaining some degree of control over emerging Android ecosystem.

Re: Another reason why Docker containers may be slow

#42

Post by dbenhur »

dingo_bat wrote: post #35

Nice debugging story but the conclusion was totally wrong! The author even knows this. If they would be logging 3-4x the usual rate they would have seen the same problem on bare metal too. Nothing to do with docker or competing containers or whatever.

Not only that, but glog authors observed the overhead of frequent fadvise and added rate limiting almost a year and half ago. https://github.com/google/glog/commit/dacd29679633c9b845708e...

+1 perf. -1 stale library use. -1 misdirected learning

Re: Another reason why Docker containers may be slow

#43

Post by asdbffg »

cjhanks wrote: post #32
pacavaca wrote: post #21

Earlier quoted context omitted.

The point was not to complain about how bad the Docker is but rather to highlight that a lot of unexpected things may come from the fact that the kernel is shared. "Logging too much" was just a reason for the posix_fadvise being called too often but this becomes a problem ONLY when the kernel is shared. In case of virtualization, everyone gets its own version of fadvise (the kernel) and the conflict doesn't happen.

If your VM call to `fadvise` is not calling the underlying host kernel operation, is it even working?

VM certainly does "call underlying host kernel operation", it just does so indirectly — the guest userspace calls fadvise(), kernel implementation of fadvise() asks the virtio disk driver to perform particular read/writes, the virtio driver asks underlying kernel disk driver to read/write individual disk sectors (without knowing, that they are related to specific file in guest filesystem).

This specific bug was caused by putting high load on "kernel dentry cache", e.g. a contention for memory structure, present in kernel memory. Guests normally don't share memory, so contending for it was avoided.

Incidentally, there are situations, when different guests can compete for same memory — when VM uses so-called "memory deduplication" techniques. Which is why enabling that stuff on production systems may be a bad idea.

Re: Another reason why Docker containers may be slow

#44

Post by asdbffg »

deathanatos wrote: post #17
dstroot wrote: post #15

Better title: “nother reason why my code is slow and I’m logging too much”

Yeah. The whole article seems to be a bug in the logging library that's in use here, and an area of contention in the kernel? (How many times is his application logging, to cause that much contention?) I'm not sure how Docker figures into it, aside from it easily lets one run multiple instances of an app on the same hardware, but stuff like supervisord will also do that? I'm not entirely clear as to why a logging lib…

Using O_APPEND does not imply, that kernel needs to purge the pages from cache ASAP, does it? Removing pages from cache may be expensive operation by itself, so I presume, that it is avoided by default.

More importantly, if the disk can not catch up, the log data is going to end up waiting in page cache anyway (typical case of bufferbloat). Linux kernel does not have telepathic abilities to balance needs of crazy logger and other applications in system, so without resolving underlying issue (bufferbloat), those writes would take up too much cache, potentially bringing down disk performance of other applications.

fadvise() may schedule quicker eviction, effectively acting as syscall version of vm.dirty_ratio. Of cause, that does not resolve the problem, — just moves it to different layer. The real solution is either

1) blocking the apps until their logs are fully written (for example, by using O_DIRECT)

2) showing those apps middle finger and throwing away some of their logs (AFAIK, this is occasionally done by syslog).

Re: Another reason why Docker containers may be slow

#45

Post by ofrzeta »

lurker9 wrote: post #2

I don't understand why more people don't use Solaris Zones, they seem to me to be the superior solution by far, and with work done by Joyent you now have modern LX-branded zones also. Is the lack of adoption mainly due to the fact that it's Solaris, and not Linux? (Solaris lives on in Illumos et al)

It lives but it's not exactly thriving. OpenIndiana struggled, OmniOS gave up (and got picked up again) but still all that struggle makes me expect that Illumos will eventually fade out. At the latest when/if Joyent goes out of business. It's a shame because I like SmartOS a lot but it nowhere has the momentum Linux has. I don't see who will write all the device drivers for the next generation of hardware coming out.

Re: Another reason why Docker containers may be slow

#47

Post by oneweekwonder »

asdbffg wrote: post #41
aaronchall wrote: post #38

Earlier quoted context omitted.

I remember being told Java is FOSS too... now tell that to Google...

Java is FOSS. Oracle used excuse, that it's current OpenJDK license (GPLv2) is incompatible with license, used by Google's runtime (Apache 2). If Google re-licensed it's Java implementation under GPL, some of arguments, used by Oracle lawyers (code reuse and patent (?) violations), would have been void, and arguing about reuse of APIs would have been a lot harder. Of course, this does not really matter, because the w…

Can google pull a swift move?

Re: Another reason why Docker containers may be slow

#49

Post by rickycook »

crcl wrote: post #13
rmrfrmrf wrote: post #10

Isn't this kind of thing part of the reason why you log to stdout instead of handling disk writes in your app?

A comment in the Medium article asked the same thing. The author's response: > I agree that piping all logs to stdout would be the best solution in case of Dockerized microservices. It’s just that in our case we were porting an existing system, which a) already heavily relied on logging to files b) consisted of many microservices itself, which we couldn’t yet split into separate Docker containers but also couldn’t pi…

well that’s just an argument to use a pluggable logging system like in python/log4j where you don’t configure logging in your service/library and leave that up to the implementor. also let’s you log to a structured format like JSON and pipe that to a logging aggregator

Re: Another reason why Docker containers may be slow

#50

Post by jordanrobinson »

oneweekwonder wrote: post #47
asdbffg wrote: post #41

Earlier quoted context omitted.

Java is FOSS. Oracle used excuse, that it's current OpenJDK license (GPLv2) is incompatible with license, used by Google's runtime (Apache 2). If Google re-licensed it's Java implementation under GPL, some of arguments, used by Oracle lawyers (code reuse and patent (?) violations), would have been void, and arguing about reuse of APIs would have been a lot harder. Of course, this does not really matter, because the w…

Can google pull a swift move?

Many would say they already are with Kotlin.
Post reply on HN