Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Should Ditch Nginx

hipyoungstartup.com

Post reply on HN
Print view View original

71–80 of 176 posts

Re: We Should Ditch Nginx

#71

Post by rdtsc »

dknecht wrote: post #37

CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx. access_log /dev/nginx_access log_format_name buffer=64k flush=10s; Since…

> CloudFlare generates 50gb/s of logs globally

Wow, mind blowing. That's 4pb/day, 1+eb/year.

Re: We Should Ditch Nginx

#72

Post by jacques_chester »

venus wrote: post #64
jacques_chester wrote: post #57

Earlier quoted context omitted.

I am not disagreeing that in this specific case that the licence structure might lead to unhappy arithmetic. I do however stand by my original point that syslog support is an excellent cleavage point to segment the customer base.

Yeah, I do agree with that. I'm slightly less happy about the HTTP live streaming being in the Plus package, though. I've been playing around a lot with a media app using HLS and I can't even trial using nginx for it - not that I'd be too happy paying $1300 a year since I don't have a single user yet. I can see how they are trying to get money out of the big streaming companies, but it's not too friendly to the littl…

Segmentation is a lossy function, unfortunately. If you try too hard to fit the price/feature curve by creating too many segments, you wind up with nonsense like Windows Home Professional Premium Plus Ultimate Edition.

Re: We Should Ditch Nginx

#73

Post by embro »

It's open source. If your company lives from the web traffic, hire a coder to patch it or just pay for it.

There is also other solutions like Cherokee web server and Lighthttpd.

Re: We Should Ditch Nginx

#74

Post by Mikeb85 »

twelve40 wrote: post #68
Mikeb85 wrote: post #9

Then ditch it and move on. If everyone made a post every time they chose not to buy X product but instead to use Y product then HN would be unreadable. You have the source, patch/fork it. If you're too lazy/not skilled enough, switch to Apache and don't tell us about it.

Although I share your sentiment, noc and devops people are legitimately paranoid about stability, so no, he's neither lazy nor unskilled - this part I can easily believe. What I can't understand is, if they have tons of traffic and he admittedly uses a product he likes, why doesn't he just pay for it. FOSS does not enslave its authors to perpetually provide free infinite scalability for every use case.

Very true. Forgot to mention that part. Why not pay for a great product if you find it useful. I've paid for FOSS stuff before, and supported distros I use...

Re: We Should Ditch Nginx

#75

Post by earino »

dknecht wrote: post #37

CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx. access_log /dev/nginx_access log_format_name buffer=64k flush=10s; Since…

50gb/s ofCloudFlare logging data to "data science" at sounds like the most interesting damned problem I've ever heard. Well, probably second to how you manage that flow into a pipeline in the first place.

Re: We Should Ditch Nginx

#76

Post by [deleted] »

earino wrote: post #75
dknecht wrote: post #37

CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx. access_log /dev/nginx_access log_format_name buffer=64k flush=10s; Since…

50gb/s ofCloudFlare logging data to "data science" at sounds like the most interesting damned problem I've ever heard. Well, probably second to how you manage that flow into a pipeline in the first place.

[deleted]

Re: We Should Ditch Nginx

#77

Post by kul_ »

krenoten wrote: post #69

If you don't want it to hit disk, why don't you just not log to disk? Are you running a unix variant? There are several ways to do this. Why not log to /dev/shm or a pipe, instead of changing nginx? Rsyslog is also probably not the best choice for high volume log aggregation. It performs an rpc per message. That sucks. Use something that batches, like scribe.

I agree, If you are aggregating logs so small that they can fit into memory, most likely that you do not care where it goes. Otoh if the logs are enormous you must have robust periodic log aggregation jobs like map reduce to do that and it is must to have logs on disk.

Moreover 50m/day is ~ 578/sec , which isn't huge for a cluster of servers.

Re: We Should Ditch Nginx

#78

Post by icambron »

Karunamon wrote: post #53
icambron wrote: post #34

Earlier quoted context omitted.

This. From the article: >I don’t blame NGINX for wanting money in exchange for what is admittedly amazing software. I just don’t want any part of it. Why not? You're using this awesome software whose value you appreciate and which is, you admit, open source. It's just not convenient enough for you. So why can't you pony up the money for your huge, high-traffic website?

I can sympathize with this: >Yep, they took a feature that Apache httpd has had literally forever and put it behind a pay wall. Charging for support is one thing, but this seems like it should be a basic feature. And if I were running a server farm on FOSS software and then realized I'd need to cough up ($1350 * ~100) around $135k for the ability to disable obnoxious behavior , because the corporation arbitrarily dec…

It's perfectly reasonable to cringe at a $135K price tag. But if that's so onerous, why not spend a day compiling the source and setting it up for distribution on your servers? The answer is that you (or at least the hypothetical you) feel entitled to this software packaged how you want it with the features you want.

If you want to use Apache, use Apache. That's totally your right. Or you can fork NGINX and do whatever you want. It's even got a BSDish license to make that easy. You could even use the last version of the software that baked this feature into the standard distro. But you're using the official NGINX distro for a reason, and I don't see why the people who make it shouldn't demand a bunch of money for whatever that reason is.

To be clear, it's perfectly fine to say, "I don't think this is a good feature to put behind the paywall." I'm just balking at the whiny and entitled tone of the original post. "How dare they ask me to pay for the things I want!" If I were trying to monetize an open source project, per Jacques's comment above, this is exactly the kind of high-scale feature I'd target for that.

Re: We Should Ditch Nginx

#79

Post by rodgerd »

davidu wrote: post #16

I'm curious what business model the author of the post thinks would be possible for the author of nginx and not result in this scenario for feature x/y/z? Also, as an aside, it seems like the author of the post pays for NetApp and the very, very expensive support contracts that come with it, nginx contracts are a walk in the park, financially, in comparison.

> I'm curious what business model the author of the post thinks would be possible for the author of nginx and not result in this scenario for feature x/y/z?

The author presumably belongs to the same cohort who download films and music without paying and sneers "how you feed yourself isn't my problem, go sell some t-shirts or something". I doubt the author gives a shit, they just want stuff for free, so they can continue to make money with it.

Re: We Should Ditch Nginx

#80

Post by sbierwagen »

chrissnell wrote: post #56
dknecht wrote: post #37

CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx. access_log /dev/nginx_access log_format_name buffer=64k flush=10s; Since…

Apologies everyone for the somewhat o/t question here but, what do you guys do with your 50Gb of logs every second? Where do they go after they leave nginx?

To the NSA, presumably: http://exiledonline.com/read-yasha-levines-introduction-to-t...
Post reply on HN