Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grepping logs is terrible

asylum.madhouse-project.org

Post reply on HN
Print view View original

101–105 of 105 posts

Re: Grepping logs is terrible

#101

Post by falcolas »

616c wrote: post #97
falcolas wrote: post #88

Earlier quoted context omitted.

In addition to logging, you can send out a statsd[0] message, graph it, and use something like Skyline[1] for alerting based on trend issues. You can also use logstash to generate metrics on logs when sending them up to Elasticsearch. [0] https://github.com/etsy/statsd [1] https://github.com/etsy/skyline

Excellent sample projects, especially Skyline. This seems the closest thing to what I had envisioned sofar. Very cool stuff. Do you use it?

Skyline was a bit too much overhead, but we took the concept and adapted it to our needs.

When I say too much overhead, I'm referring to the carbon proxy and redis requirements. We found that just using the json output from graphite was sufficient to feed a trend monitoring system.

The output is pretty sensitive, moreso than Icinga2 (Nagios) expects, so we had to turn down a few of the "is this really down" re-checks, since it would silence legitimate trend alerts.

Re: Grepping logs is terrible

#102

Post by datenwolf »

TheLoneWolfling wrote: post #98
datenwolf wrote: post #95

Earlier quoted context omitted.

> Frequency-sweeping... How are you doing that? The basic principle is called FDML; there's a short description of how it works on our company website: http://optores.com/index.php/technology/5-fourier-domain-mod... A much more thorough description is found in the paper that introduced FDML for the first time: https://www.osapublishing.org/oe/abstract.cfm?URI=oe-14-8-32... > Is the laser itself able to frequency swee…

Interesting. What you're doing sounds a lot like time-domain spectroscopy in an odd sort of way. What are the advantages of this versus just chirping a pulsed supercontinuum source?

    > What you're doing sounds a lot like time-domain
    > spectroscopy in an odd sort of way.
The measurement principle is definitely related to TDS.

    > What are the advantages of this versus just
    > chirping a pulsed supercontinuum source?
Output power: Our system can emit >100mW

Sweep uniformity: The phase evolution of the sweeps is very stable; the mean deviation in phase differences between sweeps is in the order of millirad. Which means that for the time→k-space mapping the phase evolution has to be determined only one time and can then be used for hours of operation; in fact the system operates to repeatable that even after being powered off over the night, the next morning you can often reuse the phase calibration of the previous day. Without that, you'd have to use a second interferometer and sample a k-space reference signal for each and every sweep in parallel and use that for k-space remapping.

Ease of synchronization: Trigger signals have very small jitter. Also the jitter between electrical and optical synchronization is in the order of few ps, which is important for things like Doppler-OCT.

Coherence: Supercontinuum Sources have issues with coherence stability, which degrades the imaging range.

Sentisivity issued: Chirping Pulsed Supercontinuum Sources (which are actually used for OCT) is challenging. It requires a lot of dispersion. High dispersion means a lot of loss, which in turn means it requires another output amplification stage, which in turn will also produce significant optical noise. And optical noise is the bane of OCT, since that reduces the sensitivity. In contrast to that if properly dispersion compensated an FDML laser will exhibit very little noise.

Price: Pulsed Supercontinuum Sources suitable for chirping and OCT applications are quite expensive. Our laser is not cheap as well, but it's still more price effective.

Re: Grepping logs is terrible

#103

Post by TheLoneWolfling »

datenwolf wrote: post #102
TheLoneWolfling wrote: post #98

Earlier quoted context omitted.

Interesting. What you're doing sounds a lot like time-domain spectroscopy in an odd sort of way. What are the advantages of this versus just chirping a pulsed supercontinuum source?

> What you're doing sounds a lot like time-domain > spectroscopy in an odd sort of way. The measurement principle is definitely related to TDS. > What are the advantages of this versus just > chirping a pulsed supercontinuum source? Output power: Our system can emit >100mW Sweep uniformity: The phase evolution of the sweeps is very stable; the mean deviation in phase differences between sweeps is in the order of mill…

All good answers. Thank you.

Re: Grepping logs is terrible

#104

Post by babuskov »

madhouse wrote: post #70
babuskov wrote: post #38

If you need to grep logs on regular basis, you're doing it wrong. Store important data in the database so that you can query it efficiently. Keep logs for random searches when something unexpected happens. I log gigabytes per day, but only grep maybe once-twice a year.

Agreed. But if I keep my logs in a database, I may aswell use the database to query my logs, instead of grepping in them. (And voila, you have binary log storage.)

Separate the data you are sure you need often and only store that in the database. Store everything else in the textual logs.
Post reply on HN