Zenoss and Syslog catching
I got Zenoss to gather messages from syslog today. Actually, I had it up a few days ago, but was having a problem that I finally resolved today. We already have a centralized syslog server in the data center using syslog-ng, so everything is hitting the monitor station. So logically it should be easy to feed the messages into Zenoss. But of course it never is. I ran into 2 problems:(1) zenoss defaults to listening on the primary syslog port, UDP 514. Obviously that conflicts with the existing server. Zenoss provides an option to give a different port, but the issue is that there's no consistent way (that I've found yet) to specify that. The documentation hints that you can add to $ZENOSS_HOME/etc/zensyslog.conf, but it also says you have to specify ALL possible options, without a clear explanation of what all is required. So for now the options are in $ZENOSS_HOME/bin/zensyslog, tacked onto the end of the call (see below for specifics).(2) The other problem I had, after fixed the first, is that a lot of the messages were being tagged to the nonexistent host "127.0.0.1". After a lot of digging, I figured out that it's because zenoss was setting the device to the host that originates the syslog packet, which because it was being re-routed by syslog-ng, was localhost.From the syslog-ng docu, there's an option to the destination parameter, spoof_source(yes), that will tell it to write the forwarding packet with the original sender's info. When syslog-ng started complaining about the option, I had to search some more, finally downloading the source for syslog-ng 2.0x. This option is mentioned in the docs still, but nothing actually using it in the source code. So I grabbed v 1.6x, and found it there. Apparently BalaBit (vendor for syslog-ng) rewrote the source in moving to 2.0, stripping out this feature, but hasn't yet updated the docs. Unfortunately Sarge (including backports) can't compile the 1.6 code anymore, so I had to forget that route. <sigh>So after kicking myself for wasting that couple of hours, I looked back at the options for zensyslog. Not surprisingly, there's another option that can be passed to zensyslog, --parsehosts, to tell it to use what's in the actual message instead of the packet. What was surprising is that it didn't work! Watching the debug output showed it still assigning to localhost. After some more digging, it turns out there's a bug in the function that handles this, where the parsed host gets saved to the wrong variable, thus dropping it (and of course Python merrily allows the assignment to a non-existent member variable without a peep!)Once I fixed and reported the bug (ticket #999), logs started flowing into the right buckets. Yeah!Now for the specifics on what all I did:
- Add options to zensyslog startup:
- Add entries to syslog-ng.conf to setup the forwarding:
<br /><pre><br /><br /># make sure these options are set, to keep it simple for zenoss<br />options {<br /> use_fqdn(no);<br /> chain_hostnames(no);<br /> normalize_hostnames(yes);<br /> log_msg_size(4096);<br /> keep_hostname(yes);<br />};<br />source s_all {<br /> internal();<br /> unix-stream("/dev/log" keep-alive(yes) max-connections(100));<br /> pipe("/proc/kmsg");<br /> tcp(keep-alive(yes) max-connections(150));<br /> udp();<br />};<br />filter f_warn { priority(warning..emerg); };<br />destination d_zenoss { udp("localhost" port(5514)); };<br />log { source(s_all); filter(f_warn); destination(d_zenoss); };<br /><br /></pre><br /> - Fix parsehosts bug:
- Restart zensyslog and syslog-ng:
<br /><pre><br />zenoss@localhost:/opt/zenoss$ ./bin/zensyslog restart<br /><br />(change to root)<br /><br />[root@localhost] # /etc/init.d/syslog-ng restart<br />Stopping system logging: syslog-ng.<br />Starting system logging: syslog-ng.<br /><br /><br /></pre><br />
<br /><pre><br />zenoss@localhost:/opt/zenoss$ more bin/zensyslog<br />#! /usr/bin/env bash<br /><br />. $ZENHOME/bin/zenfunctions<br /><br />SUDO=sudo<br />PRGHOME=$ZENHOME/Products/ZenEvents<br />PRGNAME=zensyslog.py<br />CFGFILE=$CFGDIR/zensyslog.conf<br />PIDFILE=$VARDIR/$PRGNAME.pid<br /><br />generic "$@" --syslogport 5514 --parsehost<br /></pre><br />
<br /><pre><br />*** $ZENOSS_HOME/Products/ZenEvents/SyslogProcessing.py.orig 2007-01-19 15:32.000000000 -0500<br />--- $ZENOSS_HOME/Products/ZenEvents/SyslogProcessing.py 2007-02-16 16:13:54.000000000 -0500<br />***************<br />*** 136,143 ****<br /> msg = m.group(2).strip()<br /> msglist = msg.split()<br /> if self.parsehost and not self.notHostSearch(msglist[0]):<br />! evt.hostname = msglist[0]<br />! slog.debug("parseHEADER hostname=%s", evt.hostname)<br /> msg = " ".join(msglist[1:])<br /> return evt, msg<br /><br />--- 136,143 ----<br /> msg = m.group(2).strip()<br /> msglist = msg.split()<br /> if self.parsehost and not self.notHostSearch(msglist[0]):<br />! evt.device = msglist[0]<br />! slog.debug("parseHEADER hostname=%s", evt.device)<br /> msg = " ".join(msglist[1:])<br /> return evt, msg<br /></pre><br />
Comments
Post new comment