[dokuwiki-teams] Re: Server Admin

2012/4/6 Andreas Gohr <andi@xxxxxxxxxxxxxx>:
> Hi Admin-Team,
>
> I just had to reset the server as it became unreachable around 12:40
> (Europe/Berlin) - no ping, ssh or http on IPv4 and IPv6.
>
> A couple of things I'd like you to have a look at:
>
> - check logs and munin if you can find the cause

I've been looking into this and it looks like the zabbix agent has
been running amok. SInce April 2nd i've found more than 12 million(!!)
entries like these:

Apr  6 11:51:07 dib kernel: [8436349.936983]
Shorewall:fw2net:ACCEPT:IN= OUT=eth0 SRC=46.4.55.201 DST=212.37.47.123
LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=39506 DF PROTO=TCP SPT=42873
DPT=10051 WINDOW=5840 RES=0x00 SYN URGP=0

Which to me looks as if the agent tries to connect to the zabbix
trapper (port 10051 on the zabbix server). Today alone almost 2
million connection attempts were made between 05.00 and noon - when
the server stopped writing to the logs. What exactly the zabbix agent
is trying to do is hard to say as the agent log is truncated at
restart.

My best guess is that the server maxed out the TCP port space or
perhaps too many open file descriptors - allthough I can't find
annything in the logs to back up my theory - heck, I can't find much
else than entries like the one above :-(

> - check if any mysql tables need repairs

I don't think I've ever known the root password for the mysql server -
I'd need that to check. If somebody could help me with that I'd be
happy to check

> - check filesystem and running process for unusual stuff (just to be
> sure this was not an intrusion)

Checked it and didn't find anything out of the ordinary.

> - adjust the firewall logs. we have so much unnecessary crap from the
> firewall in the system logs that is hard to see any relevant things

To get rid of firewall entries in the syslog/messages log we should
probably make shorewall log to ulogd instead of rsyslogd. I you all
agree, I could look into that sometime next week.

> - check server monitoring (I did not get an alert)

I have to confess I'm a bit of a zabbix novice, could someone with
better understanding of the setup please look into this. Thanks.

-- 
Med venlig hilsen / Best Regards
Frank M.G. Jørgensen

Other related posts: