From: Subject: Watching your log files Date: Sun, 17 Jun 2001 11:54:42 +0900 MIME-Version: 1.0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Location: http://www.enteract.com/~lspitz/swatch.html X-MimeOLE: Produced By Microsoft MimeOLE V5.00.3018.1300 Watching your log files How to automate your log filtering =
Watching Your=20 Logs
Lance=20 Spitzner
Last Modified: 19 July, 2000=20

Logs are a critical asset to successfully running your systems. = They tell=20 us what is and what is not happening. However, logs can be extremely = copious,=20 quickly overwhelming us with information. Soon they become useless files = that=20 just fill up disk space. This article will cover how to solve this by = automating=20 the filtering of your logs, freeing up your time while alerting you with = the=20 information you need.
 =20

Filtering
Logs are an incredible asset,=20 unfortunately they are often ignored. You have too little time to review = too=20 much information. Wouldn’t it be nice to automate the process, a = process that=20 reviews the logs for you, then notifies you with only the information = you need.=20 Well, we are going to do just that. I am going to cover how to filter = your logs=20 for the information you need, then implement a notification system.=20

The first part of this article will cover developing a plan on what = you want=20 to filter, and what you want to be notified of. The second half will be = on=20 implementing the filter. For this article, I will be using swatch as a = filter,=20 written by Todd Atkins. We will also be using sendmail logs as an = example of the=20 filtering process. However, you can apply these guidelines to any type = of=20 logging.=20

Where = to=20 begin
The best place to start is with a plan. = There are=20 three steps to planning for automated logging. The first step is define = what you=20 want to know. Determine what information  you need out of your = system logs.=20 The second step is to identify which logs contain that information. The = third=20 step is identifying the trigger, what defines the critical information?=20

For example, lets say you are concerned about the security of your = sendmail,=20 specifically you want to know if someone attempts to use your mail = server as a=20 spam relay. You also want to know if anyone is attempting to gain = unauthorized=20 information with SMTP commands, such as expn. We have completed the = first step=20 by determining what we want to know.=20

The second step is identifying the source, or what log contains this=20 information. The best place to find that is /etc/syslog.conf, this = configuration=20 file will show you what information is logged where. For mail, we see = that all=20 mail information is logged to /var/log/syslog on our Solaris system=20 (/var/log/maillog for Linux).=20

goalith #cat = /etc/syslog.conf=20 | grep mail
 mail.debug ifdef(`LOGHOST', /var/log/syslog,=20 @loghost)=20

The final step is defining the trigger. What specific entries in the = logs=20 define the information we are looking for. For sendmail, we are looking = for two=20 triggers.=20

  1. Trigger that shows un-authorized IP addresses attempting to use = our mail=20 server as a mail relay.=20
  2. Trigger that shows someone is attempting to use the expand = command, which=20 we have turned off.


The best way to define the trigger is to recreate the incident = while=20 monitoring the log with /usr/bin/tail –f. If you can’t do = this on a production=20 system, find a lab system you can replicate the trigger on. First, lets = recreate=20 the incident for the first trigger, unauthorized use of our system as a = mail=20 relay . From an un-authorized IP address, attempt to use your mailserver = as a=20 relay. With /usr/bin/tail –f you see the log entry in = /var/log/syslog (Refer to = Figure=20 A).=20

There we see the error message of someone at moo.com attempting to = relay=20 email, potentially a sign of spam. This is the trigger for unauthorized = mail=20 relay. Notice how the error also includes the IP address, verifying the = domain.=20

Now, lets recreate the second trigger, unauthorized use of the = expn =20 command. Telnet to the SMTP port and execute expn. Meanwhile monitor the = /var/log/syslog with tail –f (Refer to = Figure=20 B).=20

There we see the error message of someone at moo.com attempting to = expand the=20 username root. This is the trigger if anyone attempts to exploit the = "expn"=20 command. Notice how the error also includes the IP address, verifying = the=20 domain.=20

We have now completed the three steps in planning for automated log=20 filtering. We first identified the information important to us, = unauthorized=20 attempts to use our mail server as a mail relay and use of the expn=20 command.  We then identified the logs that contain this = information,=20 /var/log/syslog. Last, we identified the triggers for this information = by=20 recreating the incidents. We are now ready to build our automated = filter.=20
 =20

SWATCH
SWATCH, "The Simple WATCHer and = filter", is=20 a perl program developed by Todd Atkins that monitors your logs in real = time.=20 Swatch monitors your logs for specific triggers, when those triggers are = matched=20 swatch notifies you in a pre-determined manner. In our case, we are = going to=20 implement swatch to alert us whenever someone is messing with our = sendmail.=20

The program is extremely simple to install.  Swatch comes with a = useful=20 install script that will copy all the libraries, man pages, and perl = files to=20 their respective directories. You might need to compile and install = several perl=20 modules, the installation script will let you know. Once done = installing, all=20 that is left is creating a configuration file and then launching the = program.=20 You can download swatch here.=20

The configuration file, called swatchrc, is the heart of the swatch = program.=20 This text file tells swatch what logs to monitor, what triggers to look = for, and=20 what to do if triggered. Swatch works by looking for regular expressions = that=20 match the triggers defined in swatchrc. When it finds a match, it = executes the=20 notification procedure defined in swatchrc. Swatch monitors the files in = real=20 time, using /usr/bin/tail –f.=20

We will now create a swatchrc file for our sendmail logging we = discussed=20 above. The goal is to have sendmail email us whenever someone is messing = with=20 our email system. We defined this earlier as anyone attempting = unauthorized mail=20 relay or the expn command. The syntax of a swatchrc file is as follows. = It=20 consists of four fields, the first two fields are required, the last two = fields=20 are optional. The first field is=20

/pattern/pattern/=20

where pattern is a regular expression that swatch is looking for. = This is our=20 trigger. The second field is=20

Action,action…=20

where action is what to do if the pattern is matched. Swatch has = various=20 options for actions, including email, paging, or executing any file you=20 select.  The third field 'throttle' (which is optional) is a time = interval=20 defined as=20

HH:MM:SS=20

HH is for hours, MM for minutes, and SS for seconds. This time = interval is=20 the amount of time swatch will ignore identical matched patterns that = repeat=20 themselves. For example, if you define this period as 5 minutes, swatch = will=20 only report one identical matched pattern over that time period, even = though it=20 might have matched 20 identical entries.=20

The fourth field (required if you are using the third field) is a = timestamp,=20 defined as
start:length. This defines the location and length of the = timestamp in the notification message.=20

 For our sendmail example, we want to create a swatchrc file = that looks=20 for patterns matching our two triggers (See Figure = A and Figure = B). When=20 it matches either of these patterns, we want it to notify via email=20 abuse@ourcompany.com and to include the matched pattern in the email. = However,=20 we have to be careful not to be flooded with warnings. For example, if = someone=20 attempts to relay off us with 1000 emails a minute, we would be = overwhelmed with=20 notifications. So, we will set a time interval of 5 minutes. Regardless = of how=20 many identical patterns are matched in a five minute period, we will = receive=20 only one warning. Our swatchrc file would look as follows:
 =20

watchfor /Relaying = denied|expn/
        =20 echo=3Dnormal
        =20 mail=3Dabuse@ourcompany.net,subject=3D--- Sendmail Alert! = ---=20
         throttle 5:00 = 0:16=20

 The first field has "/Relaying denied|expn/". If swatch matches = either pattern=20 in the regular expression, it will send an alert. The first pattern = "Relaying=20 denied" is found in trigger #1 (Figure = A), this=20 log is the result of someone attempting an unauthorized mail relay. The = pattern=20 "expn" is found in trigger #2 (Figure = B), it is=20 the result of someone attempting to use the expn command. You will find = both=20 expressions in the triggers we covered in the first part of the article. =

The second filed has "echo=3Dnormal,mail=3Dabuse@company.net"
This = field states=20 email a warning to abuse@ourcompany.net, and echo the matched log entry. =

The third and fourth field (which are optional), have "5:00 0:16". = This states=20 do not repeat any warning for identical patterns matched within 5 = minutes. The=20 last field states the location and length of the timestamp.=20

We now have a properly configured swatchrc file. The last step is = starting=20 swatch itself. Swatch can be launched with a variety of options. = However, we=20 will launch with the following syntax.=20

/usr/local/bin/swatch -c=20 /var/log/syslogrc -t /var/log/syslog &=20

The –c option points to the configuration file, and the = –t option monitors=20 the log file in realtime. The "&" runs the swatch in the background. = Once=20 launched, swatch forks a child, so swatch will be running as two = processes. Be=20 sure to kill both processes in any stop/start scripts you create. = That’s it.=20 Your sendmail logs will be automatically filtered. Whenever someone = messes with=20 your sendmail system, you will be instantly notified via email, with the = matched=20 trigger in the log included (See Figures = A and Figure = B).=20

Example = swatchrc=20 config file

Conclusion
Logs are powerful tools, yet = they can=20 easily overwhelm us with data. When this happens, we start ignoring this = valuable source because we don’t have time to scan through megs of = data.=20 Automating the filtering of such logs solves the problem. These = automated=20 filters do the work for us, alerting us in real time with the = information we=20 need. Hopefully this article has given you ideas on how to automate the=20 filtering of your log files.=20

Figure=20 A
Trigger for=20 anyone attempting un-authorized mail relay from your sendmail=20 server.
Oct 3=20 14:48:51 homer sendmail[6704]: OAA06704:=20 ruleset=3Dcheck_rcpt,arg1=3Dbsmith@domain.com, relay=3Dfoo@moo.com=20 [206.54.252.1],reject=3D550 root@domain.com... Relaying = denied=20
 =20

Figure=20 B
Trigger for=20 anyone attempting to utilize the expn command on your sendmail=20 server.
Oct 2=20 20:28:37 homer sendmail[5453]: NOQUEUE: foo@moo.com[206.54.252.1]: expn = root=20 [rejected]
 =20

Author’s=20 bio
Lance Spitzner enjoys learning by = blowing up=20 his Unix systems at home. Before this, he was an Officer in the = Rapid=20 Deployment Force, where he blew up things of a different nature. You = can=20 reach him at lance@honeynet.org .=20
 
 =20

Whitepapers /=20 = Publications

 =20