The goal of this document is to present a succinct overview of Linux security related subjects, tips & tricks that can help an aspiring sysadmin to create Linux workstations that are as secure as the state of the art allows.

This HOWTO does not go into depth. Where possible, instead of spelling out everything it contains pointers to further information. Such, it relies on the natural intelligence of the implementor (the local sysadmin). More comprehensive documents are referenced below.

This document is not portable to different operating systems, it assumes Linux. Partly it even assumes Redhat of the same version on each computer in the cluster. It assumes fast net connection between cluster members. It assumes cluster members trust each other. It partly assumes NFS: at least one central shared partition (may be read-only).

Requirements: this documents assumes a sysadmin who has already taken the first hurdles: knows how to read and interpret the output of netstat and ps, edit a file, install packages etc. A great introductory text is the Security-Quickstart HOWTO, which comes in a general and a RedHat-specific version. See below for the reference.

Quicklinks: Generic | Tools | Apache | SPAM | Windows | Firewalls | Cron | Organisations | Articles | Misc

Objectives of good sysadmin work

• Generate secure systems.
• Only little maintenance after installation: cron scripts.
• Automatic configuration.
• Only configure and upgrade one host. The others get their upgrade and configuration from a central repository.
• The reporting and loging system generates little noise: only report real suspicious events. Otherwise the admin will start ignoring it.
• As much free/open-source software as possible
• Inform and educate users.
• General configuration goals:
  • divulge as little as possible version + software info to the outside
  • keep abreast of security updates
  • good and easy sysadmin work makes for more security
  • extensive reporting and analysis: if the human immune systems spends a big amount of body resources, why can't the computer immune system?

Generic

• only install essential network tools
• constantly update and secure individual tools
• have a tight file permission system
• strong password requirements
• firewalls

The 2 most important layers are not covered in this document: physical and social security. If somebody can enter your facility and tack an axe to your server, you just suffered a physical DOS (denial-of-service) attack. And if your users are willing to give their password to the "technician from QWest" on the telephone, you are susceptible to social attacks.

nmap needs to be installed and run against each computer to make sure no unneeded ports are open. Use netstat or lsof to see what programs are behind each of the open ports and determine if they are really necessary. Run nmap explicitely for each port (1-65536)

Unnecessary services include: swat, ident, auth, linuxconf-web. Remove the packages and disable the services in /etc/xinetd.d

No telnet, rlogin, rsh, rexec. Ever.

Only anonymous outgoing FTP. As little uploading as possible.

Enable passwordless ssh connections so users find it convenient

NFS is good, though sometimes locking issues need to be worked out. Keep updating kernels to get NFS fixes.

Passwords should by shadow and MD5.

Syslog: have central syslog server that writes directly to secure storage

In especially vulnerable systems, see if you can mount things read-only, so even if they are broken into the attacker can't destroy anything. This can be done if the partition we want to secure is NFS mounted and the server's /etc/exports does not allow read-write access

Automatic and manual backup systems - hd, tape, CDs. DVDs once DVD-writer standards stabilize.

Run all daemons as their own user and group, NOT as root.

Buy a service contract from redhat (RHN) and use it. up2date is invaluable. 60$/year is a steal.

If your ftp server is wu-ftpd, in /etc/ftpaccess, set stat terse. This disables ftpd software + version information to the outside. But a better ftp server is vsftpd

The Xserver option -nolisten tcp disables listening on port 6000. If you do want to access X over the net you still can, as long as you are using ssh forwarding. This is automatic if you are loged into the remote host with openssh.

In the mysql configuration file /etc/my.cnf, you can set the option skip-networking in the [mysqld] section if all database access is only from the local machine. This disables mysql listening on port 3306.

Tools

http://www.insecure.org/tools.html
contains other security tools worth trying out.

nessus
Run nessus each month. Test each port explicitely. Use all strong options. Better kill your server then don't know about holes.

snort
Parses network traffic for suspicious patterns. Highly configurable. You need to configure it so it does not generate too many superfluous warning messages which would drown out the real emergencies. If you get portscanned every 5 minutes you do not want to know about each instance.

ACID
Postprocesses snort's output to put it into a nicer form, with webinterface and alert management.

logcheck
Is an example of tools that sift through the syslog stream and search for uncommon messages. Such a tool should be installed. It needs to be configured to only output truly uncommon messages, so the sysadmin does not get into the habit of ignoring it.

GPG
Encrypt communications absolutely secure.

Openwall kernel patches
Secure the Linux kernel further, for instance by making the stack unexecutable, thwarting buffer overflow exploits.

tripwire
Once initialized with a known good, unhacked, unbreached state, this tools alerts the sysadmin to any changes to this state.

Pro:

• Very secure and comprehensive

Con:

• If the system changes regularly, re-initialisation must be performed often, or many false positives will be encountered. In other words: each time anything on the system changes (files, perms, access times) reinitialisation is necessary. This can quickly become tedious.
• Due to security, passwords must be entered interactively. This means tripwire can't be scripted and automated. Especially bad if you try to automate administration of many hosts.

Run RedHat's up2date tool at least once a week.

ethereal
Shows you what's really going on in your network. Have a user run a telnet connection and show him how the whole session (including password, pine email-reading etc) is transmitted in the clear. Learn how TCP/IP work. Learn how DNS, ARP and many more protocols work. See the packets fly by.

chkrootkit
Simple scripts that check the system for signs of installed root kits. Very basic but easy to install and use and better than nothing.

Center for Internet Security
Distributes a tool to benchmark a workstations security

John the Ripper
A password cracker/strength checker. Run every few month to verify your passwords can not easily be cracked.

Apache

Read up on CGI security and audit all CGI scripts on your web server. Perl's taint mode is your friend. Secure scripts are at http://nms-cgi.sourceforge.net/

Go through logs regularly and look for

• People who abuse the service by taking too much bandwidth - block them
• Errors - fix them
• Missing files people expect - create them or redirect somewhere else
  

      Redirect permanent /favicon.ico http://microsoft.com/junk/your/favicon/stuff.ico
      

• Learn from attack attempt (i.e. people trying to access formmail.pl or some such)

Run apache as user apache, group apache, not user root or user nobody.

Provide as little info as possible to clients:

ServerSignature off
ServerTokens Prod

SPAM

spamassassin-milter
Binds spamassassin together with sendmail, so mails can be filtered for a whole site or domain.

Pertinent parts of the sendmail config file /etc/mail/sendmail.mc

Windows

Never leave a vanilla Windows computer on an unprotected connection to the internet. See the Worms under Windows article.

Even if protected, Windows computer can be broken into through e-mail viruses or IE exploits. No administrator action can help there.

Choose one of these solutions:

• Run Windows only for short periods of time
• Insulate the rest of your cluster from your Windows computers and protect them through a firewall
• Windows computers have no connection to the net

putty
An excellent ssh client for Windows. Delete telnet.exe from the Windows users computers and install putty instead. Don't forget to set the defaults of putty to use SSH connections, not telnet.

A list of ssh and scp clients and servers for Windows

Windows computers should only access shared discs read-only

Windows computers should not export services to non-Windows computers. No SQL, mail- or web servers under Windows!

The applications with the biggest security risks seem to be IE and Outlook/Outlook Express. Try to move your users over to Mozilla and Pegasus mail/ Eudora mail.

Educate the users. Tell them (or force, if need be) to switch off scriptability.

Give them options. Make it easy for them to move to Unix: install X Servers on the Windows computers so they can begin running unix applications on them. Install Linux software that replicates Windows functionality. I.e. openoffice, mozilla, kmail etc.

Firewalls

The linux iptables firewall configuration language and toolset is very comfortable and understandable, worth learning. The older ipchains language and tool is less developed and comfortable. It should be avoided.

The homepage to the iptables (netfilter) system
Contains a wealth of information, tutorials and FAQs. Recommanded are the packet-filtering HOWTO and Oscar Andreasson's iptables tutorial.

The NFS-HOWTO
Contains information on securing NFS and running NFS through a firewall.

Tools for configuring a firewall under Linux:

• lokkit
  Comes standard with RedHat. Very basic tool, only for starters.
• firestarter
  Nice wizard interface to create firewall, NAT and TOS rules. Creates a very nice bash-shell script that can be further customized by hand - great way to learn by example.

Usefull scripts to be run by cron

cron.weekly: rpm --rebuilddb

cron.daily: 

• up2date -p
  Tells RHN your present software base, so alerts are correct.
• cd $REPOSITORY && rpm --freshen *.rpm
  Keep an RPM repository and daily, update all your RPMs. This way, the repository could be NFS mounted and all hosts in your cluster upgrade themselves automatically.
• /usr/bin/updatedb -f "proc,devpts" -e "/tmp,/var/tmp,/usr/tmp,/afs,/net"
  Tell slocate to include NFS in its index, so people see their home directories. Just take the stock /etc/cron.daily/slocate.cron and edit it.

/etc/cron.hourly: 

• Have each host ping each other. Maybe attempt an ssh login. This way, you will be notified if any host becomes unresponsive.
• Run df -h and check that no harddisc partition is above 90% usage. This way you will get notified if any runaway processes (or users) begin filling up the disc.
• Check that loadavg is below 10, otherwise warn:

       cat /proc/loadavg |cut -f1 -d' '|perl -n -e'print "loadavg=$_" if $_>10'
     

• Check that memory usage is below 80%, otherwise warn:

       free -t|tail -1|perl -n -e'@a=split;$u=$a[2]/$a[1];print "used mem=$u" if $u>0.79'
     

• Check for defunct processes, print those and their parent and cousin processes: here
• Check some critical file permissions, for example in /dev
• Run ldconfig -v and look for error messages
• Check that the rpm package list is sane: no duplicates; no packages where we have two different versions installed (except kernel)
• Run a daemon monitoring service that checks that all necessary daemons are running: lockd, rpc.statd, portmap, syslogd, klogd, sendmail, etc

Organisations

Linux security community site

CERT Advisory mailing list - subscribe!

Bugtraq mailing list - subscribe!

Learn about tools, tactics and motives of the blackhat community

Articels / Documents

Worms under Windows. Shines a light at Windows' bad security

Overview of admining a redhat 9 system. Worth checking out.

The SSH FAQ. You should know everything about SSH.

Linux-Security HOWTO

Security-Quickstart HOWTO

Additional software and references that you might want to check out

Tagged Message Delivery Agent
Is a very powerfull but also quite complicated mail filtering solution.

MonMotha's Firewall
A comprehensive firewall creation script written in bash. Good example.

IPCop
A Linux distribution for dedicated firewalls

Counterpane Internet Security, Inc
Bruce Schneier's "CryptoGram" missives are worth reading and subscribing to

DEF CON: largest security conference

Security tools, papers, magazines
Example: http://packetstorm.linuxsecurity.com/groups/thc/

Information Systems Security Association
Local chapters with workshops and lectures

Home Network Security

The Honeynet project explained

Secure Programming for Linux and Unix HOWTO

SSH tunneling explained on the example of tunneling WiFi

This overview was written by Hans Ecke using experience gathered administrating a cluster of Linux workstations at the Colorado School of Mines 2000-2003.