In part 2, I laid out the hardware for my SAN and recommended that no one follow my example. In this part, I want to talk about preparing the operating system. I lied about covering iSCSI Enterprise Target and DRBD.
Since we need to differentiate between the two SANs, I will use “local device” to refer to the SAN that will go on the local network and “remote device” to refer to the remote replication target.
Installing the base system mostly consists of stripping out things that you don’t need and disabling what you can’t get rid of. I’m a big fan of turning off unneeded services as a way to increase security; I just wish that CentOS felt the same way.
I wrote these instructions as part of formal documentation for the poor guy (or girl) who comes after me who has to maintain this system and fix it when it breaks. Despite my efforts, the documentation was out of date by the time the system went live. I’ve included some notes in italics where I don’t have exact instructions since I changed the configuration on the fly.
Finally, an apology about this post's formatting. I wish the default Blogger.com tools were a bit better; as it is, everything but bold and italic markups disappear when I publish the post.
Collect settings
There are a handful of settings you’ll want to know before you install the software. Here is the list I came up with:
local-fqdn The fully-qualified domain name of the local device. This is used by DRBD to decide what part of its configuration file applies to which computer. This does not necessarily need to match the domain name of the local SAN as resolved by DNS.
local-ip The IP address of the local SAN within the office network. This is probably a private IP address.
remote-temp-ip The temporary IP address of the remote SAN while it is being built and tested within the office network.
local-gateway The gateway address of the office network.
local-subnet The IP subnet of the office network.
local-public-ip The public IP address of the office gateway.
remote-fqdn The fully-qualified domain name of the remote SAN. This is used by DRBD to decide what part of its configuration file applies to which computer. This does not necessarily need to match the domain name of the remote SAN as resolved by DNS.
remote-ip The IP address of the remote SAN. This is a public IP address
remote-gateway The gateway address used by the remote SAN.
remote-subnet The IP subnet to which the remote device belongs.
remote-netmask The IP subnet mask of the office network in CIDR notation.
third-party-ip A public IP address of a third party that will manage the remote SAN.
ini-user The user name given by the office server that will use the iSCSI volume.
ini-password The password given by the office server that will use the iSCSI volume. Due to various operating system restrictions, this should be exactly 12 characters long.
admin-email The email address of the SAN administrator, for event notifications
host-email The email address of each SAN, which may not necessarily correspond to a real mailbox
mail-server The name of your email server or mail exchanger
Install the operating system
The CentOS installer can use either text or graphics mode. The graphics mode installer may have problems with some Intel onboard video controllers. These instructions follow a text-mode installation, and use an FTP site as the install source. You can use arrow keys to move among the fields and options in the text-mode interface. Press the spacebar to select check boxes and radio buttons.
Boot to the CentOS 5.6 install disc. At the boot prompt, enter
linux text
Choose your language and keyboard layout. In the installation method dialog, choose FTP. Under Enable IPv4 support, select Manual Configuration and deselect Enable IPv6 support. Enter the local IP address if this is the local SAN, or the temporary remote IP address if this is the remote SAN. Enter the local netmask, gateway, and DNS server. Enter the name of a server that mirrors the CentOS distribution and the FTP directory of the installation files, which should end with /os/i386 or /os/x86_64.
I can't think of a good reason to use a 32-bit operating system, especially if you are using new hardware.
Select text mode instead of VNC.
If this is the first time the hard disks have been used, the installer will prompt to initialize each empty disk. Select Yes for each disk on which you will install the operating system. Select No for the data disks.
Choose Create custom layout.
These instructions configure the device to use the swap volume on each OS drive as a separate volume. This is not very bright, since it meant that the OS could crash if one of the OS drives drops out. The right way to do this is to configure the physical swap partitions as software RAID volumes, configure a mirror on top of them, and format that mirror as a swap volume.
Select New. Change the file system type to swap. For allowable drives, choose just the first operating system drive. Set the size to half the size of physical RAM. Select OK. Select New. Change the file system type to software RAID. For allowable drives, choose just the first operating system drive. Select Fill all available space. Select OK.
Create the same two partitions on the second operating system drive.
Select RAID. Set the mount point to /. Leave the file system type as ext3. Set the RAID level to RAID1. Select OK.
Accept the defaults in each Boot Loader Configuration dialog. When prompted for the location of the boot loader, choose the first operating system hard drive (probably /dev/sda) rather than the RAID volume (/dev/md0).
Choose to configure network interface eth0. The default settings should be to activate the interface on boot and enable IPv4. Verify the IP, netmask, gateway and DNS addresses. In the Hostname Configuration dialog, enter the fully qualified domain name of the system.
In Time Zone Selection, select the correct time zone or geographical area. Most likely the system does not use UTC, so deselect that option.
Enter the root password. For ease of maintenance, this should be the same for both devices.
In the Package selection dialog, deselect each package.
Keep choosing Next until the installer starts. The installer should format the root volume and install the operating system. When the installer reports that the installation is complete, click Reboot. Remove the installer CD and wait for the system to reboot.
Disable SELinux
On the first boot, CentOS will display a Setup Agent dialog. Choose Firewall configuration. Under SELinux, choose Disabled. The Setup Agent will disappear if you don’t touch any keys after a couple of minutes - you can get it back by typing:
firstboot
Remove unneeded packages and update the system
At this point, you can remotely connect to the system with SSH. Log in as root. At the shell prompt, enter the following to remove unneeded packages:
yum -y erase fetchmail NetworkManager bluez* ccid desktop-file-utils
yum -y erase dnsmasq ifd-egate irda-utils isdn4k-utils mutt pcmciautils
yum -y erase slrn talk wpa_supplicant yp*
To update the system, enter the following:
yum -y update
reboot
Disable unneeded services
Disable unneeded services, including gpm, netfs, nfslock, pcscd, portmap, rpcgssd, rpcidmapd, and rpcsvcgssd. Use the following one-line command:
for f in gpm netfs nfslock portmap rpcgssd rpcidmapd rpcsvcgssd; do chkconfig --del $f; service $f stop; done
Configure the firewall
Replace the firewall configuration in /etc/sysconfig/iptables with the following configuration. Fill in the IP addresses for the local and remote private networks and the local and remote public IP addresses.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:AllowedServices - [0:0]
:AllowedHosts - [0:0]
# Always allow traffic on loopback, filter all other incoming
-A INPUT -i lo -j ACCEPT
-A INPUT -j AllowedServices
-A FORWARD -j AllowedServices
## List of services to allow
# Existing connections
-A AllowedServices -m state --state ESTABLISHED,RELATED -j ACCEPT
# NTP
-A AllowedServices -p udp -m udp --dport 123 -j AllowedHosts
# SSH, iSCSI, DRBD Proxy, DRBD
-A AllowedServices -m state --state NEW -p tcp -m multiport --dports 22,3260,7788,7789 -j AllowedHosts
# ICMP
-A AllowedServices -p icmp --icmp-type any -j AllowedHosts
# Reject everything else
-A AllowedServices -j DROP
## List of hosts to allow
# The local private network
-A AllowedHosts -s local-subnet/local-netmask -j ACCEPT
# The local device's public IP
-A AllowedHosts -s local-public-ip -j ACCEPT
# The remote device’s IP
-A AllowedHosts -s remote-ip -j ACCEPT
# Optional: allow some third party to administer the devices
-A AllowedHosts -s third-party-ip -j ACCEPT
# Reject everyone else
-A AllowedHosts -j DROP
COMMIT
Note that this firewall works for both the local and remote SANs. This makes recovery easier if you have to bring the remote SAN into the local network.
Install NTP
It is not critical to synchronize time on the NTP server, but it is useful. Run the following commands to install and configure NTP:
yum -y install ntp
chkconfig ntpd on
service ntpd start
Configure Logwatch
Install the sendmail configuration compiler.
yum -y install sendmail-cf
Edit /etc/mail/sendmail.mc. Find the line
dnl define(`SMART_HOST', `smtp.your.provider')dnl
Replace it with:
define(`SMART_HOST', `mail-server')
Note that the dnl phrases are removed from the beginning and ending of the line, effectively uncommenting it. Save the file and run the following commands to compile the configuration and enable the sendmail service:
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf
chkconfig --level 235 sendmail on
service sendmail start
Edit /etc/logwatch/conf/logwatch.conf and add the following lines:
MailTo=admin-email
MailFrom=host-email
Limited possibilities
You know how every high school valedictorian speech includes a phrase along the lines of “unlimited possibilities?” I have found that the best way to get things done is to limit the possibilities to exactly what you need. If you are designing a SAN, forget about also making it a NAS, and a web server, and a French fry chopper. And if you’re just going to use it for backup, don’t bother with LVM. Anything beyond what you actually need is a distraction.
In the next part, I promise to get to iSCSI Enterprise Target and DRBD.
