Well, that was a bit of a hiatus. On the plus side, I've learned a tremendous amount from over a year of actually operating the system I created. More on that much later.
One thing to note, if for some insane reason you're following along while building your own system: don't use IET - it's obsolete. Use the native Linux SCSI Target, which is much more reliable and flexible. I'll continue to transcribe my notes, adding caveats as I go.
Create the data array
Create the array with mdadm. The following command creates a RAID-5 array with 7 disks and one spare; the disks are named sdc through sdj. Note that this is a single-line command.
mdadm --create /dev/md1 -n 7 -x 1 -l raid5 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
Get the UUID of the array from any of the member disks:
mdadm -E /dev/sdc
Do not reboot the computer until the array finishes initializing.
An array created with mdadm will not automatically be available when the computer reboots. You need to describe the array in /etc/mdadm.conf. The following sample file describes two arrays: the boot array and the data array. Because device names may change, the configuration file tells mdadm to scan all available partitions and disks for the components of each array.
DEVICE partitions
MAILADDR admin-email
MAILFROM host-email
ARRAY /dev/md0 level=raid1 num-devices=2 uuid=boot-array-uuid
ARRAY /dev/md1 level=raid5 num-devices=7 spares=1 uuid=data-array-uuid
Install and configure iSCSI Enterprise Target
Download and install the iSCSI Enterprise Target source code. As of this writing the latest version was 1.4.20.2.
yum -y install kernel-devel openssl-devel gcc rpm-build
cd /usr/src
wget http://downloads.sourceforge.net/project/iscsitarget/iscsitarget/1.4.20.2/iscsitarget-1.4.20.2.tar.gz
tar xzvf iscsitarget-1.4.20.2.tar.gz
cd iscsitarget-1.4.20.2
make
make install
Replace the contents of /etc/iet/ietd.conf:
# Discovery authentication
IncomingUser ini-user ini-password
# List of iSCSI targets
Target iscsi-qualified-name
Lun 0 path=/dev/drbd0,Type=fileio
IncomingUser ini-user ini-password
Run the following command to protect the contents of the file:
chmod 640 /etc/iet/ietd.conf
Install and configure DRBD
The package repository for CentOS 5.6 offers a very old version of DRBD, version 8.0.16. DRBD version 8.3.10 comes with features that make remote replication much easier.
Download and compile the 8.3.10 or later source code:
yum -y install flex
cd /usr/src
wget http://oss.linbit.com/drbd/8.3/drbd-8.3.10.tar.gz
tar xzvf drbd-8.3.10.tar.gz
cd drbd-8.3.10
./configure --with-km --without-heartbeat --without-pacemaker --without-xen --sysconfdir=/etc
make
make install
During the make you will see a lot of scary compiler warnings. Complain to the developers.
To verify that the driver loads, type:
modprobe drbd
cat /proc/drbd
You should see version and build information.
Fix driver load order
iSCSI Enterprise Target needs to load after DRBD to ensure it properly detects the DRBD volume.
Enter the following commands to alter the load order of DRBD and iSCSI:
chkconfig --del iscsi-target
cd /etc
ln -s ../init.d/drbd rc0.d/K52drbd
ln -s ../init.d/drbd rc1.d/K52drbd
ln -s ../init.d/drbd rc2.d/S48drbd
ln -s ../init.d/drbd rc3.d/S48drbd
ln -s ../init.d/drbd rc4.d/S48drbd
ln -s ../init.d/drbd rc5.d/S48drbd
ln -s ../init.d/drbd rc6.d/K52drbd
ln -s ../init.d/iscsi-target rc0.d/K51iscsi-target
ln -s ../init.d/iscsi-target rc1.d/K51iscsi-target
ln -s ../init.d/iscsi-target rc2.d/S49iscsi-target
ln -s ../init.d/iscsi-target rc3.d/S49iscsi-target
ln -s ../init.d/iscsi-target rc4.d/S49iscsi-target
ln -s ../init.d/iscsi-target rc5.d/S49iscsi-target
ln -s ../init.d/iscsi-target rc6.d/K51iscsi-target
Create the replication block device
Replace the configuration file /etc/drbd.conf. Set the sync rate to a setting that will not overwhelm either end of the Internet connection. The following configuration file is for a DRBD setup without DRBD Proxy:
global {
usage-count no;
}
resource r0 {
protocol A;
device /dev/drbd0;
disk /dev/md1;
meta-disk internal;
syncer {
rate 4M;
csums-alg sha1;
}
net {
data-integrity-alg sha1;
on-congestion pull-ahead;
congestion-fill 1M;
}
startup {
become-primary-on local-fqdn;
wfc-timeout 60;
degr-wfc-timeout 60;
outdated-wfc-timeout 60;
}
on local-fqdn {
address local-ip:7789;
}
on remote-fqdn {
address remote-temp-ip:7789;
}
}
If you are using DRBD proxy, use the following setup:
global {
usage-count no;
}
resource r0 {
protocol A;
device /dev/drbd0;
disk /dev/md1;
meta-disk internal;
proxy {
compression on;
memlimit 1024M;
}
syncer {
rate 4M;
csums-alg sha1;
}
net {
data-integrity-alg sha1;
on-congestion pull-ahead;
congestion-fill 5M;
}
startup {
become-primary-on local-fqdn;
wfc-timeout 60;
degr-wfc-timeout 60;
outdated-wfc-timeout 60;
}
on local-fqdn {
address 127.0.0.1:7789;
proxy on local-fqdn {
inside 127.0.0.1:7788;
outside local-ip:7788;
}
}
on remote-fqdn {
address 127.0.0.1:7789;
proxy on remote-fqdn {
inside 127.0.0.1:7788;
outside remote-temp-ip:7788;
}
}
}
Create the local copy of the replication volume with the following command:
drbdadm create-md r0
You should see initialization messages followed by the message “success”.
Load the DRBD kernel module and start the replication device:
drbdadm attach r0
drbdadm syncer r0
drbdadm connect r0
At this point, each SAN has its own independent copy of the replication device.
Reboot the device.
Things get interesting...
Up to this point, these instructions apply equally to both devices. From now on, the servers are configured differently. The local device needs to be configured with a primary (active) DRBD volume with iSCSI on top. The remote device needs to be configured with a secondary (passive) DRBD volume, and iSCSI should remain off.
Thursday, August 8, 2013
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment