Apache+MySQL+DRBD-8.0.6 (Another setup)
OS : Centos 5 , Kernel 2.6.18-53.1.4.el5
Notes : DRBD is of newer version in this setup. A little sharp edged! Be careful!
This is a partial extract from a setup I did recently. (now a year ago, when I am uploading it).
DB Servers:
hostname : db1.example.com
Public IP = 10.1.1.130
Private IP = 192.168.1.130
hostname : db2.example.com
Public IP = 10.1.1.131
Private IP = 192.168.1.131
Webmail Servers:
Public IP: 10.1.1.132
Private IP = 192.168.1.132
Public IP: 10.1.1.133
Private IP = 192.168.1.133
.134 = DB VIP
.135 = WEB VIP
The DB servers are the DNS as well.
On both servers:
yum install bind bind-chroot caching-nameserver
cp /etc/named.conf /var/named/chroot/etc
rm -f /etc/named.conf
ln -s /var/named/chroot/etc/named.conf /etc/named.conf
vi /etc/named.conf
. . .
. . .
view "external"
{
/* This view will contain zones you want to serve only to "external" clients
* that have addresses that are not on your directly attached LAN interface subnets:
*/
match-clients { !localnets; !localhost; };
match-destinations { !localnets; !localhost; };
recursion no;
you’d probably want to deny recursion to external clients, so you don’t
end up providing free DNS service to all takers
all views must contain the root hints zone:
include “/etc/named.root.hints”;
These are your “authoritative” external zones, and would probably
contain entries for just your web and mail servers:
zone "my.external.zone" {
type master;
file "my.external.zone.db";
};
zone "example.com" {
type master;
file "example.com.zone";
};
};
vi /var/named/chroot/var/named/example.com.zone
$ORIGIN example.com.
$TTL 3600
@ IN SOA ns1.example.com. kamran.wbitt.com. (
20071214 ; serial (d. adams)
3H ; refresh
15M ; retry
1W ; expiry
1D ) ; minimum
@ IN NS ns1.example.com.
@ IN NS ns2.example.com.
@ IN MX 10 mail1.example.com.
@ IN MX 20 mail2.example.com.
ns1.example.com. IN A 10.1.1.130
ns2.example.com. IN A 10.1.1.131
webmail1.example.com. IN A 10.1.1.132
webmail2.example.com. IN A 10.1.1.133
db1.example.com. IN A 10.1.1.130
db2.example.com. IN A 10.1.1.131
mail1.example.com. IN CNAME webmail1.example.com.
mail2.example.com. IN CNAME webmail2.example.com.
Disable SELINUX on all 4 servers
chkconfig --level 35 named on
chkconfig --level 35 sendmail off
chkconfig --level 35 cups off
chkconfig --level 35 pcmcia off
[root@db1 ~]# service named start
Starting named:
Error in named configuration:
/etc/named.conf:58: open: /etc/named.root.hints: file not found
[FAILED]
[root@db1 ~]#
!OHHH!
[root@db1 ~]# ls -l /var/named/chroot/var/named/named.ca
-rw-r----- 1 root named 2518 Nov 10 07:22 /var/named/chroot/var/named/named.ca
[root@db1 ~]#
Now either I can copy the file found above, as in /var/named/chroot/etc/named.root.hints file.
And create it’s symbolic link in /etc/
Or:
I can simply query the latest name servers and put them directly in the file /var/named/chroot/etc/named.root.hints and create a symbolic link in /etc.
dig ns . > /var/named/chroot/etc/named.root.hints
ln -s /var/named/chroot/etc/named.root.hints /etc/
[root@db1 ~]# service named restart
Stopping named: [FAILED]
Starting named:
Error in named configuration:
/etc/named.root.hints:2: '}' expected near ';'
[FAILED]
[root@db1 ~]#
I think there is a bug in /etc/named.conf. The # should not be used for a comment . rather a
all views must contain the root hints zone:
include “/etc/named.root.hints”;
But even correcting that gives this error.
[root@db1 ~]# service named restart
Stopping named: [FAILED]
Starting named:
Error in named configuration:
/etc/named.root.hints:2: '}' expected near ';'
[FAILED]
[root@db1 ~]#
OK I removed the following two lines from the top of hints file:
; DiG 9.3.3rc2 ns .
Still the same problem:
[root@db1 ~]# service named restart
Stopping named: [FAILED]
Starting named:
Error in named configuration:
/etc/named.root.hints:1: '}' expected near ';'
[FAILED]
[root@db1 ~]#
Problem is in /etc/named.conf, NOT root hints file.
Removed all views and placed a simple zone:
zone "example.com" {
type master;
file "example.com.zone";
};
OK New problem:
[root@db1 ~]# service named restart
Stopping named: [FAILED]
Starting named: [FAILED]
[root@db1 ~]#
Dec 13 22:46:09 db1 named[3511]: /etc/named.conf:46: configuring key 'ddns_key': bad base64 encoding
There is a section in /etc/named.conf
key ddns_key
{
algorithm hmac-md5;
secret "use /usr/sbin/dns-keygen to generate TSIG keys";
};
So ran the program :
[root@db1 ~]# /usr/sbin/dns-keygen
9lc0vhBNGHgFpZ7k7rH7KjgSEg7uUZCRt4PZZYpsLqctWnlZMd7WQdCFMBo7
[root@db1 ~]#
Now lets copy and paste this key in /etc/named.conf . The code looks like this now:
key ddns_key
{
algorithm hmac-md5;
secret "9lc0vhBNGHgFpZ7k7rH7KjgSEg7uUZCRt4PZZYpsLqctWnlZMd7WQdCFMBo7";
};
[root@db1 ~]# service named restart
Stopping named: [FAILED]
Starting named: [ OK ]
[root@db1 ~]#
Alhumdulillah.
The /etc/resolv.conf now looks like this:
nameserver 127.0.0.1
nameserver 10.2.2.160.16
nameserver 10.2.2.170.17
Zone transfers were not working. Turned out to be firewall problem. Turned off firewall on both DNS servers.
DNS done. Alhumdulillah. The traditional way.
DRBD setup:
yum -y install kernel kernel-devel kmod-drbd drbd
This will install newer kernel as well. Which will need a reboot.
The kernel on both servers is now:-
[root@db1 ~]# uname -r
2.6.18-53.1.4.el5
[root@db1 ~]#
[root@db2 ~]# uname -r
2.6.18-53.1.4.el5
[root@db2 ~]#
On both DB servers (Watch for the DRBD version):-
cp /usr/share/doc/drbd-8.0.6/drbd.conf /etc/
Then , on both DB servers:
vi /etc/drbd.conf
global { usage-count yes; }
common { syncer { rate 10M; } }
resource r0 {
protocol C;
net {
}
on db1.example.com {
device /dev/drbd0;
disk /dev/sda6;
address 192.168.1.130:7789;
flexible-meta-disk /dev/sda5;
}
on db2.example.com {
device /dev/drbd0;
disk /dev/sda6;
address 192.168.1.131:7789;
flexible-meta-disk /dev/sda5;
}
}
Then, on both DB servers:
modprobe drbd
This time modprobe was successful
echo "modprobe drbd" >> /etc/rc.local
Then on both DB servers:
drbdadm up all
. . .
Failure: (119) No valid meta-data signature found.
==> Use ‘drbdadm create-md res’ to initialize meta-data area. <==
Command ‘drbdsetup /dev/drbd0 disk /dev/sda6 /dev/sda5 flexible –set-defaults –create-device’ terminated with exit code 10
[root@db1 ~]#
OK let’s create md devices first on both DB servers.
drbdadm create-md all
[root@db1 ~]# drbdadm create-md all
v08 Magic number not found
v07 Magic number not found
About to create a new drbd meta data block
on /dev/sda5.
==> This might destroy existing data! <==
Do you want to proceed?
[need to type 'yes' to confirm] yes
Creating meta data...
initialising activity log
NOT initialized bitmap (1003995 KB)
New drbd meta data block sucessfully created.
success
[root@db1 ~]#
[root@db2 ~]# drbdadm create-md all
v08 Magic number not found
v07 Magic number not found
About to create a new drbd meta data block
on /dev/sda5.
==> This might destroy existing data! <==
Do you want to proceed?
[need to type 'yes' to confirm] yes
Creating meta data...
initialising activity log
NOT initialized bitmap (1003995 KB)
New drbd meta data block sucessfully created.
success
[root@db2 ~]#
Now, again on both servers, turn on the drbd devices:
drbdadm up all
Once this command is issued, both servers will have the status of Secondary/Secondary:
[root@db1 ~]# cat /proc/drbd
version: 8.0.6 (api:86/proto:86)
SVN Revision: 3048 build by buildsvn@c5-i386-build, 2007-12-08 01:17:51
0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent C r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
[root@db1 ~]#
[root@db2 ~]# cat /proc/drbd
version: 8.0.6 (api:86/proto:86)
SVN Revision: 3048 build by buildsvn@c5-i386-build, 2007-12-08 01:17:51
0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent C r---
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:0 misses:0 starving:0 dirty:0 changed:0
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
[root@db2 ~]#
You see that both DB servers say that they are secondary and that the data is inconsistant. This is because no initial sync has been made yet.
I want to make DB1 the primary DB/NFS server and DB2 the “hot-standby”, If DB1 fails, DB2 takes over, and if DB1 comes back then all data that has changed in the meantime is mirrored back from DB2 to DB1 so that data is always consistent.
Perfrom this step ONLY on DB1 (This will start the sync process and may take hours, depending on the IO speed of your setup):-
drbdadm -- --overwrite-data-of-peer primary all
Now you can keep checking the progress of both PCs, by monitoring the /proc/drbd file. Optionally you can watch for this file on both servers.
[root@db1 ~]# cat /proc/drbd
version: 8.0.6 (api:86/proto:86)
SVN Revision: 3048 build by buildsvn@c5-i386-build, 2007-12-08 01:17:51
0: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
ns:466944 nr:0 dw:0 dr:466944 al:0 bm:28 lo:0 pe:0 ua:0 ap:0
[>...................] sync'ed: 0.3% (194865/195321)M
finish: 4:48:41 speed: 11,460 (10,376) K/sec
resync: used:0/31 hits:29155 misses:29 starving:0 dirty:0 changed:29
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
[root@db1 ~]#
Started process on:
[root@db1 ~]# date
Fri Dec 14 02:46:16 PST 2007
[root@db1 ~]#
[root@db2 ~]# cat /proc/drbd
version: 8.0.6 (api:86/proto:86)
SVN Revision: 3048 build by buildsvn@c5-i386-build, 2007-12-08 01:17:51
0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
ns:0 nr:1799168 dw:1799168 dr:0 al:0 bm:109 lo:0 pe:0 ua:0 ap:0
[>...................] sync'ed: 1.0% (193564/195321)M
finish: 4:46:45 speed: 11,460 (10,280) K/sec
resync: used:0/31 hits:112338 misses:110 starving:0 dirty:0 changed:110
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
[root@db2 ~]#
After around 4 hours, I notice the status as:
[root@db1 ~]# cat /proc/drbd
version: 8.0.6 (api:86/proto:86)
SVN Revision: 3048 build by buildsvn@c5-i386-build, 2007-12-08 01:17:51
0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---
ns:200009218 nr:0 dw:0 dr:200009218 al:0 bm:12208 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:12488370 misses:12208 starving:0 dirty:0 changed:12208
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
[root@db1 ~]#
[root@db2 ~]# cat /proc/drbd
version: 8.0.6 (api:86/proto:86)
SVN Revision: 3048 build by buildsvn@c5-i386-build, 2007-12-08 01:17:51
0: cs:Connected st:Secondary/Primary ds:UpToDate/UpToDate C r---
ns:0 nr:200009218 dw:200009218 dr:0 al:0 bm:12208 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:12488370 misses:12208 starving:0 dirty:0 changed:12208
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
[root@db2 ~]#
I also see the status as “Connected” .
As per my understanding the following was only required in previous versions of DRBD.
# drbdadm -- connect all
It is not required in newer versions.
And if you issue this command in this state, you will get the following error:
[root@db1 ~]# drbdadm -- connect all
Failure: (125) Device has a net-config (use disconnect first)
Command 'drbdsetup /dev/drbd0 net 192.168.1.130:7789 192.168.1.131:7789 C --set-defaults --create-device' terminated with exit code 10
drbdsetup exited with code 10
[root@db1 ~]#
And if you are “too determined” to issue this command in any case, then you need to disconnect the devices first:
# drbdadm -- disconnect all
# drbdadm -- connect all
Time to format the DRBD device.
Only on DB1:
mke2fs -v -j /dev/drbd0
Then on both servers, create a mount point:
mkdir /data
Now mount /dev/drbd0 on /data ONLY on DB1. Make sure that you DO NOT mention the mounting in /etc/fstab as that will be controlled through heartbeat.
mount -t ext3 /dev/drbd0 /data # only on DB1
Verify by using “mount” and df -h on DB1.
[root@db1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 19G 1.3G 17G 7% /
tmpfs 1014M 0 1014M 0% /dev/shm
/dev/drbd0 188G 188M 179G 1% /data
[root@db1 ~]# mount
...
/dev/sda1 on / type ext3 (rw)
/dev/drbd0 on /
...
Alhumdulillah.
Question: Do I need to format the same device on DB2 too ????
Answer: No. The DRBD is handling it. And on slave node, which this DB2 is, it will be readonly anyway.
Time to create the directories in /data, only on DB1, where your drbd0 device is mounted currently.
mkdir -p /data/nfs # This will serve for correct functionality of DB server
mkdir -p /data/mysql # The /var/lilb/mysql will be linked here. Will be used for locally running MySQL server.
mkdir -p /data/home # This will be mounted on /home on host1 and host2.
mkdir -p /data/www # This will be mounted on /var/www on host1 and host2.
You will need to change ownership of /data/mysql directory as it will be used at /var/lib/mysql. The uid and gid of user/group mysql MUST be same on both DB servers.
You may need to make sure that mysql-server package is installed. You may need to install it before moving ahead.
yum install mysql-server
chown mysql:mysql /data/mysql
Setup NFS exports file on BOTH DB servers:
vi /etc/exports
put the ip addresses of the webmail servers here
/data/home 10.1.1.132(rw,no_root_squash) 10.1.1.133(rw,no_root_squash)
/data/www 10.1.1.132(rw,no_root_squash) 10.1.1.133(rw,no_root_squash)
Make sure that the NFS service is stopped on both NFS servers.
service nfs stop
Both servers:
cd /var/lib
mv nfs nfs.old
Now link the nfs from the /data/nfs to this place on both servers. This will create a valid link on DB1 and invalid link on DB2.
On both servers:
ln -s /data/nfs nfs
only on DB1:
cp -r nfs.old/* nfs/
If you start NFS service on DB1 only, now, then NFS will create new files in /var/lib/nfs -> /data/nfs for itself. Good.
But I am not going to start NFS now.
HEARTBEAT:
Lets install Heartbeat and IPVSADM , etc on both DB servers.
Lets install heartbeat and related packages on BOTH DB servers:
yum install curl perl-Crypt-SSLeay perl-HTML-Parser perl-LDAP perl-Net-DNS perl-libwww-perl libidn perl-Convert-ASN1 perl-Digest-HMAC perl-HTML-Tagset perl-URI perl-XML-SAX perl-Digest-SHA1 perl-XML-NamespaceSupport perl-Mail-IMAPClient heartbeat ipvsadm
Now, on both DB servers, create this file:
vi /etc/ha.d/ha.cf
logfacility local0
keepalive 2
#deadtime 30 # USE THIS!!!
deadtime 10
bcast eth1
#serial /dev/ttyS0
#baud 19200
auto_failback off
node db1.example.com db2.example.com
Now create haresources file on both DB servers. 10.1.1.134 will be the cluster IP.
DB1:
vi /etc/ha.d/haresources
db1.example.com IPaddr::10.1.1.134/27/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfs
DB2:
vi /etc/ha.d/haresources
db2.example.com IPaddr::10.1.1.134/27/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfs
Both servers:
vi /etc/ha.d/authkeys
auth 3
3 md5 redhat
Both DB servers:
chmod 600 /etc/ha.d/authkeys
Time to start DRBD and Heartbeat services on both servers.
[root@db1 ~]# service drbd start
Starting DRBD resources: [ ].
[root@db1 ~]#
[root@db2 ~]# service drbd start
Starting DRBD resources: [ ].
[root@db2 ~]#
[root@db1 ~]# service heartbeat start
Starting High-Availability services:
2007/12/15_11:38:45 INFO: Resource is stopped
[ OK ]
[root@db1 ~]#
[root@db2 ~]# service heartbeat start
Starting High-Availability services:
2007/12/15_11:40:13 INFO: Resource is stopped
[ OK ]
[root@db2 ~]#
Ifconfig reveals the following on each server:
[root@db1 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:20:30:61:AF:94
inet addr:10.1.1.130 Bcast:10.1.1.159 Mask:255.255.255.224
inet6 addr: fe80::230:48ff:fe61:af94/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:132587 errors:0 dropped:0 overruns:0 frame:0
TX packets:115896 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:32074701 (30.5 MiB) TX bytes:11049930 (10.5 MiB)
Base address:0x2000 Memory:d8920000-d8940000
eth0:0 Link encap:Ethernet HWaddr 00:20:30:61:AF:94
inet addr:10.1.1.134 Bcast:10.1.1.159 Mask:255.255.255.224
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Base address:0x2000 Memory:d8920000-d8940000
...
[root@db2 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:20:30:61:AF:4E
inet addr:10.1.1.131 Bcast:10.1.1.159 Mask:255.255.255.224
inet6 addr: fe80::230:48ff:fe61:af4e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:132652 errors:0 dropped:0 overruns:0 frame:0
TX packets:117073 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:33537810 (31.9 MiB) TX bytes:10732645 (10.2 MiB)
Base address:0x2000 Memory:d8920000-d8940000
...
Now on both DB servers:
chkconfig --level 35 drbd on
chkconfig --level 35 heartbeat on
Now create an account for webmaster on all 4 servers, with same uid/gid
Both NFS servers:
groupadd -g 509 webmaster
useradd -u 509 -g 509 -s /bin/bash -d /var/www/vhosts webmaster
passwd webmaster
**Lets check and then mount the exported directories on web servers:**
[root@webmail1 ~]# showmount -e 10.1.1.134
Export list for 10.1.1.134:
/data/www 10.1.1.133,10.1.1.132
/data/home 10.1.1.133,10.1.1.132
[root@webmail2 ~]# showmount -e 10.1.1.134
Export list for 10.1.1.134:
/data/www 10.1.1.133,10.1.1.132
/data/home 10.1.1.133,10.1.1.132
Now mount the /data/www on /var/www/ on each webserver:
mount -t nfs 10.1.1.134:/data/www /var/www/
[root@webmail1 ~]# mount -t nfs 10.1.1.134:/data/www /var/www/
[root@webmail1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 20G 1.1G 18G 6% /
tmpfs 1.5G 0 1.5G 0% /dev/shm
10.1.1.134:/data/www
188G 188M 179G 1% /var/www
[root@webmail1 ~]#
[root@webmail2 ~]# mount -t nfs 10.1.1.134:/data/www /var/www/
[root@webmail2 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 20G 1.1G 18G 6% /
tmpfs 1.5G 0 1.5G 0% /dev/shm
10.1.1.134:/data/www
188G 188M 179G 1% /var/www
[root@webmail2 ~]#
Now on both WEB Servers:
vi /etc/fstab
. . .
. . .
10.1.1.134:/data/www /var/www nfs defaults 0 0
Alright. It’s time to sync the old webserver’s content with this new one. I will log on to the old server at liquid web as root and sync content via rsync to the primary db server, here.
On host1.example.com, as root, do the following:
rsync -avz -e ssh /var/www/ root@10.1.1.130:/data/www/
MYSQL:
If you create the a symbolic link from /data/mysql to /var/lib/mysql. It will NOT work and everytime mysql WILL FAIL TO START. MySQL expects this to be a directory, not a symbolic link. The solution is to edit the /etc/my.cnf and change the directives from from “/var/lib” to “/data” . So when mysqld is started, it will automatically create the /data/mysql directory. And will start fine. Thanks to Naveed Ahmad (naveed@wbitt.com) for pointing this out.
On both Servers:
cp /etc/my.cnf /etc/my.cnf.orig
vi /etc/my.cnf
[mysqld]
# datadir=/var/lib/mysql
datadir=/data/mysql
#socket=/var/lib/mysql/mysql.sock
socket=/data/mysql/mysql.sock
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
[mysql.server]
user=mysql
#basedir=/var/lib
basedir=/data
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
[mysql]
socket=/data/mysql/mysql.sock
Before actually starting the mysql service, do the following:
The /etc/init.d/mysqld script has a section as :
- Spin for a maximum of N seconds waiting for the server to come up.
- Rather than assuming we know a valid username, accept an “access
- denied” response as meaning the server is functioning.
echo "result of starting mysqld_safe is $ret"
if [ $ret -eq 0 ]; then
STARTTIMEOUT=30
while [ $STARTTIMEOUT -gt 0 ]; do
RESPONSE=`/usr/bin/mysqladmin -uUNKNOWN_MYSQL_USERping 2>&1` && break
echo "$RESPONSE" | grep -q "Access denied for user" && break
sleep 1
let STARTTIMEOUT=${STARTTIMEOUT}-1
done
if [ $STARTTIMEOUT -eq 0 ]; then
echo "Timeout error occurred trying to start MySQL Daemon."
action $"Starting $prog: " /bin/false
else
action $"Starting $prog: " /bin/true
fi
else
action $"Starting $prog: " /bin/false
fi
The cause of error / FAILED is this line:
RESPONSE=/usr/bin/mysqladmin -uUNKNOWN_MYSQL_USERping 2>&1 && break
The problem is that the mysqld_safe program line (not shown here) is given an argument to load the variable values from a default file “/etc/my.cnf”, in this script file. However, the RESPONSE line shown here does not account for socket path and keeps looking for a socket at default location /var/lib/mysql/mysql.sock . Whereas in our situation, we have moved the entire /var/lib/mysql to /data , or so to speak . The solution is to pass the “–socket $socketfile” or “-S $socketfile” parameter to the RESPONSE line above. Changing it to look like:
RESPONSE=/usr/bin/mysqladmin -S $socketfile -uUNKNOWN_MYSQL_USERping 2>&1 && break
After saving this file you can easily start and stop mysql . Make sure that you copy the modified script to the other node as well.
Note : Do this only on the primary DB server, not on the slave one!
[root@db1 ~]# service mysqld start
Starting MySQL: [ OK ]
[root@db1 ~]#
Once it is tested, stop it again. Also make sure it does not start at boot time. As we want to control it through heartbeat.
[root@db1 var]# service mysqld stop
Stopping MySQL: [ OK ]
[root@db1 var]# chkconfig --level 35 mysqld off
Add mysqld to /etc/ha.d/haresources file on both NFS servers.
vi /etc/ha.d/haresources
db1.example.com IPaddr::10.1.1.134/27/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfs mysqld
vi /etc/ha.d/haresources
db2.example.com IPaddr::10.1.1.134/27/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfs mysqld
Also updated the DNS zone for db.example.com against the cluster IP 10.1.1.134, on master DNS .
vi /var/named/chroot/var/named/example.com.zone
And restarted DNS service on both nodes.
service named restart
Now it is time to add users/hosts to mysql server:
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.1.1.130' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.1.1.131' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.1.1.132' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.1.1.133' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.1.1.134' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.1.1.135' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'db1.example.com' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'db2.example.com' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'db.example.com' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'www.example.com' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'webmail1.example.com' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'webmail2.example.com' IDENTIFIED BY 'redhat' with grant option;
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
mysql>
VSFTPD:
Install VSFTPD on both DB nodes.
yum install vsftpd
We already have setup an os user “webmaster” on both DB nodes. And they have their home directories setup as “/var/www/vhosts”. However, on the DB nodes , it is not “/var/www/vhosts” , rather, it is “/data/www/vhosts” , so update the os user on DB nodes as :
usermod -d /data/www/vhosts webmaster
Add vsftpd to HighAvailability config file haresources on both servers:
[root@db1 ~]# vi /etc/ha.d/haresources
db1.example.com IPaddr::10.1.1.134/27/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfs mysqld vsftpd
[root@db2 ~]# vi /etc/ha.d/haresources
db2.example.com IPaddr::10.1.1.134/27/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfs mysqld vsftpd
Apache:
It would be better to instal httpd and httpd-devel packages at this time on both DB servers:
yum install httpd httpd-devel
Setup Apache to be controlled through heartbeat:
[root@db1 ~]# vi /etc/ha.d/haresources
db1.example.com IPaddr::10.1.1.134/27/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfs mysqld vsftpd httpd
[root@db2 ~]# vi /etc/ha.d/haresources
db2.example.com IPaddr::10.1.1.134/27/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfs mysqld vsftpd httpd
And make sure you have php and related packages installed too on both DB servers:
yum install php php-mysql php-gd php-mbstring
service httpd restart
LOAD BALANCER SETUP:
Load Balancer Setup on already available db1 and db2. I will install Linux LB packages on both db1 and db2 and use another cluster IP / Vritual IP as the target for www and @ for example.com.
yum install heartbeat-ldirectord perl-TimeDate perl-Net-SSLeay perl-IO-Socket-SSL pygtk2 libglade2
modprobe ip_vs_dh
modprobe ip_vs_ftp
modprobe ip_vs
modprobe ip_vs_lblc
modprobe ip_vs_lblcr
modprobe ip_vs_lc
modprobe ip_vs_nq
modprobe ip_vs_rr
modprobe ip_vs_sed
modprobe ip_vs_sh
modprobe ip_vs_wlc
modprobe ip_vs_wrr
Also put it in /etc/rc.local
vi /etc/rc.local
modprobe ip_vs_dh
modprobe ip_vs_ftp
modprobe ip_vs
modprobe ip_vs_lblc
modprobe ip_vs_lblcr
modprobe ip_vs_lc
modprobe ip_vs_nq
modprobe ip_vs_rr
modprobe ip_vs_sed
modprobe ip_vs_sh
modprobe ip_vs_wlc
modprobe ip_vs_wrr
Both LBs:
vi /etc/sysctl.conf
Enables packet forwarding
net.ipv4.ip_forward = 1
sysctl -p
Both LB nodes:
vi /etc/ha.d/ha.cf
logfacility local0
keepalive 2
#deadtime 30 # USE THIS!!!
deadtime 10
bcast eth1
#serial /dev/ttyS0
#baud 19200
auto_failback off
respawn hacluster /usr/lib/heartbeat/ipfail
node host3.example.com host4.example.com
Now on both LBs but with a bit different on each one:
LB Node1:
vi /etc/ha.d/haresources
db1.example.com ldirectord::ldirectord.cf LVSSyncDaemonSwap::master IPaddr2::10.1.1.135/27/eth0 IPaddr::10.1.1.134/27/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfs mysqld vsftpd httpd
LB Node2:
vi /etc/ha.d/haresources
db2.example.com ldirectord::ldirectord.cf LVSSyncDaemonSwap::master IPaddr2::10.1.1.135/27/eth0 IPaddr::10.1.1.134/27/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext3 nfs mysqld vsftpd httpd
both LBs
vi /etc/ha.d/ldirectord.cf
checktimeout=10
checkinterval=2
autoreload=no
logfile=”local0”
quiescent=yes
virtual=10.1.1.135:80
real=10.1.1.132:80 gate
real=10.1.1.133:80 gate
fallback=127.0.0.1:80 gate
service=http
request=”ldirector.html”
receive=”Test Page”
scheduler=rr
protocol=tcp
checktype=negotiate
Both LBs
chkconfig --level 35 ldirectord off
chkconfig --level 35 heartbeat on
service ldirectord stop
WWW1 and WWW2 (disabled these settings )
vi /etc/sysctl.conf
. . .
. . .
Setup for LB
- When an arp request is received on eth0, only respond if that address is
- configured on eth0. In particular, do not respond if the address is
- configured on lo
net.ipv4.conf.eth0.arp_ignore = 1
Ditto for eth1, add for all ARPing interfaces
1net.ipv4.conf.eth1.arp_ignore = 1
Enable configuration of arp_announce option
net.ipv4.conf.all.arp_announce = 2
When making an ARP request sent through eth0 Always use an address that
is configured on eth0 as the source address of the ARP request. If this
is not set, and packets are being sent out eth0 for an address that is on
lo, and an arp request is required, then the address on lo will be used.
As the source IP address of arp requests is entered into the ARP cache on
the destination, it has the effect of announcing this address. This is
not desirable in this case as adresses on lo on the real-servers should
be announced only by the linux-director.
net.ipv4.conf.eth0.arp_announce = 2
Ditto for eth1, add for all ARPing interfaces
net.ipv4.conf.eth1.arp_announce = 2
Then:
sysctl -p # on BOTH WWWs
WWW1 and WWW2
REDHAT / CentOS / Fedora
cd /etc/sysconfig/network-scripts/
cp ifcfg-lo ifcfg-lo:0
vi ifcfg-lo:0
DEVICE=lo:0
IPADDR=10.1.1.135
NETMASK=255.255.255.255
ONBOOT=yes
NAME=loopback:0
Both WWWs
echo "Test Page" > /var/www/vhosts/example.com/ldirector.html