Apache High Availability Howto, using LVS and Heartbeat
Assalam-u-alaikum,
This howto is also a bit old now. But I thought of uploading it in the howtos section, as it still works.
Created : Mid 2007
Last updated: Mid 2007
The following link is very nice tutorial:
http://www.howtoforge.com/high_availability_loadbalanced_apache_cluster
but uses Debian!
I have used CentOS 4.3 and all heartbeat and ipvs packages from centos43 extras. Nothing needs to be downloaded from ANYWHERE. No need to go to Ultra monkey site, etc.
Make sure that example.com has proper Zone entries on your DNS server for:
| www1.example.com. | IN | A | 192.168.0.101 |
| www2.example.com. | IN | A | 192.168.0.102 |
| lb1.example.com. | IN | A | 192.168.0.103 |
| lb2.example.com. | IN | A | 192.168.0.104 |
| www.example.com. | IN | A | 192.168.0.105 |
| 192.168.0.103 | lb1.example.com | lb1 |
HOSTNAME=lb1.example.com
GATEWAY=192.168.0.254
IPADDR=192.168.0.103
NETMASK=255.255.225.0
DEVICE=eth0
BOOTPROTO=static
HWADDR=00:40:26:5F:5C:60
ONBOOT=yes
TYPE=Ethernet
IPADDR=192.168.0.102
NETMASK=255.255.255.0
both Webservers:
mkdir /media/nfs ; mkdir /media/extras
mount -t nfs 192.168.0.254:/data/cdimages/centos43nfs /media/nfs
echo "mount -t nfs 192.168.0.254:/data/cdimages/centos43nfs /media/nfs" >> /etc/rc.local
mount -t nfs 192.168.0.254:/data/cdimages/centos43extrasnfs /media/extras/
echo "mount -t nfs 192.168.0.254:/data/cdimages/centos43extrasnfs /media/extras/" >> /etc/rc.local
rpm -ivh httpd-2.0.52-22.ent.centos4.i386.rpm apr-0.9.4-24.5.c4.1.i386.rpm httpd-suexec-2.0.52-22.ent.centos4.i386.rpm apr-util-0.9.4-21.i386.rpm
Load Balancers:
/etc/modules does not exist on RedHAT and clones.
On both load balancers (also placed the same in /etc/rc.local)
/etc/modprobe.conf can be used but I do not know how to use it, so /etc/rc.local is the easy way out.
modprobe ip_vs_dh
modprobe ip_vs_ftp
modprobe ip_vs
modprobe ip_vs_lblc
modprobe ip_vs_lblcr
modprobe ip_vs_lc
modprobe ip_vs_nq
modprobe ip_vs_rr
modprobe ip_vs_sed
modprobe ip_vs_sh
modprobe ip_vs_wlc
modprobe ip_vs_wrr
both LBs:
rpm -ivh curl-7.12.1-8.rhel4.i386.rpm perl-Crypt-SSLeay-0.51-5.i386.rpm perl-HTML-Parser-3.35-6.i386.rpm perl-LDAP-0.31-5.noarch.rpm perl-Net-DNS-0.48-1.i386.rpm perl-libwww-perl-5.79-5.noarch.rpm libidn-0.5.6-1.i386.rpm perl-Convert-ASN1-0.18-3.noarch.rpm perl-Digest-HMAC-1.01-13.noarch.rpm perl-HTML-Tagset-3.03-30.noarch.rpm perl-URI-1.30-4.noarch.rpm perl-XML-SAX-0.12-7.noarch.rpm perl-Digest-SHA1-2.07-5.i386.rpm perl-XML-NamespaceSupport-1.08-6.noarch.rpm
rpm -ivh perl-Mail-IMAPClient-2.2.9-1.rf.noarch.rpm heartbeat-* ipvsadm-1.24-6.i386.rpm
Install Heartbeat packages:
vi /etc/sysctl.conf
# Enables packet forwarding
net.ipv4.ip_forward = 1
sysctl -p
USE “UNAME -n” for node names
vi /etc/ha.d/ha.cf
logfacility local0
bcast eth0 # Linux
mcast eth0 225.0.0.1 694 1 0
auto_failback off
node lb1.example.com
node lb2.example.com
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster
USE “UNAME -n” for node names
LB1:
vi /etc/ha.d/haresources
lb1.example.com
ldirectord::ldirectord.cf
LVSSyncDaemonSwap::master
IPaddr2::192.168.0.105/24/eth0/192.168.0.255
LB2:
lb2.example.com
ldirectord::ldirectord.cf
LVSSyncDaemonSwap::master
IPaddr2::192.168.0.105/24/eth0/192.168.0.255
both LBs
vi /etc/ha.d/authkeys
auth 3
3 md5 redhat
both LBs
chmod 600 /etc/ha.d/authkeys
both LBs
vi /etc/ha.d/ldirectord.cf
checktimeout=10
checkinterval=2
autoreload=no
logfile=”local0”
quiescent=yes
virtual=192.168.0.105:80
real=192.168.0.101:80 gate
real=192.168.0.102:80 gate
fallback=127.0.0.1:80 gate
service=http
request=”ldirector.html”
receive=”Test Page”
scheduler=rr
protocol=tcp
checktype=negotiate
Both LBs
chkconfig –level 35 ldirectord off
chkconfig –level 35 heartbeat on
service ldirectord stop
service heartbeat start
+++++++++++++++++++++++++
[root@lb1 /]# service ldirectord stop
Stopping ldirectord [ OK ]
[root@lb1 /]# service heartbeat start
Starting High-Availability services:
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[FAILED]
heartbeat: 2007/02/16_04:12:30 ERROR: Current node [lb1.example.com] not in configuration!
heartbeat: 2007/02/16_04:12:30 info: By default, cluster nodes are named by `uname -n` and must be declared with a 'node' directive in the ha.cf file.
heartbeat: 2007/02/16_04:12:30 ERROR: Configuration error, heartbeat not started.
[root@lb1 /]# uname -n
lb1.example.com
[root@lb1 /]#
OHHHHH
Files corrected with uname -n output . i.e. instead of lb1, I have now changed that to lb1.example.com
[root@lb1 /]# service heartbeat start
Starting High-Availability services:
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[ OK ]
[root@lb1 /]#
[root@lb2 /]# service ldirectord stop
Stopping ldirectord [ OK ]
[root@lb2 /]# service heartbeat start
Starting High-Availability services:
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[ OK ]
[root@lb2 /]#
Good. Alhumdulilah.
LB1:
[root@lb1 /]# ip addr sh eth0
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:08:c7:69:13:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.0.103/24 brd 192.168.0.255 scope global eth0
inet 192.168.0.105/24 brd 192.168.0.255 scope global secondary eth0
inet6 fe80::208:c7ff:fe69:139d/64 scope link
valid_lft forever preferred_lft forever
[root@lb1 /]#
Notice LB1 has two IPs , which DO NOT SHOW in ifconfig -a output.
LB2:
[root@lb2 /]# ip addr sh eth0
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:50:8b:60:c8:f8 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.104/24 brd 192.168.0.255 scope global eth0
inet6 fe80::250:8bff:fe60:c8f8/64 scope link
valid_lft forever preferred_lft forever
[root@lb2 /]#
[root@lb1 /]# ldirectord ldirectord.cf status
ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 3978
[root@lb1 /]#
NOTE: The above output (running) will be displayed on ACTIVE LB
[root@lb2 /]# ldirectord ldirectord.cf status
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[root@lb2 /]#
[root@lb1 /]# ipvsadm -L -n
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@lb1 /]#
[root@lb2 /]# ipvsadm -L -n
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@lb2 /]#
NOTE: If LB1 does not show as below, try restarting heartbeat on LB1 and LB2. In the same order I have written here.
[root@lb1 /]# ipvsadm -L -n
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.105:80 rr
-> 192.168.0.101:80 Route 0 0 0
-> 192.168.0.102:80 Route 0 0 0
-> 127.0.0.1:80 Local 1 0 0
[root@lb1 /]#
----
[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master running
(ipvs_syncmaster pid: 4704)
[root@lb1 /]#
NOTE: Master running on LB1
[root@lb2 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master stopped
(ipvs_syncbackup pid: 4488)
[root@lb2 /]#
---
WWW1 and WWW2
vi /etc/sysctl.conf
# When an arp request is received on eth0, only respond if that address is
# configured on eth0. In particular, do not respond if the address is
# configured on lo
net.ipv4.conf.eth0.arp_ignore = 1
# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_ignore = 1
# Enable configuration of arp_announce option
net.ipv4.conf.all.arp_announce = 2
# When making an ARP request sent through eth0 Always use an address that
# is configured on eth0 as the source address of the ARP request. If this
# is not set, and packets are being sent out eth0 for an address that is on
# lo, and an arp request is required, then the address on lo will be used.
# As the source IP address of arp requests is entered into the ARP cache on
# the destination, it has the effect of announcing this address. This is
# not desirable in this case as adresses on lo on the real-servers should
# be announced only by the linux-director.
net.ipv4.conf.eth0.arp_announce = 2
# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_announce = 2
sysctl -p on BOTH WWWs
---
Note : /etc/network/interfaces exists in debian only it does not exist in RedHat. For that you need to use the /etc/sysconfig/network-scripts/ifcfg-* files.
DEBIAN
vi /etc/network/interfaces
auto lo:0
iface lo:0 inet static
address 192.168.0.105
netmask 255.255.255.255
pre-up sysctl -p > /dev/null
---
WWW1 and WWW2
REDHAT / CentOS / Fedora
cd /etc/sysconfig/network-scripts/
cp ifcfg-lo ifcfg-lo:0
vi ifcfg-lo:0
DEVICE=lo:0
BOOTPROTO=static
IPADDR=192.168.0.105
NETMASK=255.255.255.255
ONBOOT=yes
NAME=loopback
---
both WWWs
[root@www1 network-scripts]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0A:5E:05:97:B4
inet addr:192.168.0.101 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::20a:5eff:fe05:97b4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:7914 errors:472 dropped:0 overruns:0 frame:472
TX packets:2875 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2257584 (2.1 MiB) TX bytes:293073 (286.2 KiB)
Interrupt:5 Base address:0x7080
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:26 errors:0 dropped:0 overruns:0 frame:0
TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1820 (1.7 KiB) TX bytes:1820 (1.7 KiB)
[root@www1 network-scripts]# ifup lo:0
[root@www1 network-scripts]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0A:5E:05:97:B4
inet addr:192.168.0.101 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::20a:5eff:fe05:97b4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8001 errors:472 dropped:0 overruns:0 frame:472
TX packets:2923 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2266978 (2.1 MiB) TX bytes:299149 (292.1 KiB)
Interrupt:5 Base address:0x7080
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:26 errors:0 dropped:0 overruns:0 frame:0
TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1820 (1.7 KiB) TX bytes:1820 (1.7 KiB)
lo:0 Link encap:Local Loopback
inet addr:192.168.0.105 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
[root@www1 network-scripts]#
[root@www2 network-scripts]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:40:26:5F:5C:60
inet addr:192.168.0.102 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::240:26ff:fe5f:5c60/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8349 errors:0 dropped:0 overruns:0 frame:0
TX packets:4236 errors:0 dropped:0 overruns:0 carrier:0
collisions:132 txqueuelen:1000
RX bytes:2187763 (2.0 MiB) TX bytes:450715 (440.1 KiB)
Interrupt:5 Base address:0x7080
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:53 errors:0 dropped:0 overruns:0 frame:0
TX packets:53 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5347 (5.2 KiB) TX bytes:5347 (5.2 KiB)
[root@www2 network-scripts]# ifup lo:0
[root@www2 network-scripts]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:40:26:5F:5C:60
inet addr:192.168.0.102 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::240:26ff:fe5f:5c60/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8412 errors:0 dropped:0 overruns:0 frame:0
TX packets:4275 errors:0 dropped:0 overruns:0 carrier:0
collisions:132 txqueuelen:1000
RX bytes:2193922 (2.0 MiB) TX bytes:455943 (445.2 KiB)
Interrupt:5 Base address:0x7080
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:53 errors:0 dropped:0 overruns:0 frame:0
TX packets:53 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5347 (5.2 KiB) TX bytes:5347 (5.2 KiB)
lo:0 Link encap:Local Loopback
inet addr:192.168.0.105 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
[root@www2 network-scripts]#
---
Both WWWs
vi /var/www/html/ldirector.html
Test Page
---
Both WWWs
Configure the ServerName Directive repectively for each node:
WWW1
ServerName www1.example.com
WWW2
ServerName www2.example.com
Both WWWs:
cd /etc/httpd/conf.d/
mv welcome.conf welcome.conf.orig
service httpd start
chkconfig --level 35 httpd on
WWW1 (to represent some web content):
[root@www1 conf.d]# cd /var/www/html/
[root@www1 html]# vi index.html
WWW1 Website page
WWW2 (to represent some web content):
[root@www2 conf.d]# cd /var/www/html/
[root@www2 html]# vi index.html
WWW2 Website page
NOTE: I have used different webpage contents on both servers to actually notice the difference if one node fails. In real life, of-course the web content will be same.
Now try from some webclient / browser (192.168.0.22) the following:
http://www.example.com
OR
http://192.168.0.105
You will see WWW1 one time you press GO or ENTER, and will see WWW2 the second time. And it will keep switching like this proving that the load balancer is working properly and is sending connections to the web server which has lesser load. Since there are only two web servers abnd one client to check both servers one by one, the Load Balancer takes you to www1.example.com and to www2.example.com turn by turn.
Great Alhumdulillah.
Try stopping apache on WWW1, you would see the WWW2 page on your browser and if you keep clicking GO/ refresh, you will keep seeing WWW2 page. As soon as you turn up WWW1 server, and press refresh on client. You will see WWW1 page. Same goes for WWW2 server. Great.
---
Lets Fail the LB now. First check which one is active?
[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master running
(ipvs_syncmaster pid: 4704)
[root@lb1 /]#
So LB1 is ACTIVE LB. Let's fail that and see if heartbeat takes over or not. So we still see our web page on 192.168.0.105 (www.example.com) or not?
[root@lb1 /]# service heartbeat stop
Stopping High-Availability services:
[ OK ]
[root@lb1 /]#
Check status again:
[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master stopped
(ipvs_syncbackup pid: 5125)
[root@lb1 /]#
Check to make sure if LB2 has taken over as master:
[root@lb2 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master running
(ipvs_syncmaster pid: 4743)
[root@lb2 /]#
NOTICE the master has stopped on LB1. Check browser on client, Working perfectly. We still get our page.
If I start heartbeat on LB1 again, LB1 must remain inactive and LB2 must still remain ACTIVE. Lets see:
[root@lb1 /]# service heartbeat start
Starting High-Availability services:
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[ OK ]
[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master stopped
(ipvs_syncbackup pid: 5125)
[root@lb1 /]#
Notice that LB2 is still ACTIVE.
Alhumdulillah.
---
================================================================
Lets connect null modem wire to serial ports of the two LBs and change appropriate settings in various config files on both LBs , so the too much broadcast for heartbeat signals on the switch is removed / minimized.
Both LBs :
vi /etc/ha.d/ha.cf
logfacility local0
serial /dev/ttyS0
baud 19200
#bcast eth0 # Linux
#mcast eth0 225.0.0.1 694 1 0
auto_failback off
node lb1.example.com
node lb2.example.com
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster
[root@lb2 ~]# service heartbeat stop
Stopping High-Availability services:
[ OK ]
[root@lb1 ~]# service heartbeat stop
Stopping High-Availability services:
[ OK ]
```
Checked through browser. Found everything working. Success. Alhumdulillah.