WBITT

We Bring In Tomorrows Technology

  • Full Screen
  • Wide Screen
  • Narrow Screen
  • Increase font size
  • Default font size
  • Decrease font size

Apache High Availability Howto, using LVS and Heartbeat

E-mail Print PDF
User Rating: / 6
PoorBest 

Assalam-u-alaikum,

This howto is also a bit old now. But I thought of uploading it in the howtos section, as it still works.

Created : Mid 2007

Last updated: Mid 2007

The following link is very nice tutorial :

http://www.howtoforge.com/high_availability_loadbalanced_apache_cluster
but uses Debian!

I have used CentOS 4.3 and all heartbeat and ipvs packages from centos43 extras. Nothing needs to be downloaded from ANYWHERE. No need to go to Ultra monkey site, etc.


Make sure that example.com has proper Zone entries on your DNS server for:


www1.example.com.       IN      A       192.168.0.101
www2.example.com.       IN      A       192.168.0.102
lb1.example.com.        IN      A       192.168.0.103
lb2.example.com.        IN      A       192.168.0.104
www.example.com.        IN      A       192.168.0.105



192.168.0.103           lb1.example.com         lb1

HOSTNAME=lb1.example.com
GATEWAY=192.168.0.254

IPADDR=192.168.0.103
NETMASK=255.255.225.0

DEVICE=eth0
BOOTPROTO=static
HWADDR=00:40:26:5F:5C:60
ONBOOT=yes
TYPE=Ethernet
IPADDR=192.168.0.102
NETMASK=255.255.255.0

--------------------------------------
both Webservers

mkdir /media/nfs ; mkdir /media/extras
mount -t nfs 192.168.0.254:/data/cdimages/centos43nfs /media/nfs
echo "mount -t nfs 192.168.0.254:/data/cdimages/centos43nfs /media/nfs" >> /etc/rc.local
mount -t nfs 192.168.0.254:/data/cdimages/centos43extrasnfs /media/extras/
echo "mount -t nfs 192.168.0.254:/data/cdimages/centos43extrasnfs /media/extras/" >> /etc/rc.local
-------

rpm -ivh httpd-2.0.52-22.ent.centos4.i386.rpm apr-0.9.4-24.5.c4.1.i386.rpm httpd-suexec-2.0.52-22.ent.centos4.i386.rpm apr-util-0.9.4-21.i386.rpm

------------------------------------------------
Load Balancers:
/etc/modules does not exist on RedHAT and clones.

On both load balancers (also placed the same in /etc/rc.local)
/etc/modprobe.conf can be used but I do not know how to use it, so /etc/rc.local is the easy way out.


modprobe ip_vs_dh
modprobe ip_vs_ftp
modprobe ip_vs
modprobe ip_vs_lblc
modprobe ip_vs_lblcr
modprobe ip_vs_lc
modprobe ip_vs_nq
modprobe ip_vs_rr
modprobe ip_vs_sed
modprobe ip_vs_sh
modprobe ip_vs_wlc
modprobe ip_vs_wrr


--------------------
both LBs


rpm -ivh curl-7.12.1-8.rhel4.i386.rpm perl-Crypt-SSLeay-0.51-5.i386.rpm perl-HTML-Parser-3.35-6.i386.rpm perl-LDAP-0.31-5.noarch.rpm perl-Net-DNS-0.48-1.i386.rpm perl-libwww-perl-5.79-5.noarch.rpm libidn-0.5.6-1.i386.rpm perl-Convert-ASN1-0.18-3.noarch.rpm perl-Digest-HMAC-1.01-13.noarch.rpm perl-HTML-Tagset-3.03-30.noarch.rpm perl-URI-1.30-4.noarch.rpm perl-XML-SAX-0.12-7.noarch.rpm perl-Digest-SHA1-2.07-5.i386.rpm perl-XML-NamespaceSupport-1.08-6.noarch.rpm

rpm -ivh perl-Mail-IMAPClient-2.2.9-1.rf.noarch.rpm heartbeat-* ipvsadm-1.24-6.i386.rpm


------------------

Install Heartbeat packages

-------------------

vi /etc/sysctl.conf

# Enables packet forwarding
net.ipv4.ip_forward = 1

sysctl -p

-------------------

USE "UNAME -n" for node names
vi /etc/ha.d/ha.cf

logfacility        local0
bcast        eth0                # Linux
mcast eth0 225.0.0.1 694 1 0
auto_failback off
node        lb1.example.com
node        lb2.example.com
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster


-------------------
USE "UNAME -n" for node names

LB1:

vi /etc/ha.d/haresources

lb1.example.com        \
ldirectord::ldirectord.cf \
LVSSyncDaemonSwap::master \
IPaddr2::192.168.0.105/24/eth0/192.168.0.255

LB2:

lb2.example.com        \
ldirectord::ldirectord.cf \
LVSSyncDaemonSwap::master \
IPaddr2::192.168.0.105/24/eth0/192.168.0.255

------------
both LBs

vi /etc/ha.d/authkeys

auth 3
3 md5 redhat

----------------------
both LBs

chmod 600 /etc/ha.d/authkeys

-----------
both LBs

vi /etc/ha.d/ldirectord.cf

checktimeout=10
checkinterval=2
autoreload=no
logfile="local0"
quiescent=yes

virtual=192.168.0.105:80
real=192.168.0.101:80 gate
real=192.168.0.102:80 gate
fallback=127.0.0.1:80 gate
service=http
request="ldirector.html"
receive="Test Page"
scheduler=rr
protocol=tcp
checktype=negotiate


---------
Both LBs

chkconfig --level 35  ldirectord off
chkconfig --level 35  heartbeat  on


service ldirectord stop
service heartbeat start
+++++++++++++++++++++++++

[root@lb1 /]# service ldirectord stop
Stopping ldirectord                                        [  OK  ]
[root@lb1 /]# service heartbeat start
Starting High-Availability services:
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[FAILED]
heartbeat: 2007/02/16_04:12:30 ERROR: Current node [lb1.example.com] not in configuration!
heartbeat: 2007/02/16_04:12:30 info: By default, cluster nodes are named by `uname -n` and must be declared with a 'node' directive in the ha.cf file.
heartbeat: 2007/02/16_04:12:30 ERROR: Configuration error, heartbeat not started.

[root@lb1 /]# uname -n
lb1.example.com
[root@lb1 /]#

OHHHHH


Files corrected with uname -n output . i.e. instead of lb1, I have now changed that to lb1.example.com

[root@lb1 /]# service heartbeat start
Starting High-Availability services:
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[  OK  ]
[root@lb1 /]#   

[root@lb2 /]# service ldirectord stop
Stopping ldirectord                                        [  OK  ]
[root@lb2 /]# service heartbeat start
Starting High-Availability services:
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[  OK  ]
[root@lb2 /]#


Good. Alhumdulilah.


LB1:

[root@lb1 /]# ip addr sh eth0
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:08:c7:69:13:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.0.103/24 brd 192.168.0.255 scope global eth0
inet 192.168.0.105/24 brd 192.168.0.255 scope global secondary eth0
inet6 fe80::208:c7ff:fe69:139d/64 scope link
valid_lft forever preferred_lft forever
[root@lb1 /]#

Notice LB1 has two IPs , which DO NOT SHOW in ifconfig -a output.


LB2:
[root@lb2 /]# ip addr sh eth0
2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:50:8b:60:c8:f8 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.104/24 brd 192.168.0.255 scope global eth0
inet6 fe80::250:8bff:fe60:c8f8/64 scope link
valid_lft forever preferred_lft forever
[root@lb2 /]#


----------------------------

[root@lb1 /]# ldirectord ldirectord.cf status
ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 3978
[root@lb1 /]#

NOTE: The above output (running) will be displayed on ACTIVE LB


[root@lb2 /]# ldirectord ldirectord.cf status
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[root@lb2 /]#

-------------------

[root@lb1 /]# ipvsadm -L -n
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@lb1 /]#


[root@lb2 /]# ipvsadm -L -n
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@lb2 /]#

NOTE: If LB1 does not show as below, try restarting heartbeat on LB1 and LB2. In the same order I have written here.


[root@lb1 /]# ipvsadm -L -n
IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.105:80 rr
-> 192.168.0.101:80             Route   0      0          0
-> 192.168.0.102:80             Route   0      0          0
-> 127.0.0.1:80                 Local   1      0          0
[root@lb1 /]#      


--------------

[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master running
(ipvs_syncmaster pid: 4704)
[root@lb1 /]#  

NOTE: Master running on LB1


[root@lb2 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master stopped
(ipvs_syncbackup pid: 4488)
[root@lb2 /]#

--------------------

WWW1 and WWW2

vi /etc/sysctl.conf

# When an arp request is received on eth0, only respond if that address is
# configured on eth0. In particular, do not respond if the address is
# configured on lo
net.ipv4.conf.eth0.arp_ignore = 1

# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_ignore = 1


# Enable configuration of arp_announce option
net.ipv4.conf.all.arp_announce = 2

# When making an ARP request sent through eth0 Always use an address that
# is configured on eth0 as the source address of the ARP request.  If this
# is not set, and packets are being sent out eth0 for an address that is on
# lo, and an arp request is required, then the address on lo will be used.
# As the source IP address of arp requests is entered into the ARP cache on
# the destination, it has the effect of announcing this address.  This is
# not desirable in this case as adresses on lo on the real-servers should
# be announced only by the linux-director.
net.ipv4.conf.eth0.arp_announce = 2

# Ditto for eth1, add for all ARPing interfaces
#net.ipv4.conf.eth1.arp_announce = 2


sysctl -p on BOTH WWWs

---------------------

Note :  /etc/network/interfaces exists in debian only it does not exist in RedHat. For that you need to use the /etc/sysconfig/network-scripts/ifcfg-* files.

DEBIAN

vi /etc/network/interfaces

auto lo:0
iface lo:0 inet static
address 192.168.0.105
netmask 255.255.255.255
pre-up sysctl -p > /dev/null

------------------

WWW1 and WWW2

REDHAT / CentOS / Fedora


cd /etc/sysconfig/network-scripts/

cp ifcfg-lo ifcfg-lo:0

vi ifcfg-lo\:0

DEVICE=lo:0
BOOTPROTO=static
IPADDR=192.168.0.105
NETMASK=255.255.255.255
ONBOOT=yes
NAME=loopback


-----------------
both WWWs

[root@www1 network-scripts]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0A:5E:05:97:B4
inet addr:192.168.0.101  Bcast:192.168.0.255  Mask:255.255.255.0
inet6 addr: fe80::20a:5eff:fe05:97b4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:7914 errors:472 dropped:0 overruns:0 frame:472
TX packets:2875 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2257584 (2.1 MiB)  TX bytes:293073 (286.2 KiB)
Interrupt:5 Base address:0x7080

lo        Link encap:Local Loopback
inet addr:127.0.0.1  Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING  MTU:16436  Metric:1
RX packets:26 errors:0 dropped:0 overruns:0 frame:0
TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1820 (1.7 KiB)  TX bytes:1820 (1.7 KiB)

[root@www1 network-scripts]# ifup lo:0


[root@www1 network-scripts]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0A:5E:05:97:B4
inet addr:192.168.0.101  Bcast:192.168.0.255  Mask:255.255.255.0
inet6 addr: fe80::20a:5eff:fe05:97b4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:8001 errors:472 dropped:0 overruns:0 frame:472
TX packets:2923 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2266978 (2.1 MiB)  TX bytes:299149 (292.1 KiB)
Interrupt:5 Base address:0x7080

lo        Link encap:Local Loopback
inet addr:127.0.0.1  Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING  MTU:16436  Metric:1
RX packets:26 errors:0 dropped:0 overruns:0 frame:0
TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1820 (1.7 KiB)  TX bytes:1820 (1.7 KiB)

lo:0      Link encap:Local Loopback
inet addr:192.168.0.105  Mask:255.255.255.255
UP LOOPBACK RUNNING  MTU:16436  Metric:1

[root@www1 network-scripts]#



[root@www2 network-scripts]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:40:26:5F:5C:60
inet addr:192.168.0.102  Bcast:192.168.0.255  Mask:255.255.255.0
inet6 addr: fe80::240:26ff:fe5f:5c60/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:8349 errors:0 dropped:0 overruns:0 frame:0
TX packets:4236 errors:0 dropped:0 overruns:0 carrier:0
collisions:132 txqueuelen:1000
RX bytes:2187763 (2.0 MiB)  TX bytes:450715 (440.1 KiB)
Interrupt:5 Base address:0x7080

lo        Link encap:Local Loopback
inet addr:127.0.0.1  Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING  MTU:16436  Metric:1
RX packets:53 errors:0 dropped:0 overruns:0 frame:0
TX packets:53 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5347 (5.2 KiB)  TX bytes:5347 (5.2 KiB)

[root@www2 network-scripts]# ifup lo:0
[root@www2 network-scripts]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:40:26:5F:5C:60
inet addr:192.168.0.102  Bcast:192.168.0.255  Mask:255.255.255.0
inet6 addr: fe80::240:26ff:fe5f:5c60/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:8412 errors:0 dropped:0 overruns:0 frame:0
TX packets:4275 errors:0 dropped:0 overruns:0 carrier:0
collisions:132 txqueuelen:1000
RX bytes:2193922 (2.0 MiB)  TX bytes:455943 (445.2 KiB)
Interrupt:5 Base address:0x7080

lo        Link encap:Local Loopback
inet addr:127.0.0.1  Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING  MTU:16436  Metric:1
RX packets:53 errors:0 dropped:0 overruns:0 frame:0
TX packets:53 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5347 (5.2 KiB)  TX bytes:5347 (5.2 KiB)

lo:0      Link encap:Local Loopback
inet addr:192.168.0.105  Mask:255.255.255.255
UP LOOPBACK RUNNING  MTU:16436  Metric:1

[root@www2 network-scripts]#  

-----------------------
Both WWWs

vi /var/www/html/ldirector.html
Test Page


-------------

Both WWWs

Configure the ServerName Directive repectively for each node:

WWW1
ServerName www1.example.com

WWW2
ServerName www2.example.com

Both WWWs:
cd /etc/httpd/conf.d/
mv welcome.conf welcome.conf.orig


service httpd start
chkconfig --level 35 httpd on


WWW1 (to represent some web content):

[root@www1 conf.d]# cd /var/www/html/
[root@www1 html]# vi index.html
WWW1 Website page


WWW2 (to represent some web content):

[root@www2 conf.d]# cd /var/www/html/
[root@www2 html]# vi index.html
WWW2 Website page


NOTE: I have used different webpage contents on both servers to actually notice the difference if one node fails. In real life, of-course the web content will be same.


Now try from some webclient / browser (192.168.0.22) the following:

http://www.example.com

OR

http://192.168.0.105


You will see WWW1 one time you press GO or ENTER, and will see WWW2 the second time. And it will keep switching like this proving that the load balancer is working properly and is sending connections to the web server which has lesser load. Since there are only two web servers abnd one client to check both servers one by one, the Load Balancer takes you to www1.example.com and to www2.example.com turn by turn.

Great Alhumdulillah.


Try stopping apache on WWW1, you would see the WWW2 page on your browser and if you keep clicking GO/ refresh, you will keep seeing WWW2 page. As soon as you turn up WWW1 server, and press refresh on client. You will see WWW1 page.  Same goes for WWW2 server. Great.


-----------------------------

Lets Fail the LB now. First check which one is active?

[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master running
(ipvs_syncmaster pid: 4704)
[root@lb1 /]#


So LB1 is ACTIVE LB. Let's fail that and see if heartbeat takes over or not. So we still see our web page on 192.168.0.105 (www.example.com) or not?

[root@lb1 /]# service heartbeat stop
Stopping High-Availability services:
[  OK  ]
[root@lb1 /]#


Check status again:

[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master stopped
(ipvs_syncbackup pid: 5125)
[root@lb1 /]#

Check to make sure if LB2 has taken over as master:

[root@lb2 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master running
(ipvs_syncmaster pid: 4743)
[root@lb2 /]#


NOTICE the master has stopped on LB1. Check browser on client, Working perfectly. We still get our page.


If I start heartbeat on LB1 again, LB1 must remain inactive and LB2 must still remain ACTIVE. Lets see:

[root@lb1 /]# service heartbeat start
Starting High-Availability services:
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[  OK  ]
[root@lb1 /]# /etc/ha.d/resource.d/LVSSyncDaemonSwap master status
master stopped
(ipvs_syncbackup pid: 5125)
[root@lb1 /]#

Notice that LB2 is still ACTIVE.

Alhumdulillah.


-----------------------------------------------------------------
================================================================

Lets connect null modem wire to serial ports of the two LBs and change appropriate settings in various config files on both LBs , so the too much broadcast for heartbeat signals on the switch is removed / minimized.

Both LBs :


vi /etc/ha.d/ha.cf

logfacility        local0
serial /dev/ttyS0
baud 19200

#bcast        eth0                # Linux
#mcast eth0 225.0.0.1 694 1 0
auto_failback off
node        lb1.example.com
node        lb2.example.com
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster



[root@lb2 ~]# service heartbeat stop
Stopping High-Availability services:
[  OK  ]

[root@lb1 ~]# service heartbeat stop
Stopping High-Availability services:
[  OK  ]

Checked through browser. Found everything working. Success. Alhumdulillah.

 

You are here How To / Tutorials Apache High Availability Howto, using LVS and Heartbeat