keepalived+nginx实现nginx的高可用
=================================
nginx的高可用
nginx实现后端realserver的负载均衡
==================================
实验环境:
OS:Centos 6.4(redhat 6.4)
yum源:
1
2
3
4
5
6
7
8
9
10
11
[centos]
name=sohu-centos
baseurl=
http://mirrors.sohu.com/centos/$releasever/os/$basearchgpgcheck=1
enable=0
gpgkey=
http://mirrors.sohu.com/centos/RPM-GPG-KEY-CentOS-6[epel]
name=sohu-epel
baseurl=
http://mirrors.sohu.com/fedora-epel/$releasever/$basearch/enable=1
gpgcheck=0
拓扑图:
拓扑图的规划:
IP地址
软件
Master
172.16.22.1(VIP:172.16.22.100)
keepalived+nginx
Backup
172.16.22.2(VIP:172.16.22.100)
keepalived+nginx
apache1
172.16.22.3
httpd
apache2
172.16.22.4
httpd
此架构需考虑的问题
1)、Master没挂,则Master占有vip且nginx运行在Master上
2)、Master挂了,则backup抢占vip且在backup上运行nginx服务
3)、如果master服务器上的nginx服务挂了,则vip资源转移到backup服务器上
4)、检测后端服务器的健康状态
Master和Backup两边都开启nginx服务,无论Master还是Backup,当其中的一个keepalived服务停止后,vip都会漂移到keepalived服务还在的节点上,
如果要想使nginx服务挂了,vip也漂移到另一个节点,则必须用脚本或者在配置文件里面用shell命令来控制。
首先必须明确后端服务器的健康状态检测keepalived在这种架构上是无法检测的,后端服务器的健康状态检测是有nginx来判断的,但是nginx 的检测机制有一定的缺陷,后端服务器某一个宕机之后,nginx还是会分发请求给它,在一定的时间内后端服务响应不了,nginx则会发给另外一个服务 器,然后当客户的请求来了,nginx会一段时间内不会把请求分发给已经宕机的服务器,但是过一段时间后,nginx还是会把分发请求发给宕机的服务器 上。
一、安装keepalived+nginx
Master:
1、安装keepalived和编译安装nginx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[root@jie1 ~]# yum -y install keepalived
[root@jie1 ~]#tar xf nginx-1.4.2.tar.gz
[root@jie1 ~]#yum -y groupinstall "Development tools" "Server Platform Development"
[root@jie1 ~]#yum -y install pcre-devel
[root@jie1 ~]# cd nginx-1.4.2
[root@jie1 nginx-1.4.2]# groupadd nginx
[root@jie1 nginx-1.4.2]# useradd -r -g nginx nginx
[root@jie1 nginx-1.4.2]#./configure \
--prefix=/usr\
--sbin-path=/usr/sbin/nginx\
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx/nginx.pid \
--lock-path=/var/lock/nginx.lock \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_flv_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--http-client-body-temp-path=/var/tmp/nginx/client/\
--http-proxy-temp-path=/var/tmp/nginx/proxy/\
--http-fastcgi-temp-path=/var/tmp/nginx/fcgi/\
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi\
--http-scgi-temp-path=/var/tmp/nginx/scgi\
--with-pcre
[root@jie1 nginx-1.4.2]# make && make install
2、提供nginx的system V服务脚本文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
[root@jie1 nginx-1.4.2]# vim /etc/rc.d/init.d/nginx
#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig: - 85 15
# description: Nginx is an HTTP(S) server, HTTP(S) reverse \
# proxy and IMAP/POP3 proxy server
# processname: nginx
# config: /etc/nginx/nginx.conf
# config: /etc/sysconfig/nginx
# pidfile: /var/run/nginx.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING"= "no"] && exit0
nginx="/usr/sbin/nginx"
prog=$(basename$nginx)
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
[ -f /etc/sysconfig/nginx] && . /etc/sysconfig/nginx
lockfile=/var/lock/subsys/nginx
make_dirs() {
# make required directories
user=`nginx -V 2>&1 | grep"configure arguments:"| sed's/[^*]*--user=\([^ ]*\).*/\1/g'-`
options=`$nginx -V 2>&1 | grep'configure arguments:'`
foropt in$options; do
if[ `echo$opt | grep'.*-temp-path'` ]; then
value=`echo$opt | cut-d "="-f 2`
if[ ! -d "$value"]; then
# echo "creating" $value
mkdir-p $value && chown-R $user $value
fi
fi
done
}
start() {
[ -x $nginx ] || exit5
[ -f $NGINX_CONF_FILE ] || exit6
make_dirs
echo-n $"Starting $prog: "
daemon $nginx -c $NGINX_CONF_FILE
retval=$?
echo
[ $retval -eq0 ] && touch$lockfile
return$retval
}
stop() {
echo-n $"Stopping $prog: "
killproc $prog -QUIT
retval=$?
echo
[ $retval -eq0 ] && rm-f $lockfile
return$retval
}
restart() {
configtest || return$?
stop
sleep1
start
}
reload() {
configtest || return$?
echo-n $"Reloading $prog: "
killproc $nginx -HUP
RETVAL=$?
echo
}
force_reload() {
restart
}
configtest() {
$nginx -t -c $NGINX_CONF_FILE
}
rh_status() {
status $prog
}
rh_status_q() {
rh_status >/dev/null2>&1
}
case"$1"in
start)
rh_status_q && exit0
$1
;;
stop)
rh_status_q || exit0
$1
;;
restart|configtest)
$1
;;
reload)
rh_status_q || exit7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit0
;;
*)
echo$"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
exit2
esac
[root@jie1 nginx-1.4.2]# chmod +x /etc/rc.d/init.d/nginx
[root@jie1 nginx-1.4.2]# service nginx start
Starting nginx: [ OK ]
[root@jie1 nginx-1.4.2]# scp -p /etc/rc.d/init.d/nginx 172.16.22.2:/etc/rc.d/init.d #把nginx的服务脚本复制到backup上,-p是保持原有的权限
3、修改配置文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[root@jie1 ~]# cd /etc/keepalived/
[root@jie1 keepalived]# vim keepalived.conf
global_defs {
notification_email {
root@localhost
}
notification_email_from admin@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LTT
}
vrrp_script chk_nginx { #检测nginx服务是否在运行有很多方式,比如进程,用脚本检测等等
script "killall -0 nginx" #用shell命令检查nginx服务是否存在
interval 1 #时间间隔为1秒检测一次
weight -2 #当nginx的服务不存在了,就把当前的权重-2
fall 2 #测试失败的次数
rise 1 #测试成功的次数
}
vrrp_instance IN_1 {
state MASTER
interface eth0
virtual_router_id 22
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass aaaa
}
virtual_ipaddress {
172.16.22.100
}
track_script {
chk_nginx #引用上面的vrrp_script定义的脚本名称
}
}
[root@jie1 keepalived]#scp keepalived.conf 172.16.22.2:/etc/keepalived #把配置文件copy到Backup服务器上,copy之前要保证Backup服务器上面已经安装了keepalived
4、开启keepalived和nginx的服务
1
2
3
4
5
6
7
8
[root@jie1 keepalived]# service keepalived start
Starting keepalived: [ OK ]
[root@jie1 keepalived]# chkconfig --add keepalived
[root@jie1 keepalived]# chkconfig keepalived on
[root@jie1 ~]# service nginx start
Starting nginx: [ OK ]
[root@jie1 ~]# chkconfig --add nginx
[root@jie1 ~]# chkconfig nginx on
Backup:
1、安装keepalived和编译安装nginx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[root@jie2 ~]# yum -y install keepalived
[root@jie2 ~]#tar xf nginx-1.4.2.tar.gz
[root@jie2 ~]#yum -y groupinstall "Development tools" "Server Platform Development"
[root@jie2 ~]#yum -y install pcre-devel
[root@jie2 ~]# cd nginx-1.4.2
[root@jie2 nginx-1.4.2]# groupadd nginx
[root@jie2 nginx-1.4.2]# useradd -r -g nginx nginx
[root@jie2 nginx-1.4.2]#./configure \
--prefix=/usr \
--sbin-path=/usr/sbin/nginx \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx/nginx.pid \
--lock-path=/var/lock/nginx.lock \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_flv_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--http-client-body-temp-path=/var/tmp/nginx/client/ \
--http-proxy-temp-path=/var/tmp/nginx/proxy/ \
--http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \
--http-scgi-temp-path=/var/tmp/nginx/scgi \
--with-pcre
[root@jie2 nginx-1.4.2]# make && make install
2、之前 已经从Master复制了nginx的system V服务脚本文件,启动nginx服务
1
2
3
4
[root@jie2 ~]# service nginx start
Starting nginx: [ OK ]
[root@jie2 ~]# chkconfig --add nginx
[root@jie2 ~]# chkconfig nginx on
3、修改配置文件
1
2
3
4
[root@jie2 ~]# cd /etc/keepalived/
[root@jie2 keepalived]# vim keepalived.conf #此配置文件是从Master服务器上copy过来,只需小小改动
state BACKUP #把这里原先的MASTER改成BACKUP
priority 99 #把这里原先的100改成99
4、开启服务
1
2
3
4
[root@jie2 keepalived]# service keepalived start
Starting keepalived: [ OK ]
[root@jie2 keepalived]# chkconfig --add keepalived
[root@jie2 keepalived]# chkconfig keepalived on
apache1:
1、安装(博主这里用rpm包安装,各位朋友可以用源码编译安装)
1 [root@jie3 ~]# yum -y install httpd
2、建立测试网页文件
1
2
3
[root@jie3 ~]# cd /var/www/html/
[root@jie3 html]# cat index.html #建一个测试网页
this is apache1
3、开启服务
1
2
3
4
[root@jie3 html]# service httpd start
Starting httpd: [ OK ]
[root@jie3 html]# chkconfig --add httpd
[root@jie3 html]# chkconfig httpd on
apache2:
1、安装(博主这里用rpm包安装,各位朋友可以用源码编译安装)
1 [root@jie4 ~]# yum -y install httpd
2、建立测试网页文件
1
2
3
[root@jie4 ~]# cd /var/www/html/
[root@jie4 html]# cat index.html #建一个测试网页
this is apache2
3、开启服务
1
2
3
4
[root@jie4 html]# service httpd start
Starting httpd: [ OK ]
[root@jie4 html]# chkconfig --add httpd
[root@jie4 html]# chkconfig httpd on
此致所有安装已经完成。
二、nginx实现后端realserver的负载均衡,由于两边的配置文件必须保持一致,所以在Master配置完后直接copy到Backup上
Master:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@jie1 ~]# cd /etc/nginx/
[root@jie1 nginx]# grep -v "#" nginx.conf | grep -v "^$"
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream apacheweb { #定义负载均衡的模块
server 172.16.22.3:80 max_fails=3 fail_timeout=2s;
server 172.16.22.4:80 max_fails=3 fail_timeout=2s;
}
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
location ~ \.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
root /var/www/html; #此处定义后端服务器网页存放路径
proxy_pass
http://apacheweb; }
}
}
[root@jie1 nginx]# scp nginx.conf 172.16.22.2:/etc/nginx