简述
前端时间尝试着搭建了airflow的高可用（HA）环境，为避免自己遗忘，现将安装过程整理成文档。公司环境没有外网，只是配置了本地yum源，安装之前将所有的用到的包都预先整理了一下。为了方便大家阅读我稍后会将包结构给出，大家在阅读的时候有些地方请根据自己的环境进行替换。
airflow_ha_install/
├── airflow-packages //该目录下主要存放了airflow依赖的三方包，没有外网只能先下载下来搞咯
├── airflow-scheduler-failover-controller-master
│ ├── CHANGELOG.md
│ ├── License.md
│ ├── README.md
│ ├── scheduler_failover_controller
│ ├── scripts
│ ├── setup.cfg
│ └── setup.py
├── pip-9.0.1.tar.gz
├── rabbitmq
│ ├── erlang-19.3.6.4-1.el7.x86_64.rpm //这个包是我编译之后的包，大家可以通过下载源码自行编译
│ └── rabbitmq-server-3.7.4-1.el7.noarch.rpm
└── systemd
├── airflow
├── airflow.conf
├── airflow-flower.service
├── airflow-kerberos.service
├── airflow-scheduler.service
├── airflow-webserver.service
├── airflow-worker.service
└── README
环境规划
| host | IP | service |
|---|---|---|
| airflow-01 | 192.168.3.191 | airflow-worker/webserver/ASFC/rabbitmq |
| airflow-02 | 192.168.3.192 | airflow-worker/webserver/ASFC/rabbitmq |
| airflow-03 | 192.168.3.193 | airflow-worker/webserver/Haproxy |
准备环境
centos7
准备环境的操作需要以root用户完成
关闭防火墙和selinux
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
新建用户airflow 并赋予sudo权限
useradd airflow
安装依赖包
yum groupinstall "Development tools"
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel python-devel wget cyrus-sasl-devel.x86_64 libffi-devel python-psycopg2
安装pip
上传安装包到目标环境中的每台服务器
以root身份登录目标环境中的每台服务器执行如下命令
yum groupinstall "Development tools" -y
yum install python-setuptools python-devel -y
tar xzvf pip-9.0.1.tar.gz
cd pip-9.0.1
python setup.py install
安装rabbitmq_ha
单机上安装
分别在192.168.3.191 192.168.3.192 节点上安装rabbitmq server
- Install erlang
yum install airflow_ha_install/rabbitmq/erlang-19.3.6.4-1.el7.x86_64.rpm - Install RabbitMQ Server
yum install airflow_ha_install/rabbitmq/rabbitmq-server-3.7.4-1.el7.noarch.rpm - 启动
rabbitmq-server start - 设置开机自启动
systemctl enable rabbitmq-server - 启用插件rabbitmq management
rabbitmq-plugins enable rabbitmq_management 输入http://ip:15672可以登录管理界面，默认账户guest/guest只能使用http://localhost:15672登录，要想远程登录，需要添加一个新的用户： # rabbitmqctl add_user admin admin #用户设置为administrator才能远程访问 rabbitmqctl set_user_tags admin administrator rabbitmqctl set_permissions -p / admin ".*" ".*" ".*" //该命令使用户admin具有‘/’这个virtual host中所有资源的配置、写、读权限以便管理其中的资源,查看所有用户#rabbitmqctl list_users
集群搭建
#将192.168.3.191上的/var/lib/rabbitmq/.erlang.cookie复制到192 var/lib/rabbitmq/.erlang.cookie, 即服务器必须具有相同的cookie，如果不相同的话，无法搭建集群.
#192.168.3.192节点上分别执行命令，加入到集群
systemctl restart rabbitmq-server
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@airflow-01
rabbitmqctl start_app
#其中--ram代表是内存节点，如果希望是磁盘节点则不用加--ram，在rabbitmq集群中，至少需要一个磁盘节点
#查看集群的状态
rabbitmqctl cluster_status
设置成镜像队列
在192.168.3.191上执行
rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode": "automatic"}'
安装HAproxy
在192.168.3.191上安装HAproxy
yum install haproxy
vi /etc/haproxy/haproxy.cfg
#1.修改默认的模式为tcp
#2.在文件末尾添加如下内容：
# port forwarding from 8080 to the airflow webserver on 8080
listen impala
bind 0.0.0.0:8080
balance roundrobin
server airflow_webserver_1 auto-nn-01.embrace.com:8080 check
server airflow_webserver_2 auto-nn-02.embrace.com:8080 check
listen rabbitmq-web-ui
bind 0.0.0.0:15677
balance roundrobin
server rabbitmq_server_1 auto-cn-01.embrace.com:15672 check
server rabbitmq_server_2 auto-nn-01.embrace.com:15672 check
listen rabbitmq-ui
bind 0.0.0.0:5677
balance roundrobin
server rabbitmq_server_1 auto-cn-01.embrace.com:5672 check
server rabbitmq_server_2 auto-nn-01.embrace.com:5672 check
# This sets up the admin page for HA Proxy at port 1936.
listen stats :1936
mode http
stats enable
stats uri /
stats hide-version
stats refresh 30s
启动
systemctl start haproxy
关闭
systemctl stop haproxy
加入系统服务
systemctl enbale haproxy
安装配置airflow
安装
以root用户登录目标环境中的一台服务器，并执行如下命令进行安装
cd airflow_ha_install/airflow-packages
pip install --no-index --find-links . apache-airflow
pip install --no-index --find-links . apache-airflow[celery]
pip install --no-index --find-links . apache-airflow[crypto]
配置环境变量（AIRFLOW_HOME）并创建必要的目录
以airflow用户登录目标环境中的每一台服务器，执行如下命令
mkdir ~/airflow
# vi ~/.bash_profile 添加如下内容
export AIRFLOW_HOME=~/airflow
#载入环境变量
source ~/.bash_profile
#创建必须的目录
cd $AIRFLOW_HOME
mkdir dags
mkdir logs
配置airflow
该部分操作只需要在目标环境中的某一台服务器上执行即可，执行完成之后需将配置文件同步到其他服务器上
- 初始化数据库
airflow initdb
2.修改配置文件（$AIRFLOW_HOME/airflow.cfg）
* 修改执行器为CeleryExecutor
excutor=CeleryExcutor
* 修改数据库链接信息
sql_alchemy_conn = postgresql+psycopg2://airflow:airflow@$ip4addr:5432/airflow
* 设置dags初始化后为暂停状态
dags_are_paused_at_creation = True
* 不引用例子
load_examples = False
* 设置消息队列链接信息
broker_url = amqp://admin:admin@192.168.3.191:5677/
* 设置celery执行pg
celery_result_backend = db+postgresql+psycopg2://postgres:postgres@192.168.3.191:5432/airflow
* 再次运行 airflow initdb，创建相关表结构
airflow initdb
- 下发airflow.cfg
将airflow.cfg到集群中的每台机器上
将airflow加入系统服务
以root用户登录目标环境中每一台服务器，执行如下命令
- 拷贝service文件
cd airflow_ha_install/systemd cp airflow*.service /usr/lib/systemd/system chown airflow:airflow /usr/lib/systemd/system/airflow*.service #确保文件中airflow命令的路径是正确的 - copy the airflow.conf to /etc/tmpfiles.d/ or /usr/lib/tmpfiles.d/
cd airflow_ha_install/systemd cp airflow.conf /etc/tmpfiles.d/ - 确保 /run/airflow 存在，且所有者和所属组正确 (0755 airflow airflow)
- 拷贝airflow至/etc/sysconfig/airflow 并修改 AIRFLOW_HOME
cd airflow_ha_install/systemd cp airflow /etc/sysconfig/airflow - 加入到开机自启动
systemctl enable airflow-webserver systemctl enable airflow-wroker systemctl enable airflow-flower
安装airflow-scheduler-failover-controller
-
Install ASFC
cd airflow_ha_install/airflow-scheduler-failover-controller-master pip install -e . --no-index --find-links ../airflow-packages -
初始化
scheduler_failover_controller init -
更新airflow.cfg
scheduler_nodes_in_cluster = auto-nn-01,auto-nn-02 #it is recommended that you use the value printed from the following command:scheduler_failover_controller get_current_host 启动airflow免密登录
-
测试联通行
scheduler_failover_controller test_connection -
确保运行正常
scheduler_failover_controller metadata -
加入到系统服务
Login to each of the machines acting as Scheduler Failover Controllers
Login as root
Copy the scheduler_failover_controller.service file to the systemd directory. See the bellow list for the correct location based off your environment.
* /usr/lib/systemd/system/ for CentOS
* /lib/systemd/system/ for UbuntuEdit the scheduler_failover_controller.service and change any configurations you would like
a. Change user and group options as needed to ensure that it is the same as the user and group the main airflow processes are running asEnable the service to be ran on startup of the machine
systemctl enable scheduler_failover_controllerYou're done!
