doris数据库详解（生产Doris系列之集群部署完整版）

生产Doris系列之集群部署完整版1.准备IPHostName角色192.168.1.141ruozedata001fe(leader)、be192.168.1.142ruozedata002fe(follower)、be192.16。

1.准备

IP	HostName	角色
192.168.1.141	ruozedata001	fe(leader)、be
192.168.1.142	ruozedata002	fe(follower)、be
192.168.1.143	ruozedata003	fe(follower)、be
192.168.1.144	ruozedata004	fe(observer)、be

1.1 查看centos系统 7.1及以上[root@ruozedata001 ~]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@ruozedata001 ~]# 1.2 JDK版本在1.8及以上[root@ruozedata001 ~]# mkdir /usr/java[root@ruozedata001 ~]# tar -xzvf jdk-8u181-Linux-x64.gz -C /usr/java/切记必须修正所属⽤户及⽤户组[root@ruozedata001 ~]# chown -R root:root /usr/java/jdk1.8.0_181配置[root@ruozedata001 ~]# vi /etc/proFileexport JAVA_HOME=/usr/java/jdk1.8.0_181export PATH=/usr/java/jdk1.8.0_181/bin:${PATH}生效[root@ruozedata001 ~]# source /etc/profile[root@ruozedata001 ~]# which java/usr/java/jdk1.8.0_181/bin/java[root@ruozedata001 ~]# 1.3 GCC版本在4.8.2 及以上[root@ruozedata001 ~]# gcc --version-bash: gcc: command not found如果没有gcc命令，执行以下命令查看版本[root@ruozedata001 ~]# ll/usr/libexec/gcc/x86_64-redhat-linux/total 4drwxr-xr-x. 2 root root 4096 Feb32021 4.8.2lrwxrwxrwx1 root root5 Dec72020 4.8.5 -> 4.8.21.4 文件系统最好是ext4在安装操作系统的时候，请选择ext4文件系统。[root@ruozedata001 ~]# df -TFilesystemType1K-blocksUsed Available Use% Mounted ondevtmpfsdevtmpfs8124216081242160% /devtmpfstmpfs8133996081339960% /dev/shmtmpfstmpfs8133996884481251521% /runtmpfstmpfs8133996081339960% /sys/fs/cgroup/dev/vda2ext4(这里)37024300 235228441161422467% /tmpfstmpfs1626800016268000% /run/user/0cm_processestmpfs8133996681281271841% /run/cloudera-scm-agent/process1.5 时钟同步Doris 的元数据要求时间精度要小于5000ms，所以所有集群所有机器要进行时钟同步，避免因为时钟问题引发的元数据不一致导致服务出现异常。#所有节点安装ntp[root@ruozedata001 ~]# yum install -y ntp#选取ruozedata001为ntp的主节点[root@ruozedata001 ~]# vi /etc/ntp.conf #timeserver 0.asia.pool.ntp.orgserver 1.asia.pool.ntp.orgserver 2.asia.pool.ntp.orgserver 3.asia.pool.ntp.org#当外部时间不可⽤时，可使⽤本地硬件时间server 127.127.1.0 iburst local clock #允许哪些⽹段的机器来同步时间restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap#开启ntpd及查看状态[root@ruozedata001 ~]# systemctl start ntpd[root@ruozedata001 ~]# systemctl status ntpd● ntpd.service - Network Time Service Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-05-11 10:15:00 CST; 11min ago Main PID: 18518 (ntpd) CGroup: /system.slice/ntpd.service!"18518 /usr/sbin/ntpd -u ntp:ntp -gMay 11 10:15:00 ruozedata001 systemd[1]: Starting Network Time Service...May 11 10:15:00 ruozedata001 ntpd[18518]: proto: precision = 0.088 usecMay 11 10:15:00 ruozedata001 ntpd[18518]: 0.0.0.0 c01d 0d kern kernel time sync enabledMay 11 10:15:00 ruozedata001 systemd[1]: Started Network Time Service.#验证[root@ruozedata001 ~]# ntpq -p remote refid st t when poll reach delay offset jitter============================================================================== LOCAL(0) .LOCL. 10 l 726 64 0 0.000 0.000 0.000#其他从节点停⽌禁⽤ntpd服务[root@ruozedata002 ~]# systemctl stop ntpd[root@ruozedata002 ~]# systemctl disable ntpdRemoved symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.[root@ruozedata002 ~]# /usr/sbin/ntpdate ruozedata00111 May 10:29:22 ntpdate[9370]: adjust time server 192.168.1.141 offset 0.000867 sec#每天凌晨同步ruozedata001节点时间[root@ruozedata002 ~]# crontab -e00 00 * * * /usr/sbin/ntpdate ruozedata001 [root@ruozedata003 ~]# systemctl stop ntpd[root@ruozedata003 ~]# systemctl disable ntpdRemoved symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.[root@ruozedata003 ~]# /usr/sbin/ntpdate ruozedata00111 May 10:29:22 ntpdate[9370]: adjust time server 192.168.1.141 offset 0.000867 sec#每天凌晨同步ruozedata001节点时间[root@ruozedata003 ~]# crontab -e00 00 * * * /usr/sbin/ntpdate ruozedata001[root@ruozedata004 ~]# systemctl stop ntpd[root@ruozedata004 ~]# systemctl disable ntpdRemoved symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.[root@ruozedata003 ~]# /usr/sbin/ntpdate ruozedata00111 May 10:29:22 ntpdate[9370]: adjust time server 192.168.1.141 offset 0.000867 sec#每天凌晨同步ruozedata001节点时间[root@ruozedata004 ~]# crontab -e00 00 * * * /usr/sbin/ntpdate ruozedata0011.6 swap关闭Linux交换分区会给Doris带来很严重的性能问题，需要在安装之前禁用交换分区不重启电脑，禁用启用swap，立刻生效[root@ruozedata001 ~]# free -mtotalusedfreesharedbuff/cacheavailableMem:15886107717281643874851Swap:4095 04095[root@ruozedata001 ~]# swapoff -a[root@ruozedata001 ~]# free -mtotalusedfreesharedbuff/cacheavailableMem:15886107697301643864853Swap:0 0 0重启，永久禁用Swapswap这行前面加上#注释掉[root@ruozedata001 ~]# vi /etc/fstab## /etc/fstab# Created by anaconda on Tue Sep 12 00:20:17 2017## Accessible filesystems, by reference, are maintained under '/dev/disk'# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info#UUID=51c8bb02-2d2a-4f2f-b823-f8c974300184 /ext3defaults1 1#UUID=4104bb80-c9e6-440c-a9e0-5606951284f9 swapswapdefaults0 0重启生效，查看[root@ruozedata001 ~]# reboot[root@ruozedata001 ~]# free -m1.7 系统优化参数设置系统最大打开文件句柄数和进程数echo "* soft nofile 960000" >> /etc/security/limits.confecho "* hard nofile 960000" >> /etc/security/limits.confecho "* soft nproc 960000" >> /etc/security/limits.confecho "* hard nproc 960000" >> /etc/security/limits.conf重启生效[root@ruozedata001 ~]# reboot[root@ruozedata001 ~]# ulimit -n 960000[root@ruozedata001 ~]#

2.下载

wget https://archive.apache.org/dist/doris/1.1/1.1.0-rc05/apache-doris-1.1.0-bin-x86-jdk8.tar.gz

3.部署

3.1 解压，重命名[root@ruozedata001 ~]# tar -xzvf apache-doris-1.1.0-bin-x86-jdk8.tar.gz -C /opt/[root@ruozedata001 ~]# cd /opt/[root@ruozedata001 opt]# ln -sapache-doris-1.1.0-bin-x86-jdk8 doris[root@ruozedata001 opt]# lltotal 16drwxr-xr-x710081008 4096 Jul 26 11:04 apache-doris-1.1.0-bin-x86-jdk8drwxr-xr-x8 cloudera-scm cloudera-scm 4096 Jul 25 15:28 clouderadrwxr-xr-x3 rootroot4096 Jul 25 15:02 cloudera-managerlrwxrwxrwx1 rootroot 31 Jul 26 11:04 doris -> apache-doris-1.1.0-bin-x86-jdk8drwxr-xr-x. 2 rootroot4096 Oct 312018 rh[root@ruozedata001 opt]# cd doris3.2 创建meta文件夹及编辑fe参数文件[root@ruozedata001 doris]# mkdir meta[root@ruozedata001 doris]# cd fe/conf/[root@ruozedata001 conf]# vi fe.conf #元数据路径meta_dir = /opt/doris/meta#当前机器的IPpriority_networks = 192.168.1.141/243.3 scp整个文件夹到另外2个机器[root@ruozedata001 opt]# scp -r doris ruozedata002:/opt/[root@ruozedata001 opt]# scp -r doris ruozedata003:/opt/[root@ruozedata001 opt]# scp -r doris ruozedata004:/opt/3.4 ruozedata002机器修改fe配置文件[root@ruozedata002 ~]# cd /opt/doris/fe/conf[root@ruozedata002 conf]# vi fe.conf #元数据路径meta_dir = /opt/doris/meta#当前机器的IPpriority_networks = 192.168.1.142/243.5 ruozedata003机器修改fe配置文件[root@ruozedata003 ~]# cd /opt/doris/fe/conf[root@ruozedata003 conf]# vi fe.conf #元数据路径meta_dir = /opt/doris/meta#当前机器的IPpriority_networks = 192.168.1.143/243.6 ruozedata004机器修改fe配置文件[root@ruozedata004 ~]# cd /opt/doris/fe/conf[root@ruozedata004 conf]# vi fe.conf #元数据路径meta_dir = /opt/doris/meta#当前机器的IPpriority_networks = 192.168.1.144/24---------------------------------------------------------3.6 ruozedata001~ruozedata004配置be[root@ruozedata001 doris]# mkdir data01 data02 data03#3块磁盘[root@ruozedata001 doris]# vi be/conf/be.conf #当前机器的IPpriority_networks = 192.168.1.141/24#数据存储目录storage_root_path = ${DORIS_HOME}/data01;${DORIS_HOME}/data02;${DORIS_HOME}/data03[root@ruozedata002 doris]# mkdir data01 data02 data03#3块磁盘[root@ruozedata002 doris]# vi be/conf/be.conf #当前机器的IPpriority_networks = 192.168.1.142/24#数据存储目录storage_root_path = ${DORIS_HOME}/data01;${DORIS_HOME}/data02;${DORIS_HOME}/data03[root@ruozedata003 doris]# mkdir data01 data02 data03#3块磁盘[root@ruozedata003 doris]# vi be/conf/be.conf #当前机器的IPpriority_networks = 192.168.1.143/24#数据存储目录storage_root_path = ${DORIS_HOME}/data01;${DORIS_HOME}/data02;${DORIS_HOME}/data03[root@ruozedata004 doris]# mkdir data01 data02 data03#3块磁盘[root@ruozedata004 doris]# vi be/conf/be.conf #当前机器的IPpriority_networks = 192.168.1.144/24#数据存储目录storage_root_path = ${DORIS_HOME}/data01;${DORIS_HOME}/data02;${DORIS_HOME}/data03

4.FE FOLLOWER节点启动，作为master

第一台作为master[root@ruozedata001 ~]# start_fe.sh --daemon客户端登录，添加FOLLOWER和OBSERVER节点[root@ruozedata001 ~]# /usr/local/MySQL/bin/mysql -h ruozedata001 -P 9030 -uroot #直接回车刚开始无密码即可登录，后续可设置Welcome to the mysql monitor.Commands end with ; or \g.Your MySQL connection id is 0Server version: 5.7.37 Doris version 1.1.0-rc05-UnknownCopyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.#添加mysql>ALTER SYSTEM ADD FOLLOWER "ruozedata002:9010";mysql>ALTER SYSTEM ADD FOLLOWER "ruozedata003:9010";mysql>ALTER SYSTEM ADD OBSERVER "ruozedata004:9010";#查看mysql> show proc '/frontends'; ---------------------------------- --------------- -------------- ------------- ---------- ----------- --------- ---------- ---------- ----------- ------- ------- ------------------- --------------------- ---------- -------------------------------------------------------------------- -------------------- ------------------ | Name| IP| HostName| EditLogPort | HttpPort | QueryPort | RpcPort | Role| IsMaster | ClusterId | Join| Alive | ReplayedJournalId | LastHeartbeat| IsHelper | ErrMsg | Version| CurrentConnected | ---------------------------------- --------------- -------------- ------------- ---------- ----------- --------- ---------- ---------- ----------- ------- ------- ------------------- --------------------- ---------- -------------------------------------------------------------------- -------------------- ------------------ | 192.168.1.141_9010_1658812495496 | 192.168.1.141 | ruozedata001 | 9010| 8030| 9030| 9020| FOLLOWER | true| 80066555| true| true| 322| 2022-07-26 14:32:12 | true|| 1.1.0-rc05-Unknown | Yes|| 192.168.1.142_9010_1658817055629 | 192.168.1.142 | ruozedata002 | 9010| 8030| 0| 0| FOLLOWER | false| 80066555| false | false | 0| NULL| true| java.net.ConnectException: Connection refused (Connection refused) | NULL| No|| 192.168.1.143_9010_1658817055629 | 192.168.1.143 | ruozedata003 | 9010| 8030| 0| 0| FOLLOWER | false| 80066555| false | false | 0| NULL| true| java.net.ConnectException: Connection refused (Connection refused) | NULL| No|| 192.168.1.144_9010_1658817081961 | 192.168.1.144 | ruozedata004 | 9010| 8030| 0| 0| OBSERVER | false| 80066555| false | false | 0| NULL| false| java.net.ConnectException: Connection refused (Connection refused) | NULL| No| ---------------------------------- --------------- -------------- ------------- ---------- ----------- --------- ---------- ---------- ----------- ------- ------- ------------------- --------------------- ---------- -------------------------------------------------------------------- -------------------- ------------------

5.FE FOLLOWER、OBSERVER节点启动

[root@ruozedata002 ~]# start_fe.sh --helper ruozedata001:9010 --daemon[root@ruozedata003 ~]# start_fe.sh --helper ruozedata001:9010 --daemon[root@ruozedata004 ~]# start_fe.sh --helper ruozedata001:9010 --daemon--helper 参数仅在 follower 和 observer 第一次启动时才需要。查看这2台的fe状态mysql> show proc '/frontends'; ---------------------------------- --------------- -------------- ------------- ---------- ----------- --------- ---------- ---------- ----------- ------ ------- ------------------- --------------------- ---------- -------- -------------------- ------------------ | Name| IP| HostName| EditLogPort | HttpPort | QueryPort | RpcPort | Role| IsMaster | ClusterId | Join | Alive | ReplayedJournalId | LastHeartbeat| IsHelper | ErrMsg | Version| CurrentConnected | ---------------------------------- --------------- -------------- ------------- ---------- ----------- --------- ---------- ---------- ----------- ------ ------- ------------------- --------------------- ---------- -------- -------------------- ------------------ | 192.168.1.141_9010_1658812495496 | 192.168.1.141 | ruozedata001 | 9010| 8030| 9030| 9020| FOLLOWER | true| 80066555| true | true| 387| 2022-07-26 14:35:47 | true|| 1.1.0-rc05-Unknown | Yes|| 192.168.1.142_9010_1658817055629 | 192.168.1.142 | ruozedata002 | 9010| 8030| 9030| 9020| FOLLOWER | false| 80066555| true | true| 385| 2022-07-26 14:35:47 | true|| 1.1.0-rc05-Unknown | No|| 192.168.1.143_9010_1658817055629 | 192.168.1.143 | ruozedata003 | 9010| 8030| 9030| 9020| FOLLOWER | false| 80066555| true | true| 385| 2022-07-26 14:35:47 | true|| 1.1.0-rc05-Unknown | No|| 192.168.1.144_9010_1658817081961 | 192.168.1.144 | ruozedata004 | 9010| 8030| 9030| 9020| OBSERVER | false| 80066555| true | true| 385| 2022-07-26 14:35:47 | false|| 1.1.0-rc05-Unknown | No| ---------------------------------- --------------- -------------- ------------- ---------- ----------- --------- ---------- ---------- ----------- ------ ------- ------------------- --------------------- ---------- -------- -------------------- ------------------ 3 rows in set (0.02 sec)mysql>

6.客户端登录新增BE记录

mysql> ALTER SYSTEM ADD BACKEND "ruozedata001:9050";mysql> ALTER SYSTEM ADD BACKEND "ruozedata002:9050";mysql> ALTER SYSTEM ADD BACKEND "ruozedata003:9050";mysql> ALTER SYSTEM ADD BACKEND "ruozedata004:9050";

7.BE启动及查看

[root@ruozedata001 ~]# start_be.sh --daemon[root@ruozedata002 ~]# start_be.sh --daemon[root@ruozedata003 ~]# start_be.sh --daemon[root@ruozedata004 ~]# start_be.sh --daemon客户端登录查看mysql> show proc '/backends'; ----------- ----------------- --------------- -------------- --------------- -------- ---------- ---------- --------------------- --------------------- ------- ---------------------- ----------------------- ----------- ------------------ --------------- --------------- --------- ---------------- -------------------------- -------- -------------------- ------------------------------------------------------------------------------------------------------------------------------- | BackendId | Cluster| IP| HostName| HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime| LastHeartbeat| Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | Tag| ErrMsg | Version| Status| ----------- ----------------- --------------- -------------- --------------- -------- ---------- ---------- --------------------- --------------------- ------- ---------------------- ----------------------- ----------- ------------------ --------------- --------------- --------- ---------------- -------------------------- -------- -------------------- ------------------------------------------------------------------------------------------------------------------------------- | 11001| default_cluster | 192.168.1.141 | ruozedata001 | 9050| 9060| 8040| 8060| 2022-07-26 15:13:05 | 2022-07-26 15:15:06 | true| false| false| 0| 0.000| 31.134 GB| 105.927 GB| 70.61 % | 70.61 %| {"location" : "default"} || 1.1.0-rc05-Unknown | {"lastSuccessReportTabletsTime":"2022-07-26 15:14:13","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} || 11002| default_cluster | 192.168.1.142 | ruozedata002 | 9050| 9060| 8040| 8060| 2022-07-26 15:14:57 | 2022-07-26 15:15:06 | true| false| false| 0| 0.000| 61.977 GB| 105.927 GB| 41.49 % | 41.49 %| {"location" : "default"} || 1.1.0-rc05-Unknown | {"lastSuccessReportTabletsTime":"2022-07-26 15:15:02","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} || 11003| default_cluster | 192.168.1.143 | ruozedata003 | 9050| 9060| 8040| 8060| 2022-07-26 15:13:54 | 2022-07-26 15:15:06 | true| false| false| 0| 0.000| 61.979 GB| 105.927 GB| 41.49 % | 41.49 %| {"location" : "default"} || 1.1.0-rc05-Unknown | {"lastSuccessReportTabletsTime":"2022-07-26 15:14:59","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} || 11004| default_cluster | 192.168.1.144 | ruozedata004 | 9050| 9060| 8040| 8060| 2022-07-26 15:13:54 | 2022-07-26 15:15:06 | true| false| false| 0| 0.000| 61.979 GB| 105.927 GB| 41.49 % | 41.49 %| {"location" : "default"} || 1.1.0-rc05-Unknown | {"lastSuccessReportTabletsTime":"2022-07-26 15:14:59","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | ----------- ----------------- --------------- -------------- --------------- -------- ---------- ---------- --------------------- --------------------- ------- ---------------------- ----------------------- ----------- ------------------ --------------- --------------- --------- ---------------- -------------------------- -------- -------------------- ------------------------------------------------------------------------------------------------------------------------------- 3 rows in set (0.00 sec)

8.web

open http://ruozedata001:8030 admin/无密码

注: 外网，先去安全组开放8030端口号