impala+kudu

虚拟机环境配置

虚拟机设置启动代理
软件安装都需要使用apt-get 需要用到代理，设置代理如下：
vim /etc/profile.d/proxy.sh
export httpprxoy=http://dev-proxy.oa.com:8080/
export httpsprxoy=http://dev-proxy.oa.com:8080/
export ftp_prxoy=http://dev-proxy.oa.com:8080/
设置JDK信息
查看已有的JDK信息若系统自带则卸载重装
查看当前系统JDK情况

whereis java
which java （java执行路径）
echo $JAVA_HOME echo$ PATH
卸载openJDK

sudo apt-get remove openjdk*
2.在oracle官网上下载目标JDK文件，将下载的tar.gz文件移动到想存放的目录下，输入下面命令即可 tar -xzvf jdk-XXX-linux-x64.tar.gz
安装配置JDK

vim /etc/profile 然后source /etc/profile export JAVAHOME=/usr/local/java/jdk1.8.0191 export JREHOME= ${JAVAHOME}/jre export CLASSPATH=.:$ {JAVAHOME}/lib: ${JREHOME}/lib export PATH=$ {JAVA_HOME}/bin:$PATH
大数据架构搭建
kudu安装
若系统级代理没有设置，可以直接apt-get install是单独设置
sudo apt-get install kudu -o Acquire::http::proxy="http://dev-proxy.oa.com:8080/"
sudo apt-get install kudu-master -o Acquire::http::proxy="http://dev-proxy.oa.com:8080/"
sudo apt-get install kudu-tserver -o Acquire::http::proxy="http://dev-proxy.oa.com:8080/"
sudo apt-get install libkuduclient0 -o Acquire::http::proxy="http://dev-proxy.oa.com:8080/"
sudo apt-get install libkuduclient-dev -o Acquire::http::proxy="http://dev-proxy.oa.com:8080/"
impala安装（未完待续）
impala依赖hive依赖MySQL、hadoop，Hadoop依赖ssh、git等（依赖环境搭建好之后后面有相关的配置内容）
流程
impala的配置JDK地址：日志信息：
/var/log/impala/(默认目录)
配置脚本：

vim /etc/default/bigtop-utils
export JAVAHOME=/home/local/jdk1.8.0131/
vim /etc/default/impala
IMPALACATALOGSERVICEHOST=192.168.0.116
IMPALASTATESTOREHOST=192.168.0.116
cd /etc/impala/conf.dist
cp -a /opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop/core-site.xml
cp -a /opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop/hdfs-site.xml
MySQL安装
5.7社区版本安装
sudo apt-get install mysql-server mysql-client
(5.7社区版本用户名密码root 1234)
mysql -u root -p 登录验证 mysql-connector配置
sudo cp mysql-connector-java8.0.12-1ubuntu16.04all.deb将对应的jar包复制到 /usr/share/java/ 放置到对应目录
Hive安装
安装hive-metastore 并且配置mysql为数据库hive3.00版本的metastore
1、apache-hive-3.0.0-bin.tar.gz* http://mirror.bit.edu.cn/apache/hive/
2、文档路径https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-Requirements
3、脚本执行&环境变量配置
tar -xzvf hive-xyztar.gz
cd hive-xyz
export HIVEHOME={{/home/sherrichen/dataplatform/hive/apache-hive-3.0.0-bin}}
export PATH= $HIVE_HOME/bin:$ PATH
echo $PATH 查看修改在mysql中建立相应的元数据数据库，执行相应的sql脚本，并且建立供hive连接mysql的用户，赋予其权限。$ mysql -u root –p 1234
mysql> CREATE DATABASE metastore;
mysql> USE metastore;
mysql> SOURCE /home/sherrichen/data_platform/hive/apache-hive-metastore-3.0.0-bin/scripts/metastore/ upgrade/mysql/hive-schema-3.0.0.mysql.sql; //执行hive相关的建表内容
根据apache-hive-metastore-3.0.0-bin/scripts/metastore/upgrade/mysql/目录里面创建用户的脚本创建用户
CREATE USER 'sherrichen'@'localhost' IDENTIFIED BY '1234';
CREATE USER 'sherrichen'@'%' IDENTIFIED BY '1234';
REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'sherrichen'@'localhost';
REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'sherrichen'@'%';
GRANT ALL PRIVILEGES ON metastore.* TO 'sherrichen'@'localhost';
GRANT ALL PRIVILEGES ON metastore.* TO 'sherrichen'@'%';
FLUSH PRIVILEGES;
-Red 碰到问题
问题1：
CREATE USER 'sherrichen'@'localhost' IDENTIFIED BY '1234'; ERROR 3009 (HY000): Column count of mysql.user is wrong. Expected 45, found 42. Created with MySQL 50561, now running 50723. Please use mysql_upgrade to fix this error
解决方式:
exit mysql 然后 mysql_upgrade -u root -p XXX 进行更新DB
Hadoop安装
流程
1、下载hadoop（hive依赖hadoop的环境运行）
2、检查JDK版本和hadoop匹配性https://wiki.apache.org/hadoop/HadoopJavaVersions
3、hadoop版本和hive版本的匹配关系：http://hive.apache.org/downloads.html
4、必须安装ssh并且必须运行sshd才能使用管理远程Hadoop守护程序的Hadoop脚本（如果要使用可选的启动和停止脚本）。此外，建议还安装pdsh以实现更好的ssh资源管理
sudo apt-get install ssh
sudo apt-get install pdsh
5、修改Java_home的环境
etc/hadoop/hadoop-env.sh
JAVAHOME=/usr/local/java/jdk1.8.0191
6、bin/hadoop 可以查看hadoop操作使用
7、配置独立部署模式
$mkdir input$ cp etc/hadoop/.xml input $bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar grep input output 'dfs[a-z.]+'$ cat output/
7、检查无密码登录SSH
ssh localhost
8、免密设置脚本
ssh-keygen -t rsa -P '' -f ~/.ssh/idrsa cat ~/.ssh/idrsa.pub >> ~/.ssh/authorizedkeys chmod 0600 ~/.ssh/authorizedkeys
9、格式化&启动
bin/hdfs namenode -format
10、启动NameNode守护程序和DataNode守护程序：
sbin / start-dfs.sh 脚本里面依赖项配置
export HADOOPHOME=/home/sherrichen/dataplatform/hadoop/hadoop-3.1.1
export HADOOPCONFDIR= ${HADOOPHOME}/etc/hadoop export HADOOPCOMMONLIBNATIVEDIR=$ {HADOOPHOME}/lib/native
export HADOOPOPTS="-Djava.library.path= ${HADOOPHOME}/lib" export HADOOPMAPREDHOME=$ {HADOOPHOME} export HIVEHOME=/home/sherrichen/dataplatform/hive/apache-hive-3.0.0-bin
export HIVECONFDIR= ${HIVEHOME}/conf export CLASSPATH=.:$ {HIVEHOME}/lib: $CLASSPATH export PATH=.:$ {HADOOPHOME}/sbin: ${HIVE_HOME}/bin:$ PATH
JAVAHOME,HADOOPCOMMONHOME,HADOOPHDFSHOME,HADOOPCONFDIR,CLASSPATHPREPEND_DISTCACHE,HADOOPYARNHOME,HADOOPMAPREDHOME<
启动成功之后可以通过浏览器访问网页，可以看到相应的资源管理界面
NameNode - http://localhost:9870/ 默认 hdfs://localhost:9000
11、YARN on a Single Node
配置HADOOPMAPREDHOME 信息 export HADOOP_MAPREDHOME= ${HADOOPHOME} 碰到问题问题1： hadoop-env.sh: line 36: hdfs: command not found 解决方式: vim /etc/profile 然后source /etc/profile export PATH=/home/sherrichen/data_platform/hadoop/hadoop-3.1.1/bin:$ PATH
问题2：
sherrichen@ubuntu:~/dataplatform/hadoop/hadoop-3.1.1 $sudo sbin/start-dfs.sh Starting namenodes on [localhost] ERROR: Attempting to operate on hdfs namenode as root ERROR: but there is no HDFSNAMENODEUSER defined. Aborting operation. Starting datanodes ERROR: Attempting to operate on hdfs datanode as root ERROR: but there is no HDFSDATANODEUSER defined. Aborting operation. Starting secondary namenodes [ubuntu] ERROR: Attempting to operate on hdfs secondarynamenode as root ERROR: but there is no HDFSSECONDARYNAMENODE_USER defined. Aborting operation. 解决方式: 修改sbin/start-dfs.sh sbin/stop-dfs.sh 增加内容如下 HDFSDATANODEUSER=root HDFSDATANODESECUREUSER=hdfs HDFSNAMENODEUSER=root HDFSSECONDARYNAMENODE_USER=root 问题3： sherrichen@ubuntu:~/dataplatform/hadoop/hadoop-3.1.1$ sudo sbin/start-dfs.sh
WARNING: HADOOPSECUREDNUSER has been replaced by HDFSDATANODESECUREUSER. Using value of HADOOPSECUREDNUSER.
Starting namenodes on [localhost]
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
localhost: Permission denied (publickey,password).
pdsh@ubuntu: localhost: ssh exited with exit code 255
Starting datanodes
localhost: Permission denied (publickey,password).
pdsh@ubuntu: localhost: ssh exited with exit code 255
Starting secondary namenodes [ubuntu]
ubuntu: Warning: Permanently added 'ubuntu' (ECDSA) to the list of known hosts.
ubuntu: Permission denied (publickey,password).
pdsh@ubuntu: ubuntu: ssh exited with exit code 255
解决方式:
sbin/start-dfs.sh 不能使用sudo 方式启动会导致用户不识别
问题4：
pdsh@xxx: localhost: connect: Connection refused
解决方式:
原因是pdsh默认采用的是rsh登录，修改成ssh登录即可，在环境变量/etc/profile里加入 export PDSHRCMDTYPE=ssh 再source profile生效。然后再启动查看jps即可看到hdfs服务启动。
问题5：
sudo sbin/start-dfs.sh
问题：WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
解决方式:
export HADOOPOPTS="-Djava.library.path= $HADOOPHOME/lib:$ HADOOPCOMMONLIBNATIVEDIR"
问题6：
执行 hadoop fs -ls 异常
问题sherrichen@ubuntu:~/data_platform/hadoop/hadoop-3.1.1 $hadoop fs -lsls: Call From ubuntu/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 解决方式: 1、查看netstat -an|grep 9000 是否启动，实际是服务未启动导致 2、最初碰到这个问题，是根据网上的错误资料导致将端口9000和9870配置到/etc/ssh/sshdconfig，然后telnet可以公共connect，但最后服务启动只有，导致访问localhost:9000出现了重定向到www.localhost.com 并且提示protocol mismatch 导致的Error信息：SSH-2.0-OpenSSH6.6.1p1 Ubuntu-2ubuntu2.10 Protocol mismatch. 问题7： sherrichen@ubuntu:~/data_platform/hadoop/hadoop-3.1.1$ sbin/stop-dfs.sh
Stopping namenodes on [localhost]
pdsh@ubuntu: localhost: rcmd: socket: Permission denied
Stopping datanodes
pdsh@ubuntu: localhost: rcmd: socket: Permission denied
Stopping secondary namenodes [ubuntu]
解决方式:
vim ~/.bashrc,add export PDSHRCMDTYPE=ssh and source ~/.bashrc
问题8：
wget http://localhost:9870/ --2018-10-18 04:30:01-- http://localhost:9870/ Resolving dev-proxy.oa.com (dev-proxy.oa.com)... 10.14.87.100 Connecting to dev-proxy.oa.com (dev-proxy.oa.com)|10.14.87.100|:8080... connected. Proxy request sent, awaiting response... 504 Gateway Time-out 2018-10-18 04:30:02 ERROR 504: Gateway Time-out.
解决方式:
/etc/profile.d/proxy.sh中增加代理
export no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"
问题9：
hadoop上传文件错误org.apache.hadoop.ipc.RemoteException(java.io.IOException)
需要配置datanode\namenode信息，否则datanode实际是启动失败的目录可以自己建立然后配置即可

core-site.xml
fs.defaultFS hdfs://localhost:9000 hadoop.tmp.dir /home/sherrichen/data_platform/hadoop/hadoop-3.1.1/tmp

hdfs-site.xml
dfs.replication 1 dfs.namenode.name.dir /home/sherrichen/dataplatform/hadoop/hadoop-3.1.1/namenode dfs.datanode.data.dir /home/sherrichen/dataplatform/hadoop/hadoop-3.1.1/datanode