360SDN.COM

首页/Hadoop/列表

Hortonworks Hadoop 大数据平台构建

来源:时空大数据社区  2017-09-19 09:36:52    评论:0点击:

一、环境准备

1、在VMware中安装CentOS-7-x86_64-Minimal-1708 虚拟机,

。。。。。。。自行百度!

2、安装JDK,

sudo apt-get purge openjdk*

tar -zxvf jdk*

mv jdk* jdk

cd /usr/local/

mkdir java

cd ~

mv jdk /usr/local/java/

export JAVA_HOME=/usr/local/java/jdk

export JRE_HOME=${JAVA_HOME}/jre

exportCLASSPATH=.:${JRE_HOME}/lib:${JAVA_HOME}/lib

export PATH=$PATH:${JRE_HOME}/bin:${JAVA_HOME}/bin

echo "exportJAVA_HOME=$JAVA_HOME">>/etc/profile

echo "exportJRE_HOME=$JRE_HOME">>/etc/profile

echo "exportCLASSPATH=$CLASSPATH">>/etc/profile

echo "exportPATH=$PATH">>/etc/profile

update-alternatives --remove-all"java"

update-alternatives --remove-all"javac"

update-alternatives --remove-all"javaws"

update-alternatives --install"/usr/bin/java" "java""/usr/local/java/jdk/bin/java" 1

update-alternatives --install"/usr/bin/javac" "javac""/usr/local/java/jdk/bin/javac" 1

update-alternatives --install"/usr/bin/javaws" "javaws""/usr/local/java/jdk/bin/javaws" 1

update-alternatives --set java/usr/local/java/jdk/bin/java

update-alternatives --set javac/usr/local/java/jdk/bin/javac

update-alternatives --set javaws/usr/local/java/jdk/bin/javaws

java -version

3、规划主机名称和IP地址

        Master.hufeihu.com 192.168.248.128

       Server1.hufeihu.com192.168.248.129

       Server2.hufeihu.com192.168.248.130

4、设置Master虚拟机的静态IP地址,其他的两台机器同样的道理

    IPADDR=192.168.248.130

    NETMASK=255.255.255.0

    GATEWAY=192.168.248.2

    DNS1=210.77.69.1

    DNS2=210.77.69.2

    重启网卡:systemctlrestart  network.service

以上参数在VMware的网络设置中查看

5、然后安装yum -yinstall vim, yum -y install net-tools,主要是查看网络连接

6、克隆三台虚拟机,然后取名为Master,Server1,Server2

7、永久改变主机名,分别在三台机器上在/etc/sysconfig/network中写入一下内容

    NETWORKING=yes

    HOSTNAME=Master.hufeihu.com

    NETWORKING=yes

    HOSTNAME=Server1.hufeihu.com

    NETWORKING=yes

    HOSTNAME=Server2.hufeihu.com

8、然后在三台机器的/etc/hosts中配置主机名和IP地址对应

     168.248.130Server2.hufeihu.com

     168.248.129Server1.hufeihu.com

     168.248.128Master.hufeihu.com

9、防火墙,SELinux,时间同步,三台机器同时执行一下命令

关闭防火墙:systemctl disable firewalld

            servicefirewalld stop 

                   systemctlstop firewalld.service #停止firewall

systemctl disable firewalld.service #禁止firewall开机启动

       关闭SELinux

                  永久关闭:vi/etc/selinux/config 

SELINUX=enforcing改为SELINUX=disabled

临时关闭:setenforce 0

安装时间同步:yum install -y ntp,

systemctl is-enabled ntpd

systemctl enable ntpd

systemctl start ntpd

查看时钟同步结果:ntpdate pool.ntp.org

10、设置umask

umask 0022

永久有效:echo umask 0022 >>/etc/profile

11、ssh免密钥登陆

ssh-keygen

cat id_rsa.pub >> authorized_keys

chmod 700 ~/.ssh

chmod 600 ~/.ssh/authorized_keys

(1)    先自身免密钥登陆

(2)    相互之间免密钥登陆,把id_rsa.pub追加到对方的authorized_keys中。

(3)    cat id_rsa.pub >> ~/.ssh/authorized_keys

上述问题中,重启电脑之后有可能要重新启动的服务,

systemctl enable ntpd

systemctl start ntpd

 

 二、安装Ambari-server

1、配置yum源

Centos 7:

yum install wget

wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.1.0/ambari.repo-O /etc/yum.repos.d/ambari.repo

2、安装Ambari-server

yum install ambari-server

3、设置Ambari-server

ambari-server setup

4、启动Ambari-server

ambari-server start

ambari-server status

ambari-server stop

5、安装MySQL

配置yum 源:

yum localinstall \

https://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm

安装MySQL

yum installmysql-community-server

启动MySQL

systemctl startmysqld.service

获得随机root密码

grep 'Atemporary password is generated for root@localhost' \

/var/log/mysqld.log |tail -1

获取密码后登陆MySQL

mysql -u root -p

重新设置root用户密码

set global validate_password_policy=0;

set global validate_password_length=1;

ALTER USER 'root'@'localhost' IDENTIFIED BY 'hufeihu';

6、配置MySQL生成SAM and Schema相关元数据表

mysql -u root -p

create database registry;

create database streamline;

 CREATE USER 'registry'@'%' IDENTIFIED BY 'hufeihu';

CREATE USER 'streamline'@'%' IDENTIFIED BY 'hufeihu';)

(注意此处的密码必须改为自己的设置的MySQL root用户密码)

GRANT ALL PRIVILEGES ON registry.* TO 'registry'@'%' WITH GRANTOPTION ;

GRANT ALL PRIVILEGES ON streamline.* TO 'streamline'@'%'WITH GRANT OPTION ;

7、配置MySQL 产生Druid and Superset元数据表

mysql -u root -p

 CREATE DATABASE druid DEFAULT CHARACTER SET utf8;

CREATE DATABASE superset DEFAULT CHARACTER SET utf8;

 CREATE USER 'druid'@'%' IDENTIFIED BY 'hufeihu';

CREATE USER 'superset'@'%' IDENTIFIED BY 'hufeihu';

 GRANT ALL PRIVILEGES ON *.* TO 'druid'@'%' WITH GRANT OPTION;

GRANT ALL PRIVILEGES ON *.* TO 'superset'@'%' WITH GRANTOPTION;

8、输入master IP

http://192.168.248.128:8080进入登录ambari

输入密码和用户:admin/admin,登入安装界面。

9、添加主机,这一步至关重要,这一步一定要选ssh私钥登陆,要不然后面没法注册机器,把私钥从相关服务器下载下来,然后添加到框中,可以直接复制粘贴私钥内容。

这一个步骤中Master节点也必须添加到主机中。

10、等待安装安装ambari-agent在相关的机器上

11、选择3台机器开始安装

12、选择要安装的服务

13、部署节点安装位置

14、红色标注的为手动输入项,输入相关的密码,注意Druid输入的是之前的MySQL密码

15、根据所有的警告信息配置相关的存储大小,一般情况为设置的值的大小。

16、开始整个集群的安装过程,采用网络源速度较慢。

17、终于大公告成,可以嗨起来了,先启动HDFS, MapReduce, HBase体验一下吧。

18、最屌的大数据可视化分析平台,Apache Zepplin

永远相信,美好即将来临!!

阅读原文

为您推荐

友情链接 |九搜汽车网 |手机ok生活信息网|ok生活信息网|ok微生活
 Powered by www.360SDN.COM   京ICP备11022651号-4 © 2012-2016 版权