大数据（十六）Hue

阅读量
× 文章目录
  1. 1. Quick Guide
    1. 1.1. 一、用途
    2. 1.2. 二、架构
    3. 1.3. 三、安装
      1. 1.3.1. 1.前置
      2. 1.3.2. 2.安装
      3. 1.3.3. 3.hue集成hadoop
      4. 1.3.4. 4.hue集成hive配置
      5. 1.3.5. 5.hue集成spark配置
      6. 1.3.6. 6.MySql初始化
      7. 1.3.7. 7.启动hue

Hue （Hadoop User Experience Hadoop用户体验）是一个开源的 Apache Hadoop UI系统，由Cloudera Desktop演化而来，最后Cloudera公司将其贡献给Apache基金会的Hadoop社区，它是基于Python Web框架Django实现的。通过使用HUE我们可以在浏览器端的Web控制台上与Hadoop集群进行交互，来分析处理数据。

本文章主要介绍 Hue 和 在centos环境下如何安装 。

Quick Guide

一、用途

  • 访问HDFS和文件浏览
  • 通过web调试和开发hive以及数据结果展示
  • 查询solr和结果展示，报表生成
  • 通过web调试和开发impala交互式SQL Query
  • spark调试和开发
  • Pig开发和调试
  • oozie任务的开发，监控，和工作流协调调度
  • Hbase数据查询和修改，数据展示
  • Hive的元数据（metastore）查询
  • MapReduce任务进度查看，日志追踪
  • 创建和提交MapReduce，Streaming，Java job任务
  • Sqoop2的开发和调试
  • Zookeeper的浏览和编辑
  • 数据库（MySQL，PostGres，SQlite，Oracle）的查询和展示

二、架构

从总体上来讲，Hue应用采用的是B/S架构，该web应用的后台采用python编程语言别写的。大体上可以分为三层，分别是前端view层、Web服务层和Backend服务层。Web服务层和Backend服务层之间使用RPC的方式调用。

三、安装

1.前置

  • 1.安装 python 和 maven
  • 2.安装 hadoop 和 hive
  • 3.创建应用类服务专有用户
1
useradd -g hadoop hue
  • 4.安装依赖包
1
yum -y install ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi gcc gcc-c++ krb5-devel libtidy libxml2-devel libxslt-devel openldap-devel python-devel sqlite-devel openssl-devel mysql-devel gmp-devel

2.安装

  • 2.1.下载安装包(官网或者github)
1
2
3
4
5
6
7
yum install git -y
cd /hadoop/install
git clone https://github.com/cloudera/hue.git branch-4.7


# 重命名
mv branch-4.7 hue

  • 2.2.切换用户 hue

    1
    2
    chown -R  hue:hadoop hue
    su hue

  • 2.3.编译

    1
    2
    cd hue
    make apps

  • 2.4.修改 /usr/local/hue/desktop/conf/pseudo-distributed.ini

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[desktop]
    # 安全秘钥，存储session的加密处理
    secret_key=dfsahjfhflsajdhfljahl
    # Time zone name
    time_zone=Asia/Shanghai
    # Enable or disable debug mode.
    django_debug_mode=false
    # Enable or disable backtrace for server error
    http_500_debug_mode=false
    # This should be the hadoop cluster admin
    ## default_hdfs_superuser=hdfs
    default_hdfs_superuser=root
    # 不启用的模块
    # app_blacklist=impala,security,rdbms,jobsub,pig,hbase,sqoop,zookeeper,metastore,indexer
[[database]]
       # 数据库引擎类型
       engine=mysql
       # 数据库主机地址
       host=master
       # 数据库端口
       port=3306
       # 数据库用户名
       user=root
       # 数据库密码
       password=123456
       # 数据库库名
       name=hue

3.hue集成hadoop

  • 3.1.修改配置文件，增加代理用户hue
    1
    vi $HADOOP_HOME/etc/hadoop/core-site.xml

新增内容：

1
2
3
4
5
6
7
8
<property>
<name>hadoop.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.hue.groups</name>
    <value>*</value>
</property>

  • 3.2.配置hue:修改 desktop/conf/pseudo-distributed.ini文件

修改标签 [[hdfs_clusters]] 如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[hadoop]

  # Configuration for HDFS NameNode
  # ------------------------------------------------------------------------
  [[hdfs_clusters]]
    # HA support by using HttpFs

    [[[default]]]
      # Enter the filesystem uri
      fs_defaultfs=hdfs://master:54310

      # NameNode logical name.
      logical_name=master

      # Use WebHdfs/HttpFs as the communication mechanism.
      # Domain should be the NameNode or HttpFs host.
      # Default port is 14000 for HttpFs.
      ## webhdfs_url=http://localhost:50070/webhdfs/v1
      webhdfs_url=http://master:9870/webhdfs/v1

      # Change this if your HDFS cluster is Kerberos-secured
      ## security_enabled=false

      # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs
      # have to be verified against certificate authority
      ## ssl_cert_ca_verify=True

      # Directory of the Hadoop configuration
      ## hadoop_conf_dir=$HADOOP_CONF_DIR when set or '/etc/hadoop/conf'
      hadoop_conf_dir=$HADOOP_CONF_DIR

修改标签 [[yarn_clusters]] 如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# Configuration for YARN (MR2)
  # ------------------------------------------------------------------------
  [[yarn_clusters]]

    [[[default]]]
      # Enter the host on which you are running the ResourceManager
      ## resourcemanager_host=localhost
      resourcemanager_host=master

      # The port where the ResourceManager IPC listens on
      ## resourcemanager_port=8032

      # Whether to submit jobs to this cluster
      submit_to=True

      # Resource Manager logical name (required for HA)
      ## logical_name=

      # Change this if your YARN cluster is Kerberos-secured
      ## security_enabled=false

      # URL of the ResourceManager API
      ## resourcemanager_api_url=http://localhost:8088
      resourcemanager_api_url=http://master:8088

      # URL of the ProxyServer API
      ## proxy_api_url=http://localhost:8088
      proxy_api_url=http://master:8088

      # URL of the HistoryServer API
      ## history_server_api_url=http://localhost:19888
      history_server_api_url=http://master:19888

      # URL of the Spark History Server
      ## spark_history_server_url=http://localhost:18088
      spark_history_server_url=http://master:18080

      # Change this if your Spark History Server is Kerberos-secured
      ## spark_history_server_security_enabled=false

      # In secure mode (HTTPS), if SSL certificates from YARN Rest APIs
      # have to be verified against certificate authority
      ## ssl_cert_ca_verify=True

4.hue集成hive配置

  • 4.1.拷贝配置 hive-site.xml 到目录

    1
    2
    3
    mkdir -p /usr/local/hue/hive/conf
    cp /hadoop/install/apache-hive-3.1.2/conf/hive-site.xml /usr/local/hue/hive
    /conf/

  • 4.2.修改desktop/conf/pseudo-distributed.ini的标签[beeswax]

1
2
3
4
5
6
7
[beeswax]
    # hiveServer2 服务地址(填主机名，kerberos要用)
    hive_server_host=master
    # hiveServer2服务端口
    hive_server_port=10000
    # hiveServer2 hive-site.xml配置文件存放位置
    hive_conf_dir=/usr/local/hue/hive/conf

5.hue集成spark配置

  • 5.1.启动spark的thrift server

    1
    2
    cd /hadoop/install/spark/sbin
    start-thriftserver.sh --master yarn --deploy-mode client

  • 5.2.安装livy

    1
    2
    3
    4
    5
    cd /hadoop/software
    wget http://livy.incubator.apache.org/download/apache-livy-0.6.0-incubating-bin
    .zip
    unzip apache-livy-0.6.0-incubating-bin.zip -C 
    mv apache-livy-0.6.0-incubating-bin /hadoop/install/livy-0.6.0

  • 5.3.创建livy的 livy-env.sh 配置文件和log目录

    1
    2
    3
    cd livy-0.6.0/conf/
    cp livy-env.sh.template livy-env.sh
    mkdir -p /data/livy/logs

  • 5.4.在 livy-env.sh 新建配置

    1
    2
    3
    4
    export HADOOP_CONF_DIR=/hadoop/install/hadoop/etc/hadoop
    export SPARK_HOME=/hadoop/install/spark
    export LIVY_LOG_DIR=/data/livy/logs
    export LIVY_PID_DIR=/data/livy/pid

  • 5.5.配置 livy.conf

    1
    cp livy.conf.template livy.conf

  • 5.6.在livy.conf文件中加入以下内容

    1
    2
    3
    4
    5
    6
    # What port to start the server on.
    livy.server.port = 8998
    # What spark master Livy sessions should use.
    livy.spark.master = yarn
    # What spark deploy mode Livy sessions should use.
    livy.spark.deploy-mode = client

  • 5.7.启动livy

    1
    /hadoop/install/livy-0.6.0/bin/livy-server start

  • 5.8.修改 desktop/conf/pseudo-distributed.ini 文件的标签[spark]

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    ###########################################################################
    # Settings to configure the Spark application.
    ###########################################################################

    [spark]
      # The Livy Server URL.
      ## livy_server_url=http://localhost:8998
      livy_server_url=http://master:8998

      # Configure Livy to start in local 'process' mode, or 'yarn' workers.
      ## livy_server_session_kind=yarn
      livy_server_session_kind=yarn

      # Whether Livy requires client to perform Kerberos authentication.
      ## security_enabled=false

      # Whether Livy requires client to use csrf protection.
      ## csrf_enabled=false

      # Host of the Sql Server
      ## sql_server_host=localhost
      sql_server_host=master

      # Port of the Sql Server
      ## sql_server_port=10000
      sql_server_port=10000

      # Choose whether Hue should validate certificates received from the server.
      ## ssl_cert_ca_verify=true

    ###########################################################################

6.MySql初始化

  • 1.建一个名为hue的库

    1
    2
    3
    4
    5
    6
    7
    8
    9
    # 登录mysql数据库
    mysql -u root -p
    # 创建数据库hue
    create database hue;
    # 创建用户
    create user 'hue'@'%' identified by '123456';
    # 授权
    grant all privileges on hue.* to 'hue'@'%';
    flush privileges;

  • 2.生成表

    1
    2
    build/env/bin/hue syncdb
    build/env/bin/hue migrate

7.启动hue

  1. 启动hue
    1
    build/env/bin/supervisor &

扫一扫，分享到微信

微信分享二维码

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

A Code Farmer.