博客
关于我
强烈建议你试试无所不能的chatGPT，快点击我
E-MapReduce集群如何使用Phoenix on HBase
阅读量：7062 次
发布时间：2019-06-28

本文共 9410 字，大约阅读时间需要 31 分钟。

        E-MapReduce的emr-2.0.0以下的HBase集群中并不包含Phoenix，也没有启动yarn，下面介绍一种通过创建集群时设置的引导操作来完成集群上Phoenix的部署。

  • HBase+Phoenix部署
  • HBase+Yarn+Phoenix部署

HBase+Phoenix部署

1.引导操作shell脚本(phoenix_bootstrap.sh)

入参:

参数名 取值 备注
regionId cn-hangzhou/cn-beijing 创建集群所在的regionId
isVpc 0/1 创建的是否是vpc集群 
#!/bin/bashregionId="$1"isVpc=$2echo $regionIdecho $isVpcif [ ! $regionId ]; then    regionId="cn-hangzhou"fiif [ ! $isVpc ]; then    isVpc=0fiisMaster=`hostname --fqdn | grep emr-header-1`masterIp=`cat /etc/hosts | grep emr-header-1 | awk '{print $1}'`bucket=""if [[ $regionId == "cn-hangzhou" ]]; then    bucket="emr-agent-pack"elif [[ $regionId == "cn-beijing" ]]; then    bucket="emr-bj"fiphoenixpackUrl="http://emr-agent-pack.oss-cn-hangzhou-internal.aliyuncs.com/phoenix/phoenix-4.7.0-HBase-1.1-bin.tar.gz"if [[ isVpc -eq 1 ]]; then    phoenixpackUrl="http://$bucket.vpc100-oss-$regionId.aliyuncs.com/phoenix/phoenix-4.7.0-HBase-1.1-bin.tar.gz"else    phoenixpackUrl="http://$bucket.oss-$regionId-internal.aliyuncs.com/phoenix/phoenix-4.7.0-HBase-1.1-bin.tar.gz"ficd /opt/appswget $phoenixpackUrltar xvf phoenix-4.7.0-HBase-1.1-bin.tar.gzrm -rf /opt/apps/phoenix-4.7.0-HBase-1.1-bin.tar.gzchown -R hadoop:hadoop /opt/apps/phoenix-4.7.0-HBase-1.1-bincp phoenix-4.7.0-HBase-1.1-bin/phoenix-4.7.0-HBase-1.1-server.jar /opt/apps/hbase-1.1.1/lib/chown hadoop:hadoop /opt/apps/hbase-1.1.1/lib/phoenix-4.7.0-HBase-1.1-server.jarif [[ $isMaster != "" ]]; then    echo "$masterIp emr-cluster" >>/etc/hosts    sed -i '$i
    
     \n
     
      hbase.master.loadbalancer.class
     \n
     
      org.apache.phoenix.hbase.index.balancer.IndexLoadBalancer
     \n
    \n
    \n
    
     \n
     
      hbase.coprocessor.master.classes
     \n
     
      org.apache.phoenix.hbase.index.master.IndexMasterObserver
     \n
    ' /etc/emr/hbase-conf/hbase-site.xmlelse  sed -i '$i
    \n
    
     \n
     
      hbase.regionserver.wal.codec
     \n
     
      org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
     \n
    \n
    \n
    
     \n
     
      hbase.region.server.rpc.scheduler.factory.class
     \n
     
      org.apache.hadoop.hbase.ipc.PhoenixRpcSchedulerFactory
     \n
     
      Factory to create the Phoenix RPC Scheduler that uses separate queues for index and metadata updates
     \n
    \n
    
     \n
     
      hbase.rpc.controllerfactory.class
     \n
     
      org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory
     \n
     
      Factory to create the Phoenix RPC Scheduler that uses separate queues for index and metadata updates
     \n
    \n
    \n
    
     \n
     
      hbase.coprocessor.regionserver.classes
     \n
     
      org.apache.hadoop.hbase.regionserver.LocalIndexMerger
     \n
    ' /etc/emr/hbase-conf/hbase-site.xmlfised -i '$a# ensure that we receive traces on the server\nhbase.sink.tracing.class=org.apache.phoenix.trace.PhoenixMetricsSink\n# Tell the sink where to write the metrics\nhbase.sink.tracing.writer-class=org.apache.phoenix.trace.PhoenixTableMetricsWriter\n# Only handle traces with a context of "tracing"\nhbase.sink.tracing.context=tracing' /etc/emr/hbase-conf/hadoop-metrics2-hbase.propertiesecho "export PATH=/opt/apps/phoenix-4.7.0-HBase-1.1-bin/bin:\$PATH" >>/etc/profile.d/hadoop.shecho "export HADOOP_CLASSPATH=/opt/apps/phoenix-4.7.0-HBase-1.1-bin/*:\$HADOOP_CLASSPATH" >>/etc/profile.d/hadoop.sh

2.OSS存储phoenix_bootstrap.sh

将1中的phoenix_bootstrap.sh脚本上传到OSS中，创建集群的时候需要从OSS中选择

3.创建HBase集群(添加引导操作)

在E-MapReduce中创建HBase集群，在创建集群的基本信息页面，点击添加引导操作，选择2中OSS的phoenix_bootstrap.sh脚本，并且根据需求填写可选参数(即1中介绍的脚本入参)，如下图所示(在杭州region创建classic集群)

phoenixbootstrap

phoe

4.验证

集群创建完成，状态显示为空闲后，登陆集群的master节点进行验证

1)示例1

执行:

sudo su hadoopperformance.py localhost 1000

输出(截取部分):

CSV Upsert complete. 1000 rows upsertedTime: 1.298 sec(s)                                COUNT(1)----------------------------------------                                    2000Time: 0.437 sec(s)HO--CSEUNATime: 0.2 sec(s)DOMAIN----------------------------------------Apple.comGoogle.comSalesforce.comTime: 0.079 sec(s)DAY-----------------------2016-06-13 00:00:00.000Time: 0.076 sec(s)                                COUNT(1)----------------------------------------                                      45Time: 0.068 sec(s)

2)示例2

执行:

sudo su hadoopcd ~echo '100,Jack,Doe' >>example.csvecho '200,Tony,Poppins' >>example.csv
sqlline.py localhost0: jdbc:phoenix:localhost> CREATE TABLE example (    my_pk bigint not null,    m.first_name varchar(50),    m.last_name varchar(50)     CONSTRAINT pk PRIMARY KEY (my_pk));psql.py -t EXAMPLE localhost example.csv

验证

sqlline.py localhost0: jdbc:phoenix:localhost> select * from example;
+--------+-------------+------------+| MY_PK  | FIRST_NAME  | LAST_NAME  |+--------+-------------+------------+| 100    | Jack        | Doe        || 200    | Tony        | Poppins    |+--------+-------------+------------+

HBase+Yarn+Phoenix部署

Phoenix也可以结合MapReduce来使用，需要额外部署Yarn。

1.引导操作shell脚本(yarn_bootstrap.sh)

E-MapReduce的HBase集群目前没有启动yarn,可通过引导操作来启动，脚本如下：

#!/bin/bashHADOOP_SOFT_LINK_CONF_DIR="/etc/emr/hadoop-conf"HDFS_HBASE_CONF_DIR="/etc/emr/hadoop-conf-2.6.0.hbase"if test -d ${HADOOP_SOFT_LINK_CONF_DIR};then    rm -rf ${HADOOP_SOFT_LINK_CONF_DIR}filn -s ${HDFS_HBASE_CONF_DIR} ${HADOOP_SOFT_LINK_CONF_DIR}isMaster=`hostname --fqdn | grep emr-header-1`masterIp=`cat /etc/hosts | grep emr-header-1 | awk '{print $1}'`
headerecsid=`cat /etc/hosts | grep 'emr-header-1' |awk '{print $4}'`sed -i "/
    
     /a 
     
      \n
      
       master_hostname
      \n
      
       $headerecsid
      \n
     " /etc/emr/hadoop-conf-2.6.0.hbase/core-site.xml sed -i 's/#yarn_nodemanager_resource_cpu_vcores#/6/g' /etc/emr/hadoop-conf-2.6.0.hbase/yarn-site.xmlsed -i 's/#yarn_nodemanager_resource_memory_mb#/13107/g' /etc/emr/hadoop-conf-2.6.0.hbase/yarn-site.xmlsed -i 's/#yarn_scheduler_minimum_allocation_mb#/2184/g' /etc/emr/hadoop-conf-2.6.0.hbase/yarn-site.xmlsed -i 's/\/opt\/apps\/extra-jars\/*/\/opt\/apps\/extra-jars\/*,\n\/opt\/apps\/phoenix-4.7.0-HBase-1.1-bin\//g' /etc/emr/hadoop-conf-2.6.0.hbase/yarn-site.xmlsed -i 's/#mapreduce_reduce_java_opts#/-Xmx2184m/g' /etc/emr/hadoop-conf-2.6.0.hbase/mapred-site.xmlsed -i 's/#mapreduce_map_memory_mb#/3208/g' /etc/emr/hadoop-conf-2.6.0.hbase/mapred-site.xmlsed -i 's/#mapreduce_reduce_memory_mb#/3208/g' /etc/emr/hadoop-conf-2.6.0.hbase/mapred-site.xmlsed -i 's/#mapreduce_map_java_opts#/-Xmx2184m/g' /etc/emr/hadoop-conf-2.6.0.hbase/mapred-site.xml
    
if [[ $isMaster != "" ]]; then    sed -i 's/#loop#{disk_dirs}\/mapred#,#/file:\/\/\/mnt\/disk1\/mapred/g' /etc/emr/hadoop-conf-2.6.0.hbase/mapred-site.xml    sed -i 's/#loop#{disk_dirs}\/yarn#,#/file:\/\/\/mnt\/disk1\/yarn/g' /etc/emr/hadoop-conf-2.6.0.hbase/yarn-site.xml    sed -i 's/#loop#{disk_dirs}\/log\/hadoop-yarn\/containers#,#/file:\/\/\/mnt\/disk1\/log\/hadoop-yarn\/containers/g' /etc/emr/hadoop-conf-2.6.0.hbase/yarn-site.xml     chmod 777 /mnt/disk1/log/     su -l hadoop -c '/opt/apps/hadoop-2.6.0/sbin/yarn-daemon.sh start resourcemanager'else    sed -i 's/#loop#{disk_dirs}\/mapred#,#/file:\/\/\/mnt\/disk4\/mapred,file:\/\/\/mnt\/disk1\/mapred,file:\/\/\/mnt\/disk3\/mapred,file:\/\/\/mnt\/disk2\/mapred/g' /etc/emr/hadoop-conf-2.6.0.hbase/mapred-site.xml    sed -i 's/#loop#{disk_dirs}\/yarn#,#/file:\/\/\/mnt\/disk4\/yarn,file:\/\/\/mnt\/disk1\/yarn,file:\/\/\/mnt\/disk3\/yarn,file:\/\/\/mnt\/disk2\/yarn/g' /etc/emr/hadoop-conf-2.6.0.hbase/yarn-site.xml    sed -i 's/#loop#{disk_dirs}\/log\/hadoop-yarn\/containers#,#/file:\/\/\/mnt\/disk4\/log\/hadoop-yarn\/containers,file:\/\/\/mnt\/disk1\/log\/hadoop-yarn\/containers,file:\/\/\/mnt\/disk3\/log\/hadoop-yarn\/containers,file:\/\/\/mnt\/disk2\/log\/hadoop-yarn\/containers/g' /etc/emr/hadoop-conf-2.6.0.hbase/yarn-site.xml    chmod 770 /mnt/disk1    chmod 770 /mnt/disk2    chmod 770 /mnt/disk3    chmod 770 /mnt/disk4    chmod 777 /mnt/disk1/log/    su -l hadoop -c '/opt/apps/hadoop-2.6.0/sbin/yarn-daemon.sh start nodemanager'    sed -i '$aif test -d "\/mnt\/disk1";then\nchmod 770 \/mnt\/disk1\nfi\nif test -d "\/mnt\/disk2";then\nchmod 770 \/mnt\/disk2\nfi\nif test -d "\/mnt/disk3";then\nchmod 770 /mnt\/disk3\nfi\nif test -d "\/mnt/disk4";then\nchmod 770 \/mnt\/disk4\nfi\nif test -d "\/mnt\/disk1\/log";then\nchmod 777 \/mnt\/disk1\/log\nfi' /usr/local/emr/emr-bin/script/hbase/sbin/4_start_hdfs.shfi

2.引导操作shell脚本(phoenix_bootstrap.sh)

跟HBase+Phoenix部署中的phoenix_bootstrap.sh一致。

3.OSS存储yarn_bootstrap.sh

将1中的yarn_bootstrap.sh脚本上传到OSS中，创建集群的时候需要从OSS中选择

4.创建HBase集群(添加引导操作)

在E-MapReduce中创建HBase集群，在创建集群的基本信息页面，点击添加引导操作先添加yarn_bootstrap.sh脚本，然后再点击添加引导操作来添加phoenix_bootstrap.sh，如下所示：

yarnbootstrap

phoenixyarn

5.验证

集群创建完成，状态显示为空闲后，登陆集群的master节点进行验证

1)示例1

执行:

sudo su hadoopcd ~echo '100,Jack,Doe' >>example.csvecho '200,Tony,Poppins' >>example.csvhadoop dfs -put example.csv /
sqlline.py localhost0: jdbc:phoenix:localhost> CREATE TABLE example (    my_pk bigint not null,    m.first_name varchar(50),    m.last_name varchar(50)     CONSTRAINT pk PRIMARY KEY (my_pk));
hadoop jar /opt/apps/phoenix-4.7.0-HBase-1.1-bin/phoenix-4.7.0-HBase-1.1-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool --input  /example.csv -z localhost -t example

验证

sqlline.py localhost0: jdbc:phoenix:localhost> select * from example;
+--------+-------------+------------+| MY_PK  | FIRST_NAME  | LAST_NAME  |+--------+-------------+------------+| 100    | Jack        | Doe        || 200    | Tony        | Poppins    |+--------+-------------+------------+

转载地址：http://acbll.baihongyu.com/

你可能感兴趣的文章
ssh事务回滚，纪念这几个月困扰已久的心酸
查看>>
jQuery中的编程范式
查看>>
比较快速排序，冒泡排序，双向冒泡排序的执行效率
查看>>
还没被玩坏的robobrowser(5)——Beautiful Soup的过滤器
查看>>
Linux 精准获取进程pid--转
查看>>
Servlet、Filter、Listener总结
查看>>
[翻译] JTBorderDotAnimation
查看>>
关于Thread类中三个interrupt方法的研究与学习（转）
查看>>
mysql 加入列，改动列，删除列。
查看>>
UML的学习
查看>>
置顶十三问
查看>>
备忘：spring jdbc事务代码 mybatis, nhibernate
查看>>
PHPCMS如何实现后台访问限制？
查看>>
网络直播电视之M3U8解析篇 （下）
查看>>
ContentProvider简单介绍
查看>>
SQL SERVER获取数据库中所有表名 XTYPE类型
查看>>
java设计模式3--单例模式（Singleton）
查看>>
MongoDB aggregate 运用篇(转)
查看>>
C#开发微信门户及应用(27)-公众号模板消息管理
查看>>
Java日志最佳实践
查看>>
喝酒易醉，品茶养心，人生如梦，品茶悟道，何以解忧？唯有杜康！-- 愿君每日到此一游！
当前时间: 2025-02-07 17:20:27   当前IP: 3.129.73.98   联系邮箱:javaeecc@qq.com     Copyright © 2020 - 2022 baihongyu.com 京ICP备2021015314号-2
强烈建议你试试无所不能的CHAT-GPT，快点击我
强烈建议你试试无所不能的CHAT-GPT，快点击我