Install Oozie - you must know

When I want to install Oozie for my hadoop environment , I encounter many problem issues. So I decide to write down here to record.

Install Oozie, something you must do first.

I highly recommend you adduser different from root, and add this user  to sudo group, because oozie can't accept you start hdfs using root user.

1.Install Java, I recommend you use JDK1.7+
export your JAVA_HOME & PATH

2.Install Maven, I use version 3.3.3
export your MAVEN_HOME & PATH

3.Install Hadoop, I use version 2.7
export your HADOOP_HOME & PATH

Someone maybe say it needs to install pig, but I try it without using pig.

OK, now we just start install Oozie.

tar -zxvf oozie-4.1.0.tar.gz

cd oozie-4.1.0

vi pom.xml

change it from 1.6 to 1.7(your version)

<targetJavaVersion>1.7</targetJavaVersion>

bin/mkdistro.sh -DskipTests -Dhadoop.version=2.3.0
(skip test, because build oozie install tar need about 10 minutes.)

Oh No, you execute this command and last it tells you failure.
because some jars are empty.

In my case, I missed three jars.
There are jackson-core-asl-1.5.2.jar & jackson-mapper-asl-1.5.2.jar & jsr311-api-0.11.jar

please go to download 

http://www.java2s.com/Code/Jar/j/Downloadjacksoncoreasl152jar.htm

http://www.java2s.com/Code/Jar/j/Downloadjacksonmapperasl152jar.htm

http://www.java2s.com/Code/Jar/j/Downloadjsr311api011jar.htm

and put them to correct path under oozie

Now, you can rebuild to execute command (bin/mkdistro.sh -DskipTests -Dhadoop.version=2.3.0)

If you build it successful, you can find oozie-4.1.0-distro.tar.gz under /oozie-4.1.0/distro/target , this file is a real oozie intall file.

go to /oozie-4.1.0/distro/target and continue to do

tar -zxvf  oozie-4.1.0-distro.tar.gz

Now you can find oozie-4.1.0 folder under /oozie-4.1.0/distro/target

and you must copy hadoop reference lib to oozie-4.1.0(this folder under /oozie-4.1.0/distro/target)

I recommend you to move this folder to other path, ex: /usr/local/oozie-4.1.0 (to diff from first you tar oozie-4.1.0.tar.gz)

go to /usr/local/oozie-4.10 (tar from oozie-4.1.0-distro.gz)

mkdir libext  (/usr/local/oozie-4.1.0/libext)

cp /usr/local/hadoop/share/hadoop/*/*.jar libext/
cp /usr/local/hadoop/share/hadoop/*/lib/*.jar libext/

cd libext

rm -f  jsp-api-2.1.jar  (because this is oozie tomcat lib, but hadoop also have so to conflict.)

download  ext-2.2.zip to libext

wget wget http://dev.sencha.com/deploy/ext-2.2.zip

#Be careful , this download link from apache is fail, your must use I give you above.

This file is used to build oozie Web UI.

Now go to your $HADOOP_HOME/etc/hadoop
Add properties to core-site.xml

<property>
   <name>hadoop.proxyuser.oozie.hosts</name>
   <value>*</value>
</property>
<property>
   <name>hadoop.proxyuser.oozie.groups</name>
   <value>*</value>

</property>

#Maybe /usr/local/oozie-4.10/conf/hadoop-conf also has core-site.xml, you can add again.

cd /usr/local/oozie-4.1.0

bin/oozie-setup.sh prepare-war
(must be successful, because this is oozie web war)

bin/oozie-setup.sh db create -run
(this step is create database and table for oozie using derby including on install file)

Now , start HDFS

start-dfs.sh
(you must can see namenode,datanode,secondarynamenode process, and start by other user as you install oozie that is diff root)

bin/oozie-setup.sh  sharelib  create -fs hdfs://hostname:8020
(change hostname or port for mapping your configuration on your environment )

Now you can start Oozie Server
bin/oozied.sh start

bin/oozie admin -oozie http://localhost:11000/oozie -status
util appear System mode : Normal

Yes, try to open your browser to connect http://hostname:11000/oozie
util you can see web user interface.

Congratulations 

Oracle + Sqoop + HDFS + Hive + Derby + Spark 實際測試範例

建置完所有需要的系統之後，我們終於可以實際的進入測試階段了
當然，每個系統建置完都先實際的測試一下才繼續下一個安裝

以下我將Demo一個實際的範例，內容包括
1. Sqoop import Oracle DB table and data into HDFS & Hive
2.Test Data in Hive and HDFS
3.Query Data in Hive
4.Spark call Hive Data to be a DataFrame, and using DataFrame to show data

在實際執行範例之前，我們必須先把 HDFS , YARN, Derby, Hive測試能成功啟動
Hadoop: 至  $HADOOP_HOME/sbin 執行  start-all.sh (會啟動 HDFS & YARN)
Derby:至  $DERBY_HOME/bin 執行 startNetworkServer -noSecurityManager (可背景執行)
(建議把這些執行的command放到 bashrc的 PATH中，否則一直切換目錄很麻煩)

測試 HDFS and YARN 是否成功:
執行 hadoop dfsadmin -report ($HADOOP_HOME/bin下)或是至 http://nameNodeIP:50070 可觀察是否有2個節點
(註: 如果只用jps 觀察是不夠精準的，原因是NameNode server can start DataNode, but sometimes DataNode can't connect NameNode)

測試 Hive是否可以connect derby後正常啟動

結果如下:


 



確定環境均可正常服務後，我們準備開始測試嘍!

1. Sqoop import oracle database table and data to HDFS &Hive
利用 Sqoop 將 Oracle 裡的 Table & Data 同時 import 至 HDFS & Hive
需要先做兩件事
(1)建立 HDFS 目錄 (執行 sqoop command 時需指定目的地的 HDFS 目錄，附上check是否建成功的command)


(2)Download ojdb6_g.jar 或 ojdbc6.jar 至 $SQOOP_HOME/lib



執行 Sqoop import 嘍! 以後指令以我的測試資料為準，請依你的資料修改指令

sqoop import --connect jdbc:oracle:thin:ds_ams/ds_ams@192.168.11.6:1521:tcffsn --username ds_ams -P --table ams_rank --hive-import --columns "RANK_ID,RANK_NAME,RANK_LEVEL,CREATE_USER,CREATE_TIME,MODIFY_USER,MODIFY_TIME" --target-dir /user/hive/warehouse/ams_rank -m 1

以下過程只附上幾個較為重要的關鍵





2.Test Data in Hive and HDFS




3. Query Data in Hive



4.Spark call Hive Data to be a DataFrame, and using DataFrame to show data










恭喜你，成功嘍~*

Install Hive and derby on HDFS 注意事項

安裝 Hive 需要安裝一個database，參考以下2個網址均是使用 Derby
http://www.tutorialspoint.com/hive/hive_installation.htm
https://cwiki.apache.org/confluence/display/Hive/HiveDerbyServerMode

但若照著做總是會出現出乎意料的問題

1. Derby 無法啟動
    原因: 假設出現 package steal timer 等字眼，很有可能是你的 ClassPath 重複使用了
derbytools.jar ，根據 第二個網址 請您把 derbyclient.jar & derbytools.jar from $DERBY_HOME/lib to $HADOOP_HOME/lib
這個動作不需要做，因為第一個網址已經把 derbytools.jar 放置在 ~/.bashrc 的 CLASSPATH

2.Derby 無法啟動
    原因:因為找不到 jdbc driver
請至 hive-site.xml，加入 如下(第一篇教學網址沒說)




<property>

  <name>javax.jdo.option.ConnectionDriverName</name>

  <value>org.apache.derby.jdbc.ClientDriver</value>

  <description>Driver class name for a JDBC metastore</description>

</property>

  

2.啟動 Hive 的時候 出現 connection refuse by port 1527
    原因: 這並不是因為 Derby is not running，而是因為在 java 預設啟動時有Security policy
，請至 $JAVA_HOME/jre/lib/security 目錄下，編輯  java.policy ，在最下面加入一行

    permission java.net.SocketPermission "localhost:1527", "listen,resolve";

如圖


參考 http://stackoverflow.com/questions/21154400/unable-to-start-derby-database-from-netbeans-7-4

3.在 $HIVE_HOME下執行 bin/hive 仍無法 work
 出現 

java.net.URISyntaxException when starting HIVE

原因是因為，原本我們的 hive-site.xml本來就是複製而來，Hive啟動時需要一個 tmp的目錄

因此修改檔案內容為，當然我假設你在 hdfs 已有建立一個名為 /tmp 的目錄

<name>hive.exec.scratchdir</name>
<value>/tmp/hive-${user.name}</value>

 <name>hive.exec.local.scratchdir</name>
 <value>/tmp/${user.name}</value>

<name>hive.downloaded.resources.dir</name>
<value>/tmp/${user.name}_resources</value>

<name>hive.scratch.dir.permission</name>
    <value>733</value>

參考 http://stackoverflow.com/questions/27099898/java-net-urisyntaxexception-when-starting-hive

注意事項:啟動 Hive 之前 HDFS 要可以 work  , Hive 會 connect 至 NameNode

安裝Spark on Hive注意事項:
1.複製 $Hive_HOME/conf/hive-site.xml to $SPARK_HOME/conf
啟動 spark-shell後會出現

Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.apache.derby.jdbc.AutoloadedDriver40

Solution:這個原因是 spark call hive , but hive need derby 缺少 derby.jar 檔在 Hive 的 Libery
請至$DERBY_HOME/lib下 複製 derby.jar 至 $HIVE_HOME/lib下 重新啟動 spark-shell

CentOS 建置 HDFS 注意事項 different from Ubuntu

最近用CentOS 建置 HDFS 時，遇到了在Ubuntu沒過到的問題，在此記錄一下

1. Install JDK
    如果安裝後，在~/.bashrc下加入了Java_Home & PATH後
即使 source ~/.bashrc後，java路徑仍是舊的路徑時，應該使用

sudo update-alternatives --install "/usr/bin/java" "java" "/usr/local/java/jdk1.8.0_51/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/local/java/jdk1.8.0_51/bin/javac" 1
sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/local/java/jdk1.8.0_51/bin/javaws" 1

sudo update-alternatives --set java /usr/local/java/jdk1.8.0_51/bin/java
sudo update-alternatives --set javac /usr/local/java/jdk1.8.0_51/bin/javac
sudo update-alternatives --set javaws /usr/local/java/jdk1.8.0_51/bin/javaws

2. SSH issue
    一般來說 在 Ubuntu 是用指令

1. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
2. 但是在 CentOS 雖然上面指令可以執行，但實際卻無法 work, ssh登入別台仍需password

應改為使用 ssh-copy-id hadooop@yourhostname 

該指令一樣會將 public key 寫入一個自動新增的 authorized_key 檔案

3.即使整個 HDFS  Conf 都設置無誤後，在  start-dfs.sh 及 start-yarn.sh 後

雖然 每台的 service 均有啟動(使用 jps monitor)，但卻發生 datanode can't connect server(NameNode)

首先在  NameNode使用 指令

netstat -ntulp | grep java

確定 Hadoop 的 IPC service 是否有啟動，一般預設為9000 or 8020(Cloudera),請觀察你的  core-site.xml 

 此時 binding 的 IP 應該為 yourIP 而不是 0.0.0.0. or 127.0.0.1

如果確定 服務是有啟動的且IP為真實的IP，則很有可能是firewall的問題，

因為 CentOS Linux 預設 firewall 是啟動的 ，請下指令試著將 firewall 關閉

service iptables stop

PS:使用 hdfs put file 至 DataNode時，DataNode的  firewall 也需關掉

如果每次都要改覺得麻煩的話，請至

vi /etc/rc.d/rc.local
新增 service iptables stop

#此設定會在開機時去關閉firewall

當然 更好的做法是，你會設定firewall ，讓 hdfs 彼此溝通的 port 可以 pass就好

如此一來，NameNode在 http://yourIP:50070 應該可以Monitor到你的  DataNode 了

1.