HIVE - HADOOP : INSTALLATION , EXECUTION OF SAMPLE PROGRAM - Part II

HIVE - HADOOP : INSTALLATION , EXECUTION  OF SAMPLE PROGRAM - Part II

Hive Clients

To Start Hive Server use following command .. 

$hive --service hiveserver

Following are different Hive Clients, to connect to this Server

1) CommandLine

2) Thrift Client

3) JDBC Driver

4) ODBC Driver

Courtesy : HadoopDefinativeGuide

1) CommandLine : Operates in embedded mode only, it needs to have access to the hive libraries.

This is section is briefly described in my previous article ( Hive - Part I ) .

2) Thrift Client : 

The Apache Thrift software framework, for scalable cross-language services development, combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, OCaml and Delphi and other languages. Thrift Software

Hive Server is implemented in java, so to query hive server using ruby,php,C++  language then you need to build thrift client specific to that language.

Steps to build Thrift Client for Ruby

a) Install Thrift Thrift

b)  Download Thrift Source Thrift Source

c) Navigate to Hive Source directory 

thrift —gen rb -I service/include metastore/if/hive_metastore.thrift 
thrift —gen rb -I service/include -I . service/if/hive_service.thrift 
thrift —gen rb service/include/thrift/fb303/if/fb303.thrift 
thrift —gen rb serde/if/serde.thrift 
thrift —gen rb ql/if/queryplan.thrift 
thrift —gen rb service/include/thrift/if/reflection_limited.thrift

Or you can download Thrift client for ruby from Thrift Ruby Client

Thrift Java Client : operates in embedded mode and standalone server

Thrift C++ Client : operated only in embedded mode

3) JDBC Driver :  

For embedded mode, uri is just "jdbc:hive://". 

For standalone server, uri is "jdbc:hive://host:port/dbname" where host and port are determined by where the hive server is run. 

Example :  "jdbc:hive://localhost:10000/default". Currently, the only dbname supported is "default".

JDBC Client Sample Code JDBC Client Sample Code

4) ODBC Driver : 

The Hive ODBC Driver is a software library that implements the Open Database Connectivity (ODBC) API standard for the Hive database management system, enabling ODBC compliant applications to interact seamlessly (ideally) with Hive through a standard interface. This driver will NOT be built as a part of the typical Hive build process and will need to be compiled and built separately ODBC Client

The Metastore :


The metastore is the central repository of Hive metadata. The metastore is divided into two pieces: a service and the backing store for the data. By default, the metastore service runs in the same JVM as the Hive service and contains an embedded Derby database instance backed by the local disk. This is called the embedded metastore configuration

Disadvantage with Embedded metastore is only one Hive Session can be opened against Hive Server, For multiple Hive Session support move metastore to a Relational Database which supports JDO (Derby Server,MYSQL ..) 

MetaStore : Derby Server

wget http://archive.apache.org/dist/db/derby/db-derby-10.4.2.0/db-derby-10.4.2.0-bin.tar.gz

tar -xzf db-derby-10.4.2.0-bin.tar.gz


mv db-derby-10.4.2.0-bin derby


mkdir derby/data


Login as root, add follwing variables in /etc/profile.d/derby.sh



export DERBY_INSTALL=/home/HADOOP/derby
export DERBY_HOME=/home/HADOOP/derby



cd /home/HADOOP/derby/bin/

./startNetworkServer -h 0.0.0.0 &

/scratch/rjuluri/HADOOP/hive/conf

vi hive-site.xml

  hive.test.mode.nosamplelist

 

  if hive is running in test mode, dont sample the above comma seperated list of tables

  hive.metastore.local

  true

  controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM

  javax.jdo.option.ConnectionURL

jdbc:derby://hostname:port/metastore_db;create=true

  JDBC connect string for a JDBC metastore

  javax.jdo.option.ConnectionDriverName

org.apache.derby.jdbc.ClientDriver

  Driver class name for a JDBC metastore

Add following file 

vi /home/HADOOP/hive/conf/jpox.properties

javax.jdo.PersistenceManagerFactoryClass=org.jpox.PersistenceManagerFactoyImpl

org.jpox.autoCreateSchema=false

org.jpox.validateTables=false

org.jpox.validateColumns=false

org.jpox.validateConstraints=false

org.jpox.storeManagerType=rdbms

org.jpox.autoCreateSchema=true

org.jpox.autoStartMechanismMode=checked

org.jpox.transactionIsolation=read_committed

javax.jdo.option.DetachAllOnCommit=true

javax.jdo.option.NontransactionalRead=true

javax.jdo.option.ConnectionDriverName=org.apache.derby.jdbc.ClientDriver

javax.jdo.option.ConnectionURL=jdbc:derby://hostname:port/metastore_db;create=true

javax.jdo.option.ConnectionUserName=APP

javax.jdo.option.ConnectionPassword=mine

cp /home/HADOOP/derby/lib/derbyclient.jar  /home/HADOOP/hive/lib

cp  /home/HADOOP/derby/lib/derbytools.jar  /home/HADOOP/hive/lib

Hive> show tables;

OK

Time taken: 0.048 seconds

Execute max_cgpa.q script (Previous article Hive - Part I )

[root@slc01mcd hive-0.9.0]# hive -f max_cgpa.q


Hadoop job information for null: number of mappers: 0; number of reducers: 0
2012-07-18 07:01:23,161 null map = 0%,  reduce = 0%
2012-07-18 07:01:26,174 null map = 100%,  reduce = 0%
2012-07-18 07:01:29,186 null map = 100%,  reduce = 100%
Ended Job = job_local_0001
Execution completed successfully
Mapred Local Task Succeeded . Convert the Join into MapJoin
OK


cse     8.6
ece     9.0


Time taken: 10.476 seconds

Hive> show tables;

OK

maxcgpa1

Time taken: 2.769 seconds