Impala build steps on CentOS 6.3

• Build boost-1.42.0
• Before building impala
  • change be/CMakeLists.txt. I removed all boost RPMs and built boost libraries from sources. boost_date_time will be /usr/local/lib/libboost_date_time-mt.*. The build failed without this change. If you have boost RPMs 1.41 installed, you may not need to change this. But the build will fail with other issues.

    diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
    index c14bd31..cd5abac 100644
    --- a/be/CMakeLists.txt
    +++ b/be/CMakeLists.txt
    @@ -224,7 +224,7 @@ set (IMPALA_LINK_LIBS
       ${LIBZ}
       ${LIBBZ2}
       ${AVRO_STATIC_LIB}
    -  -lrt -lboost_date_time
    +  -lrt -lboost_date_time-mt
     )
     
     if ("${CMAKE_BUILD_TYPE}" STREQUAL "CODE_COVERAGE")
    

  • change build_public.sh to build release version and don't have to put -build_thirdparty in command line:

    diff --git a/build_public.sh b/build_public.sh
    index 6ea491b..28b445a 100755
    --- a/build_public.sh
    +++ b/build_public.sh
    @@ -23,8 +23,8 @@ set -e
     # Exit on reference to uninitialized variable
     set -u
     
    -BUILD_THIRDPARTY=0
    -TARGET_BUILD_TYPE=Debug
    +BUILD_THIRDPARTY=1
    +TARGET_BUILD_TYPE=Release
     
     for ARG in $*
     do
    

• After building, run shell/make_shell_tarball.sh. This can generate a shell/build dir to have all files for impala-shell.
• Prepare hadoop, hbase and hive config files, copy from /var/run/cloudera-scm-agent/process.
• change bin/set-classpath.sh like this

  CLASSPATH=\
  $HOME/hadoop/hadoop-conf:\
  $HOME/hadoop/hbase-conf:\
  $HOME/hadoop/hive-conf:\
  #$IMPALA_HOME/fe/src/test/resources:\
  #$IMPALA_HOME/fe/target/classes:\
  #$IMPALA_HOME/fe/target/dependency:\
  #$IMPALA_HOME/fe/target/test-classes:\
  $IMPALA_HOME/fe/target/impala-frontend-0.1-SNAPSHOT.jar:\
  ${HIVE_HOME}/lib/datanucleus-core-2.0.3.jar:\
  ${HIVE_HOME}/lib/datanucleus-enhancer-2.0.3.jar:\
  ${HIVE_HOME}/lib/datanucleus-rdbms-2.0.3.jar:\
  ${HIVE_HOME}/lib/datanucleus-connectionpool-2.0.3.jar:
  
  for jar in `ls ${IMPALA_HOME}/fe/target/dependency/*.jar`; do
    CLASSPATH=${CLASSPATH}:$jar
  done
  
  export CLASSPATH
  

  Otherwise, you might see if you don't include impala-frontend.jar

  Exception in thread "main" java.lang.NoClassDefFoundError: com/cloudera/impala/common/JniUtil
  Caused by: java.lang.ClassNotFoundException: com.cloudera.impala.common.JniUtil
  

  Or this if you don't have hadoop-conf in the path

  E0605 09:32:23.236434  5272 impala-server.cc:377] Unsupported file system. Impala only supports DistributedFileSystem but the LocalFileSystem was found. fs.defaultFS(file:///) might be set incorrectly
  E0605 09:32:23.236655  5272 impala-server.cc:379] Impala is aborted due to improper configurations.
  

• Impalad_flags

  -beeswax_port=21001
  -fe_port=21001
  -be_port=22001
  -hs2_port=21051
  -enable_webserver=true
  -mem_limit=-1
  -webserver_port=25001
  -state_store_subscriber_port=23001
  -default_query_options
  -log_filename=impalad
  -use_statestore=false
  -nn=5K04.corp.pivotlink.com
  -nn_port=8020
  

• create a tarball of impala build because no such a open-source script.

  tar zcvf impala.tar.gz impala --exclude="*.class" --exclude="*.o" --exclude="impala/thirdparty" --exclude="impala/.git" --exclude="*.java" --exclude="*.cpp" --exclude="*.h" --exclude="expr-test"

• start impalad

  cd impala_home
  export IMPALA_HOME=$PWD
  bin/start-impalad.sh -build_type=release --flagfile=impalad_flags_path
  

• start impala-shell

  cd impala_home
  export IMPALA_HOME=$PWD
  export IMPALA_SHELL_HOME=$PWD/shell/build/impala-shell-1.0.1
  $IMPALA_SHELL_HOME/impala-shell -i impalad-host:21001
  

Build boost for Impala in CentOS 6.3

CentOS 6.3 has only rpm for boost_1.41.0 at the time I made the build. I had to build boost from source by myself.

• Clean up the old installation. Find all boost installations, then remove all old versions.

$ rpm -qa | grep boost
$ yum remove boost boost-filesystem ...

• Download boost tarball and expand into a directory.
• Make a build. Using tagged layout, this will generate /usr/local/lib/libboost_filesystem-mt.so. If you use --layout=system, /usr/local/lib/libboost_filesystem.so is created. Don't use --build_type=complete because the build takes too long.

cd boost_1.42.0
./bootstrap
sudo ./bjam --layout=tagged install

Avro Schema Evolution

When I added a new nullable field into my avro schema, Avro reported an AvroTypeException.

SchemaEvolutionSpec:
Avro
  can read the file using the old schema
  - when adding a new field *** FAILED ***
    org.apache.avro.AvroTypeException: Found TestRecord, expecting TestRecord
    at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:231)
    at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
    at org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:127)
    at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:169)

The writer schema is:

  { "type": "record",
    "name": "TestRecord",
    "version" : 1,
    "fields": [
       { "name": "A", "type": ["null", "string"] },
       { "name": "B", "type": ["null", "int" ] } 
    ] }

The reader schema is:

  { "type": "record",
    "name": "TestRecord",
    "version" : 1,
    "fields": [
       { "name": "A", "type": ["null", "string"] },
       { "name": "B", "type": ["null", "int" ] }, 
       { "name": "C", "type": ["null", "double" ] }, 
    ] }

The error message "Found ..., expecting ..." is misleading. It is ResolvingGrammarGenerator who emits this message, not ResolvingDecoder. Here is the code:

221       for (Field rf : rfields) {
222         String fname = rf.name();
223         if (writer.getField(fname) == null) {
224           if (rf.defaultValue() == null) {
225             result = Symbol.error("Found " + writer.getFullName()
226                                   + ", expecting " + reader.getFullName());
227             seen.put(wsc, result);
228             return result;
229           } else {
230             reordered[ridx++] = rf;
231             count += 3;
232           }
233         }
234       }

The message actaully means that the reader tries to read a field that does not exist in the writer schema and does not have a default value. What? Is field "C" nullable? I think Avro should not enforce users to provide "null" for a nullable field. If you cannot wait for the fix of this issue, here is the workaround, using null as default value for field "C". This reader schema works

  { "type": "record",
    "name": "TestRecord",
    "version" : 1,
    "fields": [
       { "name": "A", "type": ["null", "string"] },
       { "name": "B", "type": ["null", "int" ] }, 
       { "name": "C", "type": ["null", "double" ], "default": null }, 
    ] }

Here is my ScalaTest code:

import org.scalatest.WordSpec
import org.scalatest.ShouldMatchers
import org.scalatest.BeforeAndAfterEach
import java.io.File

import org.apache.avro.{ Schema => AvroSchema }
import org.apache.avro.generic.{ GenericRecord, GenericRecordBuilder }
import org.apache.avro.generic.{ GenericDatumReader, GenericDatumWriter }
import org.apache.avro.file.{ DataFileReader, DataFileWriter }
import org.apache.avro.util.Utf8

import collection.JavaConversions._

class SchemaEvolutionSpec extends WordSpec with ShouldMatchers {

  "Avro" can {
    val file = new File("users.avro")

    def avroSchema(json: String): AvroSchema =
      (new AvroSchema.Parser).parse(json)

    def writeAs(schema: AvroSchema)(rec: GenericRecord) = {
      val dataFileWriter = new DataFileWriter[GenericRecord](
        new GenericDatumWriter[GenericRecord](schema))
      dataFileWriter.create(schema, file);
      dataFileWriter.append(rec);
      dataFileWriter.close();
    }

    def readAs(writeSchema: AvroSchema, readSchema: AvroSchema): GenericRecord = {
      val dataFileReader = new DataFileReader[GenericRecord](
        file, new GenericDatumReader[GenericRecord](writeSchema, readSchema));
      dataFileReader.next(null);
    }

    def record(schema: AvroSchema, data: Map[String, Any]): GenericRecord = {
      val builder = new GenericRecordBuilder(schema)
      data.foreach { kv => builder.set(kv._1, kv._2) }
      builder.build()
    }

    "read the file using the old schema" when {
      "adding a new field" in {
        val oldSchema = avroSchema("""
            { "type": "record",
              "name": "TestRecord",
              "version" : 1,
              "fields": [
             { "name": "A", "type": ["null", "string"] },
             { "name": "B", "type": ["null", "int" ] } 
              ] }
            """)
        val newSchema = avroSchema("""
            { "type": "record",
              "name": "TestRecord",
              "version": 2,
              "fields": [
             { "name": "A", "type": ["null", "string"] },
             { "name": "B", "type": ["null", "int" ] }, 
             { "name": "C", "type": ["null", "double"] } 
              ] }
            """)

        val rec = record(oldSchema, Map("A" -> "Hello", "B" -> 5))
        writeAs(oldSchema)(rec)

        val newRec = readAs(oldSchema, newSchema)

        def value[T](field: String): T =
          newRec.get(field).asInstanceOf[T]

        value[Utf8]("A").toString should be("Hello")
        value[Int]("B") should be(5)
        newRec.get("C") should be(null)
      }

      "deleting a field" in {
        pending
      }

      "renaming a field" in {
        pending
      }

      "changing data type" in {
        pending
      }
    }

  }

}

Play run in DEV mode and "ClassNotFoundException"

Play "~run" command makes development much easier. You can change the code and test it without building, packaging and deploying. But it also causes annoying "ClassNotFoundException".

Our application has a hbase filter. The filter will be packaged and deployed to HBase region servers, but it is also needed on the client side when you build a Scan. If we run the application in DEV mode, we will get "ClassNotFoundException". The java code of the filter is definitely compiled and "in the classpath" because I can find it in the output of "show full-classpath". This confusing issue forces us to use stage/dist again.

The issue is actually caused by the classloader when you start "run" command. If you use a customized filter, HBase will use "Class.forName" to load the class. Because the filter is NOT in the classpath of the classloader which loads HBase classes, "ClassNotFoundException" is thrown.

But Why the filter is NOT in the classpath? There are several classloaders when Play runs in DEV mode:

• sbtLoader, the loader loads sbt;
• applicationLoader, the loader loads the jar files in dependencyClasspath in Compile. it is also called as "SBT/Play shared ClassLoader".
• ReloadableClassLoader(v1), the loader loads the classes of the project

Because ReloadableClassLoader is a child of applicationLoader, the filter is in the classpath of ReloadableClassLoader rather than applicationLoader, and HBase library uses applicationLoader, the filter is indeed invisible to applicationLoader.

The simple workaround is to make the filter a separate project and a dependency of play. The only disadvantage is you have to build, publish, and update if you are developing the filter and the application code at the same time.

You can find the same issue: https://github.com/playframework/Play20/issues/822

Play 2.1.1 Jdbc is not compatible with hive-jdbc-0.10.0-cdh4.2.0

If you want to access a Hive/Impala server in Play 2.1.1, you will encounter a strange error saying "Cannot connect to database [xxx]. Unfortunately no further information can help you identify where is wrong. It is strange because you can using the same URL to connect to the server without any problem.

import java.sql.DriverManager

object ImpalaJdbcClient extends App {

  val impalaUrl = "jdbc:hive2://impala-host:21050/;auth=noSasl"

  val driverClass = Class.forName("org.apache.hive.jdbc.HiveDriver")

  val conn = DriverManager.getConnection(impalaUrl)

  val st = conn.createStatement()
  val rs = st.executeQuery("select count(distinct customer_id) from customers where repeat ='Y'")
  while (rs.next()) {
    println("count=%d".format(rs.getLong(1)))
  }

  conn.close
}

By digging Play and Hive-JDBC code, I figured out that Play-jdbc calls a lot of methods what hive-jdbc doesn't support. For those methods, such as setReadOnly and setCatalog, hive-jdbc just simply throws a SQLException saying "Method not supported", then Play-jdbc catch it and report "Cannot connect to database" error, but unfortunately it doesn't include the message of "Method not supported". You can fix it by removing throw statements from hive-jdbc unsupported method and recompiling. Another way is to create your own BoneCPPlugin. Just copy the source code ./src/play-jdbc/src/main/scala/play/api/db/DB.scala and remove the offending method calls:

• setAutoCommit
• commit
• rollback
• setTransactionIsolation
• setReadOnly
• setCatalog

and comment or replace this line

case mode => Logger("play").info("database [" + ds._2 + "] connected at " + dbURL(ds._1.getConnection))

to

case mode => Logger("play").info("database [" + ds._2 + "] connected at " + ds._1)

because dbURL calls conn.getMetaData.getURL and HiveDatabaseMetaData doesn't support getURL. Change dbplugin in app.configuration.getString("dbplugin").filter(_ == "disabled") to something else to avoid conflict with Play's BoneCPPlugin. Then register your own BoneCPPlugin in conf/play.plugins.

How Play set up ivy repository?

Play sets ivy repository to ${PLAY_HOME}/repository instead of using the default Ivy home $HOME/.ivy2. Checkout the file ${PLAY_HOME}/play and you will find this at the end

"$JAVA" -Dsbt.ivy.home=$dir/repository -Dplay.home=$dir/framework -Dsbt.boot.properties=$dir/framework/sbt/play.boot.properties -jar $dir/framework/sbt/sbt-launch.jar "$@"

-Dsbt.ivy.home does the trick.

Install Impala 1.0 in Cloudera Manager 4.5.0

If you don't want to upgrade to 4.5.2, you can change impala parcel URL to get the impala parcel. http://archive.cloudera.com/impala/parcels/latest/