Skip to main content

Common issues on Shark with CDH5-beta2

Issues on Shark with CDH5-beta2


IncompatibleClassChangeError: Implementing class

Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(
    at Method)
    at java.lang.ClassLoader.loadClass(
    at sun.misc.Launcher$AppClassLoader.loadClass(
    at java.lang.ClassLoader.loadClass(
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(
    at Method)
    at java.lang.ClassLoader.loadClass(
    at sun.misc.Launcher$AppClassLoader.loadClass(
    at java.lang.ClassLoader.loadClass(
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(
    at org.apache.hadoop.hive.shims.ShimLoader.createShim(
    at org.apache.hadoop.hive.shims.ShimLoader.loadShims(
    at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(
    at org.apache.hadoop.util.ReflectionUtils.setConf(
    at org.apache.hadoop.util.ReflectionUtils.newInstance(
    at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(
    at org.apache.hadoop.hive.ql.session.SessionState.start(
    at shark.SharkCliDriver$.main(SharkCliDriver.scala:128)
    at shark.SharkCliDriver.main(SharkCliDriver.scala)
Solution: One reason for this error is conflicting JARS. In shark-0.9.1 this is caused by hadoop-core-1.0.4.jar
find /opt/shark/shark/ -name hadoop-core-1.0.4.jar
#remove all the hadoop-core-1.0.4.jar


java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf

Solution : Make sure shark installation is present in worker nodes.

org/apache/hadoop/hive/cli/CliDriver : Unsupported major.minor version 51.0

Solution: shark is compiled with JDK1.7 so set JAVA_HOME to JDK1.7
export JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera


Exception in thread "main" java.lang.VerifyError:
class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$AppendRequestProto overrides final method
getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;at java.lang.ClassLoader.defineClass1(Native Method)

Solution - This is because of conflict between protobuf-java-2.4.1.jar and protobuf-java-2.5.1.jar 1. Replace protobuf-java-2.4.1.jar with protobuf-java-2.5.1.jar 2. remove protobuf classes from hive-exec*.jar
cd /opt/shark/shark/dep/hive-0.11.0-bin/lib/
jar -xf hive-exec-0.11.0-shark.jar
rm -rf com/ hive-exec-0.11.0-shark.jar
jar -cf hive-exec-0.11.0-shark.jar .


  1. I have followed your steps but I am getting the last error (4) is there any other protobuf jar that i am missing,

    1. Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient....
      Caused by: java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method getUnknownFields

    2. run this from your shark home folder

      find . -name "protobuf-java-"

      Replace all protobuf-java-2.4.1.jar with protobuf-java-2.5.1.jar

      Also verify hive-exec-0.11.0-shark.jar contains which version of protobuf

    3. find . -name "protobuf-java-*"


Post a Comment

Popular posts from this blog

spark java.lang.IllegalArgumentException: user

Today I faced an error while trying to use Spark shell. This is how I resolved.
scala> val file = sc.textFile("hdfs://...")
14/10/21 13:34:23 INFO MemoryStore: ensureFreeSpace(217085) called with curMem=0, maxMem=309225062
14/10/21 13:34:23 INFO MemoryStore: Block broadcast_0 stored as values to memory (estimated size 212.0 KB, free 294.7 MB)
file: org.apache.spark.rdd.RDD[String] = MappedRDD[1] at textFile at <console>:12

scala> file.count()
java.lang.IllegalArgumentException: user
    at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(
    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(
This error can be fixed by giving proper hostname and port

scala> file.count()14/10/21 13:44:23 INFO FileInputFormat: Total input paths to pr…

rWordCloud - An htmlwidget interface for D3 word cloud

With htmlwidget, its become easy to bind d3 scripts to R. rWordCloud is one such package.

To install rWordCloud

require(devtools) install_github('adymimos/rWordCloud')
Two main functions in rWordClouds are
d3TextCloud - this function takes strings as input, and performs word count. Before word count, it does stemming, and stop word removal.
content <- c('R is a programming language and software environment for statistical computing and graphics open source','The R language is widely used among statisticians and data miners for developing statistical software and data analysis','Polls, surveys of data miners,and studies of scholarly literature databases show that R popularity has increased substantially in recent years','languages programming study open source, analysis') label <- c('a1','a2','a3','a4') d3TextCloud(content = content, label = label )
d3Cloud - Function accepts word and its size
text <…

org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.

Recently installed the latest cloudera hadoop. First issue I faced while working with hive.
Diagnostic Messages for this Task: Container launch failed for container_1406173012885_0009_01_000021 : org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container. This token is expired. current time is 1406254943000 found 1406254938244     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)     at sun.reflect.NativeConstructorAccessorImpl.newInstance(     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(     at java.lang.reflect.Constructor.newInstance(     at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(     at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(     at org.apache.hadoop…