Saturday, March 3, 2012

Use JDB to debug beeswax

I needed to figure out who is called HiveMetaStore.get_partition_with_auth because it scans each partition one by one, and will be a performance issue when you have thousands partitions.
12/02/29 12:39:14 INFO ppd.OpProcFactory:  ((local_dt = '2011-11-01') and (local_dt < '2011-11-08'))
12/02/29 12:39:14 INFO ppd.OpProcFactory: Processing for TS(293)
12/02/29 12:39:14 INFO ppd.OpProcFactory: Pushdown Predicates of TS For Alias : lz.lz_omniture_hit
12/02/29 12:39:14 INFO ppd.OpProcFactory:  ((local_dt = '2011-11-01') and (local_dt < '2011-11-08'))
12/02/29 12:39:14 INFO metastore.HiveMetaStore: 27: get_partition_names : db=lz tbl=lz_omniture_hit
12/02/29 12:39:14 INFO HiveMetaStore.audit: ugi=platdev ip=unknown-ip-addr cmd=get_partition_names : db=lz tbl=lz_omniture_hit 
12/02/29 12:39:14 INFO metastore.HiveMetaStore: 27: get_partition_with_auth : db=lz tbl=lz_omniture_hit[2011-01-01,AT]
12/02/29 12:39:14 INFO HiveMetaStore.audit: ugi=platdev ip=unknown-ip-addr cmd=get_partition_with_auth : db=lz tbl=lz_omniture_hit[2011-01-01,AT] 
12/02/29 12:39:14 INFO metastore.HiveMetaStore: 27: get_partition_with_auth : db=lz tbl=lz_omniture_hit[2011-11-01,AR]
12/02/29 12:39:14 INFO HiveMetaStore.audit: ugi=platdev ip=unknown-ip-addr cmd=get_partition_with_auth : db=lz tbl=lz_omniture_hit[2011-11-01,AR] 
12/02/29 12:39:14 INFO metastore.HiveMetaStore: 27: get_partition_with_auth : db=lz tbl=lz_omniture_hit[2011-11-01,AT]
12/02/29 12:39:14 INFO HiveMetaStore.audit: ugi=platdev ip=unknown-ip-addr cmd=get_partition_with_auth : db=lz tbl=lz_omniture_hit[2011-11-01,AT] 
12/02/29 12:39:14 INFO metastore.HiveMetaStore: 27: get_partition_with_auth : db=lz tbl=lz_omniture_hit[2011-11-01,AU]
12/02/29 12:39:14 INFO HiveMetaStore.audit: ugi=platdev ip=unknown-ip-addr cmd=get_partition_with_auth : db=lz tbl=lz_omniture_hit[2011-11-01,AU] 
12/02/29 12:39:14 INFO metastore.HiveMetaStore: 27: get_partition_with_auth : db=lz tbl=lz_omniture_hit[2011-11-01,BE]
Unfortunately, I cannot use Eclipse on the machine where I run beeswax server because X window is not installed. JDB is the only choice, but it really not a good tool for debugging. Here is how I use JDB to debug Beeswax server:
  1. Start Beeswax server in debug mode, add this in apps/beeswax/beeswax_server.sh before hadoop jar is called
    export HADOOP_OPTS="-Dlog4j.configuration=log4j.properties -Xdebug -Xrunjdwp:transport=dt_socket,address=4444,server=y,suspend=n"
    ...
    echo Executing $HADOOP_HOME/bin/hadoop jar $BEESWAX_JAR "$@"
    exec $HADOOP_HOME/bin/hadoop jar $BEESWAX_JAR "$@"
    
  2. Start JDB using jdb -attach 4444
  3. Add source file paths
    > use /etc/hive/conf:$HOME/git/hue/apps/beeswax/src/beeswax/../../../../desktop/conf:/usr/lib/hadoop/conf:/usr/lib/hadoop:$HOME/git/hue/apps/beeswax/java/src/main:$HOME/hadoop/hive-0.7.1-cdh3u1/src/ql:$HOME/hadoop/hive-0.7.1-cdh3u1/src/metastore:$HOME/hadoop/hive-0.7.1-cdh3u1/src/common
    
  4. List all threads: threads
    Beeswax-8[1] threads
    Group system:
      (java.lang.ref.Reference$ReferenceHandler)0xec0 Reference Handler                                                          cond. waiting
      (java.lang.ref.Finalizer$FinalizerThread)0xebf  Finalizer                                                                  cond. waiting
      (java.lang.Thread)0xebe                         Signal Dispatcher                                                          running
    Group main:
      (java.lang.Thread)0xec1                         main                                                                       running
      (java.util.TimerThread)0xebd                    Timer thread for monitoring ugi                                            cond. waiting
      (java.lang.Thread)0xebc                         MetaServerThread                                                           running
      (java.lang.Thread)0xebb                         Evicter                                                                    sleeping
      (java.lang.Thread)0xeba                         pool-2-thread-1                                                            running
      (org.apache.hadoop.util.Daemon)0xeb9            LeaseChecker                                                               sleeping
      (java.lang.Thread)0xeb8                         pool-3-thread-1                                                            running
      (org.apache.hadoop.util.Daemon)0xeb7            LeaseChecker                                                               sleeping
      (org.apache.hadoop.util.Daemon)0xeb6            LeaseChecker                                                               sleeping
      (org.apache.hadoop.util.Daemon)0xeb5            LeaseChecker                                                               sleeping
      (org.apache.hadoop.util.Daemon)0xed1            LeaseChecker                                                               sleeping
      (java.lang.Thread)0xef6                         Beeswax-8                                                                  running (at breakpoint)
      (org.apache.hadoop.ipc.Client$Connection)0xef9  IPC Client (47) connection to chelhadedw002/10.184.39.97:8020 from platdev cond. waiting
      (java.lang.Thread)0xefa                         sendParams-5                                                               cond. waiting
    
  5. Go to a thread: thread 0xebc
  6. Add breakpoints:
    • Stop at the specified line:
      stop at com.cloudera.beeswax.BeeswaxServiceImpl:689
    • Stop in a method:
      stop in com.cloudera.beeswax.BeeswaxServiceImpl.query
    • Stop in a method of nested class:
      stop in org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition_with_auth
Here is what I found:
MetaServerThread[1] stop in org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition_with_auth
Set breakpoint org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition_with_auth
MetaServerThread[1] cont
>
Breakpoint hit: "thread=Beeswax-8", org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition_with_auth(), line=1,463 bci=0

Beeswax-8[1] where
  [1] org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition_with_auth (HiveMetaStore.java:1,463)
  [2] org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo (HiveMetaStoreClient.java:657)
  [3] org.apache.hadoop.hive.ql.metadata.Hive.getPartition (Hive.java:1,293)
  [4] org.apache.hadoop.hive.ql.metadata.Hive.getPartition (Hive.java:1,258)
  [5] org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune (PartitionPruner.java:229)
  [6] org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.setTaskPlan (GenMapRedUtils.java:551)
  [7] org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.setTaskPlan (GenMapRedUtils.java:514)
  [8] org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.initPlan (GenMapRedUtils.java:125)
  [9] org.apache.hadoop.hive.ql.optimizer.GenMRRedSink1.process (GenMRRedSink1.java:76)
  [10] org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch (DefaultRuleDispatcher.java:89)
  [11] org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch (DefaultGraphWalker.java:88)
My Hive version is hadoop-hive-cdh3u1. The code of PartitionPruner is not good enough and it was replaced in hadoop-hive-cdh4b1.

Wednesday, February 8, 2012

Django south migration commands

Here are commands for HUE beeswax model migration.
  • List all migrations by application
    build/env/bin/hue migrate --list
    
  • Create migration: create the migration file
    build/env/bin/hue schemamigration beeswax --auto
    
  • Migrate: create/alert tables in the database
    build/env/bin/hue migrate beeswax
    
  • Rollback: you just applied 0007, but want to roll back to 0006
    build/env/bin/hue migrate beeswax 0006

Thursday, January 5, 2012

Debug Python Process

The python process of our cloudera Hue sometimes uses 100% CPU. I found this method on stackoverflow. It worked although I still couldn't find why Hue goes into an infinite loop, but at least I can attach the process and see what the process is doing. here is the link: http://stackoverflow.com/questions/132058/showing-the-stack-trace-from-a-running-python-application
However, sometimes I need to debug a process that I didn't have the foresight to install the signal handler in. On linux, you can attach gdb to the process and get a python stack trace with some gdb macros. Put http://svn.python.org/projects/python/trunk/Misc/gdbinit in ~/.gdbinit, then:

Attach gdb: gdb -p PID
Get the python stack trace: pystack
It's not totally reliable unfortunately, but it works most of the time.
Here are the better ways to debug a python process:
  • Use strace -ppid. If the process is running as another user, use sudo -u hue strace -ppid. hue is the user account for Cloudera Hue.
  • Install python-debuginfo-2.4.3-44.el5 for Python 2.4 or python26-debuginfo-2.6.5-6.el5 for Python 2.6. And you can use this yum repository to get those packages, create /etc/yum.repos.d/debuginfo.repo :
    [debuginfo]
    name=CentOS-$releasever - DebugInfo
    # CentOS-5
    baseurl=http://debuginfo.centos.org/$releasever/$basearch/
    gpgcheck=0
    enabled=1
    # CentOS-5
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
    protect=1
    priority=1
    

Thursday, December 29, 2011

DB21018E DB2 CLP cannot start

DB2 CLP refused to start after I killed some db2 processes. And finally I found this error from db2diag.log. Actually db2 and db2bp use message queue to communicate. And somehow all queues (16) are all used. use command "ipcs -qp" to show the current message queues
2011-12-29-05.01.05.656543-480 E46743E405          LEVEL: Severe (OS)
PID     : 29100                TID  : 47033606104272PROC : db2
INSTANCE: db2c97               NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloexec, probe:20
MESSAGE : ZRC=0x870F00F2=-2029059854=SQLO_NORES
          "no resources to create process or thread"
CALLED  : OS, -, msgget                           OSERR: ENOSPC (28)
You can use "ipcrm -q msgid" to remove them. After removing all those message queues, db2 CLP starts.

Wednesday, December 7, 2011

Input DB2 password using Python openpty()

Here is how to use python openpty() to input DB2 password.
  • You need use openpty() because db2 insists reading from a terminal. You cannot use PIPE for stdin to pass the password.
  • You have to read from stdout first. If you write the password through pty before reading, db2 may not read the password because it needs time to start.
  • In python, read() will read until EOF. readline() won't work in that db2 prints "Enter " and waits for the input, no new line is present yet.
import os
import subprocess

m,s = os.openpty()
print m,s
p = subprocess.Popen("db2.sh", stdin=s, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
f = os.fdopen(m, "w")
out = p.stdout.read(5)
print "OUT: %s" % out
if out == "Enter":
  f.write("mypassword\n")
else:
  exit(-1)

print "\nSTDOUT:\n"
for line in p.stdout:
  print line
print "\nSTDERR:\n"
for line in p.stderr:
  print line
print p.returncode
#!/bin/bash
set -e
source /home/db2c97/sqllib/db2profile
db2 connect to DEVEDW user myname
echo $?
echo "loading ..."
db2 "select count(*) from players"
echo $?

Friday, December 2, 2011

Fix "The property zDeviceTemplates does not exist"

I wanted to bind my monitoring templates with my Hadoop devices programmatically. So I wrote a zendmd script like this:

...
templates = {
    "clientnode": [],
    "secondarynamenode": [ 'HadoopJVM', 'HadoopNameNode', 'HadoopDFS' ],
    "namenode":   [ 'HadoopJVM', 'HadoopNameNode', 'HadoopDFS' ],
    "jobtracker": [ 'HadoopJVM', 'HadoopJobTracker', 'HadoopFairScheduler' ],
    "datanode":   [ 'HadoopJVM', 'HadoopDataNode', 'HadoopTaskTracker' ],
    "utility":    []
    }
...

for item in dmd.Devices.Server.SSH.Linux.Ganglia.devices.objectItems():
  (name, device) = item
  bindings = set([ 'Device' ])
  rule = findRule(name, rules)
  if rule:
    device.zGangliaHost = gmond[rule["cluster"]]
    for t in rule["kinds"]:
      bindings = bindings.union(templates[t])
  device.zDeviceTemplates = list(bindings)
  print name, device.zDeviceTemplates
commit()

The basic idea is to define a list of templates for each node and set the templates list to zDeviceTemplates.

It worked after running this script in zendmd, and you can find all monitoring templates for the device. But you cannot bind templates in the WebUI any more. If you try to load objects including templates using ImportRM.loadObjectFromXML(xmlfile=f), it will throw this error "The property zDeviceTemplates does not exist".

Another problem is: zGangliaHost won't show up in "Configuration properties" after running the script, but Ganglia ZenPack works well.

I found exactly the same problem http://community.zenoss.org/thread/5812, which suggested "delete the device and create it again".

Actually you should never assign zGangliaHost and zDeviceTemplates directly. You should use device.setZenProperty('zGangliaHost', gmond[rule["cluster"]]) and device.setZenProperty('zDeviceTemplates', list(bindings)). setZenProperty actually maintains a internal property dict. If you assign zGangliaHost or zDeviceTemplates directly (using attribute directly), the property dict will not contain those properties, and you will get the error.

But you cannot call setZenProperty to set the property any more after the error is already thrown. You will kept gotten "the property doesn't exist" error. How should I fix it without delete the device?

It is actually pretty simple: delete attribute from zGangliaHost and zDeviceTemplates. Actually zenoss check if the property name is valid before set the property. If the object already has the attribute, the property name will be invalid because zenoss will add attribute to the object for each property. Unfortunately, the error message is misleading. This is my fix script:

for (id, dev) in dmd.Devices.Server.SSH.Linux.Ganglia.devices.objectItems():
  print '----- %s' % id
  
  try:
    gangliaHost = dev.zGangliaHost
    delattr(dev, 'zGangliaHost')
    dev.setZenProperty('zGangliaHost', gangliaHost)
  except:
    print 'Missing zGangliaHost'

  devTemplates = dev.zDeviceTemplates
  delattr(dev, 'zDeviceTemplates')
  dev.setZenProperty('zDeviceTemplates', devTemplates)

  print 'zGangliaHost = %s' % dev.zGangliaHost
  print 'zDeviceTemplates = %s' % dev.zDeviceTemplates
commit()

Thursday, November 17, 2011

Invalid artifact issue of Eclipselink Nexus Proxy Repository

I use Nexus as my internal maven repository. I set up a proxy repository "Eclipselink Maven Mirror" for Eclipselink. Eclipselink uses this URL http://www.eclipse.org/downloads/download.php?r=1&nf=1&file=/rt/eclipselink/maven.repo which actually redirects you to a mirror site. Just put this link into "Remote Storage Location" and create a proxy repository. I made Eclipselink repository as the last one in "Ordered Group Repository" in "Public Repository" Configuration tab. My maven settings.xml defines a mirror to use the public repository group like this
    <mirrors>
        <mirror>
            <id>nexus-public</id>
            <mirrorOf>*</mirrorOf>
            <url>http://nexus:8080/nexus/content/groups/public</url>
        </mirror>
    </mirrors>
Unfortunately it doesn't work well. When maven gets some of artifacts, especially my own artifacts, Nexus returns invalid pom or jar files in this "Eclipselink Maven Mirror", which are actually HTML pages. And the artifacts will be saved into maven local repository, and make my maven builds fail again and again. I don't know how Nexus search the public repositories to locate an artifact. But looks like Nexus tries "EclipseLink Maven Mirror", then access the Eclipselink Repo using the URL listed above. Unfortunately this URL will return a HTML page once the artifact is not found. But Nexus didn't check if it is invalid because I didn't configure it. Here is my solution:
  • Setup two mirrors: one for eclipselink and one for the others except eclipselink.
  •         
                nexus-public
                *,!eclipselink-repo
                http://nexus:8080/nexus/content/groups/public
            
            
                nexus-eclipselink
                eclipselink-repo
                http://nexus:8080/nexus/content/repositories/eclipselink-maven-mirror
            
    
  • Set "File Content Validation" to true For "Eclipselink Maven Mirror"