DB21018E DB2 CLP cannot start

DB2 CLP refused to start after I killed some db2 processes. And finally I found this error from db2diag.log. Actually db2 and db2bp use message queue to communicate. And somehow all queues (16) are all used. use command "ipcs -qp" to show the current message queues

2011-12-29-05.01.05.656543-480 E46743E405          LEVEL: Severe (OS)
PID     : 29100                TID  : 47033606104272PROC : db2
INSTANCE: db2c97               NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloexec, probe:20
MESSAGE : ZRC=0x870F00F2=-2029059854=SQLO_NORES
          "no resources to create process or thread"
CALLED  : OS, -, msgget                           OSERR: ENOSPC (28)

You can use "ipcrm -q msgid" to remove them. After removing all those message queues, db2 CLP starts.

Input DB2 password using Python openpty()

Here is how to use python openpty() to input DB2 password.

• You need use openpty() because db2 insists reading from a terminal. You cannot use PIPE for stdin to pass the password.
• You have to read from stdout first. If you write the password through pty before reading, db2 may not read the password because it needs time to start.
• In python, read() will read until EOF. readline() won't work in that db2 prints "Enter " and waits for the input, no new line is present yet.

import os
import subprocess

m,s = os.openpty()
print m,s
p = subprocess.Popen("db2.sh", stdin=s, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
f = os.fdopen(m, "w")
out = p.stdout.read(5)
print "OUT: %s" % out
if out == "Enter":
  f.write("mypassword\n")
else:
  exit(-1)

print "\nSTDOUT:\n"
for line in p.stdout:
  print line
print "\nSTDERR:\n"
for line in p.stderr:
  print line
print p.returncode

#!/bin/bash
set -e
source /home/db2c97/sqllib/db2profile
db2 connect to DEVEDW user myname
echo $?
echo "loading ..."
db2 "select count(*) from players"
echo $?

Fix "The property zDeviceTemplates does not exist"

I wanted to bind my monitoring templates with my Hadoop devices programmatically. So I wrote a zendmd script like this:

...
templates = {
    "clientnode": [],
    "secondarynamenode": [ 'HadoopJVM', 'HadoopNameNode', 'HadoopDFS' ],
    "namenode":   [ 'HadoopJVM', 'HadoopNameNode', 'HadoopDFS' ],
    "jobtracker": [ 'HadoopJVM', 'HadoopJobTracker', 'HadoopFairScheduler' ],
    "datanode":   [ 'HadoopJVM', 'HadoopDataNode', 'HadoopTaskTracker' ],
    "utility":    []
    }
...

for item in dmd.Devices.Server.SSH.Linux.Ganglia.devices.objectItems():
  (name, device) = item
  bindings = set([ 'Device' ])
  rule = findRule(name, rules)
  if rule:
    device.zGangliaHost = gmond[rule["cluster"]]
    for t in rule["kinds"]:
      bindings = bindings.union(templates[t])
  device.zDeviceTemplates = list(bindings)
  print name, device.zDeviceTemplates
commit()

The basic idea is to define a list of templates for each node and set the templates list to zDeviceTemplates.

It worked after running this script in zendmd, and you can find all monitoring templates for the device. But you cannot bind templates in the WebUI any more. If you try to load objects including templates using ImportRM.loadObjectFromXML(xmlfile=f), it will throw this error "The property zDeviceTemplates does not exist".

Another problem is: zGangliaHost won't show up in "Configuration properties" after running the script, but Ganglia ZenPack works well.

I found exactly the same problem http://community.zenoss.org/thread/5812, which suggested "delete the device and create it again".

Actually you should never assign zGangliaHost and zDeviceTemplates directly. You should use device.setZenProperty('zGangliaHost', gmond[rule["cluster"]]) and device.setZenProperty('zDeviceTemplates', list(bindings)). setZenProperty actually maintains a internal property dict. If you assign zGangliaHost or zDeviceTemplates directly (using attribute directly), the property dict will not contain those properties, and you will get the error.

But you cannot call setZenProperty to set the property any more after the error is already thrown. You will kept gotten "the property doesn't exist" error. How should I fix it without delete the device?

It is actually pretty simple: delete attribute from zGangliaHost and zDeviceTemplates. Actually zenoss check if the property name is valid before set the property. If the object already has the attribute, the property name will be invalid because zenoss will add attribute to the object for each property. Unfortunately, the error message is misleading. This is my fix script:

for (id, dev) in dmd.Devices.Server.SSH.Linux.Ganglia.devices.objectItems():
  print '----- %s' % id
  
  try:
    gangliaHost = dev.zGangliaHost
    delattr(dev, 'zGangliaHost')
    dev.setZenProperty('zGangliaHost', gangliaHost)
  except:
    print 'Missing zGangliaHost'

  devTemplates = dev.zDeviceTemplates
  delattr(dev, 'zDeviceTemplates')
  dev.setZenProperty('zDeviceTemplates', devTemplates)

  print 'zGangliaHost = %s' % dev.zGangliaHost
  print 'zDeviceTemplates = %s' % dev.zDeviceTemplates
commit()

Invalid artifact issue of Eclipselink Nexus Proxy Repository

I use Nexus as my internal maven repository. I set up a proxy repository "Eclipselink Maven Mirror" for Eclipselink. Eclipselink uses this URL http://www.eclipse.org/downloads/download.php?r=1&nf=1&file=/rt/eclipselink/maven.repo which actually redirects you to a mirror site. Just put this link into "Remote Storage Location" and create a proxy repository. I made Eclipselink repository as the last one in "Ordered Group Repository" in "Public Repository" Configuration tab. My maven settings.xml defines a mirror to use the public repository group like this

    <mirrors>
        <mirror>
            <id>nexus-public</id>
            <mirrorOf>*</mirrorOf>
            <url>http://nexus:8080/nexus/content/groups/public</url>
        </mirror>
    </mirrors>

Unfortunately it doesn't work well. When maven gets some of artifacts, especially my own artifacts, Nexus returns invalid pom or jar files in this "Eclipselink Maven Mirror", which are actually HTML pages. And the artifacts will be saved into maven local repository, and make my maven builds fail again and again. I don't know how Nexus search the public repositories to locate an artifact. But looks like Nexus tries "EclipseLink Maven Mirror", then access the Eclipselink Repo using the URL listed above. Unfortunately this URL will return a HTML page once the artifact is not found. But Nexus didn't check if it is invalid because I didn't configure it. Here is my solution:

• Setup two mirrors: one for eclipselink and one for the others except eclipselink.

        
            nexus-public
            *,!eclipselink-repo
            http://nexus:8080/nexus/content/groups/public
        
        
            nexus-eclipselink
            eclipselink-repo
            http://nexus:8080/nexus/content/repositories/eclipselink-maven-mirror
        

• Set "File Content Validation" to true For "Eclipselink Maven Mirror"

Hive Metastore Trick: "get_privilege_set failed"

Three kinds of Hive metastores are supported in Hive 0.7.1:

• Embeded
• Local
• Remote

It seems straight forward. Right? If MySQL metastore database is installed on serverA, and the Hive client is running on serverB, which one you should use? Definitely not Embedded. Should use "Remote"? Does this configuration hive-site.xml work?

<property>
  <name>hive.metastore.local</name>
  <value>false</value>
  <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>

<property>
  <name>hive.metastore.uris</name>
  <value>thrift://serverA:8003</value>
  <description>host and port for the thrift metastore server</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://chelhadedw002/metastore_dev</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>username</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>password</value>
</property>

Unfortunately this doesn't work correctly. We could run "show tables", but Hive threw the exception when we run a select statement like this

FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.metadata.HiveException(org.apache.thrift.TApplicationException: get_privilege_set failed: unknown result)
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_privilege_set failed: unknown result
        at org.apache.hadoop.hive.ql.metadata.Hive.get_privilege_set(Hive.java:1617)
        at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.authorizeUserPriv(DefaultHiveAuthorizationProvider.java:201)
        at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.authorizeUserAndDBPriv(DefaultHiveAuthorizationProvider.java:226)
        at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.authorizeUserDBAndTable(DefaultHiveAuthorizationProvider.java:259)
        at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.authorize(DefaultHiveAuthorizationProvider.java:159)
        at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:531)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:209)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:286)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:513)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: org.apache.thrift.TApplicationException: get_privilege_set failed: unknown result
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_privilege_set(ThriftHiveMetastore.java:2414)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_privilege_set(ThriftHiveMetastore.java:2379)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.get_privilege_set(HiveMetaStoreClient.java:1042)
        at org.apache.hadoop.hive.ql.metadata.Hive.get_privilege_set(Hive.java:1615)
        ... 15 more

The problem of our configuration is "We should use LOCAL metastore even the MySQL metastore locates on a different server", in other word, hive.metastore.local=true. If you want to use "Remote" metastore, you need to start the metastore service hive --service metastore. It seems working when we use "serverA:8003", because we install Hue on serverA, and beeswax starts a metastore thrift service on port 8003. That is why you can run show tables successfully, but get the above error when you run select.

Another issue is: when you use remote metastore, you will still have this issue. see https://issues.apache.org/jira/browse/HIVE-2554 and https://issues.apache.org/jira/browse/HIVE-2405.

Puppet logs

It took me hours to figure out how to make puppet write the log files. I'm using EPEL puppet-2.6.6 on CentOS 5.6 x86_64. Puppet documentation is misleading, you may find puppetdlog and masterlog in this page Puppet Configuration Reference. I tried to set them as syslog or a file in /var/log/puppet, neither worked. It turns out that you have to give --logdest option on the command line. If you run puppet service, you can set the option in /etc/sysconfig/puppet and /etc/sysconfig/puppetmaster here is /etc/sysconfig/puppet

# Where to log to. Specify syslog to send log messages to the system log.
PUPPET_LOG=/var/log/puppet/agent.log

# Autoflush logs
PUPPET_EXTRA_OPTS=--autoflush

and /etc/sysconfig/puppetmaster

PUPPETMASTER_LOG=/var/log/puppet/master.log

PUPPETMASTER_EXTRA_OPTS=--autoflush

It is better to add --autoflush. I like to use puppet kick and monitor the log. Without --autoflush, puppet seems not working because the log is not written to disk.

Hadoop Cluster Monitoring