Invalid artifact issue of Eclipselink Nexus Proxy Repository

I use Nexus as my internal maven repository. I set up a proxy repository "Eclipselink Maven Mirror" for Eclipselink. Eclipselink uses this URL http://www.eclipse.org/downloads/download.php?r=1&nf=1&file=/rt/eclipselink/maven.repo which actually redirects you to a mirror site. Just put this link into "Remote Storage Location" and create a proxy repository. I made Eclipselink repository as the last one in "Ordered Group Repository" in "Public Repository" Configuration tab. My maven settings.xml defines a mirror to use the public repository group like this

    <mirrors>
        <mirror>
            <id>nexus-public</id>
            <mirrorOf>*</mirrorOf>
            <url>http://nexus:8080/nexus/content/groups/public</url>
        </mirror>
    </mirrors>

Unfortunately it doesn't work well. When maven gets some of artifacts, especially my own artifacts, Nexus returns invalid pom or jar files in this "Eclipselink Maven Mirror", which are actually HTML pages. And the artifacts will be saved into maven local repository, and make my maven builds fail again and again. I don't know how Nexus search the public repositories to locate an artifact. But looks like Nexus tries "EclipseLink Maven Mirror", then access the Eclipselink Repo using the URL listed above. Unfortunately this URL will return a HTML page once the artifact is not found. But Nexus didn't check if it is invalid because I didn't configure it. Here is my solution:

• Setup two mirrors: one for eclipselink and one for the others except eclipselink.

        
            nexus-public
            *,!eclipselink-repo
            http://nexus:8080/nexus/content/groups/public
        
        
            nexus-eclipselink
            eclipselink-repo
            http://nexus:8080/nexus/content/repositories/eclipselink-maven-mirror
        

• Set "File Content Validation" to true For "Eclipselink Maven Mirror"

Hive Metastore Trick: "get_privilege_set failed"

Three kinds of Hive metastores are supported in Hive 0.7.1:

• Embeded
• Local
• Remote

It seems straight forward. Right? If MySQL metastore database is installed on serverA, and the Hive client is running on serverB, which one you should use? Definitely not Embedded. Should use "Remote"? Does this configuration hive-site.xml work?

<property>
  <name>hive.metastore.local</name>
  <value>false</value>
  <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>

<property>
  <name>hive.metastore.uris</name>
  <value>thrift://serverA:8003</value>
  <description>host and port for the thrift metastore server</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://chelhadedw002/metastore_dev</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>username</value>
</property>

<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>password</value>
</property>

Unfortunately this doesn't work correctly. We could run "show tables", but Hive threw the exception when we run a select statement like this

FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.metadata.HiveException(org.apache.thrift.TApplicationException: get_privilege_set failed: unknown result)
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_privilege_set failed: unknown result
        at org.apache.hadoop.hive.ql.metadata.Hive.get_privilege_set(Hive.java:1617)
        at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.authorizeUserPriv(DefaultHiveAuthorizationProvider.java:201)
        at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.authorizeUserAndDBPriv(DefaultHiveAuthorizationProvider.java:226)
        at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.authorizeUserDBAndTable(DefaultHiveAuthorizationProvider.java:259)
        at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.authorize(DefaultHiveAuthorizationProvider.java:159)
        at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:531)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:209)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:286)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:513)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: org.apache.thrift.TApplicationException: get_privilege_set failed: unknown result
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_privilege_set(ThriftHiveMetastore.java:2414)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_privilege_set(ThriftHiveMetastore.java:2379)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.get_privilege_set(HiveMetaStoreClient.java:1042)
        at org.apache.hadoop.hive.ql.metadata.Hive.get_privilege_set(Hive.java:1615)
        ... 15 more

The problem of our configuration is "We should use LOCAL metastore even the MySQL metastore locates on a different server", in other word, hive.metastore.local=true. If you want to use "Remote" metastore, you need to start the metastore service hive --service metastore. It seems working when we use "serverA:8003", because we install Hue on serverA, and beeswax starts a metastore thrift service on port 8003. That is why you can run show tables successfully, but get the above error when you run select.

Another issue is: when you use remote metastore, you will still have this issue. see https://issues.apache.org/jira/browse/HIVE-2554 and https://issues.apache.org/jira/browse/HIVE-2405.

Puppet logs

It took me hours to figure out how to make puppet write the log files. I'm using EPEL puppet-2.6.6 on CentOS 5.6 x86_64. Puppet documentation is misleading, you may find puppetdlog and masterlog in this page Puppet Configuration Reference. I tried to set them as syslog or a file in /var/log/puppet, neither worked. It turns out that you have to give --logdest option on the command line. If you run puppet service, you can set the option in /etc/sysconfig/puppet and /etc/sysconfig/puppetmaster here is /etc/sysconfig/puppet

# Where to log to. Specify syslog to send log messages to the system log.
PUPPET_LOG=/var/log/puppet/agent.log

# Autoflush logs
PUPPET_EXTRA_OPTS=--autoflush

and /etc/sysconfig/puppetmaster

PUPPETMASTER_LOG=/var/log/puppet/master.log

PUPPETMASTER_EXTRA_OPTS=--autoflush

It is better to add --autoflush. I like to use puppet kick and monitor the log. Without --autoflush, puppet seems not working because the log is not written to disk.

Hadoop Cluster Monitoring

Set zGangliaHost for a lot of servers

I realized that it was a big problem to set zGangliaHost for my Hadoop clusters, totally 55 servers. Fortunately, Zenoss is powerful if you can write some small Python code. Here is my solution: Load them using zenbatchload like this:

$ zenbatchload dev-cluster.txt

Here is the dev-cluster.txt:

/Devices/Server/SSH/Linux/Ganglia
    devnode001 comments="My Hadoop DEV cluster, client node", zGangliaHost="devnode001", setGroups='/Hadoop/DEV/ClientNode'
    devnode002 comments="My Hadoop DEV cluster, master node", zGangliaHost="devnode001", setGroups='/Hadoop/DEV/MasterNode'
    devnode003 comments="My Hadoop DEV cluster, data node", zGangliaHost="devnode001", setGroups='/Hadoop/DEV/DataNode'

You can set zGangliaHost and groups in a file. It is perfect. You can generate this file easily using a script. ZENBATCHLOAD HOW TO may be obsolete. My Zenoss version (3.1.0) doesn't use -i. My Zenoss administrator created the devices for my clusters without zGangliaHost. I don't want him to delete those devices and use zenbatchload. Here is my solution.

$ zendmd --script=set_gangliahost.py

Here is set_gangliahost.py:

import re

dev = re.compile('(hdcl001|had002|had01[0-2]).*', re.IGNORE
CASE)
test = re.compile('(hdcledw002|had001|had01[0-2]).*', re.IGNOR
ECASE)
prod = re.compile('prod.*', re.IGNORECASE)
for item in dmd.Devices.Server.SSH.Linux.Ganglia.devices.objectItems():
    (name, device) = item
    if dev.match(name):
        device.zGangliaHost = 'dev-gmond'
    elif test.match(name):
        device.zGangliaHost = 'test-gmond'
    elif prod.match(name):
        device.zGangliaHost = 'prod-gmond'
commit()

dev-gmond is the server name where you run gmond

Zenoss Ganglia ZenPack Fix

My company uses Zenoss to monitor all Linux hosts. And we want to use Ganglia ZenPack to monitor our Hadoop clusters. The ZenPack from this link doesn't work in my Zenoss, community version 3.1.0. A lot of weird things happened when I loaded the egg using "Advance -> Settings -> ZenPacks -> Install ZenPack ...". I also tried the source code of the ZenPack in github, and no luck. Finally I fixed the issue of Ganglia ZenPack and it worked perfectly in Zenoss 3.1.0. I created an ZenPack in Zenoss using "Create ZenPack ...", and used the folders under $ZENHOME/ZenPacks as a skeleton. And copied the files in the ZenPack from github except the skins. I also changed this line, but I don't remember if this is important.

diff ZenPacks/jschroeder/GangliaMonitor/datasources/GangliaMonitorDataSource.py /workplace/ws-zenpacks/ZenPacks.jschroeder.GangliaMonitor/ZenPacks/jschroeder/GangliaMonitor/datasources/GangliaMonitorDataSource.py 
72c72
<             return self.hostname
---
>             return self.host

Because I am a newbie of Zenoss, the following may be useful for you:

• Switch to zenoss sudo su - zenoss
• run zenpack --link --install=/tmp/ZenPacks.jschroeder.GangliaMonitor
• If everything is correct, you should see ZenPack in the web page.

You'd better use command line tool zenpack, it will save you a lot of time if you are new to Zenoss and Python. You probably need to run zopectl restart time to time. From my experience with Zenoss, if your ZenPack works, everything looks perfect. If you have something wrong in your ZenPack, you are doomed because Zenoss doesn't tell you too much useful information, especially when you use Zenoss web. For example, I deleted MANIFEST.in in the folder, and the egg was built without the folders libexec and objects/objects.xml. After I installed this egg, ZenPack appears in the webpage after I run "zopectl restart", but it never worked as I expected.

Puppet kick

I encountered several problems when I tried puppet kick. I did setup /etc/hosts to resolve pslave1 and could ping the host. It turns out that I have to enable tcp/8139 on pslave1's firewall.

$ sudo puppet kick -f --debug --host pslave1.puppet-test.com
Triggering pslave1.puppet-test.com
Host pslave1.puppet-test.com failed: No route to host - connect(2)
pslave1.puppet-test.com finished with exit code 2
Failed: pslave1.puppet-test.com

Then I run into another problem, I did add the following in /etc/puppet/auth.conf like this (THIS IS WRONG)

# this one is not stricly necessary, but it has the merit
# to show the default policy which is deny everything else
path /
auth any

path /run
method save
allow pmaster.puppet-test.com

And I did add run this command to create namespaceauth.conf

sudo touch /etc/puppet/namespaceauth.conf

But it is still don't allow me to kick the agent:

warning: Denying access: Forbidden request: pmaster.puppet-test.com(192.168.56.101) access to /run/pslave1.puppet-test.com [save] authenticated  at line 93
err: Forbidden request: pmaster.puppet-test.com(192.168.56.101) access to /run/pslave1.puppet-test.com [save] authenticated  at line 93

Finally I found why: because I put "path /run" after "path /". Here is the correct auth.conf

path /run
auth any
method save
allow pmaster.puppet-test.com

# this one is not stricly necessary, but it has the merit
# to show the default policy which is deny everything else
path /
auth any

You can run puppet agent like this to get the debug information:

sudo puppet agent --listen --debug --no-daemonize --verbose