Thursday, May 11, 2017

Build Git RPM on CentOS 7

I want to use core.hooksPath which supports since 2.9, but the default Git version of CentOS 7.3 is still 1.8.3:

$ yum list git

Available Packages

git.x86_64                        1.8.3.1-6.el7_2.1                         base
I have to compile from the source by myself. It is not hard to find bunch of blogs explain how to do it. For example, https://www.howtoforge.com/how-to-install-the-latest-git-version-on-centos. However, the method mentioned in those blogs is not what I want, a docker image based on centos:7 with the latest Git version. There are questions I don’t know the answers:
  • How can I clean up those required build tools like gcc? I don’t want a larger image size.
  • How can I install Git manuals? I could not remember all commands and parameters. It will be handy to reference by just typing git help xxx.
I believe the clean way is to build Git in RPM and install Git using RPM in my docker image. Sounds easy, but the first problem I had was where to get .spec to build RPM. The Git source code doesn’t have a RPM spec file. I finally found the spec from Redhat. But it is not easy like Gradle or Maven when you build from a RPM spec. You have to know the tools to pull the dependencies. Finally, my method is actually pretty simple after I figured it out all of steps because I made it in a dockerfile. Here are what I did:
  • I created a docker image git-rpm, which builds Git 2.12.2 in RPMs.
  • When running git-rpm in a container, a yum repository server starts.
  • When I build the docker image, I just simply put the local yum repository for Git, and call yum install -y git.
  • For centos:7, the default configuration turns off the manual installation tsflags=nodocs. I need to turn it on using yum --setopt tsflags='' -y install git.
If you want to get those RPMs from docker and put them to a yum repository, you can run docker cp

Monday, October 17, 2016

VNC Viewer

I recently built a desktop for development in my company’s CORP network. By using VNC, I can access the same Gnome session even from home. Just simply connect to VPN, and start a vnc viewer, I can resume what I do at office.
I have a CentOS 6 on my development desktop. I installed RPM pacakages: vnc-server. (TODO: Details about setting up VNC server.)
I tried TigerVNC Viewer, but it was really bad, especially F8 menu key because I use Eclipse, F8 is used a lot when you are debugging. And there is not a pop-up menu bar like VirtualBox when you are in full-screen mode.
RealVNC satisfies all my requirements about VNC:
  • A menu bar
  • Better configuration UI on Windows 7
  • I can write a script to start VNC viewer
    through SSH tunnel make the viewer goes into full-screen mode on my secondary display automatically.
In CORP network, just click the VNC button on Windows’ taskbard, type the password, the viewer is in full-screen mode on my secondary display.
#/bin/bash
set -e

shutdown() {
  if [ -n "$tunnel_pid" ]; then
    kill $tunnel_pid
  fi
}

trap shutdown EXIT

ssh -N -L 5902:localhost:5902 mydev.desktop &
tunnel_pid=$!

~/apps/bin/VNC-Viewer-5.2.1-Windows-64bit.exe \
  --Monitor='\\.\Display2' \
  --FullScreen=1 \
  localhost:2
I also installed cygwin on Windows so that I could use OpenSSH. The script can clean up the SSH tunnel once the script quits.
At home, the network connection is slow. And that script will fail because RealVNC gets timeout. I didn’t find how to make RealVNC wait for a longer time. I just simply run SSH tunnel and viewer separately in different screens of GNU Screen in a Cygwin terminal.
I chose JPEG encode for better performance when I at home.

Sunday, October 9, 2016

Don't set limit nofile to unlimited

After I set nofile to unlimited in /etc/security/limits.d/90-nproc.conf like this,
*          hard    nproc     unlimited
*          hard    nofile    unlimited

*          soft    nproc     unlimited
*          soft    nofile    unlimited
I could not boot into Gnome on my VM running CentOS 6.8 any more. Even I could switch to a terminal using ctrl-alt-F2, I seemed not be able to log in. I reboot the VM using a live-cd ISO. Then mount the file system, after setting to 32K as below, reboot it, the problem was fixed.
*          -    nproc     unlimited
*          -    nofile    32768
Notes:
# find out the logical volume of the hard drive
fdisk -l

# mount the logical volume
mkdir /mnt/hd
mount /dev/mapper/Volume-lv_root /mnt/hd
vi /mnt/hd/etc/security/limits.d/90-nproc.conf

Monday, September 12, 2016

Gradle show dependencies of the specified configuration

I record the method here in case I forget how to show the dependencies for ONLY one configuration like runtime again.
  • Show dependencies for one configuration ONLY: ./gradlew dependencies --configuration runtime
  • Show a task’s arguments: ./gradlew help --task dependencies

Record Request and Reponse of HTTP Samplers in JMeter

  • Add “Simple Data Writer”
  • Give the file name like “result.xml”
  • Click Configure button.
  • Check the radio buttons for
    • Save As XML
    • Save URL
    • Save Response Data (XML)
  • Start the jmeter test
  • Check the XML file
  • If you stop jmeter, then start another run, the requests and responses in the new run will be appended to the XML file.
<testResults>
<httpSample t="17" lt="17" ts="1473705874608" s="true" lb="Profile Request" rc="200" rm="OK" tn="Profile (year) 1-1" dt="text" by="253" ng="1" na="2">
  <responseData class="java.lang.String">...</responseData>
  <java.net.URL>...</java.net.URL>
</httpSample>
</testResults>

Friday, September 9, 2016

Make Spark read Teradata directly.

Spark SQL supports read JDBC resource, but the paragraph in the latest 2.0 document is not really helpful:
The JDBC driver class must be visible to the primordial class loader on the client session and on all executors. This is because Java’s DriverManager class does a security check that results in it ignoring all drivers not visible to the primordial class loader when one goes to open a connection. One convenient way to do this is to modify compute_classpath.sh on all worker nodes to include your driver JARs.
I wrote this blog to explain how to run against Teradata database:
  • in spark-shell and YARN-cluster mode
  • spark-submit and YARN-cluster mode

spark-shell and YARN mode

...
--conf spark.executor.extraClassPath=./tdgssconfig-15.10.00.22.jar:./terajdbc4-15.10.00.22.jar \
--driver-class-path $LIB_DIR/tdgssconfig-15.10.00.22.jar:$LIB_DIR/terajdbc4-15.10.00.22.jar \
...
  • for executor.extraClassPath, the path is the current directory ./, which is where YARN starts the executor.
  • for --driver-class-path, “$LIB_DIR” is the directory where those jdbc driver jars live, it is on the host where you run spark-shell.

spark-submit and YARN-cluster mode

...
--conf spark.executor.extraClassPath=./tdgssconfig-15.10.00.22.jar:./terajdbc4-15.10.00.22.jar \
--driver-class-path ./tdgssconfig-15.10.00.22.jar:./terajdbc4-15.10.00.22.jar \
--jars $LIB_DIR/tdgssconfig-15.10.00.22.jar,$LIB_DIR/terajdbc4-15.10.00.22.jar,<other jars>
...

Explanation

Two differences:
  • In spark-shell, you don’t have to put Terdata jdbc driver jars in --jars, because --driver-class-path already does. If you put them in --jars, it doesn’t hurt. But in spark-submit, you have to add them in --jars.
  • In spark-submit, --driver-class-path uses the current directory ./, not $LIB_IDR in spark-shell.
Lost? Here is why:
  • When you run spark-submit with YARN-cluster mode , the Spark app driver actually runs in a YARN container, not where you type “spark-submit”.
    • --jars makes teradata jdbc driver jars copied to the container directory where the Spark app driver (YARN appMaster) runs.
    • --driver-class-path just declares you need teradata jdbc driver jars. Be the Spark app driver runs in a YARN container, this argument tells the spark app driver JVM where to find the jdbc jars. Because the jars are copied into the YARN container’s directory, to the JVM of spark app driver, the container directory is the current directory.
    • spark-submit actually cand not find the jdbc jars in ./ in --driver-class-path, so the jdbc jars won’t be copied to the YARN container. That is why you have to use --jars to tell spark where to find those jars and ship them to the container.
  • In spark-shell, the spark app driver is started on the host where you type “spark-shell”. So you need to give the full path in --driver-class-path

Friday, May 27, 2016

Bring back google-chrome after upgrading to CentOS 6.8 and Chrome 51.

I don’t know which one is root cause: upgrading to CenOS 6.8 or Chrome 51. I used install_chrome.sh on http://chrome.richardlloyd.org.uk/ to install Google chrome on my CentOS 6 VirtualBox VM. It worked very well until this upgrade. If I ran google-chrome, the window popped up, but it is almost black.
I use the following command to investigate this problem
google-chrome --disable-plugins --disable-extensions --user-data-dir=/tmp/chrome-user-dir --enable-logging --log-level=0
  • disable all plugins and extensions
  • use a new user dir
  • enable logs
Also checking the chrome process --type=gpu-process as parameter
$ ps -ef | grep chrome
bwang  1358  1284  1 10:58 pts/6    00:00:00 /opt/google/chrome/chrome --enable-features=... --disable-features=... --type=gpu-process --channel=1284.0.1688276239 --enable-logging --log-level=0 --window-depth=24 --user-data-dir=/tmp/chrome-user-dir --supports-dual-gpus=false --gpu-driver-bug-workarounds=4,54 --gpu-vendor-id=0x80ee --gpu-device-id=0xbeef --gpu-driver-vendor=Chromium --gpu-driver-version=1.9 --user-data-dir=/tmp/chrome-user-dir --enable-logging --log-level=0 --v8-natives-passed-by-fd --v8-snapshot-passed-by-fd
The log file /tmp/chrome-user-dir/chrome_debug.log shows
...
[2945:2945:0527/111600:ERROR:texture_manager.cc(2746)] [.CommandBufferContext.DisplayCompositor-0x3d53365d63c0]GL ERROR :GL_INVALID_ENUM : glTexImage2D: <- error from previous GL command
[23:23:0527/111600:WARNING:ipc_message_attachment_set.cc(57)] MessageAttachmentSet destroyed with unconsumed descriptors: 0/1
[2945:2945:0527/111600:ERROR:gles2_cmd_decoder.cc(2167)] [.CommandBufferContext.CompositorWorker-0x3d53365d6280]GL ERROR :GL_INVALID_ENUM : GLES2DecoderImpl::DoBindTexImage2DCHROMIUM: <- error from previous GL command
[2945:2945:0527/111600:ERROR:gles2_cmd_decoder.cc(2167)] [.CommandBufferContext.CompositorWorker-0x3d53365d6280]GL ERROR :GL_INVALID_VALUE : ScopedTextureBinder::dtor: <- error from previous GL command
...
Looks like google-chrome use gpu for acceleration. So the solution is simple google-chrome --disable-gpu brings back chrome on my CentOS 6.8 VM.