Tuesday, October 24, 2017

VNC server won't start on my CentOS 7

I had to ask my boss to power down and power up the desktop in my office this morning. After the desktop came back, I could not access the vncserver because it failed to start. And I got this error when I run systemctl.
$ sudo systemctl start vncserver@\:2

Job for vncserver@:2.service failed because a configured resource limit was exceeded. See "systemctl status vncserver@:2.service" and "journalctl -xe" for details.
And I also found a Xvnc process started on :1 every time when I start the service.
I did find that `/usr/lib/systemd/system/vncserver@:2.service was missing and restored it like below:
Description=Remote desktop service (VNC)
After=syslog.target network.target


# Clean any existing files in /tmp/.X11-unix environment
ExecStartPre=-/usr/bin/vncserver -kill %i
ExecStart=/usr/bin/vncserver %i -nolisten tcp -localhost -geometry 1920x1080 -de
pth 24 
ExecStop=-/usr/bin/vncserver -kill %i


And I ran systemctl daemon-reload, systemctl disable vncserver@\:2.service, systemctl enable vncserver@\:2.service multiple times, but got the same error again and again.
I also tried using vncserver\@\:2 or vncserver@:2 for service name, it won’t fix it.
Finally when I ran vncserver from the command line, it showed this message
$ vncserver :2 -nolisten tcp -localhost -geometry 1920x1080 -depth 24

Warning: bwang.corp.rhapsody.com:2 is taken because of /tmp/.X11-unix/X2
Remove this file if there is no X server bwang.corp.rhapsody.com:2
A VNC server is already running as :2

New 'bwang.corp.rhapsody.com:1 (bwang)' desktop is bwang.corp.rhapsody.com:1

Starting applications specified in /home/bwang/.vnc/xstartup
Log file is /home/bwang/.vnc/bwang.corp.rhapsody.com:1.log
Obviously, when the deskotp was powered down, /tmp/.X11-unix/X2 was left on the hard drive, and it blocked vncserver from starting on display 2 again.
Removing /tmp/.X11-unix/X2 fix the problem.
NOTE: using vncserver@:2 without escaping : is ok when running systemctl.

Wednesday, October 4, 2017

Spark SQL AnalysisException due to data type mismatch

If you run the following code, you will encounter AnalysisException with data type mismatch.
spark.sql("create table test_table(id int, data struct<name: string, age:int>)")
spark.sql("insert overwrite table test_table select 1, struct('joe', 15)")
val d = spark.read.table("test_table")

case class UserData(name: String, age: Int)
case class User(id: Int, data: UserData)
val u = Seq(User(1, UserData("joe", 15)), User(2, UserData("mary", 10))).toDF

val missing = {
    u("id") === d("id")
    && u("data")("name") === d("data")("name")
    && u("data")("age") === d("data")("age"),
  .where( u("id").isNull || d("id").isNull)

// show this result
// +---+---------+----+----+                                                      
// |id |data     |id  |data|
// +---+---------+----+----+
// |2  |[mary,10]|null|null|
// +---+---------+----+----+

// Throws this error: org.apache.spark.sql.AnalysisException: cannot resolve '(`data` = test_table.`data`)'
// due to data type mismatch: differing types in '(`data` = test_table.`data`)'
// (struct<name:string,age:int> and struct<name:string,age:int>).;
val missing_2 = {
         u("id") === d("id") && u("data") === d("data"),
    .where(u("id").isNull || d("id").isNull)
Don’t be fooled by (struct<name:string,age:int> and struct<name:string,age:int>). The problem is caused by nullable, which is not shown in the error message. The DataFrame created from case classes has nullable=false for id and age because Scala Int cannot be null, while the SQL creates nullable fields. And if you compare a field with complex type (struct, array), Spark just thinks they are different as shown in missing_2. But if you compare field by field, there is no problem as shown in missing.
scala> u.schema.treeString
res4: String =
 |-- id: integer (nullable = false)
 |-- data: struct (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- age: integer (nullable = false)

scala> d.schema.treeString
res5: String =
 |-- id: integer (nullable = true)
 |-- data: struct (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- age: integer (nullable = true)

Thursday, August 17, 2017

Mac OS SSH Kerberose Single-Sign-On

To use Kerberos Single-Sign-On on a MacBookPro, you can do as follows:
  • Make a copy of /etc/krb5.conf from a server in your company network.
  • Save it into /etc/krb5.confon your MacBookPro. You probably need sudo.
  • Run kinit <your-login>@<realm_name>. If you don’t know what is the realm name is, take a look at /etc/krb5.conf
    default_realm = MY.COMPANY.COM
  • Add this line to ~/.ssh/config. Without this line, password method is preferred.
    GSSAPIAuthentication yes
  • Then you can ssh to any host supporting kerberos without typing a password.
  • If your kerberos ticket expires, just run kinit again. You don’t have to provide the principal <your-login>@<realm_name> at this time.
  • Use klist to show when the ticket will expire.

Gnome Terminal Profile Export

I usually run screen on a remote server in PROD environment. Whenever I want to access the server, I SSH to that host, and run screen -D -R to resume the session I leave before. Using screen, you don’t have to worry about your SSH session suddenly goes down due to the Internet connection issue.
I also adjust the terminal running the SSH connection with a title and special color scheme so that I can quickly tell this is a PROD environment and need to be careful.
I create a Gnome terminal profile so that I only need to click “File” -> “Open Terminal” -> “prod-env”. “prod-env” is the profile name with all customized attributes:
  • Title
  • Terminal window size
  • Automatically run a command ssh -t <host> screen -D -R
  • Specified color-scheme
How can I share the profile among the multiple environments, e.g., a laptop and a desktop running Gnome environment? Of course, I can set it manually on both of machines. It would be better if I can export the profile and import it on other machines.
It is actually pretty simple. The following command works on CentOS 7.3 and Gnome 3.
  • Export
    dconf dump /org/gnome/terminal/ > /tmp/terminal-profile.dconf
  • Import
    cat /tmp/terminal-profile.dconf | dconf load

Monday, June 12, 2017

Screen in docker with error "Must be connected to a terminal"

I have a running docker container with command /bin/bash --login. When I run the command
$ docker exec -it devsh /bin/bash --login
I can access the container’s Bash, but when I run screen, I got this error:
$ screen
Must be connected to a terminal.
It turns out that I should use docker attach devsh. This command allows me to access the original shell in the container.

Friday, May 26, 2017

Install Cygwin in a script

I built an Intellij docker image based on CentOS 7. To allow my colleagues to use it on Windows, I need to help them to setup a X window system. This post describe how to setup Cygwin/X using a script without user intervention.
To run in Windows, the best choice for the script is PowerShell. PowerShell is powerful, but ugly compared to Bash.
$CygwinDir=<where you want to install Cygwin>
$CygwinPkgsDir=<where the cygwin packages are cached>

function Download($uri, $outfile) {
  $webClient = New-Object System.Net.WebClient
  $Webclient.DownloadFile($uri, $outfile)

function Install-CygwinX {
  $setup = "$DownloadDir\setup-x86_64.exe"
  if (!(Test-Path "$setup")) {
    Download `
      -uri "https://cygwin.com/setup-x86_64.exe" `
      -outfile "$setup"

  Start-Process "$setup" -ArgumentList "--site ""http://mirrors.xmission.com/cygwin"" --root ""$CygwinDir"" --packages xorg-server,xhost --no-admin --local-package-dir ""$CygwinPkgsDir"" --upgrade-also --quiet-mode" -Wait -NoNewWindow
You can run this command to get the full list of command line options:
Downloads\setup-x86_64.exe --help
Most important command line options of Cygwin setup are:
  • --no-admin: Your user won’t have to be administrator
  • --quiet-mode: The script will run without asking anything
  • --packages: the packages you need to run a Cygwin/X. No package selection is needed.
The PowerShell script will download the setup executable and run it automatically. Here are some thing you need to know:
  • Don’t use Invoke-WebRequest because it is too slow. For Cygwin setup, it is not a big problem because setup-x86_64.exe is small, and Cygwin setup will download the rest package. If you download about 200MB file like ‘Docker Toolbox for Windows’, you will see how slow it is.
  • Use Start-Process to start the setup executable.
    • -Wait makes the script wait until Cygwin setup finished
    • -NoNewWindow is important if you run a .bat file. Without it, your .bat file will run in a separate window, just appear then disappear. If anything is wrong, you have no way to see what is the error message.

Install Docker Toolbox for Windows Automatically

Docker Toolbox for Windows have command line arguments which allows you to install it without user’s involvement.
You can run this command in Windows Command Prompt to get those arguments:
> Downloads\DockerToolbox.exe /HELP
The most useful arguments for automation are: /SILENT, /DIR, /COMPONENTS, and /TASKS.
To know what values for /COMPONENTS and /TASKS, you can run this command
> Downloads\DockerToolbox.exe /SAVEINF=docker_toolbox.inf
Dir=C:\Program Files\Docker Toolbox
If VirtualBox is already installed and I don’t want to install Kitematic, I can run this command, or put the command into the script to skip them. The installation will be automated.
> Downloads\DockerToolbox.exe /COMPONENTS=docker,dockermachine,dockercompose,git /TASKS=desktopicon,modifypath,upgradevm

Thursday, May 11, 2017

Build Git RPM on CentOS 7

I want to use core.hooksPath which supports since 2.9, but the default Git version of CentOS 7.3 is still 1.8.3:

$ yum list git

Available Packages

git.x86_64                                       base
I have to compile from the source by myself. It is not hard to find bunch of blogs explain how to do it. For example, https://www.howtoforge.com/how-to-install-the-latest-git-version-on-centos. However, the method mentioned in those blogs is not what I want, a docker image based on centos:7 with the latest Git version. There are questions I don’t know the answers:
  • How can I clean up those required build tools like gcc? I don’t want a larger image size.
  • How can I install Git manuals? I could not remember all commands and parameters. It will be handy to reference by just typing git help xxx.
I believe the clean way is to build Git in RPM and install Git using RPM in my docker image. Sounds easy, but the first problem I had was where to get .spec to build RPM. The Git source code doesn’t have a RPM spec file. I finally found the spec from Redhat. But it is not easy like Gradle or Maven when you build from a RPM spec. You have to know the tools to pull the dependencies. Finally, my method is actually pretty simple after I figured it out all of steps because I made it in a dockerfile. Here are what I did:
  • I created a docker image git-rpm, which builds Git 2.12.2 in RPMs.
  • When running git-rpm in a container, a yum repository server starts.
  • When I build the docker image, I just simply put the local yum repository for Git, and call yum install -y git.
  • For centos:7, the default configuration turns off the manual installation tsflags=nodocs. I need to turn it on using yum --setopt tsflags='' -y install git.
If you want to get those RPMs from docker and put them to a yum repository, you can run docker cp