Friday, December 22, 2017

Make Intellij IDEA on Linux with sharp fonts as Macbook Pro

IntelliJ IDEA supports HiDPI on Windows and MacOS. The fonts look so beautiful on a Macbook Pro Retina Display. Unfortunately IDEA doesn’t support HiDPI on Linux. Looks at the code UIUtil#isJreHiDPIEnabled. For Linux, it always return false.
You can get that sharpness through VNC by changing to the larger font size. I’m using 2017.3.1
  • Settings -> Editor -> Color Shcema -> Color Scheme Font
    • Check “Use color scheme font instead of the default`
    • Change “Size” to 22
  • Settings -> Appearance & Behavior -> Appearance -> UI Options
    • Themes: Darcular
    • Check “Override default fonts by (not recommended):”
    • Name: Dialog and size to 22
HiDPI-profiles plugin may make switching between HiDPI and lower resolution easier.

How to make VNC sharp on Retina display.

Most of time, I work on a Linux box running CentOS 7 through VNC. The largest VNC screen size is 1920x1080 (16:9). It is ok when you maximize the RealVNC viewer window on an external monitor with that resolution. On my Macbook Pro, there are two black areas on top and bottom because the Retina display is 2880x1800 (16:10), and the font looks small and blurry. I have to switch VNC screen side to 1680x1050 (16:10), the font size is proper, but still blurry. How can I make VNC sharp as a native MacOS app?
First, create a larger screen on VNC as below, then you will have the same size VNC screen as Macbook Pro.
  • Create a modeline using cvt
  • Create a new mode using the modeline
  • Add the new mode
  • Switch to the new mode
  • Set screen DPI to 220, which is the DPI of Macbook Pro’s Retina Display
[bwang@bwang ~]$ cvt 2880 1800
# 2880x1800 59.97 Hz (CVT 5.18MA) hsync: 111.84 kHz; pclk: 442.00 MHz
Modeline "2880x1800_60.00"  442.00  2880 3104 3416 3952  1800 1803 1809 1865 -hsync +vsync
[bwang@bwang ~]$ xrandr --newmode "2880x1800_60.00"  442.00  2880 3104 3416 3952  1800 1803 1809 1865 -hsync +vsync
[bwang@bwang ~]$ xrandr
Screen 0: minimum 32 x 32, current 1680 x 1050, maximum 32768 x 32768
VNC-0 connected primary 1680x1050+0+0 0mm x 0mm
   1920x1080     60.00 +
   1920x1200     60.00  
   1600x1200     60.00  
   1680x1050     60.00* 
   1400x1050     60.00  
   1360x768      60.00  
   1280x1024     60.00  
   1280x960      60.00  
   1280x800      60.00  
   1280x720      60.00  
   1024x768      60.00  
   800x600       60.00  
   640x480       60.00  
  2880x1800_60.00 (0x29f) 442.000MHz -HSync +VSync
        h: width  2880 start 3104 end 3416 total 3952 skew    0 clock 111.84KHz
        v: height 1800 start 1803 end 1809 total 1865           clock  59.97Hz
[bwang@bwang ~]$ xrandr --addmode 2880x1800_60.00
xrandr: --addmode requires two arguments
Try 'xrandr --help' for more information.
[bwang@bwang ~]$ xrandr --addmode VNC-0 2880x1800_60.00
[bwang@bwang ~]$ xrandr
Screen 0: minimum 32 x 32, current 1920 x 1080, maximum 32768 x 32768
VNC-0 connected primary 1920x1080+0+0 0mm x 0mm
   1920x1080     60.00*+
   1920x1200     60.00  
   1600x1200     60.00  
   1680x1050     60.00  
   1400x1050     60.00  
   1360x768      60.00  
   1280x1024     60.00  
   1280x960      60.00  
   1280x800      60.00  
   1280x720      60.00  
   1024x768      60.00  
   800x600       60.00  
   640x480       60.00  
   2880x1800_60.00  59.97  

[bwang@bwang ~]$ xrandr -s 2880x1800_60.00
[bwang@bwang ~]$ xrandr --dpi 220
[bwang@bwang ~]$ xdpyinfo
...
screen #0:
  dimensions:    2880x1800 pixels (332x207 millimeters)
  resolution:    220x221 dots per inch
  depths (7):    1, 4, 8, 16, 24, 32, 24
  root window id:    0x268
  depth of root window:    24 planes
  number of colormaps:    minimum 1, maximum 1
  default colormap:    0x20
  default number of colormap cells:    256
  preallocated pixels:    black 0, white 16777215
  options:    backing-store WHEN MAPPED, save-unders NO
  largest cursor:    2880x1800
  current input event mask:    0xda0003
    KeyPressMask             KeyReleaseMask           StructureNotifyMask      
    SubstructureNotifyMask   SubstructureRedirectMask PropertyChangeMask       
    ColormapChangeMask       
  number of visuals:    240
  default visual id:  0x21
...
The font size is too small to read. I tried several ways to make the fonts larger:
  • Tweak tool -> Windows -> HiDPI -> Window Scaling, change 1 to 2. This method make the font and window larger, but it is too large to me. Another issue is the font of menu is still small.
  • Run gsettings set org.gnome.desktop.interface.scaling-factor 2. Only integer is allowed, and make the font size larger on the menu bar, but not the font of terminal.
  • Scale X window xrandr --screen 0 --output VNC-0 --scale 2x2. This scaling bases on bitmap, it will cause blurry. And I ran multiple times into trouble when I set to 0.5x0.5 then 1x1, the screen zoom in with only big pixels, then I have to restart VNC server.
  • Set font scaling factor: Tweak Tool -> Fonts -> Scaling Factor. Changing to 1.5 works for me. This change makes the font larger everywhere: system menu and terminal, except Intellij IDEA. The font still look small, but it is super sharp. gsettings set org.gnome.desktop.interface text-scaling-factor 1.5
If the VNC server restarts, the new screen size will be lost. Need to find a better way to set it. Of course, I can put those commands in VNC xstartup or xinitrc.
Another thing is to make it easier to switch to lower resolution.

Monday, December 18, 2017

"No such a file or directory" when setting NETGEAR router

I got this error when I was setting up a new NETGEAR X6 router after setting up the new password. I restored the factory settings multiple times, but it either showed "No such a file or directory" for the link "http://www.routerlogin.net/genie_index.html" or pop up a dialog for user name and password, but the default "admin" and "password" didn't work.

I finally realized that it was my fault, not NETGEAR's. My laptop connected to NETGEAR X6 using a cable, but to my old NETGEAR router using WiFi. When the new router redirected the page to "http://www.routerlogin.net/genie_index.html", my laptop messed up with www.routerlogin.net because it tried to the old NETGEAR router. Turning off WiFi of my laptop solved the problem, and X6 home page showed up correctly.

Tuesday, December 12, 2017

Allow the outside world access your server in a container

I run docker containers of Nexus and httpd on a CentOS 7 host. I added a nexus.service to start the containers using docker-compose. I could access the nexus server from any machine after I started the service. But the next day, I could not access that server from other machines, running curl -k -X GET https://<host-ip> always got time out. The containers were still running and they were still bound to all interfaces because I could run curl -k -X GET https://<host-ip> on that host. The below shows those three ports 80,443,15001 of all interfaces are listened. NOTE: Proto=tcp6 doesn’t means “not listening on ipv4”.
# netstat -l -t
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp6       0      0 [::]:http               [::]:*                  LISTEN     
tcp6       0      0 [::]:15001              [::]:*                  LISTEN     
tcp6       0      0 [::]:https              [::]:*                  LISTEN
If I restart the service docker using systemctl restart docker, and service nexus systemctl start nexus, everything works again. If I reboot the server, docker starts and the containers are running, but I cannot access the Nexus server from any machine except the host.
It turns out that the host runs chef-client in the early morning and after rebooting, and set net.ipv4.ip_forward = 0. I can run sysctl net.ipv4.ip_forward=1 to make remote access to the Nexus server, and sysctl net.ipv4.ip_forward=0 to deny any access.
If I restarts the service docker, docker set net.ipv4.ip_forward=1 automatically. Check the docker document Communcating to the outside world
IP packet forwarding is governed by the ip_forward system parameter. Packets can only pass between containers if this parameter is 1. Usually you will simply leave the Docker server at its default setting —ip-forward=true and Docker will go set ip_forward to 1 for you when the server starts up. If you set —ip-forward=false and your system’s kernel has it enabled, the —ip-forward=false option has no effect. To check the setting on your kernel or to turn it on manually:

Tuesday, October 24, 2017

VNC server won't start on my CentOS 7

I had to ask my boss to power down and power up the desktop in my office this morning. After the desktop came back, I could not access the vncserver because it failed to start. And I got this error when I run systemctl.
$ sudo systemctl start vncserver@\:2

Job for vncserver@:2.service failed because a configured resource limit was exceeded. See "systemctl status vncserver@:2.service" and "journalctl -xe" for details.
And I also found a Xvnc process started on :1 every time when I start the service.
I did find that `/usr/lib/systemd/system/vncserver@:2.service was missing and restored it like below:
[Unit]
Description=Remote desktop service (VNC)
After=syslog.target network.target

[Service]
Type=forking
User=bwang

# Clean any existing files in /tmp/.X11-unix environment
ExecStartPre=-/usr/bin/vncserver -kill %i
ExecStart=/usr/bin/vncserver %i -nolisten tcp -localhost -geometry 1920x1080 -de
pth 24 
PIDFile=/home/bwang/.vnc/%H%i.pid
ExecStop=-/usr/bin/vncserver -kill %i

[Install]

WantedBy=multi-user.target
And I ran systemctl daemon-reload, systemctl disable vncserver@\:2.service, systemctl enable vncserver@\:2.service multiple times, but got the same error again and again.
I also tried using vncserver\@\:2 or vncserver@:2 for service name, it won’t fix it.
Finally when I ran vncserver from the command line, it showed this message
$ vncserver :2 -nolisten tcp -localhost -geometry 1920x1080 -depth 24

Warning: bwang.corp.rhapsody.com:2 is taken because of /tmp/.X11-unix/X2
Remove this file if there is no X server bwang.corp.rhapsody.com:2
A VNC server is already running as :2

New 'bwang.corp.rhapsody.com:1 (bwang)' desktop is bwang.corp.rhapsody.com:1

Starting applications specified in /home/bwang/.vnc/xstartup
Log file is /home/bwang/.vnc/bwang.corp.rhapsody.com:1.log
Obviously, when the deskotp was powered down, /tmp/.X11-unix/X2 was left on the hard drive, and it blocked vncserver from starting on display 2 again.
Removing /tmp/.X11-unix/X2 fix the problem.
NOTE: using vncserver@:2 without escaping : is ok when running systemctl.

Wednesday, October 4, 2017

Spark SQL AnalysisException due to data type mismatch

If you run the following code, you will encounter AnalysisException with data type mismatch.
spark.sql("create table test_table(id int, data struct<name: string, age:int>)")
spark.sql("insert overwrite table test_table select 1, struct('joe', 15)")
val d = spark.read.table("test_table")

case class UserData(name: String, age: Int)
case class User(id: Int, data: UserData)
val u = Seq(User(1, UserData("joe", 15)), User(2, UserData("mary", 10))).toDF

val missing = {
  u.join(
    d,
    u("id") === d("id")
    && u("data")("name") === d("data")("name")
    && u("data")("age") === d("data")("age"),
    "outer"
  )
  .where( u("id").isNull || d("id").isNull)
}

// show this result
// +---+---------+----+----+                                                      
// |id |data     |id  |data|
// +---+---------+----+----+
// |2  |[mary,10]|null|null|
// +---+---------+----+----+
missing.show(false)

// Throws this error: org.apache.spark.sql.AnalysisException: cannot resolve '(`data` = test_table.`data`)'
// due to data type mismatch: differing types in '(`data` = test_table.`data`)'
// (struct<name:string,age:int> and struct<name:string,age:int>).;
val missing_2 = {
  u.join(d,
         u("id") === d("id") && u("data") === d("data"),
         "outer")
    .where(u("id").isNull || d("id").isNull)
}
Don’t be fooled by (struct<name:string,age:int> and struct<name:string,age:int>). The problem is caused by nullable, which is not shown in the error message. The DataFrame created from case classes has nullable=false for id and age because Scala Int cannot be null, while the SQL creates nullable fields. And if you compare a field with complex type (struct, array), Spark just thinks they are different as shown in missing_2. But if you compare field by field, there is no problem as shown in missing.
scala> u.schema.treeString
res4: String =
"root
 |-- id: integer (nullable = false)
 |-- data: struct (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- age: integer (nullable = false)
"

scala> d.schema.treeString
res5: String =
"root
 |-- id: integer (nullable = true)
 |-- data: struct (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- age: integer (nullable = true)
"

Thursday, August 17, 2017

Mac OS SSH Kerberose Single-Sign-On

To use Kerberos Single-Sign-On on a MacBookPro, you can do as follows:
  • Make a copy of /etc/krb5.conf from a server in your company network.
  • Save it into /etc/krb5.confon your MacBookPro. You probably need sudo.
  • Run kinit <your-login>@<realm_name>. If you don’t know what is the realm name is, take a look at /etc/krb5.conf
    [libdefaults]
    default_realm = MY.COMPANY.COM
    
  • Add this line to ~/.ssh/config. Without this line, password method is preferred.
    GSSAPIAuthentication yes
    
  • Then you can ssh to any host supporting kerberos without typing a password.
  • If your kerberos ticket expires, just run kinit again. You don’t have to provide the principal <your-login>@<realm_name> at this time.
  • Use klist to show when the ticket will expire.

Gnome Terminal Profile Export

I usually run screen on a remote server in PROD environment. Whenever I want to access the server, I SSH to that host, and run screen -D -R to resume the session I leave before. Using screen, you don’t have to worry about your SSH session suddenly goes down due to the Internet connection issue.
I also adjust the terminal running the SSH connection with a title and special color scheme so that I can quickly tell this is a PROD environment and need to be careful.
I create a Gnome terminal profile so that I only need to click “File” -> “Open Terminal” -> “prod-env”. “prod-env” is the profile name with all customized attributes:
  • Title
  • Terminal window size
  • Automatically run a command ssh -t <host> screen -D -R
  • Specified color-scheme
How can I share the profile among the multiple environments, e.g., a laptop and a desktop running Gnome environment? Of course, I can set it manually on both of machines. It would be better if I can export the profile and import it on other machines.
It is actually pretty simple. The following command works on CentOS 7.3 and Gnome 3.
  • Export
    dconf dump /org/gnome/terminal/ > /tmp/terminal-profile.dconf
    
  • Import
    cat /tmp/terminal-profile.dconf | dconf load
    

Monday, June 12, 2017

Screen in docker with error "Must be connected to a terminal"

I have a running docker container with command /bin/bash --login. When I run the command
$ docker exec -it devsh /bin/bash --login
I can access the container’s Bash, but when I run screen, I got this error:
$ screen
Must be connected to a terminal.
It turns out that I should use docker attach devsh. This command allows me to access the original shell in the container.

Friday, May 26, 2017

Install Cygwin in a script

I built an Intellij docker image based on CentOS 7. To allow my colleagues to use it on Windows, I need to help them to setup a X window system. This post describe how to setup Cygwin/X using a script without user intervention.
To run in Windows, the best choice for the script is PowerShell. PowerShell is powerful, but ugly compared to Bash.
$CygwinDir=<where you want to install Cygwin>
$CygwinPkgsDir=<where the cygwin packages are cached>

function Download($uri, $outfile) {
  $webClient = New-Object System.Net.WebClient
  $Webclient.DownloadFile($uri, $outfile)
}

function Install-CygwinX {
  $setup = "$DownloadDir\setup-x86_64.exe"
  if (!(Test-Path "$setup")) {
    Download `
      -uri "https://cygwin.com/setup-x86_64.exe" `
      -outfile "$setup"
  }

  Start-Process "$setup" -ArgumentList "--site ""http://mirrors.xmission.com/cygwin"" --root ""$CygwinDir"" --packages xorg-server,xhost --no-admin --local-package-dir ""$CygwinPkgsDir"" --upgrade-also --quiet-mode" -Wait -NoNewWindow
}
You can run this command to get the full list of command line options:
Downloads\setup-x86_64.exe --help
Most important command line options of Cygwin setup are:
  • --no-admin: Your user won’t have to be administrator
  • --quiet-mode: The script will run without asking anything
  • --packages: the packages you need to run a Cygwin/X. No package selection is needed.
The PowerShell script will download the setup executable and run it automatically. Here are some thing you need to know:
  • Don’t use Invoke-WebRequest because it is too slow. For Cygwin setup, it is not a big problem because setup-x86_64.exe is small, and Cygwin setup will download the rest package. If you download about 200MB file like ‘Docker Toolbox for Windows’, you will see how slow it is.
  • Use Start-Process to start the setup executable.
    • -Wait makes the script wait until Cygwin setup finished
    • -NoNewWindow is important if you run a .bat file. Without it, your .bat file will run in a separate window, just appear then disappear. If anything is wrong, you have no way to see what is the error message.

Install Docker Toolbox for Windows Automatically

Docker Toolbox for Windows have command line arguments which allows you to install it without user’s involvement.
You can run this command in Windows Command Prompt to get those arguments:
> Downloads\DockerToolbox.exe /HELP
The most useful arguments for automation are: /SILENT, /DIR, /COMPONENTS, and /TASKS.
To know what values for /COMPONENTS and /TASKS, you can run this command
> Downloads\DockerToolbox.exe /SAVEINF=docker_toolbox.inf
[Setup]
Lang=english
Dir=C:\Program Files\Docker Toolbox
Group=Docker
NoIcons=0
SetupType=full
Components=docker,dockermachine,dockercompose,virtualbox,kitematic,git
Tasks=desktopicon,modifypath,upgradevm
If VirtualBox is already installed and I don’t want to install Kitematic, I can run this command, or put the command into the script to skip them. The installation will be automated.
> Downloads\DockerToolbox.exe /COMPONENTS=docker,dockermachine,dockercompose,git /TASKS=desktopicon,modifypath,upgradevm