WSL mv permission denied

If you get "Permission Denied" when you `mv` a file, check if you open the parent directory of the file in Windows Explorer. If so, close Windows Explorer or change it to another directory, and try again. https://github.com/microsoft/WSL/issues/1529

Automic Application Manager V9 Client: Run with Java 9

My team uses Automic AM v9.1.0 (v9.1.0_28363_28431), which is configured with Java 6 and Java 7 by default. It also works in Java 8. However, if you run it with Java 9, you will encounter network error with the followings in detail:

javax.net.ssl.SSLHandshakeException: DHPublicKey does not comply to algorithm constraints
 at java.base/sun.security.ssl.DHCrypt.checkConstraints(DHCrypt.java:237)
 at java.base/sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:837)
 at java.base/sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:320)

The reason you see this error is Java 9 requires DH key size must be larger than 1024, and the version of v9.1.0 must use DHPublicKey shorter than 1024. Check /usr/java/default/conf/security/java.security.

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224

If you update the file as below, you can run the client with Java 9.

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, \
    EC keySize < 224

The reason I want to run AM using Java 9 because Java 9 have a better support for GDK and I can get sharp font on HiDPI screen like MacBook Pro with Retina Display.

• You can run $JAVA_HOME/bin/javaws -J-Dawt.useSystemAAFontSettings=lcd http://am_host/am_engine/Client.jnlp to get better fonts with smooth edge.
• Set environment variable GDK_SCALE=2 to make the client window large on a HiDPI display.

Because this method needs to change java.security, it will be applied to every application using the Java 9 deployment, and may be a security issue. I’m running AM in a docker container, there is no such a problem, because the Java with this change is only for AM.

​

Read HAR to Spark DataFrame

When I use spark-streaming to pull JSON events from a Kafka topic and persist the data into HDFS, I have to handle a lot of tiny files because the volume of the Kafka topic is pretty small. Too many small files will cause bad performance of Hadoop NameNode. Usually you need to build Hadoop Archive (.har) to make the small files into one big archive file.

The problem is how to read the archive file (.har) into Spark DataFrame. Method text and json of Spark DataFrameReader won’t work for the path of an archive file. You have to use SparkContext#textFile and the file path needs to be ${har_path}/*.

Here is the example showing how to read the files in a HAR. DataFrameReader read nothing for all three path patterns. SparkContext.textFile successfully read the data for the patterns of dir and file.

val har = "har:///tmp/test-data/bwang/starring/tag-v1-1511170200-1511175600.har"

val paths = Map(
    "har" -> har,
    "dir" -> s"$har/tag-*",
    "file" -> s"$har/tag-*/part-*"
)

println("DataFrameReader different HAR paths")

paths.foreach {
    case (kind, path) =>
        val data = spark.read.text(path)
        println(s"--- Reading $kind using path $path.")
        data.show(2, false)
}

println("SparkContext#textFile different HAR paths")

paths.foreach {
    case (kind, path) =>
        try {
            val data = sc.textFile(path).toDF
            println(s"--- Reading $kind using path $path.")
            data.show(2, false)
        } catch {
            case e: java.io.IOException =>
                println(s" --   Failed. ${e.getMessage}")
        }
}

​

Make Intellij IDEA on Linux with sharp fonts as Macbook Pro

IntelliJ IDEA supports HiDPI on Windows and MacOS. The fonts look so beautiful on a Macbook Pro Retina Display. Unfortunately IDEA doesn’t support HiDPI on Linux. Looks at the code UIUtil#isJreHiDPIEnabled. For Linux, it always return false.

You can get that sharpness through VNC by changing to the larger font size. I’m using 2017.3.1

• Settings -> Editor -> Color Shcema -> Color Scheme Font
  • Check “Use color scheme font instead of the default`
  • Change “Size” to 22
• Settings -> Appearance & Behavior -> Appearance -> UI Options
  • Themes: Darcular
  • Check “Override default fonts by (not recommended):”
  • Name: Dialog and size to 22

HiDPI-profiles plugin may make switching between HiDPI and lower resolution easier.

​

How to make VNC sharp on Retina display.

Most of time, I work on a Linux box running CentOS 7 through VNC. The largest VNC screen size is 1920x1080 (16:9). It is ok when you maximize the RealVNC viewer window on an external monitor with that resolution. On my Macbook Pro, there are two black areas on top and bottom because the Retina display is 2880x1800 (16:10), and the font looks small and blurry. I have to switch VNC screen side to 1680x1050 (16:10), the font size is proper, but still blurry. How can I make VNC sharp as a native MacOS app?

First, create a larger screen on VNC as below, then you will have the same size VNC screen as Macbook Pro.

• Create a modeline using cvt
• Create a new mode using the modeline
• Add the new mode
• Switch to the new mode
• Set screen DPI to 220, which is the DPI of Macbook Pro’s Retina Display

[bwang@bwang ~]$ cvt 2880 1800
# 2880x1800 59.97 Hz (CVT 5.18MA) hsync: 111.84 kHz; pclk: 442.00 MHz
Modeline "2880x1800_60.00"  442.00  2880 3104 3416 3952  1800 1803 1809 1865 -hsync +vsync
[bwang@bwang ~]$ xrandr --newmode "2880x1800_60.00"  442.00  2880 3104 3416 3952  1800 1803 1809 1865 -hsync +vsync
[bwang@bwang ~]$ xrandr
Screen 0: minimum 32 x 32, current 1680 x 1050, maximum 32768 x 32768
VNC-0 connected primary 1680x1050+0+0 0mm x 0mm
   1920x1080     60.00 +
   1920x1200     60.00  
   1600x1200     60.00  
   1680x1050     60.00* 
   1400x1050     60.00  
   1360x768      60.00  
   1280x1024     60.00  
   1280x960      60.00  
   1280x800      60.00  
   1280x720      60.00  
   1024x768      60.00  
   800x600       60.00  
   640x480       60.00  
  2880x1800_60.00 (0x29f) 442.000MHz -HSync +VSync
        h: width  2880 start 3104 end 3416 total 3952 skew    0 clock 111.84KHz
        v: height 1800 start 1803 end 1809 total 1865           clock  59.97Hz
[bwang@bwang ~]$ xrandr --addmode 2880x1800_60.00
xrandr: --addmode requires two arguments
Try 'xrandr --help' for more information.
[bwang@bwang ~]$ xrandr --addmode VNC-0 2880x1800_60.00
[bwang@bwang ~]$ xrandr
Screen 0: minimum 32 x 32, current 1920 x 1080, maximum 32768 x 32768
VNC-0 connected primary 1920x1080+0+0 0mm x 0mm
   1920x1080     60.00*+
   1920x1200     60.00  
   1600x1200     60.00  
   1680x1050     60.00  
   1400x1050     60.00  
   1360x768      60.00  
   1280x1024     60.00  
   1280x960      60.00  
   1280x800      60.00  
   1280x720      60.00  
   1024x768      60.00  
   800x600       60.00  
   640x480       60.00  
   2880x1800_60.00  59.97  

[bwang@bwang ~]$ xrandr -s 2880x1800_60.00
[bwang@bwang ~]$ xrandr --dpi 220
[bwang@bwang ~]$ xdpyinfo
...
screen #0:
  dimensions:    2880x1800 pixels (332x207 millimeters)
  resolution:    220x221 dots per inch
  depths (7):    1, 4, 8, 16, 24, 32, 24
  root window id:    0x268
  depth of root window:    24 planes
  number of colormaps:    minimum 1, maximum 1
  default colormap:    0x20
  default number of colormap cells:    256
  preallocated pixels:    black 0, white 16777215
  options:    backing-store WHEN MAPPED, save-unders NO
  largest cursor:    2880x1800
  current input event mask:    0xda0003
    KeyPressMask             KeyReleaseMask           StructureNotifyMask      
    SubstructureNotifyMask   SubstructureRedirectMask PropertyChangeMask       
    ColormapChangeMask       
  number of visuals:    240
  default visual id:  0x21
...

The font size is too small to read. I tried several ways to make the fonts larger:

• Tweak tool -> Windows -> HiDPI -> Window Scaling, change 1 to 2. This method make the font and window larger, but it is too large to me. Another issue is the font of menu is still small.
• Run gsettings set org.gnome.desktop.interface.scaling-factor 2. Only integer is allowed, and make the font size larger on the menu bar, but not the font of terminal.
• Scale X window xrandr --screen 0 --output VNC-0 --scale 2x2. This scaling bases on bitmap, it will cause blurry. And I ran multiple times into trouble when I set to 0.5x0.5 then 1x1, the screen zoom in with only big pixels, then I have to restart VNC server.
• Set font scaling factor: Tweak Tool -> Fonts -> Scaling Factor. Changing to 1.5 works for me. This change makes the font larger everywhere: system menu and terminal, except Intellij IDEA. The font still look small, but it is super sharp. gsettings set org.gnome.desktop.interface text-scaling-factor 1.5

If the VNC server restarts, the new screen size will be lost. Need to find a better way to set it. Of course, I can put those commands in VNC xstartup or xinitrc.

Another thing is to make it easier to switch to lower resolution.

​

"No such a file or directory" when setting NETGEAR router

I got this error when I was setting up a new NETGEAR X6 router after setting up the new password. I restored the factory settings multiple times, but it either showed "No such a file or directory" for the link "http://www.routerlogin.net/genie_index.html" or pop up a dialog for user name and password, but the default "admin" and "password" didn't work.

I finally realized that it was my fault, not NETGEAR's. My laptop connected to NETGEAR X6 using a cable, but to my old NETGEAR router using WiFi. When the new router redirected the page to "http://www.routerlogin.net/genie_index.html", my laptop messed up with www.routerlogin.net because it tried to the old NETGEAR router. Turning off WiFi of my laptop solved the problem, and X6 home page showed up correctly.

Allow the outside world access your server in a container

I run docker containers of Nexus and httpd on a CentOS 7 host. I added a nexus.service to start the containers using docker-compose. I could access the nexus server from any machine after I started the service. But the next day, I could not access that server from other machines, running curl -k -X GET https://<host-ip> always got time out. The containers were still running and they were still bound to all interfaces because I could run curl -k -X GET https://<host-ip> on that host. The below shows those three ports 80,443,15001 of all interfaces are listened. NOTE: Proto=tcp6 doesn’t means “not listening on ipv4”.

# netstat -l -t
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp6       0      0 [::]:http               [::]:*                  LISTEN     
tcp6       0      0 [::]:15001              [::]:*                  LISTEN     
tcp6       0      0 [::]:https              [::]:*                  LISTEN

If I restart the service docker using systemctl restart docker, and service nexus systemctl start nexus, everything works again. If I reboot the server, docker starts and the containers are running, but I cannot access the Nexus server from any machine except the host.

It turns out that the host runs chef-client in the early morning and after rebooting, and set net.ipv4.ip_forward = 0. I can run sysctl net.ipv4.ip_forward=1 to make remote access to the Nexus server, and sysctl net.ipv4.ip_forward=0 to deny any access.

If I restarts the service docker, docker set net.ipv4.ip_forward=1 automatically. Check the docker document Communcating to the outside world

IP packet forwarding is governed by the ip_forward system parameter. Packets can only pass between containers if this parameter is 1. Usually you will simply leave the Docker server at its default setting —ip-forward=true and Docker will go set ip_forward to 1 for you when the server starts up. If you set —ip-forward=false and your system’s kernel has it enabled, the —ip-forward=false option has no effect. To check the setting on your kernel or to turn it on manually:

​

VNC server won't start on my CentOS 7

I had to ask my boss to power down and power up the desktop in my office this morning. After the desktop came back, I could not access the vncserver because it failed to start. And I got this error when I run systemctl.

$ sudo systemctl start vncserver@\:2

Job for vncserver@:2.service failed because a configured resource limit was exceeded. See "systemctl status vncserver@:2.service" and "journalctl -xe" for details.

And I also found a Xvnc process started on :1 every time when I start the service.

I did find that `/usr/lib/systemd/system/vncserver@:2.service was missing and restored it like below:

[Unit]
Description=Remote desktop service (VNC)
After=syslog.target network.target

[Service]
Type=forking
User=bwang

# Clean any existing files in /tmp/.X11-unix environment
ExecStartPre=-/usr/bin/vncserver -kill %i
ExecStart=/usr/bin/vncserver %i -nolisten tcp -localhost -geometry 1920x1080 -de
pth 24 
PIDFile=/home/bwang/.vnc/%H%i.pid
ExecStop=-/usr/bin/vncserver -kill %i

[Install]

WantedBy=multi-user.target

And I ran systemctl daemon-reload, systemctl disable vncserver@\:2.service, systemctl enable vncserver@\:2.service multiple times, but got the same error again and again.

I also tried using vncserver\@\:2 or vncserver@:2 for service name, it won’t fix it.

Finally when I ran vncserver from the command line, it showed this message

$ vncserver :2 -nolisten tcp -localhost -geometry 1920x1080 -depth 24

Warning: bwang.corp.rhapsody.com:2 is taken because of /tmp/.X11-unix/X2
Remove this file if there is no X server bwang.corp.rhapsody.com:2
A VNC server is already running as :2

New 'bwang.corp.rhapsody.com:1 (bwang)' desktop is bwang.corp.rhapsody.com:1

Starting applications specified in /home/bwang/.vnc/xstartup
Log file is /home/bwang/.vnc/bwang.corp.rhapsody.com:1.log

Obviously, when the deskotp was powered down, /tmp/.X11-unix/X2 was left on the hard drive, and it blocked vncserver from starting on display 2 again.

Removing /tmp/.X11-unix/X2 fix the problem.

NOTE: using vncserver@:2 without escaping : is ok when running systemctl.

​

Spark SQL AnalysisException due to data type mismatch

If you run the following code, you will encounter AnalysisException with data type mismatch.

spark.sql("create table test_table(id int, data struct<name: string, age:int>)")
spark.sql("insert overwrite table test_table select 1, struct('joe', 15)")
val d = spark.read.table("test_table")

case class UserData(name: String, age: Int)
case class User(id: Int, data: UserData)
val u = Seq(User(1, UserData("joe", 15)), User(2, UserData("mary", 10))).toDF

val missing = {
  u.join(
    d,
    u("id") === d("id")
    && u("data")("name") === d("data")("name")
    && u("data")("age") === d("data")("age"),
    "outer"
  )
  .where( u("id").isNull || d("id").isNull)
}

// show this result
// +---+---------+----+----+                                                      
// |id |data     |id  |data|
// +---+---------+----+----+
// |2  |[mary,10]|null|null|
// +---+---------+----+----+
missing.show(false)

// Throws this error: org.apache.spark.sql.AnalysisException: cannot resolve '(`data` = test_table.`data`)'
// due to data type mismatch: differing types in '(`data` = test_table.`data`)'
// (struct<name:string,age:int> and struct<name:string,age:int>).;
val missing_2 = {
  u.join(d,
         u("id") === d("id") && u("data") === d("data"),
         "outer")
    .where(u("id").isNull || d("id").isNull)
}

Don’t be fooled by (struct<name:string,age:int> and struct<name:string,age:int>). The problem is caused by nullable, which is not shown in the error message. The DataFrame created from case classes has nullable=false for id and age because Scala Int cannot be null, while the SQL creates nullable fields. And if you compare a field with complex type (struct, array), Spark just thinks they are different as shown in missing_2. But if you compare field by field, there is no problem as shown in missing.

scala> u.schema.treeString
res4: String =
"root
 |-- id: integer (nullable = false)
 |-- data: struct (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- age: integer (nullable = false)
"

scala> d.schema.treeString
res5: String =
"root
 |-- id: integer (nullable = true)
 |-- data: struct (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- age: integer (nullable = true)
"

​

Mac OS SSH Kerberose Single-Sign-On

To use Kerberos Single-Sign-On on a MacBookPro, you can do as follows:

• Make a copy of /etc/krb5.conf from a server in your company network.
• Save it into /etc/krb5.confon your MacBookPro. You probably need sudo.
• Run kinit <your-login>@<realm_name>. If you don’t know what is the realm name is, take a look at /etc/krb5.conf

  [libdefaults]
  default_realm = MY.COMPANY.COM
  

• Add this line to ~/.ssh/config. Without this line, password method is preferred.

  GSSAPIAuthentication yes
  

• Then you can ssh to any host supporting kerberos without typing a password.
• If your kerberos ticket expires, just run kinit again. You don’t have to provide the principal <your-login>@<realm_name> at this time.
• Use klist to show when the ticket will expire.

​