Miloš Pejanović

tech blog

Using macOS' built-in vnc server from X11

Published Apr 26 2020

Why

I wanted an easy way to control my macbookpro from my desktop setup.

What I found was a lot of proprietary options and a couple of open sourced ones that were okay, but always lacking some little thing that would annoy the hell out of me. Since I’m a cheap bastard and love open source I opted for the latter with modifications. Most vnc options were extremely slow, so I “vnced” my keyboard and mouse only (x2vnc), and connected the macbookpro to my monitor input.

This was working seamlessly.

The modifications I made were minor but finding the correct ones cost me some time.

The time/effort I spent here shouldn’t be repeated.

Requirements

  • a machine running macOS and “Screen Sharing” (using Mojave 10.14.16)
  • a client machine running X11 (using X.Org version 1.20.8)
  • a vnc client (using x2vnc)
  • local network access

Okay, here we go.

Setup your mac to use a vnc server

  1. System preferences > Sharing
  2. Fill Screen Sharing checkbox
  3. Computer settings… > fill password checkbox and set a password
  4. Run defaults write /Library/Preferences/com.apple.RemoteManagement VNCAlwaysStartOnConsole -bool true (this will let the vnc client connect into the currently logged in user, skipping the login. Important if youre going in without video, like in this example)

Note: you can see your vnc server address just above the “Computer settings” button (“vnc://x.x.x.x”)

Setup your X system to use your vnc server

  1. Install x2vnc (available in AUR for me, but you can always download and build it yourself).
  2. rtfm man x2vnc for basic usage and options.
  3. connect, press the hotkey (the one you supplied or if not the default one: F12)
  4. profit!

Too easy, right?

Problems

The connection is running fine, no interruptions, mouse and keyboard are working almost seamlessly. But!

Some of the keys are swapped, as macOS usually does with keyboards from other vendors (in my case those were control and super).

Some keys were not mapped. (in my case alt was missing, that is, binding the alt key to the “option” key on mac).

Solutions

Remap to the rescue!

First thing I installed was Karabiner-elements. This is a great tool to remap keys on your mac keyboard or any other keyboard you may be using directly on the mac. Ive used this tool before and it was working perfectly. Unfortunately the remaps were being ignored, possibly due to the fact I was connecting through a vnc.

Secondly I tried remapping keyboard modifier keys through macOS’ Keyboard preferences. As with karabiner, no luck.

Alright, lets remap the keys on my client side then.

Enter xmodmap. This wild thing was the most effective way to remap keys on a X system and nicely reflects the underlying system (most likely over-engineered).

I also tried using setxkbmap for remapping, but could not get the desired effect. It does however, work very well to reset your keymap to their defaults (which I used).

Configure xmodmap

My keyboard layout isn’t generic in any way, but I was happily using the generic 102 layout provided by default without any issues.

You can find out what your current layout modifier map looks like with xmodmap -pm. Heres mine:

shift       Shift_L (0x32),  Shift_R (0x3e)
lock        Caps_Lock (0x42)
control     Control_L (0x25),  Control_R (0x69)
mod1        Alt_L (0x40),  Alt_R (0x6c),  Meta_L (0xcd)
mod2        Num_Lock (0x4d)
mod3
mod4        Super_L (0x85),  Super_R (0x86),  Super_L (0xce),  Hyper_L (0xcf)
mod5        ISO_Level3_Shift (0x5c),  Mode_switch (0xcb)

Looks mostly normal: the modifiers are connected to the correct keys.

For my sessions with the mac machine I needed the following corrected:

  1. Control and super were swapped out.
  2. Alt key did the same as super, but needed it to perform as the “option” key.

Heres my take on the remapping:

remove mod4 = Super_L
remove control = Control_L
remove mod1 = Alt_L

keycode 133 = Control_L
keycode 37 = Super_L
keycode 64 = Meta_L

add mod4 = Super_L
add control = Control_L
add mod1 = Meta_L

Let me explain (im not even sure this is correct), or alternatively read man xmodmap (not light reading):

  • lines 1-3 lines are about removing the existing mapping (you wouldn’t be able to remap keys if they’re already mapped).
  • the next block is about assigning your keycodes to the actual keysym that will be used to map to the modifier in the last block (see below for keycode explanation).
  • the last block is reassigning the keysyms to the modifiers we removed in the first step (note that these now correspond to different keycodes)

Note: this example is only remapping the left side modifiers since those are the only ones I use. If your keyboard is different or you’re using the other or both sides, modify accordingly.

Keycodes explained

The keycode is the numeric representation received by the kernel when a key or a mouse button is pressed (source).

To find which keysym correspond to which keycode by default for your keyboard/layout look for them with xmodmap -pke | grep <keysym>

So for my use cases I had to find keycodes for Super_L, Control_L and Alt_L (and those were 133, 37 and 64, in the order given).

So, in the example above I’m assigning 133 to Control_L, 37 to Super_L (swap super and control), and 64 to Meta_L (the previous Alt_L). WAT?

Why Meta_L

This is the tricky part (and the one that ate most of my time spent on this). It turns out that the option key on the macbookpro keyboard via vnc corresponds to Meta keysym. Yep. Not Alt, not Mode_switch or ISO_Level3_Shift as suggested by some reading, but Meta.

dont even have that one on my keyboard :`)

Im not sure if this is on me, my keyboard, mac’s vnc server or the client I’m using, but I have to say: well done.

And how did I find this out, you ask?

I implemented a test vnc client that would send different modifiers with key combinations to check my sanity and confirm that I will/wont be able to use the “option” key while vnc-ing.

Tying the client side together with a bash script

#!/bin/bash

trap 'echo interrupted' INT
xmodmap ~/mac.xmod
x2vnc -hotkey F9 192.168.0.15:0 -passwd $HOME/.vncpasswd -edgewidth 0 -wheelhack -scrolllines 3
setxkbmap -option

Explanation

  • The first line traps the interrupt signal you will have to supply to get out of the vnc client connection, and continues running commands after it.
  • xmodmap ~/mac.xmod this applies the example mapping provided above (you can feed it into xmodmap from a file)
  • x2vnc ... connects to the vnc server. In order to switch your mouse/keyboard to the other machine, press the hotkey (the one you provided, or if not F12). Note that if you remapped your control key (as I did with Control_L) you have to use the updated mappings to issue an interrupt signal (for the example here, its Super_L+C)
  • Finally setxkbmap -option resets the keymappings to their default values

Conclusion

This was a bit of a headache but rewarding. I now have a fast, reliable and free way to control my macbookpro from my desktop setup.

Now, I need a drink.

Cheers.