I always have a hard time remembering how to set the keyboard shortcuts for umlauts on GNU/Linux.

Use the runtime utility setxkbmap:
setxkbmap -option "compose:lwin"

So now (+ just means "and then press" in this context):

Also don't forget to set ctrl+alt+backspace back to a sane setting with:
setxkbmap -option "terminate:ctrl_alt_bksp"