While at it, also do the unmap only if a specific key has been mapped.
There are some keys that are only selectively mapped and it would cause
an error if we are trying to unmap such a key (e.g. z when your
foldmethod is not manual).
fixes: #14040
Signed-off-by: Christian Brabandt <cb@256bit.org>