I encountered this error while trying to setup Wireguard on a Debian VPS:
$ ip link add dev wg0 type wireguard
RTNETLINK answers: Operation not supported
There isn’t much on the web, this can be due to a network problem, particularly if you’re using a VPS:
https://serverfault.com/questions/772648/create-vxlan-rtnetlink-answers-operation-not-supported
But luckily I found that in my case, the problem was caused by missing Linux-headers, so I did this:
sudo apt install linux-headers-$(uname --kernel-release)
It solved the error, but then I got another one, the solution was to install another version of the header (surely due to the fact that I am using backports).