Migrating from Libvirt to Qemu

virgl

Graphics in Qemu

Performance Impact of SPICE+VirGL

I have been using libvirt (virt-manager) for years, until recently I tried to run Linux VMs with Qemu directly using virgl.

Running a Linux VM with Qemu's SDL display is much smoother than SPICE+VirGL solution provided by virt-manager, it made me realize that SPICE actually has an impact on graphics performance in VMs.

SDL

I copied Qemu parameters from Quickemu project, which works for both Linux and Windows.

If you use SDL under wayland, you probably need to tell SDL to use wayland backend:

export SDL_VIDEODRIVER=wayland

There's another feature that you are going to lose if you use SDL instead of SPICE: copy/paste between host and guests. This can be partially solved with file system sharing

We can use virtio-9p:

-fsdev local,id=fsdev0,path="$HOME/Public",security_model=mapped-xattr \
    -device virtio-9p-pci,fsdev=fsdev0,mount_tag=host_share

Then mount it in guest:

sudo mount -t 9p -o trans=virtio host_share /mnt

You can also write it in /etc/fstab so it automatically mounts on start up

See https://superuser.com/questions/502205/libvirt-9p-kvm-mount-in-fstab-fails-to-mount-at-boot-time

/data   /data   9p  trans=virtio,rw,_netdev 0   0

You can also add needed modules to your initramfs

9p
9pnet
9pnet_virtio

And write to fstab like this:

# src_mnt is the mount tag
src_mnt /src 9p trans=virtio 0 0

Snapshot

Leaving the original qcow2 disk image untouched, you can:

qemu-img create -f qcow2 -b "$vdisk_file" -F qcow2 current.img

And use current.img in your qemu command

When you need to delete the snapshot (to merge it into the original disk image):

merge_current_snapshot() {
    info "Merging current snapshot to $vdisk_file"
    qemu-img rebase -b "$vdisk_file" -f qcow2 -F qcow2 "$current"
    qemu-img commit "$current"
}

Networking

Use libvirt

This requires you to install and enable libvirt service, the benefit of this is you don't need to configure NAT and stuff by yourself, just use what's provided by libvirtd.

Remember to sudo systemctl enable --now libvirtd, assuming your default network interface is virbr0:

-device virtio-net-pci,netdev=n1 \
    -netdev tap,id=n1,br=virbr0,"helper=/usr/lib/qemu/qemu-bridge-helper",vhost=on

qemu-bridge-helper can help with creating TAP devices, you don't need root to run qemu as the helper has SUID.

NOTE qemu uses the same MAC address for all guests by default, if you want them to communicate with each other you will need to change that. To generate a random MAC address:

printf -v mac "52:54:%02x:%02x:%02x:%02x" $((RANDOM & 0xff)) $((RANDOM & 0xff)) $((RANDOM & 0xff)) $((RANDOM & 0xff))

According to Arch Wiki, you can also generate a fixed unique MAC address for each guest, with this script you can make sure the MAC address doesn't change during relaunch

#!/usr/bin/env python
# usage: qemu-mac-hasher.py <VMName>

import sys
import zlib

crc = str(hex(zlib.crc32(sys.argv[1].encode("utf-8")))).replace("x", "")[-8:]
print("52:54:%s%s:%s%s:%s%s:%s%s" % tuple(crc))

We can invoke the Python code in our bash wrapper:

vm_name="ubuntu22.04"
macaddr="$(python3 -c "import zlib;crc=str(hex(zlib.crc32('$vm_name'.encode('utf-8'))).replace('x', '')[-8:]);print('52:54:%s%s:%s%s:%s%s:%s%s'%tuple(crc))")"

How to get guest IP when it starts

This is extremely useful when you want to connect to your Windows VM via RDP (which I recommend doing since the performance is really promising)

fetch_guest_ip() {
    info "Trying to obtain IP address of guest $vm_name..."
    while true; do
        guest_ip=$(ip neigh show dev virbr0 | grep "$mac" | grep -o -P "^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}")
        [[ -n "$guest_ip" ]] && {
            info "IP of $vm_name is $guest_ip"
            break
        }
        sleep 1
    done

    [[ "$is_windows" -eq 1 ]] && {
        info "Connecting to $guest_ip via RDP..."
        # use /h:1080 /w:1920 if you don't want full screen
        # change `scale` to 180 if it still looks small
        wlfreerdp /u:user /p:password /v:"$guest_ip" /scale:140 +clipboard +aero /f
    }
}

Example Wrapper Scripts

Initialization

self="$0"
vdisk_file="$1"
vm_name="$(echo "$vdisk_file" | cut -d '.' -f1)"
current="current.img"
[[ -f macaddr.txt ]] || {
    printf -v mac "52:54:%02x:%02x:%02x:%02x" $((RANDOM & 0xff)) $((RANDOM & 0xff)) $((RANDOM & 0xff)) $((RANDOM & 0xff))
    echo -ne "$mac" | tee macaddr.txt
}

is_windows=0
echo "$vm_name" | grep -i windows >/dev/null && is_windows=1
guest_ip=""
mac=$(cat macaddr.txt)
qemu_bridge_helper="/usr/libexec/qemu-bridge-helper"
[[ -f "$qemu_bridge_helper" ]] || qemu_bridge_helper="/usr/lib/qemu/qemu-bridge-helper"
[[ -f "$qemu_bridge_helper" ]] || err "$qemu_bridge_helper not found"

err() {
    echo -e "\e[31m$1\e[0m"
    exit 1
}

info() {
    echo -e "\e[36m$1\e[0m"
}

[[ -d share ]] || mkdir share

ip link | grep virbr0 || {
    info "libvirtd not running?"
    sudo systemctl status libvirtd
    sudo systemctl restart libvirtd
}

[[ "$#" -ge 1 ]] || err "$self <disk.qcow2> <snapshot/origin/win>"

command -v qemu-system-x86_64 >/dev/null || err "qemu-system-x86_64 not found"

export SDL_VIDEODRIVER=wayland

spice_port=$((3000 + RANDOM % 5000))
addr="/run/user/1000/spice-$spice_port.sock"
spice_addr="spice+unix://$addr"

Linux VM With UEFI

uefi_with_virgl_sdl() {
    disk="$current"
    [[ -n "$1" ]] && disk="$1"
    qemu-system-x86_64 -name "$vm_name",process="$vm_name" \
        -pidfile "$vm_name.pid" \
        -enable-kvm \
        -machine q35,smm=off,vmport=off \
        -cpu host,kvm=on,topoext \
        -smp cores=4,threads=2,sockets=1 \
        -m 4G \
        -device virtio-balloon \
        -vga none \
        -display sdl,gl=on \
        -device virtio-vga-gl,xres=1920,yres=1080 \
        -audiodev pa,id=audio0 \
        -device intel-hda \
        -device hda-duplex,audiodev=audio0 \
        -rtc base=localtime,clock=host,driftfix=slew \
        -device virtio-rng-pci,rng=rng0 \
        -object rng-random,id=rng0,filename=/dev/urandom \
        -device qemu-xhci,id=spicepass \
        -chardev spicevmc,id=usbredirchardev1,name=usbredir \
        -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1 \
        -chardev spicevmc,id=usbredirchardev2,name=usbredir \
        -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2 \
        -chardev spicevmc,id=usbredirchardev3,name=usbredir \
        -device usb-redir,chardev=usbredirchardev3,id=usbredirdev3 \
        -device pci-ohci,id=smartpass \
        -device usb-ccid \
        -chardev spicevmc,id=ccid,name=smartcard \
        -device ccid-card-passthru,chardev=ccid \
        -device usb-ehci,id=input \
        -device usb-kbd,bus=input.0 \
        -k en-us \
        -device usb-tablet,bus=input.0 \
        -nic tap,id=n1,br=virbr0,model=virtio-net-pci,helper="$qemu_bridge_helper",vhost=on,mac="$mac" \
        -device virtio-blk-pci,drive=SystemDisk \
        -drive id=SystemDisk,if=none,format=qcow2,file="$disk" \
        -fsdev local,id=fsdev0,path="$PWD/share",security_model=mapped-xattr \
        -device virtio-9p-pci,fsdev=fsdev0,mount_tag=host_share \
        -monitor unix:"$vm_name-monitor.socket",server,nowait \
        -global driver=cfi.pflash01,property=secure,value=on \
        -drive if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly=on \
        -drive if=pflash,format=raw,unit=1,file=OVMF_VARS.fd \
        -serial unix:"$vm_name-serial.socket",server,nowait
    # -drive media=cdrom,index=0,file="kali-linux-2023-W22-installer-amd64.iso" \
}

Linux VM Without UEFI

start_with_virgl_sdl() {
    disk="$current"
    [[ -n "$1" ]] && disk="$1"
    qemu-system-x86_64 -name "$vm_name",process="$vm_name" \
        -pidfile "$vm_name.pid" \
        -enable-kvm \
        -machine q35,smm=off,vmport=off \
        -cpu host,kvm=on,topoext \
        -smp cores=4,threads=2,sockets=1 \
        -m 4G \
        -device virtio-balloon \
        -vga none \
        -device virtio-vga-gl,xres=1920,yres=1080 \
        -display sdl,gl=on \
        -audiodev pa,id=audio0 \
        -device intel-hda \
        -device hda-duplex,audiodev=audio0 \
        -rtc base=localtime,clock=host,driftfix=slew \
        -device virtio-rng-pci,rng=rng0 \
        -object rng-random,id=rng0,filename=/dev/urandom \
        -device qemu-xhci,id=spicepass \
        -chardev spicevmc,id=usbredirchardev1,name=usbredir \
        -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1 \
        -chardev spicevmc,id=usbredirchardev2,name=usbredir \
        -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2 \
        -chardev spicevmc,id=usbredirchardev3,name=usbredir \
        -device usb-redir,chardev=usbredirchardev3,id=usbredirdev3 \
        -device pci-ohci,id=smartpass \
        -device usb-ccid \
        -chardev spicevmc,id=ccid,name=smartcard \
        -device ccid-card-passthru,chardev=ccid \
        -device usb-ehci,id=input \
        -device usb-kbd,bus=input.0 \
        -k en-us \
        -device usb-tablet,bus=input.0 \
        -nic tap,id=n1,br=virbr0,model=virtio-net-pci,helper="$qemu_bridge_helper",vhost=on,mac="$mac" \
        -device virtio-blk-pci,drive=SystemDisk \
        -drive id=SystemDisk,if=none,format=qcow2,file="$disk" \
        -fsdev local,id=fsdev0,path="$PWD/share",security_model=mapped-xattr \
        -device virtio-9p-pci,fsdev=fsdev0,mount_tag=host_share \
        -monitor unix:"$vm_name-monitor.socket",server,nowait \
        -serial unix:"$vm_name-serial.socket",server,nowait
    # -drive media=cdrom,index=0,file="kali-linux-2023-W22-installer-amd64.iso" \
}

Windows VMs

start_windows_headless_uefi() {
    disk="$current"
    [[ -n "$1" ]] && disk="$1"
    qemu-system-x86_64 -name "$vm_name",process="$vm_name" \
        -pidfile "$vm_name.pid" \
        -enable-kvm \
        -machine q35,smm=off,vmport=off \
        -cpu host,kvm=on,+hypervisor,+invtsc,l3-cache=on,migratable=no,hv_passthrough,topoext \
        -smp cores=4,threads=2,sockets=1 \
        -m 4G \
        -device virtio-balloon \
        -device virtio-vga,xres=1920,yres=1080 \
        -display none \
        -spice port="$spice_port",disable-ticketing=on \
        -audiodev pa,id=audio0 \
        -device intel-hda \
        -device hda-duplex,audiodev=audio0 \
        -rtc base=localtime,clock=host,driftfix=slew \
        -device virtio-rng-pci,rng=rng0 \
        -object rng-random,id=rng0,filename=/dev/urandom \
        -device qemu-xhci,id=spicepass \
        -chardev spicevmc,id=usbredirchardev1,name=usbredir \
        -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1 \
        -chardev spicevmc,id=usbredirchardev2,name=usbredir \
        -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2 \
        -chardev spicevmc,id=usbredirchardev3,name=usbredir \
        -device usb-redir,chardev=usbredirchardev3,id=usbredirdev3 \
        -device pci-ohci,id=smartpass \
        -device usb-ccid \
        -chardev spicevmc,id=ccid,name=smartcard \
        -device ccid-card-passthru,chardev=ccid \
        -device usb-ehci,id=input \
        -device usb-kbd,bus=input.0 \
        -k en-us \
        -device usb-tablet,bus=input.0 \
        -nic tap,id=n1,br=virbr0,model=virtio-net-pci,helper="$qemu_bridge_helper",vhost=on,mac="$mac" \
        -device virtio-blk-pci,drive=SystemDisk \
        -drive id=SystemDisk,if=none,format=qcow2,file="$disk" \
        -fsdev local,id=fsdev0,path="$PWD/share",security_model=mapped-xattr \
        -device virtio-9p-pci,fsdev=fsdev0,mount_tag=host_share \
        -monitor unix:"$vm_name-monitor.socket",server,nowait \
        -global driver=cfi.pflash01,property=secure,value=on \
        -drive if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly=on \
        -drive if=pflash,format=raw,unit=1,file=OVMF_VARS.fd \
        -serial unix:"$vm_name-serial.socket",server,nowait
    # -drive media=cdrom,index=0,file="windows.iso" \
    # -drive media=cdrom,index=1,file="virtio-win.iso" \
}

jm33_ng

Migrating from Libvirt to Qemu

Graphics in Qemu

Performance Impact of SPICE+VirGL

SDL

Snapshot

Networking

Use libvirt

How to get guest IP when it starts

Example Wrapper Scripts

Initialization

Linux VM With UEFI

Linux VM Without UEFI

Windows VMs

Comments

jm33_ng

Graphics in Qemu

Performance Impact of SPICE+VirGL

SDL

File System Sharing

Snapshot

Networking

Use libvirt

How to get guest IP when it starts

Example Wrapper Scripts

Initialization

Linux VM With UEFI

Linux VM Without UEFI

Windows VMs

Comments