PV and HVM Virtualization

Over the years, in Linux hardware virtualization (HVM) support for Xen has become much better than Paravirtualization (PV) support. Other operating systems like NetBSD or some unikernals only work well under PV mode. For new systems, use HVM if your operating system supports it.

Our legacy VPSes were all provisioned using Xen PV mode, so we’ve added a method for converting existing Linux VPSes from PV to HVM mode. This is now supported from the VPS console.

Below is a description of the differences between PV and HVM and the conversion process from PV to HVM.

Differences between PV and HVM Virtualization

Booting

PV Mode

PV systems all boot using a version of either pv-grub (legacy Grub) or Grub 2 provided.

HVM Mode

HVM systems have an emulated BIOS and can boot from either the VPS disk or netboot installer. To change this, go to the “set bootloader or rescue mode” option of the VPS console.

Serial Console

PV Mode

Paravirtualized (PV) mode systems use a paravirtualized serial console named hvc0. Because there is no virtual display by default, PV systems will default to using hvc0 if no “console=hvc0” is defined on the kernel command line.

HVM Mode

HVM systems have both an emulated hardware serial port and a paravirtualized serial port. Under Linux the emulated hardware serial port is named ttyS0 and the paravirtualized serial port is named hvc0. The bootloader only can use the emulated serial port. For the operating system, HVM systems must explicitly be told to use a serial port or there will be no output after the operating system is loaded. Linux has drivers for both ttyS0 and hvc0, however we will only expose hvc0 if needed. New installs always use ttyS0. For HVM systems you may need to press enter to get output from Linux, which is not necessary with PV mode systems.

Converting from PV to HVM

The conversion process via the VPS console does not make any disk changes, and will continue to use hvc0 if a console is specified on the kernel command line. If a console is not specified, there may not be output during booting. You will need to perform a manual conversion to avoid having to switch from the HVM to PV console during boot.

We set your VPS to boot from CDROM using the version of Grub the VPS was originally using. The bootloader uses the emulated hardware serial port ttyS0. Like before, this bootloader chain-loads the Grub configuration file from the VPS disk. If and when a bootloader is installed directly to the VPS disk, you will need to change your boot configuration from the “set bootloader or rescue mode” option of the VPS console.

When you use the out of band console from VPS console, you’ll get a different menu with options for both the PV (hvc0) and HVM (ttyS0) consoles until you do a new clean install of a preconfigured operating system.

Updating serial port

To avoid having to switch from the HVM console to the PV console during the boot process, some files to look at are:

Converting when using pv-grub

If there are issues, make sure that your bootloader configuration, /boot/grub/menu.lst, contains no splashimage lines and that hvc0 is replaced with ttyS0 in the boot entries.

Converting when using Grub 2

If you are using Grub 2, or you wish to use Grub 2 in the future, it is a good idea to make sure that your configuration files look correct. The file /etc/default/grub is the one that should be modified. Here is a sample /etc/default/grub file. Take note of the lines starting with GRUB_SERIAL_COMMAND and GRUB_TERMINAL. These insure that Grub 2 will use the serial console instead of the PV console.

GRUB_TIMEOUT=5
# GRUB_DISTRIBUTOR= # The correct value is OS-dependent
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_CMDLINE_LINUX="console=ttyS0 rootflags=barrier=0"
GRUB_DISABLE_LINUX_UUID="true"
export LINUX_ROOT_DEVICE="LABEL=PRGMRDISK1"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"

You should rebuild your Grub configuration file grub.cfg using either grub2-mkconfig or grub-mkconfig. The name of the tool and the pathname are OS-dependent. On RedHat based systems, use:

$ grub2-mkconfig -o /boot/grub2/grub.cfg

On Debian derivatives and most of the other distributions, use:

$ grub-mkconfig -o /boot/grub/grub.cfg

Distribution Errata

CentOS 6

Before starting the conversion, run the following command while booted into your installation:

$ sed -i -e 's/console=hvc0/console=ttyS0/' -e 's/xencons=tty0//' /boot/grub/menu.lst

Debian

Install the linux-image package, not linux-image-cloud.

Kernel Support

If you are using a custom-built kernel, your system should still contain the following drivers:

Here is a list, not fully cut down, of relevant Linux options as of 5.4 (Ubuntu):

CONFIG_SERIAL_8250=y
CONFIG_XEN=y
CONFIG_XEN_PV=y
CONFIG_XEN_PV_SMP=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_PVHVM_SMP=y
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_PVH=y
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=m
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_SCSI_FRONTEND=m
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_XEN_SCRUB_PAGES_DEFAULT=y
CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XENFS=m
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GNTDEV_DMABUF=y
CONFIG_XEN_GRANT_DEV_ALLOC=m
CONFIG_XEN_GRANT_DMA_ALLOC=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_PVCALLS_FRONTEND=m
CONFIG_XEN_PRIVCMD=m
CONFIG_XEN_ACPI=y

CentOS 6 (approximately 2.6.32:)

CONFIG_SERIAL_8250=y
CONFIG_XEN=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_HVC_XEN=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XENFS=m
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_PLATFORM_PCI=y

PCI, lsmod output

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)

lrwxrwxrwx 1 root root 0 Nov  1 03:35 sr0 -> ../devices/pci0000:00/0000:00:01.1/ata2/host1/target1:0:1/1:0:1:0/block/sr0
lrwxrwxrwx 1 root root 0 Nov  1 03:35 xvda -> ../devices/vbd-768/block/xvda