The bbvirt program is an attempt to take some of the pain out of what is currently required to distribute
multiple USB devices between the host and guest virtual machines. While there are several ways in which
this may be configured and managed, at present none of them actually provide a complete and coherent
solution on their own, all of them fall short of the mark in some significant and annoying way. The aim
here is to piece together enough of those hacks to actually get all of the functionality that we want
now, until the libvirt native support for this improves enough to not need it anymore.
At present this deals with libvirt managed QEMU/KVM virtual machines.
Whatdowewant?
The ideal behaviour here is pretty simple. Given some arbitrary number of BitBabbler devices, we should
be able to assign them to either the host machine, or to a guest VM running on it, and once we do that
they should behave in the normal manner expected of any USB device.
- If they are plugged in when the guest machine is started, they should be seen by that machine as they
would be by the host.
- If they are plugged in after the machine is started, they should be hotplugged into that machine as
they would be on the host.
- If they are unplugged while the machine is running, they should be cleanly removed from it, as they
would be on the host.
Whycan'twehaveit?
Right now, libvirt gives us two ways that we can assign USB devices from the host to a guest domain.
- We can assign them by their USB vendor and product ID. But that only works when there is just a single
device of that type in the host. Which is pretty useless in most of the cases that we care about here,
where the host and each of the guests are likely to have one or more BitBabbler devices of their own
assigned to them.
- We can assign them by their logical address on the USB bus. But that isn't a constant that we can
statically configure for the domain. Every time a device is plugged in, or replugged, or reset, or the
host machine is rebooted, that address is likely to change since it is dynamically allocated when the
device is enumerated on the bus.
There is a third way, but it relies on bypassing the normal libvirt configuration to make direct use of
the QEMU ability to assign a device by its physical address on the bus. Which is better, but still not a
magic bullet since it relies on plugging exactly the same devices into exactly the same ports every time
(and on having those ports enumerated in the same way by the host on every reboot, which isn't guaranteed
either). It also forces us to jump through other hoops, since we then need additional complication to
manage the access permissions of the device manually outside of libvirt, but still in coordination with
it.
The even bigger failing, which all of those methods have in common, is they all depend on the device
already being plugged in before the guest is started. If it is inserted after the guest is started, or
removed and replugged while the guest is running, or if the host bus or a hub bounces causing a
reconnect, then the device will not be (re)attached to the guest. The only way to fix that if it happens
is to manually reattach the device with an arcane incantation in XML (which relies on you knowing the new
address of the device), or to completely power down and restart the guest. Not the pinnacle of user-
friendly operation that we are looking for here.
Whatcanwedoaboutit?
There was a patch submitted to libvirt some years back which would have allowed a device to be specified
by both its USB product ID and its serial number, but that got some push-back, and so far has still not
been applied upstream. That would have gone a long way toward making this both easy and clean, leaving
us only with the hotplug aspect to deal with. We'll leave grumpy snark about that as an exercise for the
reader ...
Another alternative is we can delegate finding the device's logical address to a hotplug manager like
udev(7). This is attractive in the sense that we can know when the address of a device changes and what
it changes to, but udev itself isn't very friendly to the idea of local admin customisation (while it is
possible to do, it seems to be getting increasingly strongly discouraged) and using it still requires
some external glue to translate its events into something that libvirt can act on to configure the guest
machine.
The bbvirt program provides that glue, and a user friendly method of assigning which devices should
belong to which guest domains, and a front end that can be invoked manually or by other admin controlled
tasks to quickly and easily add or remove BitBabbler devices from any of the running guest machines.
But the limitation this approach has, is that it can't easily know when a guest machine is started which
should have devices that are already plugged in added to it. In theory we could add them to its
persistent domain definition, but that has its own problems because we can only add devices by their
ephemeral logical address, and we can't guarantee that we will get called to remove them from the domain
again when that address becomes invalid (like if the host is suddenly powered off or it is otherwise not
cleanly shut down), so we could end up with many stale entries accumulating in the persistent domain
configuration, which could later match some completely different device to what we had wanted attached to
it. Which means until that somehow gets fixed, it's only safe to add them to a live guest domain, so
that they will always be removed again when it is halted, no matter how it ended up getting halted.
Clearly we've still got some way to go to get to our ideal here.
Whatifwehititwith*two*hammers?
There appears to be only two ways that we can get notified of a guest machine being started at present.
One involves running yet another daemon process, which would do little more than just sit around waiting
for someone to start a guest so it could tell us about that. But then we'd have yet another thing to
configure, yet another process running, and yet more problems with figuring out how to ensure we don't
lose a race when the host is booted, between getting the initial set of device events, that process being
ready and active, and any guests that will be autostarted at boot actually starting.
The other way is to use a libvirt hook. Which in turn has the problem of not actually allowing us to run
any libvirt functions from it, which we need to do in order to attach the device to the host. And which
we can't guarantee that we can just install by default, because there can be only one such hook on the
system, which the local admin may already be using ...
There is a third way, but that would involve requiring the local admin to start all guest machines
through a wrapper of our own, instead of via whatever mechanism they already know and use. Which doesn't
scale to support other USB devices in the same situation, among the many ways that would be a horrible
solution to inflict on people.
But there is a loophole we can exploit. We can use the libvirt qemu hook to trigger a change event for
udev, which can in turn invoke bbvirt in much the same way that would happen if the device was really
hotplugged, which gives us the extra layer of indirection we need to be able to safely do that from the
hook. Rube Goldberg would be proud, and some of the pieces may require hand-assembly, but with all of
this in place, we can have something resembling normal USB functionality in the guest machines.
It's not pretty, but it will work with what we have to work with.
Ok,justtellmewheretohitit.
To string this together, you'll need to ensure all of the following:
- The udev(7) rules from the bit-babbler package are installed. If you installed this from the Debian
packages that should already be done. If you didn't, you will need to install the rules that are found
in debian/bit-babbler.udev from the source package to a suitable place on your system (probably
/etc/udev/rules.d).
- The bbvirt(1) script is installed in a place where the udev rules will find it. If you didn't install
this from the Debian packages, and it isn't in /usr/bin, then you'll need to tweak the udev rules to
suit.
- The devices you wish to use in guest machines, and the machines you wish to use them in, are specified
in the bbvirt configuration file. The default location for that is /etc/bit-babbler/vm.conf. If you
wish to use a different file you will need to pass its location with the --config option in the udev
rules, and update the hook script use that file too. The details of what you can put in that file are
described in the CONFIGURATIONOPTIONS section below.
- The libvirt hook file is installed. If all the above is done, then devices will be added to the
running guest machines if they get plugged in while the guest is running. This last step ensures
devices which are already plugged in will be added to newly started guests too (which includes guests
that are started automatically when the host machine boots).
Until there is some safe way we can install this without conflicting with or overwriting an existing
hook, everyone will need to do this step manually. If you have installed the Debian packages, then the
example hook script that we've provided for this can be found in
/usr/share/doc/bit-babbler/examples/qemu-hook. If you didn't it can be found in libvirt/qemu-hook of
the source package.
You will need to install that file as /etc/libvirt/hooks/qemu, or merge its content with the existing
qemu file there if you already have that hook set. If that file did not previously exist, you will
need to restart libvirtd(8) to get it to begin using it.
That should cover all of the needed automation, but you can also attach and detach devices manually at
any time too. The details of doing that will be described in the following section. Otherwise, with all
the above done, there is no other reason to need to invoke bbvirt directly.
Install Now
PNG Icons
