Commit f8194682 authored by Guido Gunther's avatar Guido Gunther

Import Upstream version 1.11

parents
*.img
*.log
*.tar.gz
*.pyc
*.egg-info
doc/_build/
man/_build/
dist/*
local/*
yarns/*.html
yarns/*.pdf
This diff is collapsed.
include NEWS
include vmextract.py
include vmsquash-tar.py
include vmdebootstrap.8.in
recursive-include examples *.sh *.txt *.py
include common/*
include doc/*
include man/*
recursive-include yarns *.yarn Makefile run-tests yarns.css shell.lib
include bin/qemu-wrapper.sh
NEWS for vmdebootstrap
======================
Version 1.11, released 2018-04-27
----------------------------------
* Fix debian/control syntax.
Version 1.10, released 2018-04-27
---------------------------------
* Change Debian maintainer address.
Version 1.9, released 2018-02-24
---------------------------------
* Warn Debian users vmdebootstrap won't be in buster.
Version 1.8, released 2017-09-17
-----------------------------------------------------------------------------
* To be filled in later.
Version 1.7-1, released 2016-09-11
---------------------------------
* NMUs by Steve McIntyre. Bug fixes.
Version 1.7, released 2016-09-11
---------------------------------
* New maintainer: Lars Wirzenius replaces Neil Williams.
* William Holland changed vmdebootstrap to allow all releases that
debootstrap does.
A lot of history missing from NEWS
----------------------------------
It's in git logs and debian/changelog, but wasn't included here.
Version 0.3
-----------------------
* Add example customisation scripts to complement the support
for other architectures which need work beyond the scope of
available packages.
Version 0.2, released 2013-11-15
--------------------------------
* Add support for building images for other architecture other
than the native one. Patch by Petter Reinholdtsen.
Version 0.1.0, released 2013-07-23
----------------------------------
* First actual release.
README for vmdebootstrap
========================
`debootstrap` installs a basic Debian system into a directory, for use
with `chroot`(8). `vmdebootstrap` is a wrapper around it to install
Debian into a disk image, which can be used with virtual machines or
real hardware.
See the manual page and `vmdebootstrap --help` for details on how to
use the program. The manual page has an example.
Limitations
-----------
`vmdebootstrap` is aimed principally at creating virtual machines, not
installers or prebuilt installation images. It is possible to create
prebuilt installation images for some devices but this depends on the
specific device. (A 'prebuilt installation image' is a single image file
which can be written to physical media in a single operation and which
allows the device to boot directly into a fully installed system - in a
similar way to how a virtual machine would behave.)
* `vmdebootstrap` assumes that all operations take place on a local image
file, not a physical block device / removable media.
* `vmdebootstrap` is intended to be used with tools like `qemu` on the
command line to launch a new virtual machine. Not all devices have
virtualisation support in hardware.
This has implications for `u-boot` support in some cases. If the device
can support reading the bootloader from a known partition, like the
Beaglebone-black, then `vmdebootstrap` can provide space for the bootloader
and the image will work as a prebuilt installation image. If the device
expects that the bootloader exists at a specific offset and therefore
requires that the bootloader is written as an image not as a binary which
can be copied into an existing partition, `vmdebootstrap` is unable to
include that bootloader image into the virtual machine image.
It is possible to wrap `vmdebootstrap` in such a way as to prepare a
*physical block device* with a bootloader image and then deploy the
bootstrap on top. However, this does require physical media to be
inserted and removed each time the wrapper is executed. Once you have
working media, an image can be created using ``dd`` to read back from
the media to an image file, allowing other media to be written with a
single image file. To do this, use the `--tarball` option to `vmdebootstrap`
instead of the `--image`` option. Then setup the physical media and
bootloader image as required for the device, redefine the partitions to
make space for the rootfs, create a filesystem on the physical media and
unpack the `vmdebootstrap` tarball onto that filesystem.
What you need
-------------
In order to use vmdebootstrap, you'll need a few things:
* debootstrap
* extlinux
* qemu-img (in the qemu-utils package in Debian)
* parted
* mbr
* kpartx
* python-cliapp (see http://liw.fi/cliapp/)
* python-distro-info
Testing vmdebootstrap from git
------------------------------
There is a strongly recommended git pre-commit hook available
for vmdebootstrap development - it requires the ``cmdtest``
package::
ln -s ../../pre-commit.sh .git/hooks/pre-commit
Running vmdebootstrap from git
------------------------------
$ sudo PYTHONPATH=. ./bin/vmdebootstrap
This has changed slightly with version 1.0 with the need for
PYTHONPATH to reference the module approach for support handlers.
vmdebootstrap modules
---------------------
The single vmdebootstrap script has been refactored to be the top
level settings parser and validator and the point where the other
modules (handlers) get to be called in a collaborative sequence.
The new modules are an attempt to work with a DRY process as well
as keeping the source code itself maintainable. Handler functions
need to check settings at the start so that calls to the handlers
can be retained in a simple flow. Where a function needs code from
multiple handlers, that function needs to be in the vmdebootstrap
script but these should, ideally, be single calls into dedicated
calls from the relevant handlers which can return True|False or
raise cliapp.AppException to affect subsequent flow. Handlers must
NOT hook into other handlers, except Base or constants, only the
vmdebootstrap script has the full set, so use function arguments to
pass variables populated by different handlers. Wherever possible,
large sections of new functionality need to be added as new handlers.
pylint
------
When using pylint, the following option is advised:
$ pylint --ignore-imports=y vmdebootstrap
(Despite the name of the option, this only ignores imports when
computing similarities and various handlers will end up needing
similar imports, it makes no sense to complain about that.)
Apart from that, vmdebootstrap uses pylint and contains comments to
disable certain pylint checks in certain areas. pylint compatibility
will make it easier to accept patches, just follow the existing pattern
of pylint usage. pylint is far from perfect but can be helpful.
Testing UEFI support
--------------------
There is EFI firmware available to use with QEMU when testing images
built using the UEFI support, but this software is in Debian non-free
due to patent concerns. If you choose to use it to test UEFI builds,
a secondary change is also needed to symlink the provided OVMF.fd to
the file required by QEMU: bios-256k.bin and then tell QEMU about the
location of this file with the -L option:
$ qemu-system-x86_64 -L /usr/share/ovmf/ -machine accel=kvm \
-m 4096 -smp 2 -drive file=amd64.img,format=raw
Note the use of -drive file=<img>,format=raw which is needed for newer
versions of QEMU.
The vmextract helper
--------------------
Once the image is built, various files can be generated or modified
during the install operations and some of these files can be useful
when testing the image. One example is the initrd built by the process
of installing a Debian kernel. Rather than having to mount the image
and copy the files manually, the vmextract helper can do it for you,
without needing root privileges.
$ /usr/share/vmdebootstrap/vmextract.py --verbose \
--image bbb/bbb-debian-armmp.img --boot \
--path /boot/initrd.img-3.14-2-armmp \
--path /lib/arm-linux-gnueabihf/libresolv.so.2
This uses python-guestfs (a Recommended package for vmdebootstrap) to
prepare a read-only version of the image - in this case with the /boot
partition also mounted - and copies files out into the current working
directory.
The integration test suite
--------------------------
To run the vmdebootstrap integration test suite, you need the yarn
tool, which is part of cmdtest. See <http://liw.fi/cmdtest/> and the
cmdtest package in Debian. The command to run the tests is:
sudo yarns/run-tests
You can skip the slow tests that actually build images, by setting the
`TESTS` variable:
sudo yarns/run-tests --env TESTS=fast
To format the test suite document:
(needs pandoc installed)
make -C yarns
Making a release
----------------
Release are best made with the bumper tool (http://liw.fi/bumper/),
or simulating that by hand:
* Update version number (`vmdebootstrap/version.py`, `NEWS`,
`debian/changelog`).
* Also update `NEWS` to to say version is released (with date).
* Commit, and tag the commit with `vmdebootstrap-x.y` for version x.y.
* Update version number and `NEWS` again to say the version is
`x.y+git`. This makes it clear when a version is from git and not
frmo a release.
* Commit.
CI will then build and upload packages. CI is currently running on
Lars's home server.
Legalese
--------
Copyright 2011-2013,2016 Lars Wirzenius
Copyright 2012 Codethink Limited
Copyright 2013-2016 Neil Williams
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
TODO
====
* document the problems of setting a default password
* first-boot customisation support (via a package) which forces a
new password, possibly new hostname, generate new ssh key etc.
* make all internal additions and operations optional
* investigate some way to support complex partitioning
* document that no-kernel can lead to images where the kernel
cannot be upgraded.
* try to support upgrading the bootloader
possibly via the config output
problems of flash-kernel
* automatically grow rootfs to media size on first boot
boot into ramdisk, umount media, move ext4 along media
expand vfat.
* document of how to add new bootloader etc.
unless the old and new bootloader exist as packages, this
is going to be manual.
* customisations as packages.
* document how the image was built with the config called.
include the config output and a copy of the script with
dependencies.
* consider limitations of only one hook script
likely to be a lack of time to implement multi-hook support.
* support method to share shell customisation scripts
#!/bin/sh
set -e
if [ -z "$1" ]; then
echo "Usage: <imagefile> <arch> [uefi_directory]"
echo "For x86_64, amd64 is also supported."
exit 1
fi
if [ -n "$2" ]; then
if [ "$2" = 'amd64' ]; then
ARCH='x86_64'
else
ARCH="$2"
fi
else
echo "Specify the architecture of the image"
echo "Usage: <imagefile> <arch>"
echo "For x86_64, amd64 is also supported."
exit 1
fi
UEFI=""
if [ -n "$3" ]; then
UEFI="-L $3 -monitor none"
fi
qemu-system-${ARCH} -m 1024 ${UEFI} -enable-kvm -drive format=raw,file=./$1
This diff is collapsed.
# Copyright 2015 Neil Williams <codehelp@debian.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
BASE_OPTS="
--owner ${WHO} --verbose \
--sudo \
--lock-root-password \
--enable-dhcp \
--configure-apt \
--log debian-cd-${SUITE}-${ARCH}.log --log-level debug \
"
TASK_PACKAGES="task-laptop task-english"
# packages which are not (yet) part of a task
EXTRA_PACKAGES="dkms locales whois telnet aptitude lsof host \
bash-completion firmware-linux-free dnsutils time rsync reportbug w3m \
ethtool ftp host lsof debian-faq debian-installer-launcher doc-debian \
"
cleanup() {
umount ${rootdir}/proc/sys/fs/binfmt_misc > /dev/null 2>&1 || true
umount ${rootdir}/proc
umount ${rootdir}/sys
}
export DEBIAN_FRONTEND=noninteractive
export LC_ALL=C
export LANG=C
export LANGUAGE=C
mount_support() {
mount proc -t proc ${rootdir}/proc
mount sys -t sysfs ${rootdir}/sys
}
disable_daemons() {
# prevent packages starting daemons inside the chroot until after boot.
# https://wiki.debian.org/chroot
cat > ${rootdir}/usr/sbin/policy-rc.d <<EOF
#!/bin/sh
exit 101
EOF
chmod a+x ${rootdir}/usr/sbin/policy-rc.d
}
# ensure you pass the replacement mirror and suite
prepare_apt_source() {
# handle the apt source
mv ${rootdir}/etc/apt/sources.list.d/base.list ${rootdir}/etc/apt/
echo "deb $1 $2 main contrib non-free" > ${rootdir}/etc/apt/sources.list
echo "deb-src $1 $2 main contrib non-free" >> ${rootdir}/etc/apt/sources.list
chroot ${rootdir} apt -qq -y update > /dev/null 2>&1
}
remove_daemon_block() {
rm ${rootdir}/usr/sbin/policy-rc.d
}
replace_apt_source() {
# Undo apt source change
rm ${rootdir}/etc/apt/sources.list
mv ${rootdir}/etc/apt/base.list ${rootdir}/etc/apt/sources.list.d/
}
blacklist_qemu_bochs() {
echo "blacklist bochs-drm" > ${rootdir}/etc/modprobe.d/qemu-blacklist.conf
}
#!/bin/sh
set -e
rootdir=$1
# common needs rootdir to already be defined.
. /usr/share/vmdebootstrap/common/customise.lib
trap cleanup 0
mount_support
disable_daemons
# prepare_apt_source
chroot ${rootdir} apt-get -q -y install ${TASK_PACKAGES} ${EXTRA_PACKAGES} \
task-xfce-desktop exim4 mutt info rpcbind pciutils \
task-ssh-server task-print-server plymouth procmail \
m4 open-vm-tools apt-listchanges at busybox nfs-common \
wamerican texinfo plymouth-themes plymouth-x11 uuid-runtime \
open-vm-tools-dkms open-vm-tools-desktop gettext-base mlocate \
irqbalance memtest86+ user-setup zerofree
remove_daemon_block
# replace_apt_source
# particular to efi builds
blacklist_qemu_bochs
echo "Customisation complete"
#!/bin/sh
set -e
# define before sourcing common
WHO=`whoami`
USER='user/live'
SUITE='jessie'
SIZE='5G'
ARCH='amd64'
SHARE_PATH='/usr/share/vmdebootstrap/common'
# needs a path for arch and task desktop
IMAGE_PATH='.'
. ${SHARE_PATH}/customise.lib
sudo vmdebootstrap \
${BASE_OPTS} --user ${USER} \
--size ${SIZE} \
--arch ${ARCH} \
--no-extlinux \
--grub --use-uefi \
--distribution ${SUITE} \
--customize "${SHARE_PATH}/${SUITE}-${ARCH}-hook.sh" \
--image ${IMAGE_PATH}/${SUITE}.img \
"$@"
# report results and check we have something valid.
ls -l ${IMAGE_PATH}/${SUITE}.img
md5sum ${IMAGE_PATH}/${SUITE}.img
#!/bin/sh
set -e
rootdir=$1
# common needs rootdir to already be defined.
. /usr/share/vmdebootstrap/common/customise.lib
trap cleanup 0
mount_support
disable_daemons
prepare_apt_source
chroot ${rootdir} apt-get -q -y install ${TASK_PACKAGES} ${EXTRA_PACKAGES} \
task-xfce-desktop exim4 mutt info rpcbind pciutils \
task-ssh-server task-print-server plymouth procmail \
m4 apt-listchanges at busybox nfs-common \
wamerican texinfo plymouth-themes plymouth-x11 uuid-runtime \
gettext-base mlocate irqbalance \
irqbalance user-setup zerofree
remove_daemon_block
replace_apt_source
# particular to efi builds
blacklist_qemu_bochs
echo "Customisation complete"
#!/bin/sh
set -e
# define before sourcing common
WHO=`whoami`
USER='user/live'
SUITE='jessie'
SIZE='5G'
ARCH='arm64'
BINFMT='/usr/bin/qemu-aarch64-static'
SHARE_PATH='/usr/share/vmdebootstrap/common'
# needs a path for arch and task desktop
IMAGE_PATH='.'
. ${SHARE_PATH}/customise.lib
sudo vmdebootstrap \
${BASE_OPTS} --user ${USER} \
--size ${SIZE} \
--arch ${ARCH} \
--foreign ${BINFMT} \
--no-extlinux \
--grub --use-uefi \
--package dosfstools \
--distribution ${SUITE} \
--customize "${SHARE_PATH}/${SUITE}-${ARCH}-hook.sh" \
--image ${IMAGE_PATH}/${SUITE}-${ARCH}.img \
"$@"
# report results and check we have something valid.
ls -l ${IMAGE_PATH}/${SUITE}-${ARCH}.img
md5sum ${IMAGE_PATH}/${SUITE}-${ARCH}.img
vmdebootstrap (1.9-1) unstable; urgency=medium
vmdebootstrap is not going to be included in the release of Debian
buster. Please switch to another tool for building Debian system
images automatically, such as vmdb2, debos, or FAI. You have until
September, 2018.
The reason for this is that vmdebootstrap has a software architecture
that makes it difficult to modify or test. Many desirable changes to
vmdebootstrap are so hard to make that its author doesn't even want to
contemplate them. Worse, fixing some of the known bugs are difficult.
Because of this, the vmdebootstrap upstream author and Debian package
maintainer will get vmdebootstrap removed from Debian before the
buster release. Bugs in vmdebootstrap are unlikely to be fixed, unless
they have no workaround and threaten user data or are a security
problem.
vmdb2 is the second attempt by the vmdebootstrap author to write a
tool for building Debian system images. It is much more flexible than
the first attempt, and much easier to modify. Unfortunately, the same
things that make vmdebootstrap hard to maintain make it hard to add
backwards compatibility to vmdb2 for command line and configuration
file syntax. vmdb2 is not a drop-in replacement. You'll have to re-do
all the image creation work with vmdb2.
There are many other tools for the same purpose. You may want to
compare several and pick the one you like best.
-- Lars Wirzenius <liw@liw.fi> Sat, 24 Feb 2018 20:48:56 +0200
vmdebootstrap (1.6-1) unstable; urgency=medium
* extlinux support is now deprecated. Grub to become default.
extlinux relies on external components to install the bootloader
inside the image which prevents vmdebootstap building the same image
on different systems. For this reason, extlinux is deprecated and
the default will change in a future release to disabling extlinux
and using grub by default.
-- Neil Williams <codehelp@debian.org> Sun, 31 Jul 2016 15:05:58 +0100
This diff is collapsed.
Source: vmdebootstrap
Section: admin
Priority: extra
Maintainer: Lars Wirzenius <liw@liw.fi>
Uploaders: Iain R. Learmonth <irl@debian.org>,
Steve McIntyre <93sam@debian.org>
Build-Depends: debhelper (>= 8.0.0), dh-python,
python-sphinx (>= 1.0.7+dfsg) | python3-sphinx,
python | python-all | python-dev | python-all-dev,
python-setuptools (>= 3)
Testsuite: autopkgtest
X-Python-Version: 2.7
Standards-Version: 3.9.8
Homepage: https://liw.fi/vmdebootstrap/
Vcs-Git: git://git.liw.fi/vmdebootstrap
Vcs-Browser: http://git.liw.fi/vmdebootstrap
Package: vmdebootstrap
Architecture: linux-any
Depends: debootstrap, qemu-utils, kpartx, parted,
${sphinxdoc:Depends}, ${python:Depends}, ${misc:Depends}
Recommends: dosfstools, grub2-common [!mips !s390x],
extlinux [amd64 i386],
squashfs-tools, python-guestfs, qemu-system, qemu-user-static
Suggests: cmdtest, mbr, pandoc, u-boot:armhf
Description: Bootstrap Debian into a (virtual machine) disk image
vmdebootstrap is a wrapper around debootstrap to install Debian
into a disk image, which can be used with a virtual machine (such as KVM).
.
syslinux and grub2 bootloaders can be supported, depending on the
distribution and the architecture. An example of setting up u-boot
for armhf on a beaglebone-black is included.
.
This package also includes documentation on using vmdebootstrap
to create installation images and live images.
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: vmdebootstrap
Source: git://git.liw.fi/vmdebootstrap
Files: *
Copyright: 2011, 2012 Lars Wirzenius
2012 Codethink Limited
License: GPL-3.0+
Files: debian/*
Copyright: 2013 Neil Williams <codehelp@debian.org>
License: GPL-3.0+
License: GPL-3.0+
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
.
This package is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
.
On Debian systems, the complete text of the GNU General
Public License version 3 can be found in "/usr/share/common-licenses/GPL-3".
README
NEWS
[DEFAULT]
overlay = True
cliapp python-cliapp; PEP386
distro-info python-distro-info; PEP386
#!/usr/bin/make -f
# -*- makefile -*-
# Sample debian/rules that uses debhelper.
# This file was originally written by Joey Hess and Craig Small.
# As a special exception, when this file is copied by dh-make into a
# dh-make output file, you may use that output file without restriction.
# This special exception was added by Craig Small in version 0.37 of dh-make.
# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1
# Prevent setuptools/distribute from accessing the internet.
export http_proxy = http://127.0.9.1:9
export LAST_CHANGE=$(shell dpkg-parsechangelog -S Date)
export BUILD_DATE=$(shell LC_ALL=C date -u "+%B %d, %Y" -d "$(LAST_CHANGE)")
# This has to be exported to make some magic below work.
export DH_OPTIONS
%:
dh $@ --with sphinxdoc,python2
override_dh_auto_build:
dh_auto_build
python setup.py build
make -C doc/ html SPHINXOPTS="-D today=\"$(BUILD_DATE)\""
make -C man/ man SPHINXOPTS="-D today=\"$(BUILD_DATE)\""
override_dh_auto_install:
dh_auto_install
dh_lintian
python setup.py install --root=$(CURDIR)/debian/vmdebootstrap/ --install-layout=deb
# move vmdebootstrap to /usr/sbin
mkdir debian/vmdebootstrap/usr/sbin/
mv debian/vmdebootstrap/usr/bin/vmdebootstrap debian/vmdebootstrap/usr/sbin/
override_dh_auto_clean:
dh_auto_clean
$(RM) -r vmdebootstrap.egg-info doc/_build dist/
[ ! -f doc/Makefile ] || make -C doc/ clean
[ ! -f doc/Makefile ] || make -C man/ clean
Tests: testsuite
Depends: cmdtest, vmdebootstrap
Restrictions: allow-stderr
#!/bin/sh
set -e
yarns/run-tests --env TESTS=fast
Document: vmdebootstrap
Title: Debian Vmdebootstrap Manual
Author: Neil Williams <codehelp@debian.org>
Abstract: This manual describes what vmdebootstap is
and how it can be used to create virtual machine
images, installation images and live images.
Section: Debian
Format: HTML
Index: /usr/share/doc/vmdebootstrap/html/index.html
Files: /usr/share/doc/vmdebootstrap/html/*.html
examples ./usr/share/vmdebootstrap/
vmextract.py ./usr/share/vmdebootstrap/
common