Thanks

See CREDITS in the project source for a list of thanks.

Preface

About the Author

I am a GNU/Linux Systems/Network Administrator/Engineer- I wear a lot of hats. I have a lot of side projects to keep me busy when I’m not working at ${dayjob}, mostly to assist in other side projects and become more efficient and proficient at those tasks. “Shaving the yak,” [1] indeed.

A lot of research went into how low-level boot operations take place when writing BDisk, both in BIOS and UEFI [2] (and corresponding concepts such as Secureboot, etc.) which is no easy task to understand and very commonly misunderstood. (For instance, a common misconception is that UEFI necessarily implies Secureboot. This is quite far from the truth and UEFI by itself is quite a useful replacement for BIOS). I invite you to do research into the specifications yourself; it’s rather fascinating.

What is BDisk?

BDisk refers to both a live distribution I use in my own uses (for rescue situations, recovery, etc.) but foremost and most importantly, it also refers to the tool I use for building that distribution. In other words, it’s both a complete GNU/Linux distribution you can run entirely from USB/CD/DVD/etc. (without needing to install it to your hard drive)…​ and also the name of a tool to create a custom GNU/Linux install. The latter is what this project and documentation refer to when the word “BDisk” is used.

This documentation was started when I rewrote BDisk in Python 3.x; versions 0.x-2.x of BDisk were written in Bash, and horribly inelegant and rigid. [3] One of my main goals was to make BDisk as easy to use as possible. This is surprisingly hard to do- it’s quite challenging to try to approach software you’ve written with the mindset of someone other than you.

It’s my hope that by releasing this utility (and documenting it), you can use it and save some time for yourself as well (and hopefully get the chance to learn a bit more in the process!).

It of course is not the only live media creator out there, but most others only focus on remastering an existing ISO, or creating an installer ISO — not creating a custom live-centric environment.

Copyright/Licensing

The BDisk code is GPLv3-licensed. This means that you can use it for business reasons, personal reasons, modify it, etc. Please be sure to familiarize yourself with the full set of terms. You can find the full license in docs/LICENSE.

GPLv3

This document, and all other associated author-generated documentation, are released under the Creative Commons CC-BY-SA 4.0 copyright. It’s essentially the GPL for non-software, so similar terms apply.

CC-BY-SA_4.0

User Manual

What good is software if nobody uses it?

BDisk was ultimately designed to make your life easier. "Why would I possibly need yet another LiveCD/LiveUSB?" Well, that’s sort of the point- by customizing a live distribution of GNU/Linux to your particular needs/desires/whimsy, you can do away with the multiple other images you keep around. It’s designed to let you create a fully customized distribution.

Using BDisk, you can:

  • Install GNU/Linux (Arch, CentOS, Debian, Gentoo, Ubuntu…​). BDisk may be Arch-based, but many if not most other distros offer ways to install from any GNU/Linux live distribution.

  • Perform disk maintenance (mdadm, fdisk / gdisk, gparted, storcli, etc.). Need to replace that disk in your RAID and you don’t have hotswap? Not a problem!

  • Rescue, recover, wipe (scalpel, nwipe, foremost, etc.). Chances are this is why you booted a live distro in the first place, yes?

  • Boot over the Internet (or LAN). Burning a new image to CD/DVD/USB is a pain. BDisk has built-in support for iPXE (and traditional PXE setups). Update the filesystem image once, deploy it everywhere.

  • And much, much more.

    • Seriously.

This manual will give you the information you need to build your very own live GNU/Linux distribution.

1. Getting Started

1.1. Downloading

If it isn’t in your distro’s repositories (It is in Arch’s AUR! Both tagged release and git master.), you can still easily get rolling. Simply visit the project’s source code web interface and download a tarball under the Download column:

cgit

If you know the tag of the commit you want, you can use curl:

curl -sL -o bdisk.tar.xz https://git.square-r00t.net/BDisk/snapshot/BDisk-3.00-BETA.tar.xz

or wget:

wget -O bdisk.tar.xz https://git.square-r00t.net/BDisk/snapshot/BDisk-3.00-BETA.tar.xz

You can use https://git.square-r00t.net/BDisk/snapshot/BDisk-master.tar.xz for the URL if you want the latest working version. If you want a snapshot of a specific commit, you can use e.g. https://git.square-r00t.net/BDisk/snapshot/BDisk-a1fe1dbc0a0ce2b2a5d1b470d30b60636f9b2efa.tar.xz and so on.

Alternatively, you can use git. Git most definitely should be in your distro’s repositories.

Tip
If you’re new to git and want to learn more, I highly recommend the book Pro Git. It is available for free download (or online reading).

You can clone via https:

git clone https://git.square-r00t.net/BDisk

or native git protocol:

git clone git://git.square-r00t.net/bdisk.git BDisk

The git protocol is much faster, but at a cost of lessened security.

Note
I also have a mirror at GitHub, but I don’t like GitHub very much and since it’s a mirror repository, it’s possible it will be out of date. For this reason, it’s recommended that you use the resources above.

1.2. Prerequisites

This is a list of software you’ll need available to build with BDisk.

Tip
Your distro’s package manager should have most if not all of these available, so it’s unlikely you’ll need to install from source.
Note
Some versions may be higher than actually needed (especially gcc).
Caution
You will need at least about 15GB of free disk space, depending on what options you enable. Each architecture chroot (i.e. x86_64, i686) is about 3.5GB after a build using the default package set (more on that later), each architecture release tarball (what we use to build the chroots) is approximately 115MB each, and each squashed image per architecture is 1.1GB (if you use the default package set). If you don’t understand what this means quite yet, don’t worry- we’ll go into more detail later on. Just know that you’ll need a fair bit of free disk space.

1.2.1. Necessary

These are needed for using BDisk.

These are required Python modules:

1.2.2. Optional

While not strictly necessary, these will greatly enhance your BDisk usage. I’ve included some reasons why you might want to install them.

Note
If you do not wish to install any of these or cannot install them, be sure to disable the relevant options in the build.ini file (we’ll talk about that later). The default extra/dist.build.ini should be sane enough to not require any of these.
  • cdrtools

    • Needed for building iPXE.

  • gcc (multilib) (>=6.x)

    • Needed for building iPXE.

  • gcc-libs (multilib) (>=6.x)

    • (Same as gcc.)

  • git

    • For autodetection of version, automatically making commits for your project, checking out source code, etc.

  • gpg/gnupg (>=2.1.11)

    • For automatically signing releases, verifying downloaded files from the Internet as part of the build process, etc. It’s okay if you don’t have a key set up!

  • rsync

    • For syncing built ISOs to a fileserver, syncing to a remote iPXE server, syncing to a traditional PXE/TFTP server, etc.

2. Important Concepts

If this is your first foray into building live distros, there are some terms and concepts we need to understand first. This will simplify the process later on.

2.1. Terms

An operating system, or OS, is what your programs (email client, web browser, etc.) run on.

There are two basic types of booting systems that communicate between the hardware (the physical computer itself and its components) and the operating system: BIOS (Basic Input/Output System) which has been around for quite some time and the newer UEFI (Unified Extensible Firmware Interface). Don’t worry, you don’t need to memorize what they’re acronyms for and there won’t be an exam — just remember that BIOS is an older technology and UEFI is the newer one (and that they operate differently).

GNU/Linux, sometimes just referred to as Linux (And there is a difference between the terminologies, but it’s nuanced. You are welcome to read up on it though!), is an example of an operating system. Other examples include Windows, macOS (previously OS X), iOS, Android, and a whole slew of others. There are many types of GNU/Linux offerings, called distributions, flavors, or distros.

A live distro, live CD, live DVD, live USB, and the like are a way of booting an operating system without installing it on the hard drive- this means the computer doesn’t even need a hard drive installed, or it doesn’t matter if the installed operating system is broken. Typically they are Linux-based, but there are several Windows-based live releases out there (usually they’re focused on rescuing broken Windows systems, so they’re not very flexible).

Hybrid ISOs are ISO files that can be burned to optical media (CDs, DVDs, etc.) and also be dd'd directly to a USB thumbdrive (for computers that support booting from USB). That means one file, multiple media types.

Architectures are different hardware platforms. This mostly refers to the CPU. Common implementations are 64-bit (also known as x86_64 or AMD64 for ones that support running both 64-bit and 32-bit software, or IA64 or Itanium for processors that only support 64-bit) and 32-bit (or i686 and the older i386 and i486 implementations). Most consumer PCs on the market today are x86_64.

Chroots, chrooting, and the like are variants on the word chroot. A chroot is a way of running a GNU/Linux install "inside" another GNU/Linux distro. It’s sort of like a virtual machine, or VM, except that it’s a lot more lightweight and it doesn’t do any actual virtualization- and uses the host’s kernel, memory mapping, etc. It’s very useful for development of operating systems.

PXE, or Pre-boot eXecution Environment, is a way of booting operating systems over a local network.

iPXE is a project that builds a very small Linux kernel, UNDI (traditional PXE) images, and the like that allow you to essentially use PXE over the Internet. It’s very flexible and customizable, and supports a custom scripting engine and such.

2.2. Why live media is necessary/Why you might want BDisk

"But Brent," I hear you ask in a voice which most likely is nothing close to what you actually sound like and entirely in my head, "Why would I need a live CD/USB/etc.? And why BDisk?"

Elementary, my dear imaginary reader! I touch on some reasons why one might want live media in the beginning of the User Manual, but here’s why you might want BDisk specifically as opposed to another live distro (or live distro creator).

  • Fully customizable

  • Works with a multitude of GNU/Linux distros — both for the host build system and as the guest. (Still under development!)

  • It performs optimizations and compression to help you get the smallest ISO possible.

  • In addition to building hybrid ISOs, it supports building iPXE hybrid ISOs (meaning you only need a very small file; the rest of the operating system boots over the Internet).

  • It supports both BIOS and UEFI systems- both the full image and the iPXE images.

  • It supports multiple architectures (x86_64, i686, possibly IA64 — untested) on the same ISO.

  • It supports automatically syncing to a web mirror, PXE boot server, etc. via rsync upon successful build.

  • It supports SecureBoot (untested!).

  • It is 100% compatible with both the Arch installation guide and the Gentoo installation guide.

  • It allows for non-interactive/automated building (i.e. nightly images).

  • It supports arbitrary file inclusion in a defined path on the ISO itself, not via some arbitrary directory as a separate partition on the media.

  • It can automatically build an accompanying "mini" ISO using iPXE — which is also a hybrid, UEFI-supported ISO.

  • Automatic versioning based on git tags (optional).

2.3. Who might want to use BDisk?

  • System builders/hardware testers

  • System Administrators/Engineers/Architects

  • Information Security professionals

  • Computer repair shops

  • Technology Consultants

  • Hobbyists

  • Home GNU/Linux users

  • Technology enthusiasts

3. Project Structure

The following is a tree of files and directories in a BDisk root directory. Note that yours may not look quite like this, as BDisk supports some directory relocation to aid in packaging for distros. These will be examined in-depth in the coming sections.

<BDisk root directory>
├── bdisk
│   ├── bchroot.py
│   ├── bdisk.py
│   ├── bGPG.py
│   ├── bSSL.py
│   ├── bsync.py
│   ├── build.py
│   ├── host.py
│   ├── ipxe.py
│   └── prep.py
├── docs
│   ├── COPYING
│   ├── LICENSE -> COPYING
│   ├── manual
│   │   └── (...)
│   ├── README
├── examples
│   └── HTTP
│       └── (...)
├── extra
│   ├── bdisk.png
│   ├── bin
│   │   └── (...)
│   ├── dist.build.ini
│   ├── external
│   │   └── (...)
│   ├── mirrorlist
│   ├── pre-build.d
│   │   ├── (...)
│   │   ├── i686
│   │   │   └── (...)
│   │   └── x86_64
│   │       └── (...)
│   └── templates
│       ├── BIOS
│       │   ├── isolinux.cfg.arch.j2
│       │   └── isolinux.cfg.multi.j2
│       ├── EFI
│       │   ├── base.conf.j2
│       │   ├── loader.conf.j2
│       │   ├── ram.conf.j2
│       │   ├── uefi1.conf.j2
│       │   └── uefi2.conf.j2
│       ├── GPG.j2
│       ├── iPXE
│       │   ├── BIOS
│       │   │   └── isolinux.cfg.j2
│       │   ├── EFI
│       │   │   ├── base.conf.j2
│       │   │   └── loader.conf.j2
│       │   ├── EMBED.j2
│       │   ├── patches
│       │   │   ├── 01.git-version.patch.j2
│       │   │   └── 02.banner.patch.j2
│       │   └── ssl
│       │       └── openssl.cnf
│       ├── overlay
│       │   ├── (...)
│       │   ├── i686
│       │   ├── x86_64
│       ├── pre-build.d
│       │   ├── (...)
│       │   ├── i686
│       │   ├── x86_64
│       ├── VARS.txt.j2
│       └── VERSION_INFO.txt.j2
└── overlay
    ├── (...)
    ├── i686
    └── x86_64

3.1. bdisk/

This directory contains the "heart" of BDisk. It essentially is a Python module package. It contains several python "subpackages" split into different files that provide different functions for BDisk. Chances are you won’t ever need to touch anything in here.

3.2. docs/

This directory contains various documentation and other helpful text.

3.2.1. COPYING

This contains BDisk’s license, the GPLv3.

3.2.2. LICENSE

This is simply a link to COPYING.

3.2.3. manual/

This directory contains the documentation source you’re reading right now! It’s written in asciidoc (well, to be more precise it’s written in/has some asciidoctor-isms). I’d recommend reading the rendered version, as the source (while perfectly human-readable) is written in a very modular fashion so it may be inconvenient to read each source file and following include directives.

3.2.4. README

This is a placeholder for common convention; it simply tells you to read the manual (and where to find it/build it).

3.3. examples/

This directory contains example filesystem layouts for e.g. your webserver (for iPXE), or your PXE server via TFTP.

3.4. extra/

This directory contains multiple "support files" for BDisk building.

3.4.1. bdisk.png

This file is used for bootloader graphics. If you change the name of the project, this can be named something different — see the section on uxname.

3.4.2. bin/

This directory contains sample code or extra tools that don’t have anything to do with BDisk normal operation but are useful in building a BDisk distribution.

3.4.3. dist.build.ini

This is the "source-provided"/upstream example build.ini. It will be sourced for any missing configuration options or the like.

3.4.4. external/

This directory contains external source code for use with extra features in BDisk that would otherwise be inconvenient to fetch and build dynamically.

3.4.5. pkg.build.ini

This is the recommended default build.ini file for packagers of distro repositories to use when packaging BDisk for inclusion in a package manager.

3.4.6. pre-build.d/

This file contains a "core" overlay. Generally these files shouldn’t be modified unless you know what you’re doing, but there are some interesting things you can do in here. Generally speaking, though, you’ll want to place your modifications in the overlay/ directory.

For files that should be included in both chroots, simply recreate the path with the desired file. For instance, if you want a file /etc/foo/bar.conf to exist in both i686 and x86_64 versions, it would exist as the path pre-build.d/etc/foo/bar.conf.

It follows the following structure:

i686/

This contains modifications that should be applied to the i686 version only. If you wanted a file to exist only in the i686 version at /etc/a/b.conf, it would be placed in pre-build.d/i686/etc/a/b.conf.

x86_64/

This contains modifications that should be applied to the x86_64 version only. If you wanted a file to exist only in the x86_64 version at /etc/z/y.conf, it would be placed in pre-build.d/x86_64/etc/z/y.conf.

3.4.7. templates/

This directory contains dynamic templates used for dynamic configuration building and other such things. They are written in Jinja2. If you haven’t used Jinja2 before, the templating documentation will prove to be very useful.

This allows you to customize low-level behaviour of BDisk without modifying the source.

BIOS/

The isolinux.cfg.arch.j2 template controls boot options for the single-arch versions of BDisk. In other words if you only build an i686 or only an x86_64 version, this is the template that would be used for BIOS boot mode.

The isolinux.cfg.multi.j2 is used for multi-arch. It manages booting for both i686 and x86_64 versions.

These files will let you change the behaviour of booting in BIOS systems. The menu colour, the menu entries, the menu default, etc.

EFI/

The files in here are systemd-boot configurations. The distributed defaults are:

base.conf.j2, which controls the "main"/default entry.

loader.conf.j2, the meta configuration file which tells the loader which entry to load by default and which entries to include.

ram.conf.j2 which allows BDisk to run entirely from RAM.

uefi1.conf.j2 which provides a UEFI shell (for older UEFI systems).

uefi2.conf.j2 which provides a UEFI shell (for newer UEFI systems).

GPG.j2

This file contains default parameters for the GPG key generation, if we need to automatically generate a key.

iPXE/

This directory holds templates for iPXE/mini builds.

The BIOS/ directory is similar to BIOS/ mentioned above, but it only needs one configuration file and is a much more minimal design (since its entire purpose is to chainload to the iPXE loader).

The EFI/ directory is similar to EFI/ above also, but needs fewer configuration files (its only purpose is to bootstrap iPXE).

EMBED.j2 is the iPXE embedded script (more info). This is what chainloads the remote resources (kernel, intird, squashed filesystem images, and so forth).

The patches/ directory largely control branding of the mini ISO. They are in unified diff (or "patch") format.

overlay/

This directory contains templated overlays. These are intended to be templated by the user. See the overlay section for more information on how to use this. Remember to suffix your template files with the .j2 extension.

pre-build.d/

This directory contains templated overlays. These are intended to not be managed by the user, as they handle configuration necessary for building an ISO. See the pre-build.d section for more information on this.

VERSION_INFO.txt.j2

This template specifies a VERSION_INFO.txt file placed in various locations throughout the builds to help identify which version, build, etc. the ISO is.

3.5. overlay/

This directory follows similar rules to the pre-build.d/ directory, except it is applied after the chroots are prepared (as it is designed to be user-centric rather than core functionality). We’ll go more into this later in-depth, as this is where most of your customizations will be done.

For files that should be included in both chroots, simply recreate the path with the desired file. For instance, if you want a file /etc/foo/bar.conf to exist in both i686 and x86_64 versions, it would exist as the path overlay/etc/foo/bar.conf.

It follows the following structure:

3.5.1. i686/

This contains modifications that should be applied to the i686 version only. If you wanted a file to exist only in the i686 version at /etc/a/b.conf, it would be placed in overlay/i686/etc/a/b.conf.

3.5.2. x86_64/

This contains modifications that should be applied to the x86_64 version only. If you wanted a file to exist only in the x86_64 version at /etc/z/y.conf, it would be placed in overlay/x86_64/etc/z/y.conf.

4. The build.ini File

This file is where you can specify some of the very basics of BDisk building. It allows you to specify/define certain variables and settings used by the build process. It uses ConfigParser for the parsing engine, and you can do some more advanced things with it than I demonstrate in the default.

It’s single-level, but divided into "sections". This is unfortunately a limitation of ConfigParser, but it should be easy enough to follow.

Blank lines are ignored, as well as any lines beginning with # and ;. There are some restrictions and recommendations for some values, so be sure to note them when they occur. Variables referencing other values in the build.ini are allowed in the format of ${keyname} if it’s in the same section; otherwise, ${section:keyname} can be used.

If you want to use your own build.ini file (and you should!), the following paths are searched in order. The first one found will be used.

  • /etc/bdisk/build.ini

  • /usr/share/bdisk/build.ini

  • /usr/share/bdisk/extra/build.ini

  • /usr/share/docs/bdisk/build.ini

  • /usr/local/etc/bdisk/build.ini

  • /usr/local/share/docs/bdisk/build.ini

  • /opt/dev/bdisk/build.ini

  • /opt/dev/bdisk/extra/build.ini

  • /opt/dev/bdisk/extra/dist.build.ini

  • <bdisk.py directory>/../build.ini

We’ll go into more detail for each section below.

4.1. Example

[bdisk]
name = BDISK
uxname = bdisk
pname = BDisk
ver =
dev = A Developer
email = dev@domain.tld
desc = A rescue/restore live environment.
uri = https://domain.tld
root_password =
user = yes
[user]
username = ${bdisk:uxname}
name = Default user
groups = ${bdisk:uxname},admin
password = $$6$$t92Uvm1ETLocDb1D$$BvI0Sa6CSXxzIKBinIaJHb1gLJWheoXp7WzdideAJN46aChFu3hKg07QaIJNk4dfIJ2ry3tEfo3FRvstKWasg/
[source_x86_64]
mirror = mirror.us.leaseweb.net
mirrorproto = https
mirrorpath = /archlinux/iso/latest/
mirrorfile = .sig
mirrorchksum = ${mirrorpath}sha1sums.txt
chksumtype = sha1
mirrorgpgsig =
gpgkey = 7F2D434B9741E8AC
gpgkeyserver =
[source_i686]
mirror = mirror.us.leaseweb.net
mirrorproto = https
mirrorpath = /archlinux/iso/latest/
mirrorfile =
mirrorchksum = ${mirrorpath}sha1sums.txt
chksumtype = sha1
mirrorgpgsig =
gpgkey =
gpgkeyserver =
[build]
dlpath = /var/tmp/${bdisk:uxname}
chrootdir = /var/tmp/chroots
basedir = /opt/dev/bdisk
isodir = ${dlpath}/iso
srcdir = ${dlpath}/src
prepdir = ${dlpath}/temp
archboot = ${prepdir}/${bdisk:name}
mountpt = /mnt/${bdisk:uxname}
multiarch = x86_64
sign = yes
ipxe = no
i_am_a_racecar = no
[gpg]
mygpgkey =
mygpghome =
[sync]
http = no
tftp = no
git = no
rsync = no
[http]
path = ${build:dlpath}/http
user = http
group = http
[tftp]
path = ${build:dlpath}/tftpboot
user = root
group = root
[ipxe]
iso = no
uri = https://domain.tld
ssldir = ${build:dlpath}/ssl
ssl_ca = ${ssldir}/ca.crt
ssl_cakey = ${ssldir}/ca.key
ssl_crt = ${ssldir}/main.crt
ssl_key = ${ssldir}/main.key
[rsync]
host =
user =
path =
iso = no

4.2. [bdisk]

This section controls some basic branding and information having to do with the end product.

4.2.1. name

This value is a "basic" name of your project. It’s not really shown anywhere end-user visible, but we need a consistent name that follows some highly constrained rules:

  1. Alphanumeric only

  2. 8 characters total (or less)

  3. No whitespace

  4. ASCII only

  5. Will be converted to uppercase if it isn’t already

4.2.2. uxname

This value is used for filenames and the like. I highly recommend it be the same as name (in lowercase) but it doesn’t need to be. It also has some rules:

  1. Alphanumeric only

  2. No whitespace

  3. ASCII only

  4. Will be converted to lowercase if it isn’t already

4.2.3. pname

This string is used for "pretty-printing" of the project name; it should be a more human-readable string.

  1. Can contain whitespace

  2. Can be mixed-case, uppercase, or lowercase

  3. ASCII only

4.2.4. ver

The version string. If this isn’t specified, we’ll try to guess based on the current git commit and tags in build:basedir. If build:basedir is not a git repository (i.e. you installed BDisk from a package manager), you MUST specify a version number.

  1. No whitespace

4.2.5. dev

The name of the developer or publisher of the ISO, be it an individual or organization. For example, if you are using BDisk to build an install CD for your distro, this would be the name of your distro. The same rules as pname apply.

  1. Can contain whitespace

  2. Can be mixed-case, uppercase, or lowercase

  3. ASCII only

4.2.6. email

An email address to use for git syncing messages, and/or GPG key generation.

4.2.7. desc

What this distribution/project is used for.

  1. Can contain whitespace

  2. Can be mixed-case, uppercase, or lowercase

  3. ASCII only

4.2.8. uri

What is this project’s URI (website, etc.)? Alternatively, your personal site, your company’s site, etc.

  1. Should be a valid URI understood by curl

4.2.9. root_password

The escaped, salted, hashed string to use for the root user.

Please see the section on passwords for information on this value. In the example above, the string 6t92Uvm1ETLocDb1D$$BvI0Sa6CSXxzIKBinIaJHb1gLJWheoXp7WzdideAJN46aChFu3hKg07QaIJNk4dfIJ2ry3tEfo3FRvstKWasg/ is created from the password test. I cannot stress this enough, do not use a plaintext password here nor just use a regular /etc/shadow file/crypt(3) hash here. Read the section. I promise it’s short.

4.2.10. user

Default: no

This setting specifies if we should create a regular (non-root) user in the live environment. See the section [user] for more options.

Note
If enabled, this user has full sudo access.
Accepts (case-insensitive) one of:

yes

no

true

false

1

0

4.3. [user]

This section of build.ini controls aspects about bdisk:user. It is only used if bdisk:user is enabled.

4.3.1. username

What username should the user have? Standard *nix username rules apply:

  1. ASCII only

  2. 32 characters or less

  3. Alphanumeric only

  4. Lowercase only

  5. No whitespace

  6. Cannot start with a number

4.3.2. name

What comment/description/real name should be used for the user? For more information on this, see the passwd(5) man page's section on GECOS.

  1. ASCII only

4.3.3. groups

What groups this user should be added to, comma-separated. They will be created if they don’t exist yet. Standard *nix group names rules apply:

  1. ASCII only

  2. 32 characters or less

  3. Can only contain lower-case letters, numeric digits, underscores, or dashes (and can end with a dollar sign)

  4. Must start with a (lower-case) letter or underscore

  5. No whitespace

4.3.4. password

The escaped, salted, hashed string to use for the non-root user.

Please see the section on passwords for information on this value. In the example above, the string 6t92Uvm1ETLocDb1D$$BvI0Sa6CSXxzIKBinIaJHb1gLJWheoXp7WzdideAJN46aChFu3hKg07QaIJNk4dfIJ2ry3tEfo3FRvstKWasg/ is created from the password test. I cannot stress this enough, do not use a plaintext password here nor just use a regular /etc/shadow file/crypt(3) hash here. Read the section. I promise it’s short.

4.4. [source_<arch>]

This section controls where to fetch the "base" tarballs.

Note
Previously, these settings were not architecture-specific, and included in the build section.

It was necessary to create this section per architecture, because Arch Linux has dropped i686 support. However, plenty of other distros also have removed support and other third-party projects have ported. (You can find the Arch Linux 32-bit/i686 port project here.)

The directives here are only covered once, however, since both sections are identical- they just allow you to specify different mirrors. Note that the two settings are [source_i686] (for 32-bit) and [source_x86_64] (for 64-bit/multilib).

Which section is used (or both) depends on what architectures you have enabled for the build.

4.4.1. mirror

A mirror that hosts the bootstrap tarball. It is highly recommended you use an Arch Linux bootstrap tarball as the build process is highly specialized to this (but patches/feature requests are welcome for other built distros). You can find a list of mirrors at the bottom of Arch’s download page.

  1. No whitespace

  2. Must be accessible remotely/via a WAN-recognized address

  3. Must be a domain/FQDN (or IP address) only; no paths (those come later!)

4.4.2. mirrorproto

What protocol should we use for the mirror?

Must be (case-insensitive) one of:

http

https

ftp

4.4.3. mirrorpath

What is the path to the tarball directory on the mirror?

  1. Must be a complete path (e.g. /dir1/subdir1/subdir2)

  2. No whitespace

4.4.4. mirrorfile

What is the filename for the tarball found in the path specified in mirrorpath ? If left blank, we will use the hash checksum file to try to guess the most recent file.

4.4.5. mirrorchksum

[optional]
default: (no hash checking done)
requires: chksumtype

The path to a checksum file of the bootstrap tarball.

  1. No whitespace

  2. Must be the full path

  3. Don’t include the mirror domain or protocol

4.4.6. chksumtype

The algorithm that mirrorchksum's hashes are in.

Accepts one of:

blake2b

blake2s

md5

sha1

sha224

sha256

sha384

sha512

sha3_224

sha3_256

sha3_384

sha3_512

shake_128

shake_256

Tip
You may have support for additional hashing algorithms, but these are the ones gauranteed to be supported by Python’s hashlib module. To get a full list of algorithms the computer you’re building on supports, you can run python3 -c 'import hashlib;print(hashlib.algorithms_available)'. Most likely, however, mirrorchksum is going to be hashes of one of the above.

4.4.7. mirrorgpgsig

[optional]
default: (no GPG checking done)
requires: gpg/gnupg
requires: gpgkey

If the bootstrap tarball file has a GPG signature, we can use it for extra checking. If it’s blank, GPG checking will be disabled.

If you specify just .sig (or use the default and don’t specify a mirrorfile), BDisk will try to guess based on the file from the hash checksum file. Note that unless you’re using the .sig "autodetection", this must evaluate to a full URL. (e.g. ${mirrorproto}://${mirror}${mirrorpath}somefile.sig)

4.4.8. gpgkey

requires: gpg/gnupg

What is a key ID that should be used to verify/validate the mirrorgpgsig?

  1. Only used if mirrorgpgsig is set

  2. Can be in "short" form (e.g. 7F2D434B9741E8AC) or "full" form (4AA4767BBC9C4B1D18AE28B77F2D434B9741E8AC), with or without the 0x prefix.

4.4.9. gpgkeyserver

default: blank (GNUPG-bundled keyservers)
requires: gpg/gnupg

What is a valid keyserver we should use to fetch gpgkey?

  1. Only used if mirrorgpgsig is set

  2. The default (blank) is probably fine. If you don’t specify a personal GPG config, then you’ll most likely want to leave this blank.

  3. If set, make sure it is a valid keyserver URI (e.g. hkp://keys.gnupg.net)

Accepts (case-insensitive) one of:

yes

no

true

false

1

0

4.5. [build]

This section controls some aspects about the host and things like filesystem paths, etc.

4.5.1. gpg

Should we sign our release files? See the [gpg] section.

Accepts (case-insensitive) one of:

yes

no

true

false

1

0

4.5.2. dlpath

Where should the release files be saved? Note that many other files are created here as well.

Warning
If you manage your project in git, this should not be checked in as it has many large files that are automatically generated!
  1. No whitespace

  2. Will be created if it doesn’t exist

4.5.3. chrootdir

Where the bootstrap tarball(s) extract to, where the chroots are built and prepped for filesystems on the live media.

Warning
If you manage your project in git, this should not be checked in as it has many large files that are automatically generated!
  1. No whitespace

  2. Will be created if it doesn’t exist

4.5.4. basedir

Where your extra/ and overlay/ directories are located. If you checked out from git, this would be your git worktree directory.

  1. No whitespace

  2. Must exist and contain the above directories populated with necessary files

4.5.5. isodir

This is the output directory of ISO files when they’re created (as well as GPG signatures if you enabled them).

Warning
If you manage your project in git, this should not be checked in as it has many large files that are automatically generated!
  1. No whitespace

  2. Will be created if it doesn’t exist

4.5.6. srcdir

This is where we save and compile source code if we need to dynamically build components (such as iPXE for mini ISOs).

  1. No whitespace

  2. Will be created if it doesn’t exist (and is needed)

4.5.7. prepdir

This is the directory we use for staging.

  1. No whitespace

  2. Will be created if it doesn’t exist

4.5.8. archboot

This directory is used to stage boot files.

Warning
This directory should not be the exact same path as other directives! If so, you will cause your ISO to be much larger than necessary. A subdirectory of another directive’s path, however, is okay.
  1. No whitespace

  2. Will be created if it doesn’t exist

4.5.9. mountpt

The path to use as a mountpoint.

  1. No whitespace

  2. Will be created if it doesn’t exist

4.5.10. multiarch

default: yes

Whether or not to build a "multiarch" image- that is, building support for both x86_64 and i686 in the same ISO.

In order to…​ Accepts (case-insensitive) one of:

build a multiarch ISO

yes

true

1

build a separate ISO for each architecture

no

false

0

only build an i686-architecture ISO

i686

32

no64

only build an x86_64-architecture ISO

x86_64

64

no32

4.5.11. ipxe

default: no

Enable iPXE ("mini ISO") functionality.

Note
This has no bearing on the [sync] section, so you can create an iPXE HTTP preparation for instance without needing to sync it anywhere (in case you’re building on the webserver itself).
Accepts (case-insensitive) one of:

yes

no

true

false

1

0

4.5.12. i_am_a_racecar

default: no

This option should only be enabled if you are on a fairly powerful, multicore system with plenty of RAM. It will speed the build process along, but will have some seriously adverse effects if your system can’t handle it. Most modern systems should be fine with enabling it.

Accepts (case-insensitive) one of:

yes

no

true

false

1

0

4.6. [gpg]

This section controls settings for signing our release files. This is only used if build:gpg is enabled.

4.6.1. mygpgkey

A valid key ID that BDisk should use to sign release files.

  1. You will be prompted for a passphrase if your key has one/you don’t have an open and authorized gpg-agent session. Make sure you have a working pinentry configuration set up!

  2. If you leave this blank we will use the key we generate automatically earlier in the build process.

  3. We will generate one if this is blank and you have selected sign as yes.

4.6.2. mygpghome

The directory should be used for the above GPG key if specified. Make sure it contains a keybox (.kbx) your private key. (e.g. /home/username/.gnupg)

4.7. [sync]

This section controls what we should do with the resulting build and how to handle uploads, if we choose to use those features.

4.7.1. http

default: no

If enabled, BDisk will generate/prepare HTTP files. This is mostly only useful if you plan on using iPXE. See the [http] section.

Accepts (case-insensitive) one of:

yes

no

true

false

1

0

4.7.2. tftp

default: no

If enabled, BDisk will generate/prepare TFTP files. This is mostly only useful if you plan on using more traditional (non-iPXE) setups and regualar PXE bootstrapping into iPXE.

Accepts (case-insensitive) one of:

yes

no

true

false

1

0

4.7.3. git

requires: git
default: no

Enable automatic Git pushing for any changes done to the project itself. If you don’t have upstream write/push access, you’ll want to disable this.

Accepts (case-insensitive) one of:

yes

no

true

false

1

0

4.7.4. rsync

requires: rsync
default: no

Enable rsync pushing for the ISO (and other files, if you choose- useful for iPXE over HTTP(S)).

Accepts (case-insensitive) one of:

yes

no

true

false

1

0

4.8. [http]

This section controls details about HTTP file preparation/generation. Only used if sync:http is enabled.

4.8.1. path

This directory is where to build an HTTP webroot.

Warning
MAKE SURE you do not store files here that you want to keep! They will be deleted!
  1. No whitespace

  2. If blank, HTTP preparation/generation will not be done

  3. If specified, it will be created if it doesn’t exist

  4. Will be deleted first

4.8.2. user

What user the HTTP files should be owned as. This is most likely going to be either 'http', 'nginx', or 'apache'.

  1. No whitespace

  2. User must exist on build system

Can be one of:

username

UID

4.8.3. group

What group the HTTP files should be owned as. This is most likely going to be either 'http', 'nginx', or 'apache'.

  1. No whitespace

  2. Group must exist on build system

Can be one of:

groupname

GID

4.9. [tftp]

This section controls details about TFTP file preparation/generation. Only used if sync:tftp is enabled.

4.9.1. path

The directory where we want to build a TFTP root.

Warning
MAKE SURE you do not store files here that you want to keep! They will be deleted!
  1. No whitespace

  2. Will be created if it doesn’t exist

  3. Will be deleted first

4.9.2. user

What user the TFTP files should be owned as. This is most likely going to be either 'tftp', 'root', or 'nobody'.

  1. No whitespace

  2. User must exist on build system

Can be one of:

username

UID

4.9.3. group

What group the TFTP files should be owned as. This is most likely going to be either 'tftp', 'root', or 'nobody'.

  1. No whitespace

  2. Group must exist on build system

Can be one of:

groupname

GID

4.10. [ipxe]

This section controls aspects of iPXE building. Only used if build:ipxe is enabled.

4.10.1. iso

default: no
requires: git

Build a "mini-ISO"; that is, an ISO file that can be used to bootstrap an iPXE environment (so you don’t need to set up a traditional PXE environment on your LAN). We’ll still build a full standalone ISO no matter what.

Accepts (case-insensitive) one of:

yes

no

true

false

1

0

4.10.2. uri

What URI iPXE’s EMBED script should use. This would be where you host an iPXE chainloading script on a webserver, for instance. See iPXE’s example of dynamic scripts for an example of the script that would be placed at this URI.

Note
If you require HTTP BASIC Authentication or HTTP Digest Authentication (untested), you can format it via https://user:password@bdisk.square-r00t.net/boot.php.
Note
This currently does not work for HTTPS with self-signed certificates.
  1. Required if iso is enabled

4.10.3. ssldir

Directory to hold SSL results, if we are generating keys, certificates, etc.

  1. No whitespace

  2. Will be created if it does not exist

4.10.4. ssl_ca

Path to the (root) CA certificate file iPXE should use. See iPXE’s crypto page for more information.

Note
You can use your own CA to sign existing certs. This is handy if you run a third-party/"Trusted" root-CA-signed certificate for the HTTPS target.
  1. No whitespace

  2. Must be in PEM/X509 format

  3. Required if iso is enabled

  4. If it exists, a matching key (ssl_cakey) must be specified

    1. However, if left blank/doesn’t exist, one will be automatically generated

4.10.5. ssl_cakey

Path to the (root) CA key file iPXE should use.

  1. No whitespace

  2. Must be in PEM/X509 format

  3. Required if iso is enabled

  4. If left blank or it doesn’t exist (and ssl_ca is also blank), one will be automatically generated

  5. Must match/pair to ssl_ca if specified/exists

  6. MUST NOT be passphrase-protected/DES-encrypted

4.10.6. ssl_crt

Path to the client certificate iPXE should use.

  1. No whitespace

  2. Must be in PEM/X509 format

  3. Required if iso is enabled

  4. If specified/existent, a matching CA cert (ssl_ca) and key (ssl_cakey) must be specified

    1. However, if left blank/doesn’t exist, one will be automatically generated

  5. Must be signed by ssl_ca/ssl_cakey if specified and already exists

4.10.7. ssl_key

Path to the client key iPXE should use.

  1. No whitespace

  2. Must be in PEM/X509 format

  3. Required if iso is enabled

  4. If left blank/nonexistent (and ssl_ca is also blank), one will be automatically generated

4.11. [rsync]

This section controls aspects of rsync pushing. Only used if sync:rsync is enabled.

4.11.1. host

The rsync destination host.

  1. Must resolve from the build server

  2. Can be host, FQDN, or IP address

4.11.2. user

This is the remote user we should use when performing the rsync push.

  1. User must exist on remote system

  2. SSH pubkey authorization must be configured

  3. The destination’s hostkey must be added to your local build user’s known hosts

4.11.3. path

This is the remote destination path we should use for pushing via rsync.

Note
You’ll probably want to set http:user and http:group to what it’ll need to be on the destination.
  1. No whitespace

  2. The path must exist on the remote host

  3. The path MUST be writable by user

4.11.4. iso

Should we rsync over the ISO files too, or just the boot files?

Accepts (case-insensitive) one of:

yes

no

true

false

1

0

5. Advanced Customization

If the build.ini file doesn’t provide enough customization to your liking, I don’t blame you! It was designed only to provide the most basic control and is primarily only used to control the build process itself.

Luckily, there are a lot of changes you can make. For all of these, you’ll want to make a copy of the basedir directory somewhere and change the basedir configuration option in the build.ini file to point to that directory.

This section isn’t going to cover every single use case, as that’s mostly an exercise for you — I can’t predict how you want to use BDisk! But we’ll cover some common cases you can use and in the process you’ll know how to implement your own customizations.

5.1. SSH Pubkey Authentication

To start with, you’ll want to secure SSH a little more than normal.

I highly recommend this article, which we’ll be following in this process.

First, create a file: <basedir>/overlay/etc/ssh/sshd_config using the following. Comments and blank lines have been stripped out for brevity.

PermitRootLogin prohibit-password
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
AuthorizedKeysFile	.ssh/authorized_keys
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
UsePAM yes
PrintMotd no # pam does that
Subsystem	sftp	/usr/lib/ssh/sftp-server
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

We’ll also want to implement a more secure ssh_config file to avoid possible leaks. The following is <basedir>/overlay/etc/ssh/ssh_config:

Host *
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
    PasswordAuthentication no
    ChallengeResponseAuthentication no
    PubkeyAuthentication yes
    HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

We’ll want to create our own moduli. This can take a long time, but only needs to be done once — it doesn’t need to be done for every build. The following commands should be run in <basedir>/overlay/etc/ssh/:

ssh-keygen -G moduli.all -b 4096
ssh-keygen -T moduli.safe -f moduli.all
mv moduli.safe moduli
rm moduli.all

Then we generate hostkeys. This isn’t strictly necessary as the live media will create them automatically when starting SSH if they’re missing, but this does provide some verification that the host you’re SSHing to is, in fact, running the BDisk instance that you yourself built. The following commands should be run in <basedir>/overlay/etc/ssh/:

ssh-keygen -t ed25519 -f ssh_host_ed25519_key -N "" < /dev/null
ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key -N "" < /dev/null

Make sure you have keys on your host workstation generated so you can SSH into BDisk. If you don’t have any ED25519 or RSA SSH keys, this will create them for you. The following should be run as the host (build machine, or what have you) user you want to be able to SSH into BDisk as:

ssh-keygen -t ed25519 -o -a 100
ssh-keygen -t rsa -b 4096 -o -a 100

The defaults are fine. Adding a password to your private key is not necessary, but recommended (though note that doing so will inhibit automated SSHing). You should now have in ~/.ssh/ the following files (assuming you kept the defaults above):

id_ed25519
id_ed25519.pub
id_rsa
id_rsa.pub
Warning
The files ending in .pub are public — they can be published anywhere. However, the ones that are not appended with .pub are your private keys and should not be shared with anyone, whether they’re password-protected or not!

Now you’ll want to get the public key of your SSH keys so you can add them to your BDisk build. The following commands should be run in <basedir>/overlay/:

mkdir -p root/.ssh
chmod 700 root/.ssh
touch root/.ssh/authorized_keys
chmod 600 root/.ssh/authorized_keys
cat ~/.ssh/id_{ed25519,rsa}.pub > root/.ssh/authorized_keys

If you decided to enable a regular non-root user in your build, you’ll want to perform the same steps above for the regular user as well (or forego the above and just enable SSH for the user you create). Remember to replace root/ with home/<username>/!

Lastly, we need to enable SSH to start on boot. Run the following command in <basedir>/overlay/etc/systemd/system/multi-user.target.wants/:

ln -s /usr/lib/systemd/system/sshd.service sshd.service

You should now have SSH automatically start once the instance boots.

5.2. VPN Configuration

For this example we’ll set up an OpenVPN client to start automatically on boot.

Setting up an OpenVPN server is outside the scope of this section, but there are a multitude of useful documentation sources out there that will help you with that.

However, once you have your client .ovpn file (in our example, we’ll call it client.ovpn) you can add it to the build relatively easily.

Copy client.ovpn as <basedir>/overlay/etc/openvpn/client/client.conf — note the changed file extension. Then, in the <basedir>/overlay/etc/systemd/system/multi-user.target.wants/ directory, issue these commands:

ln -s /usr/lib/systemd/system/openvpn-client\@.service openvpn-client\@client.service

OpenVPN will then start on boot in the built BDisk instance.

5.3. Changing the Installed Software

BDisk comes with a large list of software installed in the build instance by default, ranging from data recovery (such as foremost, scalpel, ddrescue, etc.), security and data wiping (nwipe, scrub, etc.), penetration testing (wifite, aircrack-ng, etc.) and a slew of others. Seriously, if you’re looking for a tool, changes are it’s on it.

However, this leads to a fairly long build time- even with a local repository mirror (many of the packages are from the AUR). You may want to replace the list with a smaller subset.

The iso.pkgs.* files are not files you should modify- they contain software necessary to the building of BDisk and are the basic necessary files to build a bootable image. However, the packages.* files are where you would add or remove software to be installed.

Note
The package lists can contain both Arch repository packages and AUR packages.
Note
Blank lines are ignored, and you can comment out lines by prefixing the line with the # character.

5.3.1. <basedir>/extra/pre-build.d/i686/root/packages.arch

This list contains packages to only be installed for the i686 image.

5.3.2. <basedir>/extra/pre-build.d/x86_64/root/packages.arch

This list contains packages you only want installed in the x86_64 image.

5.3.3. <basedir>/extra/pre-build.d/root/packages.both

This file contains packages for both architectures (i686 and x86_64).

5.4. Changing the Build Process

If you want to make modifications that can’t be managed by arbitrary file inclusion or changing the software package lists, you may want to introduce additional changes to the image configuration that’s run during the chroot. This is fairly easy to do. Simply modify <basedir>/extra/pre-build.d/root/pre-build.sh with the changes you desire. Note that this has a .sh file extension, but it can be any type of script you want — Bash, Perl, Python, etc. — it just needs the shebang line at the beginning of the script.

5.5. Automatic Login (TTY)

If you don’t want to have to log into the TTY on boot, BDisk can automatically log in for you with a given username.

If, for example, you want a terminal to auto-login on TTY1 with the root user, you would create the following file at <basedir>/overlay/etc/systemd/system/getty@tty1.service.d/autologin.conf:

[Service]
Type=idle
ExecStart=
ExecStart=-/usr/bin/agetty --autologin root --noclear %I 38400 linux

5.6. Starting a Desktop Environment

You can install any desktop environment or window manager you would like via package lists! From there, it’s simply a matter of setting the correct Systemd unit to start automatically. The Arch wiki has a lot of useful information here. As an example, I’ll include LXDE instructions here.

Simply create a symlink for the target. In the <basedir>/overlay/etc/systemd/system/ directory:

ln -s /usr/lib/systemd/system/lxdm.service display-manager.service

5.6.1. Autologin (LXDE)

Many desktop environments even offer an automatic login feature directly through the desktop manager (LXDM, in LXDE’s case).

Again, using LXDE as an example, create a file at <basedir>/overlay/etc/lxdm/lxdm.conf:

[base]
autologin=bdisk
greeter=/usr/lib/lxdm/lxdm-greeter-gtk
[server]
arg=/usr/bin/X -background vt1
[display]
gtk_theme=Adwaita
bottom_pane=1
lang=1
keyboard=0
theme=Industrial
[input]
[userlist]
disable=0
white=
black=

LXDE will then automatically log in with the user bdisk (note the second line, right under [base]) whenever started.

6. Building a BDisk ISO

So you finally have configured BDisk (and perhaps added further customizations. Now you’re ready to build!

Building is, thankfully, the easiest part!

Note
Due to requiring various mounting and chrooting, BDisk must be run as the root user (or via sudo).

To initiate a build, simply run <basedir>/bdisk/bdisk.py. That’s it! Everything should continue automatically.

If you’d like to specify a path to a specific build configuration, you can use <basedir>/bdisk/bdisk.py path/to/build.ini. The default is /etc/bdisk/build.ini (plus other locations).

If you’re using a packaged version you installed from your distro’s package manager, you instead should run wherever it installs to. Most likely this is going to be /usr/sbin/bdisk. (On systemd build hosts that have done the /usr merge, you can use /usr/sbin/bdisk or /sbin/bdisk.)

If you encounter any issues during the process, make sure you read the documentation — if your issue still isn’t addressed, please be sure to file a bug report!

Developer Manual

What good is software if nobody changes it?

BDisk can be sourced for other projects, as it’s written in a modular manner. Future versions may support installation as a normal Python module. This will also provide information you may need to change parts of BDisk — it is opensource, after all!

7. Layout of BDisk functions

These functions exist in bdisk/.

7.1. bchroot.py

This file controls creation of the chroots -- the directories in which BDisk builds the actual system that is booted into.

7.1.1. chroot(chrootdir, chroot_hostname, cmd = ‘/root/pre-build.sh’)

This function manages mounting the mountpoints for the chroot(s) in preparation for the images of the live media. It also runs the inner chroot preparation script. Returns chrootdir (same as the paramater provided).

chrootdir

The directory where the filesystem tree for the chroot lies. Absolute path only.

chroot_hostname

The hostname to use for the guest.

Note
This paramater may be removed in future versions.
cmd

The command to run inside the chroot once all the mountpoints are set up.

7.1.2. chrootUnmount(chrootdir)

Unmount the mounts set up in chroot().

7.1.3. chrootTrim(build)

This function performs some cleanup and optimizations to the chroot(s).

build

A dictionary of [code_build_code]'s values (with some additional keys/values added). See parseConfig().

7.2. bdisk.py

This file is a sort of "wrapper" — it pulls all the other files in this directory together into a single usable Python script. In other words, to build a BDisk distribution, you would simply run bdisk/bdisk.py — that’s it! See Building a BDisk ISO.

It contains no functions, it just contains minimal logic to tie all the other functions together.

7.3. bGPG.py

This contains functions having to do with GPG — signing files, verifying other signatures, generating a key (if one wasn’t specified), using a key (if one was specified), etc.

7.3.1. genGPG(conf)

This function controls generating (or "importing" an existing) GnuPG key for use with other operations. Returns gpg, a PyGPGME object.

conf

A dictionary of the configuration (with some additional keys/values added). See (TODO: link to host.py’s config parser).

7.3.2. killStaleAgent(conf)

This function kills off any stale GnuPG agents running. Not doing so can cause some strange behaviour both during the build process and on the host.

conf

See conf.

7.3.3. signIMG(path, conf)

This function signs a given file with the keys BDisk was either configured to use or automatically generated.

path

The full, absolute path to the file to be signed. An ASCII-armored detached signature (plaintext) will be generated at path.asc, and a binary detached signature will be generated at path.sig.

conf

See conf.

7.3.4. gpgVerify(sigfile, datafile, conf)

This function verifies a detatched signature against a file containing data. Returns True if the file verifies, or False if not.

sigfile

The detached signature file. Can be ASCII-armored or binary format. Full/absolute path only.

datafile

The file containing the data to be verified. Full/absolute path only.

conf

See conf.

7.3.5. delTempKeys(conf)

Delete automatically-generated keys (if we generated them) as well as the automatically imported verification key ([code_gpgkey_code]).

conf

See conf.

7.4. bSSL.py

Functions having to do with OpenSSL are stored here. This is used primarily for "mini" builds (via iPXE), they let you boot your BDisk distribution over the Internet. If an SSL key, CA certificate, etc. weren’t defined and you want to build a mini image, this file contains functions that will build an SSL PKI (public key infrastructure) for you automatically.

7.4.1. verifyCert(cert, key, CA = None)

This function will verify a certificate’s validity/pairing with a key, optionally against a given CA certificate. Returns True on successful verification, or False and an exit (for sanity purposes).

cert

The certificate to be validated. Must be a PyOpenSSL certificate object.

key

The key to validate against cert. Must be a PyOpenSSL key object.

CA

The CA, or certificate authority, certificate to verify against.

Note
This currently does not work, as PyOpenSSL does not currently support verifying against a specified CA certificate.

7.4.2. sslCAKey(conf)

This function imports a CA key ([code_ssl_cakey_code]) into a PyOpenSSL object (or generates one if necessary). Returns a PyOpenSSL key object.

conf

See conf.

7.4.3. sslCA(conf, key = None)

This function imports a CA certificate ([code_ssl_ca_code]) into a PyOpenSSL object (or generates one if necessary). Returns a PyOpenSSL certificate object.

conf

See conf.

key

A PyOpenSSL key object that should be used to generate the CA certificate (or is paired to the CA certificate if specified).

7.4.4. sslCKey(conf)

This function imports a client key ([code_ssl_key_code]) into a PyOpenSSL object (or generates one if necessary). Returns a PyOpenSSL key object.

conf

See conf.

7.4.5. ssslCSR(conf, key = None)

This function generates a CSR (certificate signing request).

conf

See conf.

key

A PyOpenSSL key object that should be used to generate the CSR. It should be a key that is paired to the client certificate.

7.4.6. sslSign(conf, ca, key, csr)

This function signs a CSR using a specified CA.

conf

See conf.

ca

A PyOpenSSL certificate object for the CA certificate. This certificate (object) should have signing capabilities.

key

A PyOpenSSL key object paired to ca.

csr

A PyOpenSSL CSR object. See sslCSR().

7.4.7. sslPKI(conf)

Ties all the above together into one convenient function. Returns a PyOpenSSL certificate object of the signed client certificate.

7.5. bsync.py

This file has functions relating to copying your BDisk build to various resources. For instance, if you want your ISO available to download then this file would be used to copy your finished build to an HTTP server/root you specify.

7.5.1. http(conf)

This function prepares a local HTTP directory, or webroot. See [code_http_code_2].

conf

See conf.

7.5.2. tftp(conf)

This function prepares a local TFTP directory (for traditional PXE). See [code_tftp_code_2].

conf

See conf.

7.5.3. git(conf)

This function commits (and pushes) any changes you might have made to your project ([code_basedir_code]) automatically.

conf

See conf.

7.5.4. rsync(conf)

This function syncs your builds, HTTP directory (if enabled), TFTP directory (if enabled), etc. to a remote host. See [code_rsync_code_2].

conf

See conf.

7.6. build.py

This is responsible for building the "full" ISO, building UEFI support, etc.

7.6.1. genImg(conf)

This function builds the squashed filesystem images and, if requested, signs them.

conf

See conf.

7.6.2. genUEFI(build, bdisk)

This function builds UEFI support for the ISO files. Returns the path of an embedded EFI bootable binary/ESP image.

build

The build section of the configuration.

bdisk

The bdisk section of the configuration.

7.6.3. genISO(conf)

Builds the full ISO image(s). Returns a dictionary of information about the built ISO file (see iso).

conf

See conf.

7.6.4. displayStats(iso)

Parses the output of e.g. genISO() and displays in a summary useful to the end-user.

iso

A dictionary of information about the ISO file. This is typically:

{'iso':
     {'name':<'Main' for the full ISO, 'Mini' for the mini ISO, etc.>},
     {<name>:
          'sha':<SHA256 sum of ISO file>,
          'file':<full/absolute path to ISO file>,
          'size':<size, in "humanized" format (e.g. #GB, #MB, etc.)>,
          'type':<Full or Mini>,
          'fmt':<human readable ISO type. e.g. Hybrid for an image that can be burned directly to a disk via dd or burned to optical media>
     }
}

7.6.5. cleanUp()

Currently a no-op; this function is reserved for future usage to cleanup the build process automatically.

7.7. host.py

These functions are used to perform "meta" tasks such as get information about the build host, find the build.ini file, and parse your configuration options.

7.7.1. getOS()

Returns the distribution of the build host.

7.7.2. getBits()

Returns the "bitness" of the build host (e.g. 32bit or 64bit)

7.7.3. getHostname()

Returns the hostname of the build host.

7.7.4. getConfig(conf_file = '/etc/bdisk/build.ini')

Returns a list of:

  1. the default configuration file

  2. the user-specified configuration file

conf_file

This is a full/absolute path that is searched first. If it exists and is a file, it is assumed to be the "canonical" build.ini file.

7.7.5. parseConfig(confs)

This function parses the configuration file(s) and returns a list of:

  1. A ConfigParser object

  2. The configuration as a dictionary

It performs some additional things, such as:

  • Converts "boolean" operations to true Python booleans

  • Tries to automatically detect the version if one isn’t provided

  • Establishes the build number (this is a number that should be local to the build host)

  • Forms a list of the architectures to build

  • Validates:

  • Makes prerequisite directories

confs

A list of configuration files. See getConfig().

7.8. ipxe.py

This file handles building the "mini" ISO via iPXE.

7.8.1. buildIPXE(conf)

This function builds the iPXE core files.

conf

See conf.

7.8.2. genISO(conf)

This function builds the mini ISO (if enabled). Returns a dictionary of information about the built ISO file (see iso).

conf

See conf.

7.8.3. tftpbootEnv(conf)

This function configures a TFTP boot/root directory for traditional PXE setups.

Note
This function currently is a no-op; it will be implemented in future versions.
conf

See conf.

7.9. prep.py

This contains functions that download the base tarball releases, preps them for [code_bchroot_py_code], builds necessary directory structures, and performs the overlay preparations.

7.9.1. dirChk(conf)

This function creates extra directories if needed.

conf

See conf.

7.9.2. downloadTarball(conf)

This function downloads the tarball ([code_mirrorfile_code]) from the [code_mirror_code], and performs verifications (SHA1 and GPG signature if enabled). Returns the full/absolute path to the downloaded tarball.

conf

See conf.

7.9.3. unpackTarball(tarball_path, build, keep = False)

This function extracts the tarball downloaded via downloadTarball().

tarball_path

The full/absolute path to the downloaded tarball.

build

See [build].

keep

True or False. Whether we should keep the downloaded tarball after unpacking/extracting. If your upstream tarball changes often enough, it’s recommended to set this to False. However, setting it to True can speed up the build process if you’re on a slower Internet connection.

7.9.4. buildChroot(conf, keep = False)

This incorporates donwloading and extracting into one function, as well as applying the pre-build.d/ directory (and the pre-build.d templates).

conf

See conf.

keep

See keep.

7.9.5. prepChroot(conf)

Returns a modified/updated [build]. This function:

  1. Prepares some variables that pre-build.sh needs inside the chroot(s)

  2. Builds the VERSION_INFO.txt file

  3. Updates the build number

  4. Imports the signing GPG key

conf

See conf.

7.9.6. postChroot(conf)

This function applies the overlay/ directory (and the overlay templates).

conf

See conf.

Netboot

What good is software if you can’t see it in action?

It’s possible to netboot my personal build of BDisk. I mostly keep this up for emergencies in case I need it, but it’s good to show you that yes, you can boot a 2GB+ squashed and compressed filesystem from a <50MB image file.

8. How to Netboot BDisk

I update this server with images and iPXE images you can use to netboot my personal spin of BDisk.

You can download a demo of the iPXE functionality. Note that your computer needs to be connected to a valid Internet connection via ethernet and be able to get a DHCP lease for it to work.

Note
Advanced users, you can verify it against the GPG signature (ASC, BIN). Please see this blog post for information on fetching my keys and such. Note that while this project is in flux, I may be signing with temporarily-generated throwaway keys.

Once downloaded, you can follow the appropriate steps based on your operating system:

8.1. Windows

8.1.1. CD/DVD

Simply put a blank CD/DVD-R (or RW, RW+, etc.) in your optical media drive. Find where you downloaded the above file (it should be named bdisk-mini.iso). Right-click and select Burn disc image.

8.1.2. USB

You’ll most likely want to download a program called USBWriter. Unzip it (or just open it via double-clicking) and copy the USBWriter.exe program somewhere you’ll remember- your desktop, for instance.

Next, make sure your USB thumbdrive is inserted in your computer and formatted/"initialized" already.

Warning
Formatting a disk/partition will destroy any and all data on that device! Make sure there is nothing on your USB drive you want to keep, as formatting BDisk to it will delete any data on it.

Now right-click on the USBWriter icon and select Run as administrator. You may get a warning pop up asking for permissions for USBWriter. It’s safe to click Yes.

Select the proper USB flash drive from the Target device dropdown menu. If your USB drive isn’t showing up, try clicking the Refresh button and looking again. (If it still doesn’t show up, you may need to reboot your computer.)

Click the Browse…​ button and find where you saved bdisk-mini.iso. Once you’ve found it, double-click it. Then click Write. It might take a little bit of time depending on how fast your USB interface is, so give it some time. When it finishes, click Close. You now have a bootable USB thumbdrive.

8.1.3. Booting

Booting differs depending on each and every hardware, but typically you should get a message when you first start up for "Setup" and/or "Boot options" or the like. The terminology differs here. It will probably be an F# button (usually F2, F4, F10, or F12) or the Delete key. While rebooting, try to hold or press repeatedly this key and you should come across an option somewhere with a list of devices to boot from or an order you can set. Make sure the USB (or CD/DVD, whichever media type you’re using) is set as first, and save.

8.2. Mac OS X/macOS

8.2.1. CD/DVD

Unfortunately, the OS X/macOS Disk Utility doesn’t work with hybrid ISOs (what bdisk-mini.iso is). At all. You’re out of luck, I’m afraid, unless you happen have a spare USB thumbdrive handy.

8.2.2. USB

We’ll need to get a little messy with this one.

Open Applications ⇒ Utilities ⇒ Terminal. A black box should pop up.

Insert your USB thumbdrive now (if you haven’t already) and run the following command:

diskutil list

You should see an entry, probably near the bottom, that looks something like this:

(...)
/dev/disk42 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                                                   *8.2 GB     disk42
(...)
Caution
Be sure to find the disk that matches the size of your thumbdrive! If you use the wrong disk identifier, it will break your OS X/macOS install at best and delete all your data at worst!

Now that you’ve found which disk your USB device is (the /dev/disk# part), we can continue. Make sure that it is the disk ID right above the line that contains your flash drive size! For our example, I will use /dev/disk42 as an example as it’s highly unlikely you’ll have that many disk IDs, but be sure to replace this in the following commands with the proper disk ID you found above.

Then we need to unmount the disk, in case it’s already mounted.

diskutil unmountDisk /dev/disk42

Assuming you saved BDisk Mini to your Desktop, you can do:

sudo dd if=~/Desktop/bdisk-mini.iso of=/dev/disk42
Note
The above command may prompt you for a password. This is the same password you use to log into your Mac (and unlock the screensaver, etc.). No characters will show up when you type (for security reasons, in case someone is behind you watching your screen) so it may take you a couple tries.

This will run for a couple seconds. When it finishes, you should see something similar to (but not necessarily the same numbers as) this:

0+1 records in
0+1 records out
169 bytes transferred in 0.000530 secs (318865 bytes/sec)

At this point you may get a popup warning you "The disk you inserted was not readable by this computer." If you do, just click the Ignore button.

One last step. Still in Terminal:

diskutil eject /dev/disk42

You can then close Terminal.

8.2.3. Booting

The instructions here don’t differ too much than from Windows, though it’s always the same key. From it being in a shutdown state, power on your Macbook Pro (or whatever it is you have) and hold the Option key (or the Alt key on non-Apple keyboards). The Option/Alt key should bring up a boot menu that will let you select a USB device to boot from.

Strangely enough, you should still be able to boot a BDisk Mini CD/DVD, you just can’t burn one. I’m tempted to make a cheap dig at Apple, but I’ll refrain.

8.3. GNU/Linux

8.3.1. CD/DVD

Easy. Most (if not all) of these should support burning bdisk-mini.iso to disc (I’m partial to cdrecord). If you prefer a GUI, try some of these instead (I like k3b).

8.3.2. USB

Very similar to OS X/macOS in approach. First open a terminal emulator- the ways of navigating to it depends on your window manager/desktop environment, but it’s usually under a System or Utilities menu.

Now we need to find which disk our USB thumbdrive is. Insert your USB thumbdrive now, if you haven’t already, and run in the terminal:

sudo fdisk -l

You should see a device matching your USB thumbdrive’s size. In our example, I use /dev/sdz as it’s unlikely you have that many disks attached to a system, but be sure to replace this in the following commands with the proper disk ID you find.

(...)
Disk /dev/sdz: 7.6 GiB, 8178892800 bytes, 15974400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
(...)
Caution
Be sure to find the disk that matches the size of your thumbdrive! If you use the wrong disk identifier, it will break your GNU/Linux install (or possibly Windows install if you’re dual-booting, etc.) at best and delete all your data at worst!

Make sure it isn’t mounted:

umount /dev/sdz

You should get a message that says umount: /dev/sdz: not mounted. If it was mounted before, it’s unmounted now.

Next, simply dd over the ISO file.

sudo dd if=~/Desktop/bdisk-mini.iso of=/dev/sdz
Note
The above command may prompt you for a password. This is the same password you use to log in (and unlock the screensaver, etc.). No characters will show up when you type (for security reasons, in case someone is behind you watching your screen) so it may take you a couple tries.

This will run for a couple seconds. When it finishes, you should see something similar to (but not necessarily the same numbers as) this:

75776+0 records in
75776+0 records out
38797312 bytes (39 MB, 37 MiB) copied, 9.01915 s, 4.3 MB/s

If you get a popup from your desktop environment (assuming you’re using one) about not being able to mount a disk, or that it’s unformatted, etc. and it prompts you to format, ignore/cancel/close it- do not format it! This would erase the BDisk Mini image on it.

8.3.3. Booting

Exactly the same as those for Windows. (Unless you’re running GNU/Linux on Mac hardware, in which case follow the booting instructions for Mac instead.)

Further Reading/Resources

What good is software if you can’t interact?

Here you will find further info, other resources, and such relating to BDisk.

9. Passwords

Note
If you’re specifying passwords, be sure to use a strong password!

9.1. build.ini Password Value Examples

Passwords work a little interestingly in BDisk. These aspects all apply to both the root password and the user password (if you enable a regular user).

Caution
DO NOT USE A PLAINTEXT PASSWORD IN THE build.ini! This is by design; plaintext passwords are much more insecure. If you use a plaintext password, it will not work.
Warning
Remember to escape your hash before placing it in your build.ini!
Table 1. Password Value Scheme
If you have…​ BDisk will…​

the string BLANK

give the user a blank password, allowing you to just hit <Enter> to log in

nothing set

lock the account (e.g. no non-SSH login is possible)

a properly hashed, salted, and escaped string

set the account to the password used to generate that hash.

Table 2. Password Value Examples
If the value is…​ Then BDisk…​

root_password = BLANK

will let you log into the TTY as the root user by just hitting the <Enter> key.

root_password =

will not allow the root user to log into the TTY at all.

root_password = <some salted, hashed, escaped string created from 'test'>

will let you log into the root user on a TTY with the password test.

Note
I specify "TTY login" because SSH login may still be possible. By default, SSH will allow password logins for non-root users (root user SSH password login is prohibited by default; only pubkey login for root is allowed.) — this can be overridden, however, by customization.

9.2. Generating a Password Salt/Hash

First, if you are not familiar with a salted hash that GNU/Linux uses, you may want to learn about it.

That said, there are utilities in extra/bin/ that should generate a salted hash for you. Currently only hashgen.py is distributed, but additions/examples for other languages are welcome.

 $ ./hashgen.py

 What password would you like to hash/salt?
 (NOTE: will NOT echo back!)


 Your salted hash is:
 	$6$t92Uvm1ETLocDb1D$BvI0Sa6CSXxzIKBinIaJHb1gLJWheoXp7WzdideAJN46aChFu3hKg07QaIJNk4dfIJ2ry3tEfo3FRvstKWasg/

The password test was used above. In crypt(3)-salted hashes, there are specific sections separated by USD dollar symbols ($). The first section (containing 6) marks the hash algorithm — in this case, SHA512. (The crypt man page mentions all supported hash types and their corresponding ID.) The next section, t92Uvm1ETLocDb1D, is the salt. The last section is the hash. How salted hashes work is an original piece of data is given (in our case, the word test). This data is then sent through a one-way cryptographic process that generates a new string that makes it difficult to know what the original data was. THEN a salt is added- a random string- and the process repeats. In our format, this is done 5000 times in a row. When you log in with your password, the salt is fetched and the same process is done again- predictably, the data that process goes through should then match the salted hash string stored in the password system (in this case, the /etc/shadow file).

There are other ways to generate the salted hash as well. These include:

9.2.1. Debian’s mkpasswd Utility

Part of the whois package, available in the AUR as debian-whois-mkpasswd.

mkpasswd --method=sha-512 <password>

9.2.2. Perl

The following Perl one-liner will generate a salted hash string (using the salt aBcDeFgHiJ):

perl -e 'print crypt("PASSWORD","\$6\$aBcDeFgHiJ\$") . "\n"'

9.2.3. grub-crypt

Legacy GRUB ("GRUB v1") includes grub-crypt, which will let you generate a salted hash:

/sbin/grub-crypt --sha-512

9.3. Escaping the Salted Hash

One last thing, and this is very important — failure to perform this step will cause all sorts of strange Python errors — is to escape the salted hash. Thankfully, however, this is a lot easier than it sounds.

So we have our salted hash: $6$t92Uvm1ETLocDb1D$BvI0Sa6CSXxzIKBinIaJHb1gLJWheoXp7WzdideAJN46aChFu3hKg07QaIJNk4dfIJ2ry3tEfo3FRvstKWasg/. In order to get it into a usable format, we need to make sure the configuration parsing won’t try to read sections of it as variables. To do this, we do something called escaping.

All you need to do is take the salted hash and replace every $ you see — there should be exactly three — with $$. That’s it! Count them to be sure; you should now have 6 $ symbols present instead of three. Once you’ve escaped the salted hash, you’re ready to roll.

9.4. Cheating/The Easy Way

Feeling overwhelmed? There’s an easy way to do all of this.

First, while logged into your local computer, change your password to what you want ether root_password or password to be:

passwd
Note
Remember, changing your password won’t echo the password back on the screen for security reasons!

Then get your shadow entry. This has to be done with sudo, as only the root user has access to the hashed passwords on the system. The following command will combine all steps necessary; the string it returns will be a string you can use directly in your build.ini.

sudo grep "^${SUDO_USER}:" /etc/shadow | awk -F':' '{print $2}' | sed -e 's/\$/$$/'

Don’t forget to change your password back to what it was before!

passwd

That’s it!

10. Bug Reports/Feature Requests

Note
It is possible to submit a bug or feature request without registering in my bugtracker. One of my pet peeves is needing to create an account/register on a bugtracker simply to report a bug! The following links only require an email address to file a bug (which is necessary in case I need any further clarification from you or to keep you updated on the status of the bug/feature request — so please be sure to use a valid email address).

10.1. Bugs

If you encounter any bugs in BDisk, you can file a bug report here.

If you encounter any bugs (inaccurate information, typos, misformatting, etc.) in this documentation, you can file a bug report here.

10.2. Feature Requests

If you have any features you’d like to see or you think would help BDisk become even more useful, please file a feature request here.

If you have any suggestions on how to improve this documentation or feel it’s missing information that could be useful, please file a feature request here.

10.3. Patches

I gladly welcome patches, but I deplore using GitHub (even though I have a mirror there). For this reason, please follow the same patch/pull request process for the Linux kernel and email it to bts@square-r00t.net.

Alternatively, you may attach a patch to a bug report/feature request.

11. Contact the Author

If you have any questions, comments, or concerns, you can use the following information to get in touch with me.

I am available via email. If you use GPG, you can find my pubkey and other related info here (and on most keyservers).

I occasionally write howto articles, brief tips, and other information in my dev blog.

I am on IRC as r00t^2, and am usually in the Sysadministrivia channel on Freenode. Which reminds me, I run a podcast called Sysadministrivia.

I am on Twitter as @brentsaner, though I don’t tweet very often. (I usually tweet from my podcast’s twitter.)

FAQ

What good is software if nobody understands it?

Here you will find some answers to Frequently Asked Questions I’ve received about this project. Please be sure to check this list before opening a bug report or sending a patch!

12. Why Arch Linux?

Because it’s a very easy-to-use, simple, well-documented distro. It’s no-frills and incredibly flexible/customizable, and can be made rather slim (and is out of the box, in fact). It’s also very friendly to run as a chroot inside any other distro or as a chroot host to any other distro.

Plus they release monthly tarball snapshots that are fairly small and create quick bootstrap environments.

13. Why does building take so long?

This typically occurs when you’re building from within a LiveCD/LiveUSB situation, in a VM/container/etc., or on a headless server. If this is the case, you may run into what appears to be "stalling", especially while keys are generating for the chroots. Thankfully, there is an easy fix. You can install haveged and run it (this can be done safely while a build is executing). This will show an immediate and non-negligible improvement for the above contexts. If you have extra processing power to throw at the build process (or are using a dedicated build box) as well, I recommend enabling i_am_a_racecar. BDisk will then be more aggressive with its resource consumption.

13.1. Running a local mirror

Keep in mind also that the more packages you opt to install, the longer the build process will take. This process will also use quite a fair bit of bandwidth. If you plan on building regular images (e.g. nightly builds, etc.) or are undergoing some custom change testing, I recommend running a private repository mirror on-site. This will not store AUR packages, as those will still be fetched and built (documentation on working around this is TODO) but setting up a local mirror is quite quick and easy.

First, you’ll need at least 70Gb of free disk space. Let’s say our repository clone will be at /srv/repo/arch/.

You’ll also need to find an Arch mirror, ideally one close to you that is up-to-date. The mirrorlist generator and mirror list will assist you here greatly.

Note
You’ll need to find a mirror that supports rsync.
Tip
You can use ANY distro to run a repository mirror, as long as it has rsync installed!

13.1.1. Set up the sync

Create a script and mark it as executable with the following content:

#!/bin/bash
SOURCE='rsync://your.mirror.here/archlinux'
DEST='/srv/repo/arch'
LCK_FLE='/var/run/repo-sync.lck'
PATH='/usr/bin'
if [ -e "${LCK_FLE}" ] ; then
	OTHER_PID=$(cat ${LCK_FLE})
	echo "Another instance already running: ${OTHER_PID}"
	exit 1
fi
# If e.g. /srv/repo is a mountpoint, uncomment below.
#findmnt /srv/repo > /dev/null 2>&1
#if [[ "${?}" -ne '0' ]];
#then
#	echo "External storage not mounted!"
#	exit 1
#fi
echo $$ > "${LCK_FLE}"
rsync -rvtlH --delete-after --delay-updates --safe-links --max-delete=1000 ${SOURCE}/. ${DEST}/. >> /var/log/arch.repo.sync 2>&1
date +%s > ${DEST}/lastsync
rm -f "${LCK_FLE}"

Assuming you want to run the sync script every 6 hours and it is located at /root/bin/arch.repo.clone.sh, this is the cron entry you would use (crontab -e):

0    */6  *   *   * /root/bin/arch.repo.clone.sh > /dev/null 2>&1

The first sync can take quite a while, but subsequent runs shouldn’t take more than five minutes or so.

13.1.2. Configuring the local mirror

You’ll need a way to serve this local mirror in a way pacman can understand. Luckily, it’s fairly easy. I recommend using nginx as it’s available by default in many operating systems. You can of course use others such as lighttpd, apache/httpd, etc. For the example configuration here, we’re going to use an nginx configuration file.

server {
listen [::]:80;
access_log /var/log/nginx/repo.access.log main;
error_log /var/log/nginx/repo.error.log;
#error_log /var/log/nginx/repo.error.log debug;

autoindex on;

root /srv/repo/arch;

}

The configuration may vary according to your distribution’s provided nginx default configuration, but you’ll want this configuration to be served as the default (or set an appropriate server_name directive which you would then use in <basedir>/extra/pre-build.d/etc/pacman.d/mirrorlist).

13.1.3. Configuring BDisk

You’ll then want to configure BDisk’s chroots to use your local mirror first. However, when doing so you run into an issue — in the built image, install operations will take longer than they need to because the local mirror likely won’t be available! This is a small issue as it’s unexpected that you’ll need to install software within the live environment, but I’ve run into cases where it was a necessity once or twice.

There is an easy workaround if you’re using libvirt — you can simply tell your build VM to resolve to the IP address of the box that is running the mirror for the same FQDN that the "preferred" "real" mirror on the Internet is and set that mirror at the top of <basedir>/extra/pre-build.d/etc/pacman.d/mirrorlist. However, that’s not always feasible- most notably if you’re building on a physical box and it’s the same host as the repository clone. In that case you can set the specific local resolution — e.g. http://127.0.0.1/ — at the top of <basedir>/extra/pre-build.d/etc/pacman.d/mirrorlist and then set a mirrorlist WITHOUT that entry in <basedir>/overlay/etc/pacman.d/mirrorlist. For more information on using these type of overrides, see Advanced Customization.

If you’re using the libvirt workaround, remember to configure nginx (or whatever you’re using) with a virtual host and location block that matches the "real", upstream mirror. In our example below, we use http://mirror.us.leaseweb.net/archlinux as the mirror.

server {
listen [::]:80;
access_log /var/log/nginx/repo.access.log main;
error_log /var/log/nginx/repo.error.log;
#error_log /var/log/nginx/repo.error.log debug;

server_name mirror.us.leaseweb.net;

autoindex on;

root /srv/repo/arch;

location /archlinux {
	autoindex on;
	rewrite ^/archlinux(/.*)$ /$1;
	}

}

14. Why is the ISO so large?

This actually entirely depends on what packages you have chosen to install (and if you’re building a multiarch ISO). The default list is quite large.

If you build a minimal ISO (i.e. only the necessary components required for booting and nothing else, single-arch), the ISO is only about 570MB (but work is being done to make this even smaller).

15. How do I get the version/build of an ISO?

This can be found in a multitude of places. The full-size ISO file (iso/<distname>-<git tag>-<git rev number>-(32|64|any).iso) should have the version right in the filename. If you want more detailed information (or perhaps you renamed the file), you can mount the ISO as loopback in GNU/Linux, *BSD, or Mac OS X/macOS and check /path/to/mounted/iso/VERSION_INTO.txt. Lastly, within the runtime itself (especially handy if booting via iPXE), you can check /root/VERSION_INFO.txt to get information about the build of the currently running live environment.

16. I don’t like BDisk. Are there any other alternatives?

First, I’m sorry to hear that BDisk doesn’t suit your needs. If you want any features you think are missing or encounter any bugs, please report them!

But yes; there are plenty of alternatives!

Note
Only currently maintained projects are listed here.

16.1. Archboot

Written in Bash

Pros Cons

Highly featureful

Arch-only live media

Includes an assisted Arch install script

Inaccessible to non-Arch users

Can create tarballs too

Not very customizable by default

Supports hybrid ISOs

Infrequent stable releases

Supports PXE-booting infrastructure

Requires a systemd build host

Supports SecureBoot

Not a secure setup by default

Supports GRUB2’s "ISO-loopback" mode

Builds a much larger image

Official Arch project

Some graphical bugs

Much more disk space is necessary for the tool itself

Only runs in RAM, so not ideal for RAM-constrained systems

Based on/requires an Arch build host

Requires an x86_64 build host

Has a large amount of dependencies

Manual intervention required for build process

Minimal documentation

16.2. Archiso

Written in Bash.

Pros Cons

Used to build the official Arch ISO

Requires an x86_64 build host

Supports custom local on-disk repositories

Not very featureful as far as customization goes

Supports arbitrary file placement in finished image

Requires an Arch build host

Supports hybrid ISOs

Has odd quirks with package selection

Supports Secureboot

Manual intervention required for build process

Official Arch project

Does not start networking by default

Can run in RAM or from media

Very minimal environment

Arch-only live meda

Documentation is lacking

16.3. Debian’s Simple-CDD

Written in Bash (some Python).

Pros Cons

Supports custom packages to be installed

Very limited — no customization beyond package listing

Lightweight; quick to set up

Takes a long time for preparation; requires a clone of many .deb packages first.

Doesn’t seem to work as according to the documentation

Documentation is sparse

Full featureset unknown due to ISO not building on Debian Jessie (8.0)

16.4. Fedora’s Livemedia-creator

Written in Bash.

Pros Cons

Somewhat customizable

Requires manual initialization of chroot(s) via mock

Uses kickstart configurations

Requires a kickstart configuration to be useful

Simple/easy to use

Full featureset unknown; documentation is sparse

Limited configuration/customization

16.5. LiveCD Tools

Written in Python 2, some Bash.

Pros Cons

Can use kickstarts

Requires a kickstart configuration

Simple/easy to use to use

Limited configuration/customization

Automatically builds chroots

Full featureset unknown; documentation is sparse


1. See http://catb.org/jargon/html/Y/yak-shaving.html
2. Unified Extensible Firmware Interface. UEFI is not BIOS, and BIOS is not UEFI.
3. I should take the time to note that I am still quite new to Python so expect there to be plenty of optimizations to be made and general WTF-ery from seasoned Python developers. If you encounter any bugs or improvements, please report them! It’d be much appreciated.