Documentation Overview

  1. Operation documentation
    1. UBOS Linux installation
      1. Run UBOS on a PC (64bit)
      2. Run UBOS from a boot stick on a PC (64bit)
      3. Run UBOS in a VirtualBox virtual machine (64bit)
      4. Run UBOS with Docker
      5. Run UBOS on an Amazon Web Services EC2 virtual server
      6. Run UBOS on Raspberry Pi 5
      7. Run UBOS on ESPRESSObin
      8. Run UBOS in a Linux container on a PC (64bit)
      9. Run UBOS in an AArch64 Linux container
    2. Setting up your first Site and App
    3. Setting up networking and DNS
    4. How to create a website secured by SSL/TLS
    5. How to set up a website as a Tor hidden service
    6. Managing Sites and Apps
    7. Backup and restore
    8. Upgrading and keeping your Device current
    9. The UBOS Staff
    10. App-specific notes
      1. Reliably send e-mail via Amazon Web Services’ Simple E-mail Service: amazonses
      2. Static website hosting with rsync-based upload: docroot
      3. Notes on Mastodon
      4. Notes on Nextcloud
      5. Notes on Redirect
      6. Notes on Wordpress
    11. Device-specific Notes
      1. ESPRESSObin
      2. Raspberry Pi
    12. Advanced management
      1. Enabling non-standard package repositories
      2. Migrating from one App to another
      3. Pinning resources
    13. FAQ, HOWTOs and Troubleshooting
      1. “Package not found error” when installing a new App or Accessory
      2. A UBOS container comes up degraded
      3. Booting UBOS on a PC starts out fine, but then the screen goes blank
      4. Can I use UBOS without purchasing a domain name?
      5. Cannot access MySQL database. File missing: /etc/mysql/root-defaults-ubos.cnf
      6. Cannot boot UBOS from boot stick on a PC
      7. Cannot connect to the public internet from a UBOS container
      8. Cannot create a temporary backup; the backup directory is not empty
      9. Failed to create file /sys/devices/system/cpu/microcode/reload
      10. How are the various UBOS images different from each other?
      11. How can I install more than one web App on the same Device?
      12. How do I report a bug?
      13. How do I set up WiFi?
      14. How to enable non-standard Package Repositories
      15. How to get help
      16. How to log into your UBOS Device
      17. How to modify the configuration of your Site
      18. How to report a bug or issue
      19. How to use Pagekite to reach your UBOS Device behind a firewall
      20. How to use SSH
      21. I need a Package that isn’t in UBOS
      22. I need root
      23. I own a domain name, and I’d like to use it for my UBOS Device. How do I do that?
      24. I want to move from one device to another, or from/to the cloud to/from a device
      25. I want to run ssh on a non-standard port
      26. I’m running out of disk space, what now?
      27. Installing a new Package or updating fails with a message about “invalid or corrupted package” or “key is disabled”
      28. Installing a new Package or upgrading fails with a message about “unknown trust”
      29. Is it safe to have my Site accessible from the public web?
      30. My non-English keyboard layout is all screwed up
      31. My SD card is much larger than the UBOS image. How do I use the rest of the space?
      32. Nothing happens when UBOS is supposed to be booting
      33. Problems with “IPv6 Packet Filtering Framework”
      34. UBOS is in a “degraded” state
      35. ubos-admin status reports “Systemd unit … has failed”
      36. Verify your downloaded UBOS image
      37. What is the default “root” password?
      38. What text editor can I use on UBOS?
      39. Why did you derive UBOS Linux from Arch Linux, and what is the relationship between UBOS Linux and Arch?
      40. Why does UBOS ask for a domain name when installing a new Site?
      41. Why is it called UBOS?
      42. Writing a disk image to a USB stick or SD card
        1. Writing an image to a USB stick or SD card on Linux
        2. Writing an image to a USB stick or SD card on macOS
        3. Writing an image to a USB stick or SD card on Windows
    14. Command reference
  2. Developer documentation
    1. Developer setup
      1. Developing using Docker (all Intel platforms)
      2. Developing using a systemd-nspawn container (Linux host only)
      3. Developing using Arch Linux on VirtualBox x86_64 with a systemd-nspawn container
      4. Developing using Arch Linux using UTM on Apple Silicon with a systemd-nspawn container
      5. Developing using Arch Linux using Parallels on Apple Silicon with a systemd-nspawn container
    2. Developer tutorials for standalone UBOS Gears apps (not UBOS Mesh)
      1. Build and run your first UBOS Gears App
      2. How to package UBOS Standalone Apps built with a variety of languages
        1. Hello World
        2. Glad-I-Was-Here (PHP, MySQL)
        3. An Accessory for Glad-I-Was-Here (PHP, MySQL)
        4. Glad-I-Was-Here (PHP, Postgresql)
        5. Glad-I-Was-Here (Java, MySQL)
        6. Glad-I-Was-Here (Python, MySQL)
    3. UBOS Gears Reference
      1. UBOS Manifest
        1. Structure of the UBOS Manifest
        2. Info section
        3. Roles section
        4. Customization points section
        5. Appinfo section
        6. Accessoryinfo section
        7. Variables available at deploy or undeploy
        8. Functions that may be applied to variables
        9. Creating random values
        10. Scripts in UBOS Manifests
      2. Site JSON
      3. A complex deployment example
      4. UBOS Networking
      5. Allocating and opening up non-default ports
      6. Logging
      7. UBOS state
      8. UBOS Backup format
      9. Format of the App Status JSON
      10. Testing standalone Apps with “webapptest”
      11. Understanding ubos-admin
        1. Command: ubos-admin backup
        2. Command: ubos-admin backupinfo
        3. Command: ubos-admin createsite
        4. Command: ubos-admin deploy
        5. Command: ubos-admin hostid
        6. Command: ubos-admin init-staff
        7. Command: ubos-admin list-data-transfer-protocols
        8. Command: ubos-admin listnetconfigs
        9. Command: ubos-admin listsites
        10. Command: ubos-admin read-configuration-from-staff
        11. Command: ubos-admin restore
        12. Command: ubos-admin setnetconfig
        13. Command: ubos-admin setup-shepherd
        14. Command: ubos-admin showappconfig
        15. Command: ubos-admin shownetconfig
        16. Command: ubos-admin showsite
        17. Command: ubos-admin status
        18. Command: ubos-admin undeploy
        19. Command: ubos-admin update
        20. Command: ubos-admin write-configuration-to-staff
    4. Release channels and UBOS release process
    5. Miscellaneous
      1. Potentially useful infrastructure for standalone Apps
        1. The UBOS rsync server
      2. Middleware-specific notes
        1. Node.js notes
        2. SMTP notes
      3. Setting up an Arch Linux system
        1. Prepare a PC for installing Arch Linux
        2. Prepare a VirtualBox virtual machine to develop for UBOS using Arch Linux
        3. Continuing the Arch Linux installation on a PC or virtual machine
        4. Finishing the Arch development installation by adding UBOS tools
    6. Developer FAQ
      1. Doesn’t apt / dpkg / yum / pacman etc. does what UBOS Gears does already?
      2. Doesn’t puppet / chef / ansible etc. does what UBOS Gears does already?
      3. Doesn’t Docker do what UBOS Gears does already?
      4. How to profile the UBOS Personal Data Mesh web application
      5. Is it possible to run the other UBOS components on an operating system other than UBOS Linux?
      6. How to create a UBOS development VM for VirtualBox
      7. How to create a UBOS development VM for UTM on Apple computers
      8. How to create a UBOS development VM for Parallels Desktop on Apple Silicon

The UBOS Staff

/docs/operation/shepherd-staff/

Overview

The UBOS Staff allows the secure installation and configuration of Devices without ever having to attach a keyboard or monitor to the Device. This is particularly convenient for physical hardware that is “hidden away” somewhere, and where attaching a monitor and keyboard would be cumbersome, like inside an IoT device.

Among other things, the UBOS Staff can be used:

  • to provision an administrator account on the device called the Shepherd account, into which the user can ssh over the network. Either existing ssh credentials provided by the user can be used, or UBOS can generate a new key pair automatically.

  • to connect to a WiFi access point with credentials provided by the user and saved to the UBOS Staff;

  • to automatically install and configure a Site upon boot;

The UBOS Staff is simply a standard USB flash drive whose name and directory layout follows certain conventions. A similar mechanism is available for virtualized Devices.

See also this blog post with a longer discussion.

UBOS Staff device format and name

Any standard USB flash drive can be used as the UBOS Staff. It is recommended that such a USB flash drive only be used as UBOS Staff, and not for other purposes at the same time.

The drive must have a “VFAT” (“Windows”) partition called UBOS-STAFF – otherwise UBOS will not recognize the device as a UBOS Staff. This can often be accomplished simply by inserting a new USB flash drive in a computer, and renaming the drive to UBOS-STAFF.

The USB flash drive can have any size; the amount of storage required for use as UBOS Staff is minimal. Speed is also largely irrelevant.

The UBOS Staffuses the following directory layout. For details, see below:

Directory File Description
shepherd/ssh/ id_rsa.pub SSH public key for the shepherd account.
id_rsa SSH private key for the shepherd account. You can delete it here once you have copied it to your workstation from where you will be logging into your Device.
wifi/ <ssid>.conf This directory may contain several files, one for each to-be-configured WiFi SSID. Each file must be named based on the SSID it configures, with extension .conf.
site-templates/ <name>.json This directory may contain several Site JSON Template files. Each of them will be instantiated and deployed to any Device that reads this UBOS Staff unless a Site with the same hostname exists already on the Device, or reading Site JSON Template files has been disabled on the Device.
flock/<HOSTID>/bootlog/ %lt;timestamp%gt.txt Text files containing the Device's boot log from the systemd journal. This may help with troubleshooting issues on Devices that don't have an attached monitor.
flock/<HOSTID>/device-info/ device.json A JSON file containing information about the Device.
flock/<HOSTID>/site-templates/ <name>.json This directory may contain one or more Site JSON files or Site JSON Template files. Each of them will be instantiated (if a template) and deployed to the Device with HostId <HOSTID>, but ignored on other Devices.
flock/<HOSTID>/sites/ <name>.json This directory may contain several Site JSON files, which represent the Site or Sites as they were deployed on the Device with HostId <HOSTID> at the time the UBOS Staff was written last.
flock/<HOSTID>/ssh/ ssh_host_<type>_key This directory may contain one or more of the Device's SSH host keys of different types, such as rsa or ecdsa. This can be used, in addition to the shepherd SSH key pair, to authenticate the host (not just the client) over the network.
root of UBOS Staff I-ADMINISTER-MY-UBOSBOX-MYSELF.txt If package ubos-live is installed, a UBOS Staff is present at boot and a file with this name is NOT present, the Device will register itself with Indie Computing's [UBOS Live](https://indiecomputing.com/products/ubos-live/) management service. To prevent this, do not install package ubos-live or create a file (content is irrelevant) with this name on the UBOS Staff.

<HOSTID> refers to the device’s unique HostId, which can be printed with ubos-admin hostid.

Note

All files and directories are optional and may not be present on a given UBOS Staff: an empty drive called UBOS-STAFF is entirely valid.

Virtual UBOS Staff devices

In the cloud

When booting an UBOS image on Amazon EC2, UBOS instead will take the key pair specified by the user in the instance creation wizard on the Amazon website, and configure the Shepherd account with it. No actual Staff device is required.

None of the other UBOS Staff functionality is available in the cloud.

In a Linux container

When booting UBOS in a Linux container, UBOS will treat the directory /UBOS-STAFF as the UBOS Staff, assuming it is present in the container (not the host).

It may be advantageous to bind a suitable directory into the container with the --bind flag to systemd-nspawn.

UBOS will never auto-generate a new key pair when running UBOS in a container.

Provisioning a Shepherd account

An automatically provisioned Shepherd account can be used as the primary administration account on a UBOS device. By default, it has the rights to invoke sudo ubos-admin, sudo systemctl and the like. It can also become root (see I need root).

If a Device is booted a second time with the UBOS Staff present, the ssh key will be updated. (We work under the assumption that if an attacker has the ability to physically insert a USB device into the USB port and reboot the Device, the Device should be considered compromised in any case.)

Provision a Shepherd account with an existing ssh public key

If you would like to use an existing ssh public key to log into your Device over the network as user shepherd, create the following file system layout:

shepherd/
    ssh/
        id_rsa.pub

where the file id_rsa.pub contains a valid ssh public key. You can use any existing ssh public key for which you have the corresponding private key.

I.e., the file called id_rsa.pub must be contained in a directory named ssh, which in turn must be contained in a directory called shepherd at the root level of the directory hierarchy of the UBOS Staff.

Provision a Shepherd account with a newly generated ssh key pair

If you don’t have an ssh key pair yet, and would like UBOS to generate one for you, simply use a UBOS Staff that is empty or at least does not have the shepherd directory yet at the root of the UBOS Staff.

During boot, UBOS will automatically generate the key pair, save it to the UBOS Staff, and create the shepherd account on the Device. (This behavior only occurs with a physical UBOS Staff; not with a virtual UBOS Staff in case of running UBOS in the cloud or in a Linux container.)

Once UBOS has booted and generated the ssh keys, you can unplug the UBOS Staff and insert it into the computer from which you want to log into your Device. Copy the file shepherd/ssh/id_rsa from the UBOS Staff into a secure place on your computer, as anybody who has access to this file can use it to log into your Device. Also, delete the id_rsa file from the UBOS Staff for the same reason. (The file id_rsa.pub is the public key which can be shared without harm.)

To log into a remote UBOS Device as the Shepherd

On the computer that has the private id_rsa file, execute the following command:

% ssh -i <id_rsa> shepherd@1.2.3.4

where <id_rsa> is the name of the file containing the private key from above, and 1.2.3.4 is replaced with the IP address or hostname of your Device, such as ubos-pc.local (see Setting up networking and DNS).

If you had UBOS generate the key pair, copy the private key file id_rsa to your computer first: ssh will not let you use the id_rsa directly from the UBOS Staff.

If you use a Windows workstation and PuTTY as your ssh client, you need to first convert the id_rsa file into the “PuTTY Private Key Files (.ppk)” format by running puttygen.exe. Then, use the converted file as the authentication parameter with the PuTTY-Client.

To setup WiFi

If you would like your Device to be able to connect to WiFi immediately after it boots, you can save information about one or more WiFi networks to the UBOS Staff, and UBOS will configure your Device as a WiFi client when it boots. Of course, this assumes that your Device has WiFi support and all relevant drivers have been installed (if not, this will do nothing).

To provide information on a WiFi network called ExampleWiFi, create file wifi/ExampleWiFi.conf with the following content:

ssid="ExampleWiFi"
psk="MySecret"

ssid must be the WiFi network’s SSID (here: ExampleWiFi) and psk must be the corresponding WiFi passphrase.

You can specify more than one file in directory wifi/, and your Device will be able to connect to any of those networks. If your network needs more configuration, you can add additional settings accepted by wpa_supplicant into these files: UBOS simply inserts the content of those files into the network={ ... } section of a generated wpa_supplicant.conf file, and so you can add any settings there acceptable to wpa_supplicant.

You should also create a file in directory wifi/ called wireless-regdom. Allowed WiFi frequencies are different in different countries, and this allows you to conform to radio emission regulations in your country. This file should contain a single line that, if you are based in the United States, looks like this:

WIRELESS_REGDOM="US"

If you are based in another country, use your two-letter country code instead of US.

To auto-deploy Sites upon boot

If you place one or more Site JSON files, or Site JSON Template files in the correct place on the UBOS Staff, UBOS will automatically deploy those Sites during boot. There are two places where those files may be located:

  • If placed in top-level directory site-templates/, any Device booting with the UBOS Staff will deploy the corresponding Sites. This functionality is useful if you’d like to deploy the same kind of Site running the same Apps and Accessories to several Devices.

    It is highly recommended that the files be Site JSON Template files (rather that Site JSON files) that do not contain SiteIds or AppConfigIds, in order to generate unique identifiers for Sites and AppConfigurations on different Devices.

  • If placed in directory flock/<HOSTID>/site-templates/, where <HOSTID> is the host identifier of a particular Device as determined by ubos-admin hostid, UBOS will only deploy the Sites on that Device.

Note

Sites or Site JSON Templates will not be deployed if the Device already as a Site with either the same hostname or the same SiteId or AppConfigId.

The Site JSON files of the Sites deployed through this mechanism will, once the Site has been deployed, stored in flock/<HOSTID>/sites/<SITEID>.json. This gives the user a way of knowing automatically-generated credentials, for example.

Disabling UBOS Staff functionality

To disable reading the UBOS Staff on boot at all, change the setting host.readstaffonboot to false in /etc/ubos/config.json.

To disable modifying the UBOS Staff on boot, such as by generating a new SSH keypair, change the setting host.initializestaffonboot to false in /etc/ubos/config.json.