Documentation Overview

  1. Operation documentation
    1. UBOS Linux installation
      1. Run UBOS on a PC (64bit)
      2. Run UBOS from a boot stick on a PC (64bit)
      3. Run UBOS in a VirtualBox virtual machine (64bit)
      4. Run UBOS with Docker
      5. Run UBOS on an Amazon Web Services EC2 virtual server
      6. Run UBOS on Raspberry Pi 5
      7. Run UBOS on ESPRESSObin
      8. Run UBOS in a Linux container on a PC (64bit)
      9. Run UBOS in an AArch64 Linux container
    2. Setting up your first Site and App
    3. Setting up networking and DNS
    4. How to create a website secured by SSL/TLS
    5. How to set up a website as a Tor hidden service
    6. Managing Sites and Apps
    7. Backup and restore
    8. Upgrading and keeping your Device current
    9. The UBOS Staff
    10. App-specific notes
      1. Reliably send e-mail via Amazon Web Services’ Simple E-mail Service: amazonses
      2. Static website hosting with rsync-based upload: docroot
      3. Notes on Mastodon
      4. Notes on Nextcloud
      5. Notes on Redirect
      6. Notes on Wordpress
    11. Device-specific Notes
      1. ESPRESSObin
      2. Raspberry Pi
    12. Advanced management
      1. Enabling non-standard package repositories
      2. Migrating from one App to another
      3. Pinning resources
    13. FAQ, HOWTOs and Troubleshooting
      1. “Package not found error” when installing a new App or Accessory
      2. A UBOS container comes up degraded
      3. Booting UBOS on a PC starts out fine, but then the screen goes blank
      4. Can I use UBOS without purchasing a domain name?
      5. Cannot access MySQL database. File missing: /etc/mysql/root-defaults-ubos.cnf
      6. Cannot boot UBOS from boot stick on a PC
      7. Cannot connect to the public internet from a UBOS container
      8. Cannot create a temporary backup; the backup directory is not empty
      9. Failed to create file /sys/devices/system/cpu/microcode/reload
      10. How are the various UBOS images different from each other?
      11. How can I install more than one web App on the same Device?
      12. How do I report a bug?
      13. How do I set up WiFi?
      14. How to enable non-standard Package Repositories
      15. How to get help
      16. How to log into your UBOS Device
      17. How to modify the configuration of your Site
      18. How to report a bug or issue
      19. How to use Pagekite to reach your UBOS Device behind a firewall
      20. How to use SSH
      21. I need a Package that isn’t in UBOS
      22. I need root
      23. I own a domain name, and I’d like to use it for my UBOS Device. How do I do that?
      24. I want to move from one device to another, or from/to the cloud to/from a device
      25. I want to run ssh on a non-standard port
      26. I’m running out of disk space, what now?
      27. Installing a new Package or updating fails with a message about “invalid or corrupted package” or “key is disabled”
      28. Installing a new Package or upgrading fails with a message about “unknown trust”
      29. Is it safe to have my Site accessible from the public web?
      30. My non-English keyboard layout is all screwed up
      31. My SD card is much larger than the UBOS image. How do I use the rest of the space?
      32. Nothing happens when UBOS is supposed to be booting
      33. Problems with “IPv6 Packet Filtering Framework”
      34. UBOS is in a “degraded” state
      35. ubos-admin status reports “Systemd unit … has failed”
      36. Verify your downloaded UBOS image
      37. What is the default “root” password?
      38. What text editor can I use on UBOS?
      39. Why did you derive UBOS Linux from Arch Linux, and what is the relationship between UBOS Linux and Arch?
      40. Why does UBOS ask for a domain name when installing a new Site?
      41. Why is it called UBOS?
      42. Writing a disk image to a USB stick or SD card
        1. Writing an image to a USB stick or SD card on Linux
        2. Writing an image to a USB stick or SD card on macOS
        3. Writing an image to a USB stick or SD card on Windows
    14. Command reference
  2. Developer documentation
    1. Developer setup
      1. Developing using Docker (all Intel platforms)
      2. Developing using a systemd-nspawn container (Linux host only)
      3. Developing using Arch Linux on VirtualBox x86_64 with a systemd-nspawn container
      4. Developing using Arch Linux using UTM on Apple Silicon with a systemd-nspawn container
      5. Developing using Arch Linux using Parallels on Apple Silicon with a systemd-nspawn container
    2. Developer tutorials for standalone UBOS Gears apps (not UBOS Mesh)
      1. Build and run your first UBOS Gears App
      2. How to package UBOS Standalone Apps built with a variety of languages
        1. Hello World
        2. Glad-I-Was-Here (PHP, MySQL)
        3. An Accessory for Glad-I-Was-Here (PHP, MySQL)
        4. Glad-I-Was-Here (PHP, Postgresql)
        5. Glad-I-Was-Here (Java, MySQL)
        6. Glad-I-Was-Here (Python, MySQL)
    3. UBOS Gears Reference
      1. UBOS Manifest
        1. Structure of the UBOS Manifest
        2. Info section
        3. Roles section
        4. Customization points section
        5. Appinfo section
        6. Accessoryinfo section
        7. Variables available at deploy or undeploy
        8. Functions that may be applied to variables
        9. Creating random values
        10. Scripts in UBOS Manifests
      2. Site JSON
      3. A complex deployment example
      4. UBOS Networking
      5. Allocating and opening up non-default ports
      6. Logging
      7. UBOS state
      8. UBOS Backup format
      9. Format of the App Status JSON
      10. Testing standalone Apps with “webapptest”
      11. Understanding ubos-admin
        1. Command: ubos-admin backup
        2. Command: ubos-admin backupinfo
        3. Command: ubos-admin createsite
        4. Command: ubos-admin deploy
        5. Command: ubos-admin hostid
        6. Command: ubos-admin init-staff
        7. Command: ubos-admin list-data-transfer-protocols
        8. Command: ubos-admin listnetconfigs
        9. Command: ubos-admin listsites
        10. Command: ubos-admin read-configuration-from-staff
        11. Command: ubos-admin restore
        12. Command: ubos-admin setnetconfig
        13. Command: ubos-admin setup-shepherd
        14. Command: ubos-admin showappconfig
        15. Command: ubos-admin shownetconfig
        16. Command: ubos-admin showsite
        17. Command: ubos-admin status
        18. Command: ubos-admin undeploy
        19. Command: ubos-admin update
        20. Command: ubos-admin write-configuration-to-staff
    4. Release channels and UBOS release process
    5. Miscellaneous
      1. Potentially useful infrastructure for standalone Apps
        1. The UBOS rsync server
      2. Middleware-specific notes
        1. Node.js notes
        2. SMTP notes
      3. Setting up an Arch Linux system
        1. Prepare a PC for installing Arch Linux
        2. Prepare a VirtualBox virtual machine to develop for UBOS using Arch Linux
        3. Continuing the Arch Linux installation on a PC or virtual machine
        4. Finishing the Arch development installation by adding UBOS tools
    6. Developer FAQ
      1. Doesn’t apt / dpkg / yum / pacman etc. does what UBOS Gears does already?
      2. Doesn’t puppet / chef / ansible etc. does what UBOS Gears does already?
      3. Doesn’t Docker do what UBOS Gears does already?
      4. How to profile the UBOS Personal Data Mesh web application
      5. Is it possible to run the other UBOS components on an operating system other than UBOS Linux?
      6. How to create a UBOS development VM for VirtualBox
      7. How to create a UBOS development VM for UTM on Apple computers
      8. How to create a UBOS development VM for Parallels Desktop on Apple Silicon

Backup and restore

/docs/operation/backup-restore/

UBOS backup files

To make backup and restore easy, UBOS uses standard ZIP files, with certain additional conventions. To distinguish them from arbitrary other ZIP files, UBOS backup files typically use the extension .ubos-backup.

With a single command, you can backup all the data of all the Apps installed on your Device to a single UBOS backup file. Or, you can use separate backup files for each Site on your Devices. You can also back up just a single App at a Site to a backup file.

Similarly, given a .ubos-backup file, you can restore an entire Site (same hostname, same TLS credentials, same Apps with all of their data at the same context paths) or or only parts. You can also change hostnames and context paths during restore.

UBOS keeps track inside the backup file what Apps you backed up, and how they were configured at the time they were backed up. This makes UBOS backup files essentially self-documenting, and makes it possible that backups can be interpreted even at some considerable time in the future: all information required to restore an App to the state is was in at the time the backup was created is contained in the UBOS backup file.

The details of the UBOS backup format are documented for developers.

Creating a local backup

To create a local backup of all the data of all the Apps on the Device and save that data to file <backupfile>, execute:

% sudo ubos-admin backup --tobackupfile <backupfile>

If you like UBOS to pick a suitable filename that includes the current date, only specify the directory:

% sudo ubos-admin backup --tobackupdir ~

This will create a backup file containing all installed Apps at all Sites on the local host.

If you run more than one Site on a Device, to create a local backup of all the data of only the Apps and Accessories of one particular Site with SiteId <siteid>, and to save that backup to file <backupfile>, execute:

% sudo ubos-admin backup --siteid <siteid> --tobackupfile <backupfile>

To determine the correct <siteid>, use ubos-admin listsites --detail.

Alternatively, you can specify the hostname of the Site:

% sudo ubos-admin backup --hostname <hostname> --tobackupfile <backupfile>

If you run more than one App at a Site, to create a local backup of all the data of only a single installed App and its Accessory with AppConfigId <appconfigid>, and to save that backup to file <backupfile>, execute:

% sudo ubos-admin backup --appconfigid <appconfigid> --tobackupfile <backupfile>

To determine the correct <appconfigid>, use ubos-admin listsites --detail.

If your Site uses TLS, and you do not want to store your TLS key material in the backup, execute the backup command with the --notls option.

You can also create a backup as a side effect of a ubos-admin update or ubos-admin undeploy operation: simply add option --backup <backupfile> to the command.

Creating a backup that is stored on a remote host

You can use backup destinations that contain a Data Transfer Protocol as part of their URL. Here are some examples:

  • file:/tmp/my.ubos-backup: the local file /tmp/my.ubos-backup. For convenience, you don’t need the prefix file:.

  • https://example.com/my.ubos-backup: use HTTPS to HTTP “POST” the backup file to this URL. (This requires you have to have suitable software running at example.com that knows what to do with the arriving file!)

  • s3://mybucket/my.ubos-backup: the file my.ubos-backup in Amazon Web Services' Simple Storage Service (S3), bucket mybucket. This requires the amazons3 package to be installed.

  • rsync+ssh://user@example.com/my.ubos-backup: the file my.ubos-backup uploaded to host example.com as user user, using the rsync protocol over ssh. This requires the ubos-datatransfer-rsync package to be installed.

You can find all Data Transfer Protocols currently available on your Device by executing:

% ubos-admin list-data-transfer-protocols

This will also show available options for these Data Transfer Protocols.

Each of those Data Transfer Protocols may have its own options and particularities. For example, if you use ftp, you may or may not have to turn on “passive mode” (which is a command-line option shown with ubos-admin list-data-transfer-protocols). Some may require usernames, passwords or other credentials. ubos-admin backup will either complain that a necessary option was not provided, or interactively ask you for it. For some Data Transfer Protocols, like ftp for example, it may not be obvious what options are needed for your particular situation; try out different ones until it works.

UBOS will, by default, remember the options and credentials you used for backing up to remote locations. This makes it easier to run the same backup on a regular basis – something we’d like to encourage.

Example: creating a backup that is stored on Amazon S3

As an example, let’s see how UBOS can automatically upload a backup file to your account at Amazon Web Services and store it in its Simple Storage Service (S3).

% sudo pacman -S amazons3

This makes the s3 Data Transfer Protocol available.

You need to have an existing “bucket” on S3 that you are permitted to write to. Let’s assume it is called mybucket. Then, you could invoke the backup to S3 as follows:

% sudo ubos-admin backup --backuptodir s3://mybucket

or:

% sudo ubos-admin backup --backuptofile s3://mybucket/my.ubos-backup

When you invoke this command for the first time, it will ask you for the necessary credential information so it can store the backup on your account at Amazon Web Services. This credential information will be stored on your Device, so you do not need to enter it every time you run a backup.

Specifically, you need to have the Amazon “Access Key ID” and the Amazon “Secret Access Key” for an AWS user that is permitted to create and write the S3 bucket that you specified. Creating this may involve the following steps:

  • Sign up for an Amazon Web Services (AWS) account.

  • In AWS, create an suitable Identity and Access Management (IAM) user, e.g. mybackupuser. This is a user that will only use “programmatic” access.

  • Add the needed permissions to this user by creating a policy, such as:

    • HeadBucket
    • ListBucket
    • CreateBucket
    • PutObject.
  • Create an “Access Key ID” and “Secret Access Key” for that user. Store both of them securely, as Amazon will not show you the Secret Access Key again.

Example: creating a backup that is uploaded via rsync over ssh

If you wanted to back up via rsync over ssh, for example to a home NAS device, first install the ubos-datatransfer-rsync package:

% sudo pacman -S ubos-datatransfer-rsync

This makes the rsync+ssh Data Transfer Protocol available.

Then you need to have a rsync-over-ssh endpoint (e.g. on your NAS) that can be accessed with a SSH keypair; password-based authentication is not supported.

Then you can perform the backup with a command such as:

% sudo ubos-admin backup --idfile <privatekeyfile> --backuptodir rsync+ssh://<server>/directory

where <privatekeyfile> is the SSH private key to be used.

Encrypting a backup

To automatically encrypt a backup before delivering it to its final (local or remote) location, specify --encryptid <id> as an argument to ubos-admin backup. UBOS will look in the GPG keychain of the shepherd user for a GPG public key with identifier <id>, and encrypt the backup file with it.

If you generate the GPG keypair somewhere else than as shepherd on your UBOS Device, importing the public key into the shepherd’s key ring can be as simple as executing:

% gpg --import

and copy-pasting the public key into the terminal, followed by a ^D (for end of file).

Note: Please make sure you understand public and private keys before you do this. Backups are useless if they are encrypted and you can’t decrypt them when you need to! In particular, if you make backups to be able to recover your data if your UBOS Device is lost, stolen, or destroyed, be sure you have the private key needed to decrypt your backups in a safe place that won’t be lost, stolen or destroyed at the same time!

Determining what a backup file contains

To determine the contents of a .ubos-backup file, execute:

% ubos-admin backupinfo --in <backupfile>

This will show information about the backup, such as when it was created, as well as which Sites, Apps and Accessories were backed up.

Restoring from backup

You can restore data either by specifying a local .ubos-backup file (using the --in <backupfile> command-line options) or by specifying an http or https URL from which the backup file will first be downloaded (using the --url <backupurl> command-line options). In this section, we will assume your backup file is local but all commands should work equally with remote files.

To restore all Sites with all Apps and Accessories contained in a .ubos-backup file, execute:

% sudo ubos-admin restore --in <backupfile>

This command will refuse to work if restoring the backup would cause a conflict with a Site that is already installed. Possible conflicts include the following:

  • a currently deployed Site runs at the same hostname as one to be restored;

  • a currently deployed Site has the same SiteId as one to be restored;

  • a currently deployed App has the same AppConfigId as one to be restored;

  • a currently deployed App runs at the same Context Path as one to be restored.

If you wish to restore a previous version of a currently deployed Site from backup, either back up and then undeploy the current Site first, or restore the Site at a new hostname and with new identifiers, using the --createnew options described below.

To restore a Site with a certain SiteId from a backup file <backupfile> to the current Device, but leave all other Sites unchanged, specify the SiteId:

% sudo ubos-admin restore --siteid <siteid> --in <backupfile>

Alternatively, you can use the hostname of the Site that was used at the time of the backup:

% sudo ubos-admin restore --hostname <hostname> --in <backupfile>

To restore only one App, instead of all Apps at a Site, specify the AppConfigId and the hostname of the Site to which the App shall be added:

% sudo ubos-admin restore --appconfigid <appconfigid> --tohostname <tohostname> --in <backupfile>

Alternatively you can use the SiteId of the Site to which the App shall be added:

% sudo ubos-admin restore --appconfigid <appconfigid> --tositeid <tositeid> --in <backupfile>

To copy a Site or AppConfiguration and use new identifiers and a new hostname, use one of the following:

% sudo ubos-admin restore --siteid <fromsiteid> --createnew --newhostname <newhostname> --in <backupfile>

Finally, to replace one or more Apps or Accessory with a different one during restore, use the --migratefrom <package> and --migrateto <package> options, such as:

% sudo ubos-admin restore --migratefrom owncloud --migrateto nextcloud --in <backupfile>

To see the full set of options, invoke:

% ubos-admin restore --help