Documentation Overview

  1. Operation documentation
    1. Installation
      1. Run UBOS on a PC (64bit)
      2. Run UBOS from a boot stick on a PC (64bit)
      3. Run UBOS in a VirtualBox virtual machine (64bit)
      4. Run UBOS on an Amazon Web Services EC2 virtual server
      5. Run UBOS on Raspberry Pi 5
      6. Run UBOS on ESPRESSObin
      7. Run UBOS in a Linux container on a PC (64bit)
      8. Run UBOS in an aarch64 Linux container
    2. Setting up your first Site and App
    3. Setting up networking and DNS
    4. How to create a website secured by SSL/TLS
    5. How to set up a website as a Tor hidden service
    6. Managing Sites and Apps
    7. Backup and restore
    8. Upgrading and keeping UBOS current
    9. The UBOS Staff
    10. App-specific notes
      1. Reliably send e-mail via Amazon Web Services’ Simple E-mail Service: amazonses
      2. Static website hosting with rsync-based upload: docroot
      3. Notes on Mastodon
      4. Notes on Nextcloud
      5. Notes on Redirect
      6. Notes on Wordpress
    11. Device-specific Notes
      1. ESPRESSObin
      2. Raspberry Pi
    12. Advanced management
      1. Enabling non-standard package repositories
      2. Migrating from one App to another
      3. Pinning resources
    13. Command reference
    14. FAQ, HOWTOs and Troubleshooting
      1. “Package not found error” when installing a new App or Accessory
      2. A UBOS container comes up degraded
      3. Booting UBOS on a PC starts out fine, but then the screen goes blank
      4. Can I run UBOS in a Docker container?
      5. Can I use UBOS without purchasing a domain name?
      6. Cannot access MySQL database. File missing: /etc/mysql/root-defaults-ubos.cnf
      7. Cannot boot UBOS from boot stick on a PC
      8. Cannot connect to the public internet from a UBOS container
      9. Cannot create a temporary backup; the backup directory is not empty
      10. Failed to create file /sys/devices/system/cpu/microcode/reload
      11. How are the various UBOS images different from each other?
      12. How can I install more than one web App on the same Device?
      13. How do I set up WiFi?
      14. How to enable non-standard Package Repositories
      15. How to get help
      16. How to log into your UBOS Device
      17. How to modify the configuration of your Site
      18. How to report a bug or issue
      19. How to use Pagekite to reach your UBOS Device behind a firewall
      20. How to use SSH
      21. I need a Package that isn’t in UBOS
      22. I need root
      23. I own a domain name, and I’d like to use it for my UBOS Device. How do I do that?
      24. I want to move from one device to another, or from/to the cloud to/from a device
      25. I want to run ssh on a non-standard port
      26. I’m running out of disk space, what now?
      27. Installing a new Package or updating fails with a message about “invalid or corrupted package” or “key is disabled”
      28. Installing a new Package or upgrading fails with a message about “unknown trust”
      29. Is it safe to have my Site accessible from the public web?
      30. My non-English keyboard layout is all screwed up
      31. My SD card is much larger than the UBOS image. How do I use the rest of the space?
      32. Nothing happens when UBOS is supposed to be booting
      33. Problems with “IPv6 Packet Filtering Framework”
      34. UBOS is in a “degraded” state
      35. ubos-admin status reports “Systemd unit … has failed”
      36. Verify your downloaded UBOS image
      37. What is the default “root” password?
      38. What text editor can I use on UBOS?
      39. Why did you derive UBOS Linux from Arch Linux, and what is the relationship between UBOS Linux and Arch?
      40. Why is it called UBOS?
      41. Writing a disk image to a USB stick or SD card
        1. Writing an image to a USB stick or SD card on Linux
        2. Writing an image to a USB stick or SD card on macOS
        3. Writing an image to a USB stick or SD card on Windows
  2. Developer documentation
    1. Developer setup
      1. Developing using Arch Linux on VirtualBox x86_64 with a systemd-nspawn container
      2. Developing using a UTM Arch Linux VM on Apple Silicon computers with UBOS in a systemd-nspawn container
      3. Alternate developer setups
        1. Developing using a systemd-nspawn container (Linux host only)
        2. Developing using Arch Linux using Parallels on Apple Silicon with a systemd-nspawn container
        3. Developing using a UTM Arch Linux VM on Apple x86_64 computers with UBOS in a systemd-nspawn container
    2. Developer tutorials for standalone UBOS apps (not UBOS Mesh)
      1. Build and run your first UBOS standalone App
      2. How to package UBOS standalone Apps built with a variety of languages
        1. Hello World
        2. Glad-I-Was-Here (PHP, Mariadb)
        3. An Accessory for Glad-I-Was-Here (PHP, Mariadb)
        4. Glad-I-Was-Here (PHP, Postgresql)
        5. Glad-I-Was-Here (Python, Mariadb)
    3. UBOS Gears Reference
      1. UBOS Manifest
        1. Structure of the UBOS Manifest
        2. Info section
        3. Roles section
        4. Customization points section
        5. Appinfo section
        6. Accessoryinfo section
        7. Variables available at deploy or undeploy
        8. Functions that may be applied to variables
        9. Creating random values
        10. Scripts in UBOS Manifests
      2. Site JSON
      3. A complex deployment example
      4. UBOS Networking
      5. Allocating and opening up non-default ports
      6. Logging
      7. UBOS state
      8. UBOS Backup format
      9. Format of the App Status JSON
      10. Testing standalone Apps with “webapptest”
      11. Understanding ubos-admin
        1. Command: ubos-admin backup
        2. Command: ubos-admin backupinfo
        3. Command: ubos-admin createsite
        4. Command: ubos-admin deploy
        5. Command: ubos-admin hostid
        6. Command: ubos-admin init-staff
        7. Command: ubos-admin list-data-transfer-protocols
        8. Command: ubos-admin listnetconfigs
        9. Command: ubos-admin listsites
        10. Command: ubos-admin read-configuration-from-staff
        11. Command: ubos-admin restore
        12. Command: ubos-admin setnetconfig
        13. Command: ubos-admin setup-shepherd
        14. Command: ubos-admin showappconfig
        15. Command: ubos-admin shownetconfig
        16. Command: ubos-admin showsite
        17. Command: ubos-admin status
        18. Command: ubos-admin undeploy
        19. Command: ubos-admin update
        20. Command: ubos-admin write-configuration-to-staff
    4. Release channels and UBOS release process
    5. Miscellaneous
      1. Potentially useful infrastructure for standalone Apps
        1. The UBOS rsync server
      2. Middleware-specific notes
        1. Node.js notes
        2. SMTP notes
      3. Setting up an Arch Linux system as a UBOS development system
        1. How to create a UBOS development VM for VirtualBox on x86_64
        2. How to create a UBOS development VM for UTM on x86_64 Apple computers
        3. How to create a UBOS development VM for UTM on Apple Silicon computers
        4. How to create a UBOS development VM for Parallels Desktop on Apple Silicon computers
      4. Creating cloud images
        1. Amazon Web Services EC2
    6. Developer FAQ
      1. Doesn’t apt / dpkg / yum / pacman etc. does what UBOS Gears does already?
      2. Doesn’t puppet / chef / ansible etc. does what UBOS Gears does already?
      3. Doesn’t Docker do what UBOS Gears does already?
      4. Is it possible to run UBOS Gears or Mesh on an operating system other than UBOS Linux?
      5. Can I manage apps packaged as Docker containers with UBOS?
  3. Architecture
  4. Glossary
    1. Accessory
    2. App
    3. AppConfigId
    4. AppConfigItem
    5. AppConfiguration
    6. Arch
    7. Arch Linux
    8. Attribute
    9. blessing
    10. Bot
    11. Context Path
    12. Customization Point
    13. Data Transfer Protocol
    14. Deployment
    15. Depot
    16. Device
    17. Device Class
    18. diet4j module framework
    19. EntityType
    20. Flock
    21. Gradle
    22. Handlebars
    23. History
    24. Home Server
    25. HostId
    26. Hostname
    27. IDE
    28. Installation
    29. LetsEncrypt
    30. mDNS
    31. MeshBase
    32. MeshObject
    33. MeshObjectIdentifier
    34. MeshType
    35. MeshTypeIdentifier
    36. Middleware
    37. Model
    38. Network Configuration
    39. Package
    40. Pagekite
    41. Parallels Desktop
    42. Personal Server
    43. PKGBUILD
    44. Property
    45. PropertyType
    46. Relationship
    47. RelationshipType
    48. Release Channel
    49. Repository
    50. Retention Bucket
    51. Role
    52. RoleAttribute
    53. RoleProperty
    54. RoleType
    55. Rolling Release
    56. Shepherd
    57. Site
    58. Site JSON
    59. Site JSON Template
    60. SiteId
    61. Transaction
    62. Transaction Log
    63. UBOS Gears
    64. UBOS Linux
    65. UBOS Manifest
    66. UBOS Mesh
    67. UBOS Mesh code generator
    68. UBOS Project
    69. UBOS Staff
    70. unblessing
    71. UTM
    72. VirtualBox
    73. VMWare
    74. Wildcard hostname

Roles section

/docs/development/reference-gears/manifest/roles/

The roles section in an UBOS Manifest defines how the App or Accessory needs to be deployed and related information. For example, the roles section defines:

  • which files or directories need to be created in the file system;
  • which databases need to be provisioned with which permissions;
  • which directories or databases need to be backed up;
  • which scripts to run after installation, before uninstallation, or for a version upgrade.

The roles section is structured by roles. Currently supported roles are:

  • apache2: information related to the web tier;
  • tomcat8: information related to the Java :term:Apps <App> running on Tomcat (if applicable);
  • mysql: information related to MySQL databases (if applicable);
  • postgresql: information related to Postgresql databases (if applicable);
  • generic: information not related to any of the other tiers (this is rare).

Generally, upon installation of an App or Accessory, the roles are processed in this sequence:

  1. mysql or postgresql

  2. generic

  3. tomcat8

  4. apache2

Upon uninstallation of an App or Accessory, the roles are processed in the opposite sequence.

Here are common fields for all roles:

Common fields

Depends

When the App or Accessory is deployed for this role, the field depends identifies required packages. Often, these dependencies could also be listed in the package’s PKGBUILD file, but this additional field allows the declaration of dependencies that are only required if this role is used.

Example:

"apache2" : {
  "depends" : [ "php-apache", "php-gd" ],
  ...

AppConfigItems

This section captures the items that need to be put in place before a deployment of an App or Accessory is functional. These items can be things such as files, directories, symbolic links, or databases; but also scripts that need to be run.

For example, in the apache2 role of an App the following appconfigitems section may be found:

"appconfigitems" : [
  {
    "type"         : "file",
    "name"         : "index.php",
    "source"       : "web/index.php",
  },
  {
    "type"         : "file",
    "name"         : "config.php",
    "template"     : "tmpl/config.php.tmpl",
    "templatelang" : "varsubst"
  },
  {
    "type"         : "symlink",
    "name"         : "gladiwashere.php",
    "source"       : "web/gladiwashere.php"
  }
]

Here, three items need to be put in place: two files, and a symbolic link. The following AppConfigItem types are currently supported:

  • directory: a directory to be created;
  • directorytree : a recursive directory tree, copied from somewhere else;
  • file: a file, created by copying another file, or processing another file (see below);
  • database: a database that needs to be created (only use this for database roles such as the mysql role);
  • perlscript: a Perl script that needs to be run;
  • sqlscript: a SQL script that needs to be run (only use this for the mysql role);
  • symlink: a symbolic link;
  • systemd-service: a systemd service to be running while the AppConfiguration is deployed;
  • systemd-timer: a systemd timer to be active while the AppConfiguration is deployed;
  • tcpport: a TCP port needs to be reserved for the exclusive use of this AppConfiguration;
  • udpport: a UDP port needs to be reserved for the exclusive use of this AppConfiguration.

The field name is the name of the file, directory, database, systemd service or timer to be created or operated on. names can be used as a shortcut for several AppConfigItems to which the same other settings apply.

The field template identifies a file or directory that is to be used as a template for creating the new item. The corresponding field templatelang states how the template should be used to create the item. In the example above, the varsubst (“variable substitution”) algorithm is to be applied. (See Variables available at deploy or undeploy and Scripts in UBOS Manifests.)

The field source refers to a file that is the source code for the script to be run, or the destination of the symbolic link. (Think of the original file that is either being copied, run, or pointed to with the symbolic link.)

The source field in case of directorytree, file and symlink may contain:

  • $1: it will be replaced with the value of the name or current names entry.
  • $2: it will be replaced with the file name (without directories) of the name or current names entry.

The following table shows all attributes for AppConfigItems that are defined:

JSON entry Description Relative path context Mutually exclusive with
charset Default character set for SQL database (default: Unicode) N/A N/A
collate Default collation set for SQL database N/A N/A
delimiter Delimiter for SQL scripts (default: ;) N/A N/A
dirpermissions A string containing the octal number for the chmod permissions for directories in this directory hierarchy (default: 0755) N/A N/A
filepermissions A string containing the octal number for the chmod permissions for files in this directory hierarchy (default: 0644) N/A N/A
gname The name of the Linux group that this item should belong to (default: root). N/A N/A
name The name of the created file, directory, symlink, or root of the directory tree, or the symbolic name of a database, or port. ${appconfig.apache2.dir} names
names The names of the created files, directories, symlinks, or roots of the directory trees if more than one item supposed to be processed with the same rule. ${appconfig.apache2.dir} name
permissions A string containing the octal number for the chmod permissions for this file or directory (default: "0644" for files, "0755" for directories). N/A N/A
privileges SQL privileges for a database. N/A N/A
retentionbucket If given, captures that this item contains valuable data that needs to be preserved, e.g. when a backup is performed, and gives it a symbolic name that becomes a named section in backup files. Setting this requires setting the retentionpolicy as well. N/A N/A
retentionpolicy The string "keep". All other values are reserved. Setting this requires setting the retentionbucket as well. N/A N/A
source The name of a file to copy (or execute) without change. ${package.codedir} template
template The name of a template file that will be copied after being processed according to the value of templatelang. ${package.codedir} source
templatelang Specifies the type of template processing to be performed on the content of template. N/A source
uname The name of the Linux user account that should own the created item (default: root). N/A N/A

This table shows which attributes apply to which types of AppConfigItems:

JSON entry database exec directory directory
tree
file perl
script
sql
script
symlink systemd-
service
systemd-
timer
tcpport udpport
delimiter Y
dirpermissions Y Y
filepermissions Y Y
gname Y Y Y Y
name Y Y Y Y Y Y Y Y Y Y
names Y Y Y Y
permissions Y
privileges Y
retentionbucket Y Y Y
retentionpolicy Y Y Y
source Y Y Y Y Y Y
template Y Y
templatelang Y Y
uname Y Y Y Y

Installers, Uninstallers, Upgraders

These fields identify scripts to be run when certain events occur:

  • field installers is processed when the App or Accessory is deployed;
  • field upgraders is processed after an App or Accessory has been deployed and data has been restored that potentially must be migrated to work with the current version of the App or Accessory.

Note that during software upgrades, deployment and undeployment may occur as well (see Command: ubos-admin update).

Each of these fields points to an array. Each of the elements in the array is a separate script that will be run in the sequence listed.

Here is an example for installers in the mysql role of an App that uses MySQL:

"installers" : [
  {
    "name"   : "maindb",
    "type"   : "sqlscript",
    "source" : "mediawiki/maintenance/tables.sql"
  }
]

When this section is processed, UBOS will run the script mediawiki/maintenance/tables.sql of type sqlscript against the database whose symbolic name is maindb.

Supported types are:

  • sqlscript: a SQL script (but only for the mysql role)
  • perlscript: a Perl script

Apache2 role

The apache2 role knows additional fields.

Context

Web Apps must specify one of the following two fields:

  • defaultcontext: the relative URL path at which the App is installed by default. For example, Wordpress may have a defaultcontext of /blog, i.e. if the user installs Wordpress at example.com, by default Wordpress will be accessible at http://example.com/blog. This field is to be used if the App is able to be installed at any relative URL, but this is the default.
  • fixedcontext: some web Apps can only be installed at a particular relative URL, or only at the root of a Site. Use fixedcontext to declare that relative URL.

Transport-level security (TLS) required

Some Apps require that they be accessed via HTTPS only, using SSL/TLS, and will refuse to work over HTTP. Such Apps need to declare this requirement, so UBOS can prevent that they are deployed to an HTTP-only site. To declare this requirement, use this JSON fragment:

"requirestls" : true

If not given, the default for this field is assumed to be false.

Wildcard hostname not supported

Some Apps require to know the hostname through which they are accessed, and cannot be run at the Wildcard hostname. To declare this requirements, use this JSON fragment:

"allowswildcardhostname" : false

If not given, the default for this field is assumed to be true.

Apache modules

apache2modules is a list of names of Apache2 modules that need to be activated before the App or Accessory can be successfully run. Here is an example:

"apache2modules" : [
  "php7"
]

This declaration will make sure that the php7 module is active in Apache2; if not yet, UBOS will activate it and restart Apache2 without any further work by the App or Accessory.

Note that the apache2 role still needs to declare a dependency on php7; apache2modules does not attempt to infer which packages might be needed.

PHP modules

phpmodules is a list of names of PHP modules that need to be activated before the App or Accessory can be successfully run. Here is an example:

"phpmodules" : [
  "gd"
]

This declaration will make sure that the PHP module gd has been activated; if not, UBOS will activate it and restart Apache2.

Note that the apache2 role still needs to declare a dependency on php-gd; apache2modules does not attempt to infer which packages might be needed.

Contributions to the site’s “well-known”

Sites may publish certain “well-known” files, such as robots.txt or the content of directory .well-known below the root of the Site. Subject to certain conflict resolution rules described in Site JSON, an App deployed to a Site may request to augment those entries.

For that purpose, the wellknown entry may be specified. Here is an example:

"wellknown" : {
  "carddav" : {
    "value" : "..."
  },
  "caldav" : {
    "location" : "caldav.php",
    "status" : "302 Found"
  },
  "webfinger" : {
    "proxy" : "http://localhost:1234/webfinger"
  }
}

In wellknown, each key-value pair represents an entry into the Site’s /.well-known/ context path, with the key being the name of the file and the value being a JSON object with the following potential members (Note the special rules for robots.txt and webfinger described below):

value
Static file content if there is; the value may be encoded. This field must not be used by robots.txt or webfinger entries (see additional fields below).
encoding
If given, base64 is the only valid value. It indicates that the value of value is provided using Base64 encoding and needs to be decoded first. This is useful for entries such as favicon.ico.
location
Value for the HTTP Location header when accessed. This is mutually exclusive with value: only one of these two may be provided. This field must not be used by robots.txt or webfinger entries (see below). This value may use variables (as described in variables.d), which UBOS will replace during deployment.
status
HTTP status code to return when accessed. This may only be specified when a location is provided, and the value must be a HTTP redirect status code, such as “307”. When location is provided, the default is “307” (Temporary Redirect).
allow
Only permitted for an entry whose key is robots.txt. This field must have a value of type JSON array. The members of that array are individual Allow: entries for a composite robots.txt file. Each member is prefixed by the content path to the AppConfiguration to which this App has been deployed. For example, if one of the values is /assets/, it will become Allow: /myapp/assets/ if the App has been deployed at context path /myapp.
disallow
Just like allow, but for Disallow: content for a composite robots.txt file.
proxy
Only permitted for an entry whose key is webfinger. This field must have a value of type string, containing a fully-qualified http or https URL (variable $(site.protocol) may be used). This specifies that UBOS, when a client requests the Site’s well-known webfinger URL, should access the given URL, and semantically merge the resulting JSON files obtained from all App’s defining a well-known proxy at this Site. This enables multiple App’s deployed to the Site to all publish their contribution to the Site’s webfinger well-known. This value may use variables (as described in Variables available at deploy or undeploy), which UBOS will replace during deployment.

Phases

When an AppConfiguration with an App and one ore more Accessories is deployed, generally the AppConfigItems of the App are deployed first, followed by the AppConfigItems of one Accessory at a time in the sequence the Accessories were defined in the Site JSON file.

Then, any installer or upgrader scripts are run in the sequence they were defined in the UBOS Manifest, with those defined by the App before those defined by the Accessories.

Undeploying the AppConfiguration occurs in the opposite sequence.

However, sometimes it is necessary to deviate from this default sequence, in particular if the App runs a daemon that requires that all Accessories have been deployed already at the time it starts.

For example, if an App runs a Java daemon with Accessories that contribute optional JARs, and the daemon only scans the available JARs at the time it first starts up, clearly the daemon can only start all Accessories have been deployed.

In order to support this (fairly rare) situation, the relevant AppConfigItems (in the example, of type systemd-service that starts the daemon) can be marked with an extra entry:

"phase" : "suspendresume"

This will cause the AppConfigItem to be skipped on the first pass when installing AppConfigItems, and only process it on a second pass that occurs after the Accessories have all been deployed.

No other values for phase are currently defined.

Status of the AppConfiguration

Status: proposed

Apps may optionally declare an executable or script that, when invoked, reports status information of the AppConfiguration on which it is applied. This status information is provided in JSON in a format defined in Format of the App Status JSON.

To declare such an executable or script, add the following JSON fragment in the apache2 role in its App’s UBOS Manifest JSON, assuming the Perl script status.pl in directory ${appconfig.apache2.dir} should be invoked:

"status" : {
  "source" : "status.pl",
  "type"   : "perlscript"
}

Alternatively, the exec type can be used:

"status" : {
  "source" : "status.sh",
  "type"   : "exec"
}

The script or executable will be invoked with the same arguments as other scripts or executables specified in the UBOS Manifest JSON, but the value for operation is status.

This script or executable is only invoked while the App has been deployed and is supposed to be operational. It is not invoked at any other time.