A complex deployment example

/docs/gears/developer/reference/complex-deployment-example/

In the real world, apps and sites tend to be more complex than the /docs/development/tutorials-gears/toyapps/. Let’s look at one such example, and dissect what happens in detail when such a more complex Site is deployed.

In this example, we deploy a TLS-encrypted Site as it might be used by a small business: it runs a public website, using Wordpress, with an additional plugin. It also runs Nextcloud for the company’s filesharing and calendaring needs, with the Redis cache and calendar accessories. The main site is going to be at https://example.com/. In the real world, we would probably also set up a site at https://www.example.com/ that runs the redirect App to redirect to the main site, but we won’t discuss this here.

The Site JSON file for example.com might look as follows:

{
    "hostname" : "example.com",
    "siteid" : "s6641ad1b780d2330247896c0fe74438fb1e19a33",
    "tls" : {
        "letsencrypt" : true
    },
    "appconfigs" : [
        {
           "appid" : "wordpress",
           "context" : "/blog",
           "appconfigid" : "a5830f84d072d95001a272ab7a5ee2866f93972d0",
           "accessoryids" : [
               "wordpress-plugin-photo-dropper"
           ]
        },
        {
            "appid" : "nextcloud",
            "context" : "/nextcloud",
            "appconfigid" : "a3e2e73410ad9a09fa82794783cd76eb602276736",
            "accessoryids" : [
                "nextcloud-calendar",
                "nextcloud-cache-redis"
            ]
        }
    ]
}

Some observations about the Site defined by this Site JSON file:

UBOS needs to obtain a TLS certificate from LetsEncrypt. Because letsencrypt.org performs a callback to this Site to verify ownership, UBOS needs to temporarily stage certain files with the web server that, obviously, should not conflict with any Apps installed at the same Site.
After the files for Wordpress and the Wordpress plugin have been put in the right places, the Wordpress plugin needs to be automatically activated by Wordpress.
Similarly, after the files for Nextcloud and the Nextcloud accessories have been put in the right places, the accessories need to be activated in Nextcloud as Nextcloud apps.
The Redis cache accessory just provides a connector a running Redis daemon (Redis). This deamon needs to be started and stopped at the right times.
A unique password needs to be generated that’s used to communicate between the Nextcloud Redis cache extension, and the Redis daemon.

Let’s assume we have the above Site JSON file saved to example.com.json. Then we deploy the Site with:

% sudo ubos-admin deploy --file example.com.json

Here is what UBOS does in (reasonable) detail:

Check the Site JSON file:
1. It is syntactically correct.
2. It contains all required fields with syntactically valid values.
3. It can be deployed to the current Device:
  1. The hostname of the new Site does not conflict with the hostname of any Site already deployed to the current Device.
  2. None of the AppConfigIds conflicts with an existing one. Note that the SiteId is not checked for conflicts, as it is perfectly valid to redeploy an already-deployed Site. This redeployment would cause the existing Site to be reconfigured to the new configuration.
4. Whether it is a new Site, or an update of an already-existing Site. We won’t discuss the latter here as it makes things even more complex.
Set a flag that prevents other, concurrent invocations of ubos-admin.
Download and install needed packages:
1. Download and install the App and Accessory Packages referenced in the Site JSON file, and their package dependencies. In our example, they are: wordpress, wordpress-plugin-photo-dropper, nextcloud, nextcloud-calendar, nextcloud-cache-redis, and dependencies such as php and redis.
2. Now that the Packages for the Apps and Accessories have been downloaded and installed, UBOS can examine their respective UBOS Manifest. First, UBOS downloads and installs the Packages (and, recursively, their dependencies) listed as dependencies in the various (applicable) Role sections in the UBOS Manifest of all the Apps and Accessories. Here: from Wordpress: php, php-apache; nothing from the Photo Dropper Accessory; from Nextcloud: php-apache, php-apcu, php-gd, php-systemd; from the Redis cache Accessory: php-redis.
Check the semantics of the intended configuration. This can only be done now that the UBOS Manifest files are available:
1. The AppConfigurations at the same Site may not be deployed to conflicting Context Paths. Here we have /blog (for Wordpress) and /nextcloud (for Nextcloud), which is fine.
2. Values for all required Customization Points have been provided, or have defaults. Here, the value for the Wordpress blog’s title is taken from the default specified in the UBOS Manifest of App Wordpress, and the Redis password for the Nextcloud Redis cache is automatically generated because a random password generation expression has been specified in the UBOS Manifest of the Redis cache Accessory.
3. Values of the Customization Points are valid according to the constraints specified in the respective UBOS Manifest.
4. The Accessories specified in the AppConfiguration can be used with the App in that same AppConfiguration. This would catch, for example, if the Site JSON file specified that the Nextcloud Redis cache was supposed to be used with Wordpress instead of Nextcloud.
5. The UBOS Manifest of all Apps and Accessories is valid. This includes checks for:
  1. Syntactic correctness.
  2. Semantic correctnes.
  3. The files mentioned in the UBOS Manifest actually exist in the file system. This would catch, for example, if the script to activate a Wordpress plugin, contained in the Wordpress package, had been renamed in a recent package update, but an Accessory depending on it hadn’t been updated and was still referencing the old location.
  4. If an Accessory requires the presence of another Accessory at the same AppConfiguration per its UBOS Manifest, check that it is actually present (this does not apply in our example and is rare; an example would we a Wordpress theme, packaged as an Accessory that is a child theme of another Wordpress theme, packaged as a separate Accessory).
As this is the deployment of a new Site, no existing Site needs to be suspended.
Set up a placeholder Site at the same hostname:
1. Create an Apache configuration file for this virtual host whose document directory is the “maintenance” document directory
2. Restart Apache.
Obtain the LetsEncrypt certificate:
- Invoke the certbot program to create a TLS keypair, stage the challenge files in the right location so letsencrypt.org can retrieve them via HTTP, have LetsEncrypt issue a certificate for it, and then save the certificate in the right place with the right Apache configuration directives for it.
Deploy the Site:
1. Create the directories needed by Apache2 for this Site.
2. Process all AppConfigurations:
  1. For the Wordpress AppConfiguration:
    1. Create the directory blog below the Site’s Apache document root directory.
    2. First process App Wordpress:
      1. Save the title Customization Point to a file with a well-known location so the ${installable.customizationpoints.title.file} variable can be resolved later.
      2. Process all AppConfigItems for the mysql Role of App Wordpress:
        
        Provision a new MySQL database.
        
        Provision a new MySQL database user and give it all privileges to the newly provisioned MySQL database, as that is what is specified in the UBOS Manifest.
      3. Create a symbolic link in the Apache modules directory so Apache will load the ssl Apache module upon restart.
      4. Create a symbolic link each in the Apache modules directory so Apache will load all Apache modules upon restart that are specified in Wordpress’s UBOS Manifest: php7 and rewrite.
      5. Create a file each in the PHP modules directory so Apache’s PHP module will load all PHP modules specified in Wordpress’s UBOS Manifest: gd, iconv, mysqli and pdo_mysql.
      6. Process all AppConfigItems for the apache2 Role of App Wordpress, in sequence. This includes:
        
        Recursively copy directory trees to the /blog subdirectory of the Site’s Apache document root.
        
        Create directories relative to the /blog subdirectory of the Site’s Apache document root.
        
        Run the wp-config.pl file that generates the wp-config.php file below the /blog subdirectory of the Site’s Apache document root.
        
        Copy the two htaccess files to below the /blog subdirectory of the Site’s Apache document root, while replacing the variables contained in them.
    3. Now process Accessory Photo Dropper:
      1. Process the single AppConfigItem for the apache2 Role of Accessory Photo Dropper:
        
        Recursively copy its files into wp-plugins subdirectory below the Wordpress installation
  2. For the Nextcloud AppConfiguration:
    1. Create the directory nextcloud below the Site’s Apache document root directory.
    2. First process App Nextcloud:
      1. Process all :term:AppConfigItems <AppConfigItem> for the mysql Role: of App Nextcloud:
        
        Provision a new MySQL database.
        
        Provision a new MySQL database user and give it all privileges to the newly provisioned MySQL database.
      2. Create a symbolic link each in the Apache modules directory so Apache will load all Apache modules upon restart that are specified in Nextcloud’s UBOS Manifest: php7, rewrite, headers, env and setenvif.
      3. Create a file each in the PHP modules directory so Apache’s PHP module will load all PHP modules specified in Nextcloud’s UBOS Manifest: apcu, gd, iconv, mysqli, pdo_mysql and systemd.
      4. Process all :term:AppConfigItems <AppConfigItem> for the apache2 Role: of App Nextcloud, in sequence. This includes:
        
        Recursively copy directory trees to the /nextcloud subdirectory of the Site’s Apache document root.
        
        Create directories relative to the /nextcloud subdirectory of the Site’s Apache document root.
        
        Copy files while replacing the variables contained in them.
        
        Run the Perl script fix-permissions.pl
        
        Start the Systemd timer that runs the Nextcloud background process.
    3. Now process Accessory Nextcloud Calendar:
      1. Process the single AppConfigItem for the apache2 Role: of Accessory Nextcloud Calendar:
        
        Recursively copy its files into apps subdirectory below the Nextcloud installation.
    4. Now process Accessory Nextcloud Redis Cache:
      1. Create a file each in the PHP modules directory so Apache’s PHP module will load all PHP modules specified in Nextcloud Redis Cache’s UBOS Manifest: redis and igbinary.
      2. Process all AppConfigItems for the apache2 Role: of Accessory Nextcloud Redis Cache, in sequence. This includes:
        
        Create directories below the AppConfiguration’s data directory.
        
        Copy the Redis configuration file into the right place, while replacing the variables contained in them (e.g. ${appconfig.appconfigid}, which uniquely identifies this Nextcloud installation from any other running on the same Device, thus allowing multiple Redis daemons to coexist on the same machine)
        
        Start a AppConfiguration-specific Redis Systemd service.
3. Save the Site JSON file so ubos-admin can find the configuration again.
4. Invoke the hostname callbacks for this Site. This depends on which are installed on the Device, but always includes:
  - Add the hostname of the Site to the /etc/hostname file, resolving to the local IP address.
Run the installers:
1. Run the installers for the Wordpress installation:
  1. According to Wordpress’s UBOS Manifest, run initialize.pl, which in turn invokes Wordpress’s installer script, so the user does not have to run it from the browser.
  2. According to Photo Dropper’s UBOS Manifest, run activate-plugin.pl (which is actually contained in the Wordpress Package) in order to activate the installed plugin, so the user does not have run it from the browser. As this script is invoked with the context of the Accessory’s variables, no arguments need to be specified.
2. Run the installers for the Nextcloud installation:
  1. According to Nextcloud’s UBOS Manifest, run install.pl, which in turn runs various Nextcloud command-line commands to initialize the Nextcloud installation correctly. For example, it sets up logging to the system journal instead of the default log file.
  2. According to Nextcloud Calendar’s UBOS Manifest, run activate-app.pl (which is actually contained in the Nextcloud Package) in order to activate the installed Accessory (called “app” by the Nextcloud project), so the user does not have run it from the browser.
  3. According to Nextcloud Redis Cache’s UBOS Manifest, run activate-deactivate.pl, which in turn runs various Nextcloud command-line commands to configure the Nextcloud installation to use the correct Redis instance.
Update the open ports if needed. Neither Wordpress nor Nextcloud open any non-standard ports, but if an App or Accessory requested to open up a port, UBOS would reconfigure its firewall to permit this.
Resume the Site:
1. Update the Apache virtual host configuration:
  1. Save “well-known” files, like robots.txt (none specified in the example).
  2. Create the Apache virtual host configuration file.
  3. Restart Apache.

Perhaps a good time to state that as a developer, you very rarely really have to know all of this :-)