Update 2024-11: Mattermost is no longer packaged due to low usage.
This post is a step-by-step transcript of how we packaged Mattermost for
one-command install and update on UBOS. The process turned out to be rather typical, and if you
are thinking of packaging up an app for UBOS, you can use it as a template.
In case you haven’t come across Mattermost, it is a very nice open-source implementation
of a group messaging platform similar to closed-source Slack.
Installing Mattermost manually on some other Linux distro is about middle-of-the-road in terms
of difficulty. You need to have a database management system installed (MySQL or Postgres),
provision a database, upload some files, edit a configuration file, create a user and a group
for the application, change file permissions and set up a daemon configuration. If you want
to run any other app on that same server, you also need to set up a reverse proxy, which not
many people are familiar with.
On UBOS, just like with any other app that has been packaged for UBOS, our goal is to make
it as simple as:
% sudo ubos-admin createsite
and enter mattermost as the name of the app. (Now of course Mattermost has been
packaged, so you can try that out right now on a UBOS device, cloud server or container
before continuing to read.
Instructions for how to set up UBOS are here.
But the point of this post is to document the steps we, as UBOS developers, went through to make
it so. You can see the finished work on Github.
So this is what we did:
Step 1: Read through (some of) the Mattermost documentation
To understand how to run Mattermost, we first read some of the Mattermost documentation.
We are looking for system requirements in terms of libraries and dependencies on other packages
(like databases, middleware etc.). We also look for any recommendations how the app likes
to be installed in terms of directory layout, webserver configuration etc. We find:
-
Mattermost is a daemon that talks HTTP on port 8065. Because we don’t want to expose
strange port numbers to UBOS users, this means we either have to tell Mattermost to run on
port 80 instead and thereby disallow other apps to run on the same machine (not something UBOS
users would like), or we have to configure the UBOS web server as a reverse proxy
(which we will do).
-
It can use either Postgresql or MySQL. UBOS supports both, so we don’t care which. We
decide to use MySQL because it is slightly more likely to already run on an UBOS device.
-
It wants to have its own Linux user and group.
-
By default, it wants to reside in /opt/mattermost. On UBOS, we don’t use /opt
and in any case, we like to keep all valuable data on /var/ubos, so we will need to
override this.
-
Fortunately, Mattermost is nice and allows us to override everything through its
configuration file. The default location of the configuration file is on /opt, too,
but it turns out that the daemon can be started with an option that specifies an alternate
location. This alternate invocation will go into the systemd.service file that will
start the Mattermost daemon.
-
Nothing seems to stand in the way of running multiple Mattermost instances on the same
machine, for different virtual hostnames. (This is advantageous of you want to host
apps for third parties.) As a result, we take the approach to make this possible by
default.
-
They have instructions for starting the Mattermost daemon at startup through
/etc/init.d. UBOS uses systemd, so we will have to write the systemd.service file
ourselves.
Step 2: Create the basic UBOS package structure
Armed with this information, we grab an Arch Linux development machine that has the
UBOS tools installed
(in our case, its runs on a Mac using VirtualBox), and start creating a UBOS package in a
directory called mattermost. You can see the end result
on Github).
We need a PKGBUILD file that knows how to put the mattermost package together. We
copy and paste an existing one (from the
ownCloud UBOS package and adapt it: different
name, author, license, version, download location etc.). For now, we comment out the
subroutines that actually put the package together, and just test that the package build
will download Mattermost:
% makepkg -f
The download works, but then we get a checksum error. Yes, of course, the Mattermost checksum
is going to be different than the ownCloud one, and we haven’t updated that. So we do:
% makepkg -f -g
to get the correct checksum computed, which we copy-paste into the PKGBUILD.
According to UBOS conventions, the Mattermost package’s code should end up on the
target device in /usr/share/mattermost/ubos/share/mattermost. We also see that
the download has unpacked the Mattermost files in ./src/mattermost. So we add the following
lines to the package() method:
# Code
mkdir -p ${pkgdir}/usr/share/${pkgname}/mattermost
cp -r -p ${srcdir}/mattermost/* ${pkgdir}/usr/share/${pkgname}/mattermost/
The variables in this statement come from makepkg, the Arch Linux tool we use to
create the package. They are documented on the
Arch wiki here.
Now we do:
% makepkg -f
again, and we see that a sizable file has been created:
mattermost-1.4.0-4-x86_64.pkg.tar.xz with about 17MB. Looking inside:
% tar tfJ mattermost-1.4.0-4-x86_64.pkg.tar.xz | more
shows us that the files are in the place we wanted them. It seems we have the rudiments of a
UBOS package for Mattermost.
Step 3: Define what should happen when the user decides to deploy Mattermost
If the user were to install the package we have created so far with something like:
% sudo pacman -U mattermost-1.4.0-4-x86_64.pkg.tar.xz
all they would get is a big dump of code into their /usr/share/mattermost/ubos/share/mattermost
directory. But to run the application, we need to have a database provisioned, the
daemon started, Apache reconfigured and restarted etc. So we are not done.
First, we create a Mattermost systemd.service file. As we’d like to be able to run
multiple instances of Mattermost on the same machine, we create mattermost@.service,
with the plan to instantiate it as mattermost@<appconfigid>.service, as we know that
appconfigids on UBOS are guaranteed to uniquely identify an app installation. Here’s the file:
[Unit]
Description=Mattermost
[Service]
WorkingDirectory=/usr/share/mattermost/mattermost
ExecStart=/usr/share/mattermost/mattermost/bin/platform -config=/etc/mattermost/%I.json
User=mattermost
Group=mattermost
[Install]
WantedBy=multi-user.target
The daemon, as the Mattermost documentation shows, takes an argument -config=<name>
to specify a non-default config file. As each instance of Mattermost on a given
device needs to have its own configuration, we decide to put the instance-specific
configuration files in /etc/mattermost/<appconfigid>.json. (On UBOS, we generally put
configuration files in /etc/<package>.)
How does this instance-specific configuration file get there before the daemon gets started
when the user deploys the app? Time to create the ubos-manifest.json. We do this again by
copying and pasting from another existing app. In this case, we use the gladiwashere-java
toy app (here in Github)
because it has the dependencies for an instance-specific reverse proxy setup in it,
and we need that, too.
Getting slightly distracted, because we work down the pasted ubos-manifest.json, we
ponder the default relative pathname for the app. We found nothing in the Mattermost
documentation that implied we could run it at any place other than the root of the site,
and also the app is a daemon, so we decide it’s unlikely it will work anywhere but the
root of the app. So we give it a "fixedcontext" : "" instead of a "defaultcontext".
We also try to figure out package dependencies for Mattermost from the documentation,
and we find none! (Other than the database.) This sounds strange, but we’ll go with it;
we’ll find out later for sure. We keep the Apache modules required for the reverse proxy
setup in the manifest file, except that we need proxy_http, not proxy_ajp: the Mattermost
daemon speaks HTTP, not AJP as Tomcat does in gladiwashere-java.
Back to the instance-specific setup. Mattermost needs a MySQL database with all privileges
(it seems, they aren’t quite clear about it). We will give it the symbolic name maindb,
as we usually do for databases. (UBOS will provision a unique database name upon deployment.)
We change that at the end of the ubos-manifest.json file. But then, we also need:
-
an Apache configuration fragment that performs the reverse proxy setup. We simply copy
this from the gladiwashere-java app’s tmpl directory, but change the port to
Mattermost’s 8065, and the protocol to http from afp.
(Note: because we use that single, hard-coded port, we will not be able to run multiple
instances of Mattermost on the same machine yet after all. We will be able to once
this extension request has
been implemented on UBOS.)
-
a directory for the data of the Mattermost instance. We will use
/var/lib/mattermost/${appconfig.appconfigid}/data/ubos/lib/mattermost/${appconfig.appconfigid}/data
(parameterized with the already-mentioned appconfigid) as that’s the recommended location
for valuable data on UBOS. We could have put it directly into
/var/lib/mattermost/${appconfig.appconfigid}/ubos/lib/mattermost/${appconfig.appconfigid},
but experience shows that sometimes there
is some other instance-specific data that also needs to be stored, so a subdirectory does not
hurt. This subdirectory gets a retentionpolicy so that UBOS knows it needs to back it up
and restore from backup when needed. This directory needs to be writable by the Mattermost
user.
-
Detour: we need a mattermost user and group. So we add a file mattermost.install to
our package directory, which runs a few commands to detect whether the user and group
exist already, and if not, creates them. This gets added to the PKGBUILD as
install=mattermost.install. This uses the same user and group for all instances of
Mattermost on the same machine, which may be fine. This script is the usual way Arch
Linux deals with application-specific users, and there’s no need to deviate in UBOS.
-
Now, the instance-specific configuration file. We’ll derive that from a template,
which we decide to put into our package at tmpl/config.ini.tmpl. Where do we get the
template from? Easy, we use the default Mattermost configuration file, and
parameterize it. Reviewing the file, it seems that most of the default settings will
work for us out of the box, but we need to change the data directory to the place
in /var/lib/.../ubos/lib that we picked above, and of course we need to change the
database connection string. There are a few crypto secrets in that file that should
also be changed, but we skip this for now just to get something to work, because so
far we haven’t proven we can run Mattermost on UBOS at all, which is more important.
-
And then, of course, we need to run an instance-specific daemon. So we just state, in
the manifest file, that the systemd.service mattermost@${appconfig.appconfigid}.service
needs to be run. Of course, UBOS will replace ${appconfig.appconfigid} with a
unique identifier at time of deployment.
Finally, we make sure that all the template files we just created get packaged into
the package, by adding them to the package() method in the PKGBUILD.
Rebuild the package:
% makepkg -f</pre>
Time to try it out.
Step 4: Debugging
Now we need to run UBOS and see what happens if we actually attempt to deploy Mattermost
on UBOS. For development purposes, it’s easiest to run UBOS in a Linux container with
an ephemeral virtual file system, so that every time we restart the container, it looks
like we got a brand-new copy of UBOS. Instructions are
here. For now, we assume
the tar file has been unpackaged in ~/ubos.
So far, the only place where the mattermost package exists is on our development machine.
If we simply tried to install mattermost, UBOS in the container would not be able to
find it. A simple way to solve this problem is to mount the development directory in which
we put the package together into the container. So we run systemd-nspawn with an
extra argument:
% sudo systemd-nspawn --boot --network-veth --ephemeral \
--machine ubos --directory ~/ubos --bind $(pwd):/mattermost
Once we log into that container from its console, we execute:
% ls -al /mattermost
and it will show the current directory of the outside host, i.e. the directory that contains
our just-built mattermost package.
In the container, let’s install the package. To be able to do that we first need to
loosen UBOS’ paranoid default security, as we haven’t signed our package. Edit
/etc/pacman.conf so the respective line reads like:
LocalFileSigLevel = Never
We now successfully install and check that it looks right:
% sudo pacman -U /mattermost/mattermost-*.xz
% ls -al /usr/share/mattermost
% ls -al /etc/mattermost
% ls -al /ubos/lib/mattermost
Now we create a test Site JSON file that
describes the virtual host etc. we’ll be using for testing that Mattermost installs
correctly:
% sudo ubos-admin createsite -n --out test-mattermost.json
specifying ubos as the name of the site (this should be the same as the name of
the container we used earlier, so we don’t have to setup DNS as systemd will do it for us
if we have mymachines added to /etc/nsswitch.conf on the host), mattermost as
the name of the app, and sensible other defaults.
Now comes the hour of truth, and we run it double-verbose, so we can see what is going on:
% sudo ubos-admin deploy -f test-mattermost.json -v -v
This spits out a lot of progress messages. We worry about anything labeled
WARNING, ERROR or FATAL. INFO and DEBUG are fine. And, it turns out, there have been
no errors! (Ok, we cheated. There were various syntax errors in various places on the
first iteration, but there’s no point in documenting in this blog post how we fixed
silly syntax errors.)
Now, we can point a web browser at it on our Arch host by visiting http://ubos/,
assuming the setup and naming above. And voila, we get the Mattermost front page!
Step 5: Making changes
Let’s assume it didn’t work the first time around, and we need to make changes to the
package. The easiest is to make the change, run makepkg -f again on the host, and
then in the container, say:
% sudo ubos-admin update --pkgfile /mattermost/mattermost-*.xz
This will perform a system upgrade of the container, except that – because we added the
pkgfile flag – UBOS pretends that only the mattermost package has a new version. (Even
if the version identifier has not changed.) Note that for this to work, the package can’t be
completely broken, as they sometimes are in the development process. Alternatively, we can
undeploy and redeploy our test site:
% sudo ubos-admin undeploy -s s...
% sudo ubos-admin deploy -f test-mattermost.json
or, if our container got completely borked, simply shut down and restart the container and
then deploying the Site JSON again. Because of the --ephemeral option to systemd-nspawn
systemd will automatically delete the working copy of the container’s file system.
Step 6: Some other things to test
Most importantly, backup and restore. With a running Mattermost instance that has some data
in it, do this:
% sudo ubos-admin backup -s s... -o testing1.ubos-backup
% sudo ubos-admin undeploy -s s...
Make sure that the instance is gone, e.g. by visiting its web page. Then, restore the backup:
% sudo ubos-admin restore --in testing1.ubos-backup
and Mattermost should be back at the same URL holding the restored data.
Step 7: Test script for the install
And finally, we should create a test script that allows us to easily test we haven’t
broken anything next time UBOS, or Mattermost, gets updated. For that, in UBOS, we have
webapptest.
For now, we are just going to test that Mattermost comes up correctly after install, so
we create a file called MattermostTest1.pm in subdirectory tests (it could have been
anywhere). This is a Perl fragment that specifies the configuration to be set up by the
test (e.g. which app to run), and some curl invocations against it. Once we have it,
we can run it as:
% webapptest run MattermostTest1.pm
usually with various options, such as whether to test in a container, on a physical machine
somewhere on the network, on VirtualBox, and which test strategy to use. E.g. to test it
using the UBOS container we used above, with verbose output, and stopping after each step,
we say:
% webapptest run --scaffold container:directory=$HOME/ubos MattermostTest1.pm -i -v
The end
UBOS recommends that all apps provide icons that can be shown to the user. So we add
the recommended 72x72.png and 144x144.png files to subdirectory appicons and into
the PKGBUILD file so they get added to the package.
What’s left is to add the new app into the official UBOS build. This means we add the
package to the build configuration here,
and, voila, Mattermost is now part of UBOS. Which is what it has been since
UBOS beta 7.
![[Podcast]](/images/2016-12-16/the-iot-overlords-and-their-thirst-for-your-data-imageL2.png)
![[Letsencrypt logo]](/images/2016-08-25/letsencrypt-logo-horizontal.svg){kind=logo}
