/blog/
![[RSS feed]](/images/feed-icon-28x28.png){kind=icon}
2017-11-03
It’s really easy to underestimate how much time it takes to improve your build system. This is one of those cases, but eventually, we made it to the finish line.
Welcome, UBOS beta 12!
This release does not contain app updates; those will follow shortly. Detailed release
notes are here.
2017-06-11
UBOS Beta 11 is here, and we are proud to add the Marvell ESPRESSObin to the list of supported devices.
Launched on Kickstarter earlier this year, the ESPRESSObin is an interesting board: it doesn’t have any graphics (which is fine with us because most UBOS devices are used as headless servers) but instead it has three Ethernet ports and a SATA connector. The currently available 1GB version costs only $49 on Amazon. So it’s perfect for running UBOS.
We have also verified that the Raspberry Pi Zero W (the $10 version that has WiFi) also works out of the box with UBOS.
As usual, there have also been many upgrades, bug fixes and new functionality has been added. Here are the highlights:
The smartctld daemon is now running by default. You can use this to get e-mails when your disk
is about to fail.
Nextcloud has been upgraded to current version 12, and we have added a number of frequently-used Nextcloud “apps”: calendar, contacts, mail, news, notes, spreed and tasks. Not only can you now bring your calendar and contact info home, you can also start video conferences right from your UBOS home server.
UBOS can now run Python/Django apps, just as easily as other types of apps.
Hundreds of other package upgrades.
and more! More details are in the release notes here.
This time, upgrading a device that is running a previous UBOS version is a teensy bit more complicated.
Due to some changes in Arch Linux (our upstream distro), the command sudo ubos-admin update may fail on
the first try. To proceed, execute sudo rm /etc/ssl/certs/ca-certificates.crt to delete
that file, and then say sudo ubos-admin update again. The upgrade should work then.
2017-02-12
Beta 10 has arrived with new software packages, new features and the obligatory bug fixes. The big news is this:
You can now set up web applications, like Wordpress, as a Tor hidden service with a single command. This joins previously-available functionality to set up HTTPS sites, including certificate provisioning, with a single command. Here are some examples:
sudo ubos-admin createsite creates a regular HTTP website with the apps you specify.
sudo ubos-admin createsite --tls --self-signed creates the same website, but with a
automatically generated and installed self-signed certificate.
sudo ubos-admin createsite --tls --letsencrypt creates the same website, but with an
official certificate from Letsencrypt that’s recognized by all major browsers.
sudo ubos-admin createsite --tor creates the same website, but as a Tor hidden service.
In addition to the privacy and anonymity features Tor provides to site users and
site publishers, this lets you easily run a website from behind your firewall, or
pack up your server, plug it in somewhere else and you are back up and running.
Some other things that are new:
Nextcloud 11 is now available.
Various bug fixes and improvements.
All images for all platforms has been updated.
These are just some of the highlights. The more detailed release notes are
here.
And as you probably know: to upgrade your device, all you need to say is
sudo ubos-admin update.
2016-12-16
Listen to UBOS instigator Johannes Ernst being interviewed by Bruce Sinclair of IoT Inc. As Bruce puts it:
In this episode of the IoT Business Show, I speak with Johannes Ernst about the IoT Overloads and the upcoming data showdown. (57min)
![[Podcast]](/images/2016-12-16/the-iot-overlords-and-their-thirst-for-your-data-imageL2.png)
2016-11-09
Beta 9 has arrived with new software packages, new features and the obligatory bug fixes. Here are some of the highlights:
Nextcloud 10 is now available, as well as easy ways to upgrade from Nextcloud 9 and from ownCloud.
Wordpress has gained some themes and plugins to make websites prettier.
Automatic file system snapshots protect against failing upgrades. Not that those should ever happen, but let’s be safe instead of sorry.
The development of UBOS-based appliances has become simpler and more secure.
Various bug fixes and improvements.
All images for all platforms has been updated.
These are just some of the highlights. The more detailed release notes are
here.
And as you probably know: to upgrade your device, all you need to say is
sudo ubos-admin update.
2016-09-23
The Mozilla IoT folks invited me (Johannes Ernst) to give a talk at their Connected Devices meetup in San Francisco last night. I talked about “UBOS and the Indie IoT”. It features:
Interesting? Join us on the brand-new Indie IoT mailing list at groups.io/g/indieiot.
2016-08-25
![[Letsencrypt logo]](/images/2016-08-25/letsencrypt-logo-horizontal.svg){kind=logo}
If you have a site that currently doesn’t use SSL/TLS encryption, or you use a a certificate from a different provider, UBOS lets you easily move to a Letsencrypt certificate.
Letsencrypt.org is a wonderful idea. If it didn’t exist, somebody would have to invent it! Of course, there are many certificate authorities from where you can obtain SSL/TLS certificates for your website, so the traffic to and from your website is encrypted and cannot be read or altered by anybody.
But Letsencrypt is different in some important ways:
Letsencrypt certificates are free! While over the years, certificates from other providers have come down in price, there are still certificate authorities out there even today that charge hundreds of dollars per year, for a dubious value proposition.
Installation of certificates from Letsencrypt can be much simpler than the installation
of certificates from other places. Still, it is not for the faint of heart, unless
you are running UBOS, in which case you basically have to do nothing other than saying
--tls --letsencrypt on the command-line when creating a new site with
ubos-admin createsite.
There’s an automatic renewal process for expiring certificates.
But what if you already run a site on UBOS and would like to secure it with Letsencrypt? It’s quite easy:
Let’s say your site’s hostname is example.com. First, obtain its Site JSON and save it
into a convenient place, like this:
% sudo ubos-admin showsite --json --host example.com > example.com.json
Then, edit it to say “get and use a letsencrypt certificate”. You do that by opening
up the newly saved example.com.json file in a text editor of your choice. Then, you
look for the “tls” section. If you already used SSL/TLS, there will be such a section
with all your certificate information in it. Delete the lines in the “tls” section, and
specify to use letsencrypt instead, by making it look like this:
"tls" : {
"letsencrypt" : true
}
If you hadn’t used SSL/TLS for your site yet, simply add the above section at the end of the
file, before the closing }. Make sure that there’s a comma before that added section according to
JSON syntax rules.
Then, redeploy the site:
% sudo ubos-admin deploy -f example.com.json
and voila, your new site will be protected by an automatically-provisioned SSL/TLS certificate from Letsencrypt.
P.S. If you use Firefox, restart the browser before you check whether you indeed have the new certificate; in our test Firefox thought it still used the old one.
2016-08-18
Update: UBOS now has built-in WiFi support via the UBOS Staff and the manual setup described in this post should not be necessary any more.
Unfortunately, the state of WiFi on Linux in general is a mess. We can’t fix that general mess single-handedly on UBOS I’m afraid. But when your WiFi dongle has the “right” chip set, using WiFi on UBOS can be surprisingly easy. Such as when using Element14’s Wi-Pi (datasheet) with the Raspberry Pi.
Here is how I just set up WiFi for a Raspberry Pi that is installed in my back yard, where it controls the pumps of my pool, running UBOS of course (more about that project). I needed to replace the SD Card of that Raspberry Pi, and decided to do a clean reinstall.
Boot UBOS on your Raspberry Pi, and log in (either via console or via SSH over Ethernet)
Insert the Wi-Pi dongle into the USB port
Determine its interface name, by executing:
% ip link
In directory /etc/wpa_supplicant, become root by executing:
% sudo bash
and create a file called wpa_supplicant-wlan0.conf.
This assumes that the name of the WiFi interface you found with ip link above was
wlan0. If it is something else, adjust the name of the file correspondingly.
This file needs to have the following content:
ctrl_interface=/run/wpa_supplicant
eapol_version=1
ap_scan=1
fast_reauth=1
network={
ssid="XXXXXX"
psk="YYYYYY"
}
Replace XXXXXX with the name of your WiFi network you want to connect to, and
YYYYYY with your WiFi password.
Set the correct regulatory domain. In directory /etc/conf.d, edit file wireless-regdom
and remove the comment from the line that represents your country. For example, if you
reside in the US, change line:
#WIRELESS_REGDOM="US"
to read as follows:
WIRELESS_REGDOM="US"
Start the Wifi daemon:
% sudo systemctl start wpa_supplicant@wlan0
% sudo systemctl enable wpa_supplicant@wlan0
Again, use the correct interface name.
You are done! You can check that you have an IP address with:
% ip addr
and you should be able to access any website you like, such as:
% curl -v http://ubos.net/
If it doesn’t work? Check that you got the right mix of underscores and hyphen in the
file you created. And, of course, wpa_supplicant is known to only work with some WiFi
adapters, not all. But IMHO spending a few dollars on an extra, supported WiFi adapter
beats spending hours attempting to debug WiFi drivers. At least that’s what I did.
Updated 2016-11-14 with setting the correct regulatory domain.
2016-08-12
Beta 7 was a major feature release. This time around, the improvements are more incremental. Here are the highlights:
When creating a new site with sudo ubos-admin createsite, you can now easily
install several apps at the same site. For example, you could run your
Wordpress blog at example.com/blog and your Nextcloud family file sharing
at example.com/files. The command has been restructured to allow you to
enter multiple apps.
You can now restore old backups to new apps. For example, if you’d like to
migrate your ownCloud installation to Nextcloud, all you need to do is to
create a backup of your ownCloud site with sudo ubos-admin backup and then
tell sudo ubos-admin restore that you’d like to migrate. It’s really easy.
UBOS now has the beginnings of internet protocol 6 (IPv6) support.
Various bug fixes and improvements.
These are just some of the highlights. The more detailed release notes are
here.
And as you probably know: to upgrade everything on your device, all you need to say is
% sudo ubos-admin update
2016-07-21
This post is a step-by-step transcript of how we packaged Mattermost for one-command install and update on UBOS. The process turned out to be rather typical, and if you are thinking of packaging up an app for UBOS, you can use it as a template. Reference documentation is here.
In case you haven’t come across Mattermost, it is a very nice open-source implementation of a group messaging platform similar to closed-source Slack.
Installing Mattermost manually on some other Linux distro is about middle-of-the-road in terms of difficulty. You need to have a database management system installed (MySQL or Postgres), provision a database, upload some files, edit a configuration file, create a user and a group for the application, change file permissions and set up a daemon configuration. If you want to run any other app on that same server, you also need to set up a reverse proxy, which not many people are familiar with.
On UBOS, just like with any other app that has been packaged for UBOS, our goal is to make it as simple as:
% sudo ubos-admin createsite
and enter mattermost as the name of the app. (Now of course Mattermost has been
packaged, so you can try that out right now on a UBOS device, cloud server or container
before continuing to read.
Instructions for how to set up UBOS are here.
But the point of this post is to document the steps we, as UBOS developers, went through to make it so. You can see the finished work on Github.
So this is what we did:
To understand how to run Mattermost, we first read some of the Mattermost documentation. We are looking for system requirements in terms of libraries and dependencies on other packages (like databases, middleware etc.). We also look for any recommendations how the app likes to be installed in terms of directory layout, webserver configuration etc. We find:
Mattermost is a daemon that talks HTTP on port 8065. Because we don’t want to expose strange port numbers to UBOS users, this means we either have to tell Mattermost to run on port 80 instead and thereby disallow other apps to run on the same machine (not something UBOS users would like), or we have to configure the UBOS web server as a reverse proxy (which we will do).
It can use either Postgresql or MySQL. UBOS supports both, so we don’t care which. We decide to use MySQL because it is slightly more likely to already run on an UBOS device.
It wants to have its own Linux user and group.
By default, it wants to reside in /opt/mattermost. On UBOS, we don’t use /opt
and in any case, we like to keep all valuable data on /var/ubos, so we will need to
override this.
Fortunately, Mattermost is nice and allows us to override everything through its
configuration file. The default location of the configuration file is on /opt, too,
but it turns out that the daemon can be started with an option that specifies an alternate
location. This alternate invocation will go into the systemd.service file that will
start the Mattermost daemon.
Nothing seems to stand in the way of running multiple Mattermost instances on the same machine, for different virtual hostnames. (This is advantageous of you want to host apps for third parties.) As a result, we take the approach to make this possible by default.
They have instructions for starting the Mattermost daemon at startup through
/etc/init.d. UBOS uses systemd, so we will have to write the systemd.service file
ourselves.
Armed with this information, we grab an Arch Linux development machine that has the
UBOS tools installed
(in our case, its runs on a Mac using VirtualBox), and start creating a UBOS package in a
directory called mattermost. You can see the end result
on Github).
We need a PKGBUILD file that knows how to put the mattermost package together. We
copy and paste an existing one (from the
ownCloud UBOS package and adapt it: different
name, author, license, version, download location etc.). For now, we comment out the
subroutines that actually put the package together, and just test that the package build
will download Mattermost:
% makepkg -f
The download works, but then we get a checksum error. Yes, of course, the Mattermost checksum is going to be different than the ownCloud one, and we haven’t updated that. So we do:
% makepkg -f -g
to get the correct checksum computed, which we copy-paste into the PKGBUILD.
According to UBOS conventions, the Mattermost package’s code should end up on the
target device in /usr/share/mattermost/ubos/share/mattermost. We also see that
the download has unpacked the Mattermost files in ./src/mattermost. So we add the following
lines to the package() method:
# Code
mkdir -p ${pkgdir}/usr/share/${pkgname}/mattermost
cp -r -p ${srcdir}/mattermost/* ${pkgdir}/usr/share/${pkgname}/mattermost/
The variables in this statement come from makepkg, the Arch Linux tool we use to
create the package. They are documented on the
Arch wiki here.
Now we do:
% makepkg -f
again, and we see that a sizable file has been created:
mattermost-1.4.0-4-x86_64.pkg.tar.xz with about 17MB. Looking inside:
% tar tfJ mattermost-1.4.0-4-x86_64.pkg.tar.xz | more
shows us that the files are in the place we wanted them. It seems we have the rudiments of a UBOS package for Mattermost.
If the user were to install the package we have created so far with something like:
% sudo pacman -U mattermost-1.4.0-4-x86_64.pkg.tar.xz
all they would get is a big dump of code into their /usr/share/mattermost/ubos/share/mattermost
directory. But to run the application, we need to have a database provisioned, the
daemon started, Apache reconfigured and restarted etc. So we are not done.
First, we create a Mattermost systemd.service file. As we’d like to be able to run
multiple instances of Mattermost on the same machine, we create mattermost@.service,
with the plan to instantiate it as mattermost@<appconfigid>.service, as we know that
appconfigids on UBOS are guaranteed to uniquely identify an app installation. Here’s the file:
[Unit]
Description=Mattermost
[Service]
WorkingDirectory=/usr/share/mattermost/mattermost
ExecStart=/usr/share/mattermost/mattermost/bin/platform -config=/etc/mattermost/%I.json
User=mattermost
Group=mattermost
[Install]
WantedBy=multi-user.target
The daemon, as the Mattermost documentation shows, takes an argument -config=<name>
to specify a non-default config file. As each instance of Mattermost on a given
device needs to have its own configuration, we decide to put the instance-specific
configuration files in /etc/mattermost/<appconfigid>.json. (On UBOS, we generally put
configuration files in /etc/<package>.)
How does this instance-specific configuration file get there before the daemon gets started
when the user deploys the app? Time to create the ubos-manifest.json. We do this again by
copying and pasting from another existing app. In this case, we use the gladiwashere-java
toy app (here in Github)
because it has the dependencies for an instance-specific reverse proxy setup in it,
and we need that, too.
Getting slightly distracted, because we work down the pasted ubos-manifest.json, we
ponder the default relative pathname for the app. We found nothing in the Mattermost
documentation that implied we could run it at any place other than the root of the site,
and also the app is a daemon, so we decide it’s unlikely it will work anywhere but the
root of the app. So we give it a "fixedcontext" : "" instead of a "defaultcontext".
We also try to figure out package dependencies for Mattermost from the documentation,
and we find none! (Other than the database.) This sounds strange, but we’ll go with it;
we’ll find out later for sure. We keep the Apache modules required for the reverse proxy
setup in the manifest file, except that we need proxy_http, not proxy_ajp: the Mattermost
daemon speaks HTTP, not AJP as Tomcat does in gladiwashere-java.
Back to the instance-specific setup. Mattermost needs a MySQL database with all privileges
(it seems, they aren’t quite clear about it). We will give it the symbolic name maindb,
as we usually do for databases. (UBOS will provision a unique database name upon deployment.)
We change that at the end of the ubos-manifest.json file. But then, we also need:
an Apache configuration fragment that performs the reverse proxy setup. We simply copy
this from the gladiwashere-java app’s tmpl directory, but change the port to
Mattermost’s 8065, and the protocol to http from afp.
(Note: because we use that single, hard-coded port, we will not be able to run multiple instances of Mattermost on the same machine yet after all. We will be able to once this extension request has been implemented on UBOS.)
a directory for the data of the Mattermost instance. We will use
/var/lib/mattermost/${appconfig.appconfigid}/data/ubos/lib/mattermost/${appconfig.appconfigid}/data
(parameterized with the already-mentioned appconfigid) as that’s the recommended location
for valuable data on UBOS. We could have put it directly into
/var/lib/mattermost/${appconfig.appconfigid}/ubos/lib/mattermost/${appconfig.appconfigid},
but experience shows that sometimes there
is some other instance-specific data that also needs to be stored, so a subdirectory does not
hurt. This subdirectory gets a retentionpolicy so that UBOS knows it needs to back it up
and restore from backup when needed. This directory needs to be writable by the Mattermost
user.
Detour: we need a mattermost user and group. So we add a file mattermost.install to
our package directory, which runs a few commands to detect whether the user and group
exist already, and if not, creates them. This gets added to the PKGBUILD as
install=mattermost.install. This uses the same user and group for all instances of
Mattermost on the same machine, which may be fine. This script is the usual way Arch
Linux deals with application-specific users, and there’s no need to deviate in UBOS.
Now, the instance-specific configuration file. We’ll derive that from a template,
which we decide to put into our package at tmpl/config.ini.tmpl. Where do we get the
template from? Easy, we use the default Mattermost configuration file, and
parameterize it. Reviewing the file, it seems that most of the default settings will
work for us out of the box, but we need to change the data directory to the place
in /var/lib/.../ubos/lib that we picked above, and of course we need to change the
database connection string. There are a few crypto secrets in that file that should
also be changed, but we skip this for now just to get something to work, because so
far we haven’t proven we can run Mattermost on UBOS at all, which is more important.
And then, of course, we need to run an instance-specific daemon. So we just state, in
the manifest file, that the systemd.service mattermost@${appconfig.appconfigid}.service
needs to be run. Of course, UBOS will replace ${appconfig.appconfigid} with a
unique identifier at time of deployment.
Finally, we make sure that all the template files we just created get packaged into
the package, by adding them to the package() method in the PKGBUILD.
Rebuild the package:
% makepkg -f</pre>
Time to try it out.
Now we need to run UBOS and see what happens if we actually attempt to deploy Mattermost
on UBOS. For development purposes, it’s easiest to run UBOS in a Linux container with
an ephemeral virtual file system, so that every time we restart the container, it looks
like we got a brand-new copy of UBOS. Instructions are
here. For now, we assume
the tar file has been unpackaged in ~/ubos.
So far, the only place where the mattermost package exists is on our development machine.
If we simply tried to install mattermost, UBOS in the container would not be able to
find it. A simple way to solve this problem is to mount the development directory in which
we put the package together into the container. So we run systemd-nspawn with an
extra argument:
% sudo systemd-nspawn --boot --network-veth --ephemeral \
--machine ubos --directory ~/ubos --bind $(pwd):/mattermost
Once we log into that container from its console, we execute:
% ls -al /mattermost
and it will show the current directory of the outside host, i.e. the directory that contains
our just-built mattermost package.
In the container, let’s install the package. To be able to do that we first need to
loosen UBOS’ paranoid default security, as we haven’t signed our package. Edit
/etc/pacman.conf so the respective line reads like:
LocalFileSigLevel = Never
We now successfully install and check that it looks right:
% sudo pacman -U /mattermost/mattermost-*.xz
% ls -al /usr/share/mattermost
% ls -al /etc/mattermost
% ls -al /ubos/lib/mattermost
Now we create a test Site JSON file that describes the virtual host etc. we’ll be using for testing that Mattermost installs correctly:
% sudo ubos-admin createsite -n --out test-mattermost.json
specifying ubos as the name of the site (this should be the same as the name of
the container we used earlier, so we don’t have to setup DNS as systemd will do it for us
if we have mymachines added to /etc/nsswitch.conf on the host), mattermost as
the name of the app, and sensible other defaults.
Now comes the hour of truth, and we run it double-verbose, so we can see what is going on:
% sudo ubos-admin deploy -f test-mattermost.json -v -v
This spits out a lot of progress messages. We worry about anything labeled WARNING, ERROR or FATAL. INFO and DEBUG are fine. And, it turns out, there have been no errors! (Ok, we cheated. There were various syntax errors in various places on the first iteration, but there’s no point in documenting in this blog post how we fixed silly syntax errors.)
Now, we can point a web browser at it on our Arch host by visiting http://ubos/,
assuming the setup and naming above. And voila, we get the Mattermost front page!
Let’s assume it didn’t work the first time around, and we need to make changes to the
package. The easiest is to make the change, run makepkg -f again on the host, and
then in the container, say:
% sudo ubos-admin update --pkgfile /mattermost/mattermost-*.xz
This will perform a system upgrade of the container, except that – because we added the
pkgfile flag – UBOS pretends that only the mattermost package has a new version. (Even
if the version identifier has not changed.) Note that for this to work, the package can’t be
completely broken, as they sometimes are in the development process. Alternatively, we can
undeploy and redeploy our test site:
% sudo ubos-admin undeploy -s s...
% sudo ubos-admin deploy -f test-mattermost.json
or, if our container got completely borked, simply shut down and restart the container and
then deploying the Site JSON again. Because of the --ephemeral option to systemd-nspawn
systemd will automatically delete the working copy of the container’s file system.
Most importantly, backup and restore. With a running Mattermost instance that has some data in it, do this:
% sudo ubos-admin backup -s s... -o testing1.ubos-backup
% sudo ubos-admin undeploy -s s...
Make sure that the instance is gone, e.g. by visiting its web page. Then, restore the backup:
% sudo ubos-admin restore --in testing1.ubos-backup
and Mattermost should be back at the same URL holding the restored data.
And finally, we should create a test script that allows us to easily test we haven’t
broken anything next time UBOS, or Mattermost, gets updated. For that, in UBOS, we have
webapptest (see Testing standalone Apps with “webapptest”).
For now, we are just going to test that Mattermost comes up correctly after install, so
we create a file called MattermostTest1.pm in subdirectory tests (it could have been
anywhere). This is a Perl fragment that specifies the configuration to be set up by the
test (e.g. which app to run), and some curl invocations against it. Once we have it,
we can run it as:
% webapptest run MattermostTest1.pm
usually with various options, such as whether to test in a container, on a physical machine somewhere on the network, on VirtualBox, and which test strategy to use. E.g. to test it using the UBOS container we used above, with verbose output, and stopping after each step, we say:
% webapptest run --scaffold container:directory=$HOME/ubos MattermostTest1.pm -i -v
UBOS recommends that all apps provide icons that can be shown to the user. So we add
the recommended 72x72.png and 144x144.png files to subdirectory appicons and into
the PKGBUILD file so they get added to the package.
What’s left is to add the new app into the official UBOS build. This means we add the package to the build configuration here, and, voila, Mattermost is now part of UBOS. Which is what it has been since UBOS beta 7.