Project Description

Provide image based transactional updates for MicroOS.

In the embedded world distributors (not customers) prefer image based updates, as they claim they are not able to test so many small updates (which sounds more like an excuse to not invest in test automation and not increase the automatic test coverage).

There are many solutions for image based updates (Mender, RAUC, SWUpdate, Swupd, ...), but they all require an A/B partition setup, thus are limited in the number of parallel installed systems, systems to rollback too, and the size a system can grow.

This project will try to provide a PoC to update openSUSE MicroOS (Container Host system role) via images, by combining existing tools (like kiwi, RAUC, ...) to create update-images, transfer them to a host (USB-Stick, OTA, container, ...) and apply as a new snapshot like transactional-update would update the system with zypper. Use all the tooling around it (btrfs tools, snapper, health-checker, ...), as they exist already, are proven to work and reduces the required changes to the system.

Goal for this Hackweek

  1. Build tar archives with kiwi of openSUSE MicroOS "Container Host"
  2. Use RAUC to create a verity bundle
  3. Transfer the bundle via USB Stick, OTA and/or container to the client.
  4. Follow the steps transactional-update would do by creating a new snapshot, install the image (use rsync or something similar, to not duplicate blocks, else de-duplicate) and make the new image the new default if everything is correct
  5. Reboot into the new image

Problems to solve

  1. Integrate verity bundle building into OBS
  2. How to create new users and groups used in the new image?
  3. How to handle updating of /etc? UsrEtc is one solution. There is no package manager to update/merge configuration files, but there are %pre and %post scripts accessing /etc and /var
  4. How to handle modifications to /var?
  5. At minimum update-alternative and selinux-policy install in /var. Only /boot and /usr (after UsrMove) are allowed, else /bin, /sbin/, /lib and /lib64.

Resources

Updates

Day1

  • rauc is in the current form not useable, but the idea and parts of the code can be re-used
  • There is an openSUSE MicroOS Container Host qcow2 image for testing. No transactional-update nor zypper, but rpm due to dependencies of suse-module-tools. Layout is standard MicroOS with /etc as overlay
  • There is a PoC script to build squashfs images containing the update images in casync format
  • There is a PoC script to apply the update to an btrfs snapshot, build the initrd and bootloader configuration and boot it.
  • casync has same serious bugs, as result an update of an 550MB image with the same image requires over 300MB disk space and not only a few bytes for the reflinks. Additional, it is not possible to package the data from a xfs partition and deploy it on a btrfs partition (and most likely vice versa), the handling of the '.' entry looks broken.

The biggest problem is the /etc directory. /etc cannot be part of the image, but I was not able to build a working image with /etc in a subvolume. Using the usrMove result would allow to have /etc on a read-write root filesystem and /boot plus /usr as image. But in this case, root would be no longer read-only (has advantages and disadvantages) and rollback is impossible. Putting /boot on /usr is not possible, as we have to generate initrd and files for the bootloader.

Day2

  • Start writing "tiu create" in C. Missing pieces:
    • Cleanup the subvolumes before packaging
    • Add verity checksum

Day3

was not such successful, but * Added dm-verity checksum to squashfs archive, but not yet signed (so not secure) * Start writing "tiu extract".

Day4

  • Verification of tiu archive with dm-vertiy works, only signing is missing
  • "tiu extract" can verify, mount and extract an tiu archive to a btrfs subvolume
  • Many bug fixes

Day5

  • Finish implementation of downloading the tiu archive
  • Write documentation
  • Start working on signing of the tiu archives

Looking for hackers with the skills:

Nothing? Add some keywords!

This project is part of:

Hack Week 20

Activity

  • about 3 years ago: fhouard liked this project.
  • over 3 years ago: jblainchristen liked this project.
  • over 3 years ago: ldevulder liked this project.
  • over 3 years ago: doreilly liked this project.
  • over 3 years ago: sydsb liked this project.
  • over 3 years ago: fos liked this project.
  • over 3 years ago: SLindoMansilla liked this project.
  • over 3 years ago: fcrozat liked this project.
  • over 3 years ago: llansky3 liked this project.
  • almost 4 years ago: dfaggioli liked this project.
  • almost 4 years ago: ories liked this project.
  • almost 4 years ago: Pharaoh_Atem liked this project.
  • almost 4 years ago: kukuk liked this project.
  • almost 4 years ago: aplanas liked this project.
  • almost 4 years ago: kukuk started this project.
  • almost 4 years ago: kukuk originated this project.

  • Comments

    • sydsb
      over 3 years ago by sydsb | Reply

      Would this also work for the desktop versions in the way that Silverblue does their base images? Or should we keep the current setup (based on tumbleweed) for desktops?

      • kukuk
        over 3 years ago by kukuk | Reply

        I don't know how Silverblue works, but the image based version should work for everything, as long as you don't need to make changes to the package list.

        • kukuk
          over 3 years ago by kukuk | Reply

          Ok, Silverblue is using RPM-OSTree. This is compareable with what we do already today with MicroOS, but is no real image based update. I want to have a real image based update.

    Similar Projects

    This project is one of its kind!