YaST log grouping, better visualization of the log
a project by lslezak
Updated over 2 years ago. No love. 1 follower.

Description

TL;DR: I'd like to have something like this for YaST log.

YaST logs all actions it does into the /var/log/YaST2/y2log file. The log is very detailed and contains every detail of the YaST run. And that's a problem! Even a default openSUSE Leap 15.4 installation where you basically just press Next, Next,... produces about a 13MB log file (uncompressed) with more than 80k lines! add-emoji

Navigating in such log is very difficult, you can see every tiny detail what was done, but it is difficult to find the big picture, why was that happening, what was the goal of that part.

Also some code parts repeat during the installation. For example evaluating the base product is done several times during installation, in the self-update step, when displaying the base product license, when registering the system...

So if you see in the log the base product evaluation you cannot be sure to which step it belongs, you have to scroll up and down to find it. This also means you have to start reading the log from the very beginning and go on so you know at which point you currently are. And that is terrible. add-emoji

The Idea

Let's introduce some grouping in the log, to define some structure so you can see what was happening, where it started, where it finished and what was the result.

I got inspired by the GitHub Action log grouping:

Log grouping

See more details in the GitHub documentation.

GitHub Actions do not allow nested groups, only one grouping level is possible. The YaST grouping should allow unlimited nested groups. That should allow easy navigation in the log, you should easily get into the point you are interested in, or the other way round, you could easily know where exactly you are.

Something like this:

▼ Registration step
  ▶ Enter registration data
  ▶ Register the system
  ▶ Register the "SLES-15-SP4" base product
  ▶ Select modules and extensions
  ▼ Register modules and extensions
    ▶ Register "Basesystem" module
    ▼ Register "Server Applications" module
      ▶ Contacting SCC
      ▶ Adding "Pool" repository
      ▶ Adding "Updates" repository
    ▶ Register "Desktop Applications" module
    ...

You can see an example y2log with manually added group markers to get an idea how it could look like.

Improving the YaST Logger

First we need to enhance the YaST logger to add special log markers which will mark each group. We could use the same tags as GitHub Actions (::group:: and ::endgroup::) logged as usual log messages. This would keep the backward compatibility with the current log format.

Example code:

log.group "Registering the system" do
  registration.register(...)
end

The result of the block would be also the result of the logging group. If the result is a failure (like false, :abort, Exception) we could display the log group header in red, maybe with some error icon ( add-emoji ).

Adding Result Data

It would be nice to have some result summary for each group so you not have to go into the details.

log.group "Adding online repositories" do |g|
  ...
  g.details = "added #{repos.size} repositories"
  :next
end

This would be displayed like this:

▶ Adding online repositories (added 5 repositories)

Custom Failure Reporting

The predefined error values might not be enough in some cases, allow reporting an error explicitly:

log.group "Adding online repositories" do |g|
  ...
  if ret.empty?
    g.failure = true
    g.details = "No repository added"
  end

  :skip
end

This should be displayed in the log view like this (in red)

▶ ⚠️ Adding online repositories (No repository added)

The Log Browser

Of course, the added markers allow to easily find the start and end of a block, but it would be nice to have some browser which would visualize the structure of the log.

Writing a desktop application would not be bad but generating an HTML page with the structured log would be easier. That will also allow us to easily create a web service for that so you could upload the log and directly see the result in the web browser. We could also containerize it so you could easily run it locally. add-emoji

And because YaST is written in Ruby and Ruby is also used for web applications let's write the HTML generator in Ruby.

Goal for this Hackweek

  • Add grouping support to the YaST logger
  • Use it at some places in YaST (at least in the installation workflow manager so we know where each step starts/end)
  • Create a script which can parse a single YaST y2log file and render a resulting HTML page
  • Support multiple YaST runs in the log
  • Colorized output (errors in red, ....)
  • Display nested log groups, allow expanding/collapsing groups with a ▶button
  • Basic filtering (show/hide debug messages)
  • Generate a single document HTML (with JS and CSS embedded) for easier sharing the generated document or HTML with separate JS and CSS files

Future Enhancements

  • Advanced filtering (component based: show/hide libzypp messages, libyui messages, YaST agents,...)
  • Allow reading compressed logs (like y2log-1.gz)
  • Allow reading whole tarballs created by save_y2log script
  • Create a web application, instead of running the script allow uploading it to a server running locally and directly display the rendered page
  • Extract some high level data from the log as a description of whole process (Upgrading from SLES12-SP3 to SLES15-SP4) or to detect some unusual situations (Using custom registration server http://rmt.example.com). This would be displayed at the top as the log summary
  • Containerize the script and the web application
  • Host the web application at some free hosting service

Needed Skills

  • Ruby (text parsing and processing, generating HTML, ERB)
  • Web Development (HTML, CSS, JavaScript)
  • YaST basics (optional, the y2log structure is trivial)

Join this project

Looking for hackers with the skills:

ruby html javascript css

This project is part of:

Hack Week 21

Activity

  • over 2 years ago: lslezak added keyword "ruby" to this project.
  • over 2 years ago: lslezak added keyword "html" to this project.
  • over 2 years ago: lslezak added keyword "javascript" to this project.
  • over 2 years ago: lslezak added keyword "css" to this project.
  • over 2 years ago: lslezak started this project.
  • over 2 years ago: lslezak originated this project.


    • Comments

    Be the first to comment!

    Similar Projects

    Fix RSpec tests in order to replace the ruby-ldap rubygem in OBS by enavarro_suse

    Description

    "LDAP mode is not official supported by OBS!". See: config/options.yml.example#L100-L102

    However, there is an RSpec file which tests LDAP mode in OBS. These tests use the ruby-ldap rubygem, mocking the results returned by a LDAP server.

    The ruby-ldap rubygem seems no longer maintaned, and also prevents from updating to a more recent Ruby version. A good alternative is to replace it with the net-ldap rubygem.

    Before replacing the ruby-ldap rubygem, we should modify the tests so the don't mock the responses of a LDAP server. Instead, we should modify the tests and run them against a real LDAP server.

    Goals

    Goals of this project:

    • Modify the RSpec tests and run them against a real LDAP server
    • Replace the net-ldap rubygem with the ruby-ldap rubygem

    Achieving the above mentioned goals will:

    • Permit upgrading OBS from Ruby 3.1 to Ruby 3.2
    • Make a step towards officially supporting LDAP in OBS.

    Resources

    Recipes catalog and calculator in Rails 8 by gfilippetti

    My wife needs a website to catalog and sell the products of her upcoming bakery, and I need to learn and practice modern Rails. So I'm using this Hack Week to build a modern store using the latest Ruby on Rails best practices, ideally up to the deployment.

    TO DO

    • Index page
    • Product page
    • Admin area -- Supplies calculator based on orders -- Orders notification
    • Authentication
    • Payment
    • Deployment

    Day 1

    As my Rails knowledge was pretty outdated and I had 0 experience with Turbo (wich I want to use in the app), I started following a turbo-rails course. I completed 5 of 11 chapters.

    Day 2

    Continued the course until chapter 8 and added live updates & an empty state to the app. I should finish the course on day 3 and start my own project with the knowledge from it.

    Hackweek 24

    For this Hackweek I'll continue this project, focusing on a Catalog/Calculator for my wife's recipes so she can use for her Café.

    Day 1

    Design the new UI for storage configuration at Agama by ancorgs

    Description

    We are in the process of re-designing the web user interface to configure storage at Agama. We expected to have a clear idea of what we wanted before starting Hack Week. But the idea is still not that clear. So I will use use my Hack Week time to try several prototypes since I really want this to be done.

    Goals

    Have a prototype using Patternfly components and addressing all the use-cases we want to cover. Easy for the easy cases. Capable for the complex ones.

    obs-service-vendor_node_modules by cdimonaco

    Description

    When building a javascript package for obs, one option is to use https://github.com/openSUSE/obs-service-node_modules as source service to get the project npm dependencies available for package bulding.

    obs-service-vendornodemodules aims to be a source service that vendors npm dependencies, installing them with npm install (optionally only production ones) and then creating a tar package of the installed dependencies.

    The tar will be used as source in the package building definitions.

    Goals

    • Create an obs service package that vendors the npm dependencies as tar archive.
    • Maybe add some macros to unpack the vendor package in the specfiles

    Resources

    Agama installer on-line demo by lslezak

    Description

    The Agama installer provides a quite complex user interface. We have some screenshots on the web page but as it is basically a web application it would be nice to have some on-line demo where users could click and check it live.

    The problem is that the Agama server directly accesses the hardware (storage probing) and loads installation repositories. We cannot easily mock this in the on-line demo so the easiest way is to have just a read-only demo. You could explore the configuration options but you could not change anything, all changes would be ignored.

    The read-only demo would be a bit limited but I still think it would be useful for potential users get the feeling of the new Agama installer and get familiar with it before using in a real installation.

    As a proof of concept I already created this on-line demo.

    The implementation basically builds Agama in two modes - recording mode where it saves all REST API responses and replay mode where it for the REST API requests returns the previously recorded responses. Recording in the browser is inconvenient and error prone, there should be some scripting instead (see below).

    Goals

    • Create an Agama on-line demo which can be easily tested by users
    • The Agama installer is still in alpha phase and in active development, the online demo needs to be easily rebuilt with the latest Agama version
    • Ideally there should be some automation so the demo page is rebuilt automatically without any developer interactions (once a day or week?)

    TODO

    • Use OpenAPI to get all Agama REST API endpoints, write a script which queries all the endpoints automatically and saves the collected data to a file (see this related PR).
    • Write a script for starting an Agama VM (use libvirt/qemu?), the script should ensure we always use the same virtual HW so if we need to dump the latest REST API state we get the same (or very similar data). This should ensure the demo page does not change much regarding the storage proposal etc...
    • Fix changing the product, currently it gets stuck after clicking the "Select" button.
    • Move the mocking data (the recorded REST API responses) outside the Agama sources, it's too big and will be probably often updated. To avoid messing the history keep it in a separate GitHub repository
    • Allow changing the UI language
    • Display some note (watermark) in the page so it is clear it is a read-only demo (probably with some version or build date to know how old it is)
    • Automation for building new demo page from the latest sources. There should be some check which ensures the recorded data still matches the OpenAPI specification.

    Changing the UI language

    This will be quite tricky because selecting the proper translation file is done on the server side. We would probably need to completely re-implement the logic in the browser side and adapt the server for that.

    Also some REST API responses contain translated texts (storage proposal, pattern names in software). We would need to query the respective endpoints in all supported languages and return the correct response in runtime according to the currently selected language.

    Resources

    Editor mode at Agama web interface by ancorgs

    Description

    Agama is a new Linux installer that will be very likely used for SLES 16.

    It takes a configuration (in JSON format) as input. And offers several interfaces to build that configuration in an easy and interactive way.

    I was considering the possibility to add to the web interface a "text editor" mode similar to the XML editor available at virt-manager. That could be used to see how the changes in the UI translate into changes on the configuration.

    Goals

    • Refresh my knowledge about UI development for Agama, since there was a major overhaul recently (adopting TanStack Query) and I need to learn the new way to do things.
    • Please hackers who always want to know how things work internally. :-)

    Agama Expert Partitioner by joseivanlopez

    Description

    Agama is a new Linux installer that will be very likely used for SLES 16.

    It offers an UI for configuring the target system (language, patterns, network, etc). One of the more complex sections is the storage configuration, which is going to be revamped. This project consists on exploring the possibility of having something similar to the YaST Expert Partitioner for Agama.

    Goals

    • Explore different approaches for the storage UI in Agama.