This is a follow up to https://hackweek.suse.com/projects/architecting-a-machine-learning-project-with-suse-caasp.

In the last hackweek I learned that in order to run machine learning workflows on top of SUSE CaaSP, the missing piece is to have libnvidia-containers and nvidia-containers-runtime-hook packaged.

Since then, nvidia has added the build for leap15 in libnvidia-container and nvidia-container-runtime.

However, none of them is released into the libnvidia-container repo nor nvidia-container-runtime repo.

This project is about packaging those two projects in the openSUSE Build Service for openSUSE Leap 15.1.

Looking for hackers with the skills:

nvidia machinelearning containers

This project is part of:

Hack Week 19

Activity

almost 5 years ago: drdavis liked this project.

almost 5 years ago: afesta liked this project.

almost 5 years ago: jordimassaguerpla added keyword "nvidia" to this project.

almost 5 years ago: jordimassaguerpla added keyword "machinelearning" to this project.

almost 5 years ago: jordimassaguerpla added keyword "containers" to this project.

almost 5 years ago: a_faerber liked this project.

almost 5 years ago: jordimassaguerpla started this project.

almost 5 years ago: jordimassaguerpla originated this project.

Comments

almost 5 years ago by jordimassaguerpla | Reply

First package ready: https://build.opensuse.org/package/show/home:jordimassaguerpla:nvidia_container/libnvidia-container And a Pull Request to upstream: https://github.com/NVIDIA/libnvidia-container/pull/77

almost 5 years ago by jordimassaguerpla | Reply

Second package ready: https://build.opensuse.org/package/show/home:jordimassaguerpla:nvidia_container/nvidia-container-runtime-toolkit

almost 5 years ago by jordimassaguerpla | Reply

Prove that this worked:

On a workstation with Quadro K2000 with SLE15SP1:

Installing nvidia graphics driver kernel module

zypper ar https://download.nvidia.com/suse/sle15sp1/ nvidia
zypper ref
zypper install nvidia-gfxG05-kmp-default
modprobe nvidia
lsmod | grep nvidia

Expected output:

nvidia_drm             49152  0
nvidia_modeset       1114112  1 nvidia_drm
drm_kms_helper        204800  1 nvidia_drm
drm                   536576  3 nvidia_drm,drm_kms_helper
nvidia_uvm           1036288  0
nvidia              20414464  2 nvidia_modeset,nvidia_uvm
ipmi_msghandler       110592  2 nvidia,ipmi_devintf

Installing NVIDIA driver for computing with GPUs using CUDA

zypper install nvidia-computeG05
nvidia-smi

Expected output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.59       Driver Version: 440.59       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K2000        Off  | 00000000:05:00.0 Off |                  N/A |
| 30%   43C    P0    N/A /  N/A |      0MiB /  1997MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Installing libnvidia-containers

zypper ar https://download.opensuse.org/repositories/home:/jordimassaguerpla:/nvidia_container/SLE_15_SP1/ nvidia_container
zypper install libnvidia-container
usermod -G root USER
usermod -G video USER

USER should be a user in your system which is not root

su - USER -c nvidia-container-cli info

expected output

NVRM version:   440.59
CUDA version:   10.2

Device Index:   0
Device Minor:   0
Model:          Quadro K2000
Brand:          Quadro
GPU UUID:       GPU-6a04b812-c20e-aeb6-9047-6382930eef7d
Bus Location:   00000000:05:00.0
Architecture:   3.0

> NOTE: we need to use a different user that is not root for this test because the root user does not run with the video group by default. We will fix this later when installing the toolkit. If you use root, you will see this error message

nvidia-container-cli: initialization error: cuda error: no cuda-capable device is detected

Installing nvidia-container-toolkit

zypper install nvidia-container-toolkit

Test with podman

zypper install podman podman-cni-config
podman run nvidia/cuda nvidia-smi

expected output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.59       Driver Version: 440.59       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K2000        Off  | 00000000:05:00.0 Off |                  N/A |
| 30%   43C    P0    N/A /  N/A |      0MiB /  1997MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

So it works!

Prove that this worked:

On a workstation with Quadro K2000 with SLE15SP1:

Installing nvidia graphics driver kernel module

zypper ar https://download.nvidia.com/suse/sle15sp1/ nvidia
zypper ref
zypper install nvidia-gfxG05-kmp-default
modprobe nvidia
lsmod | grep nvidia

Expected output:

nvidia_drm             49152  0
nvidia_modeset       1114112  1 nvidia_drm
drm_kms_helper        204800  1 nvidia_drm
drm                   536576  3 nvidia_drm,drm_kms_helper
nvidia_uvm           1036288  0
nvidia              20414464  2 nvidia_modeset,nvidia_uvm
ipmi_msghandler       110592  2 nvidia,ipmi_devintf

Installing NVIDIA driver for computing with GPUs using CUDA

zypper install nvidia-computeG05
nvidia-smi

Expected output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.59       Driver Version: 440.59       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K2000        Off  | 00000000:05:00.0 Off |                  N/A |
| 30%   43C    P0    N/A /  N/A |      0MiB /  1997MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Installing libnvidia-containers

zypper ar https://download.opensuse.org/repositories/home:/jordimassaguerpla:/nvidia_container/SLE_15_SP1/ nvidia_container
zypper install libnvidia-container
usermod -G root USER
usermod -G video USER

USER should be a user in your system which is not root

su - USER -c nvidia-container-cli info

expected output

NVRM version:   440.59
CUDA version:   10.2

Device Index:   0
Device Minor:   0
Model:          Quadro K2000
Brand:          Quadro
GPU UUID:       GPU-6a04b812-c20e-aeb6-9047-6382930eef7d
Bus Location:   00000000:05:00.0
Architecture:   3.0

nvidia-container-cli: initialization error: cuda error: no cuda-capable device is detected

Installing nvidia-container-toolkit

zypper install nvidia-container-toolkit

Test with podman

zypper install podman podman-cni-config
podman run nvidia/cuda nvidia-smi

expected output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.59       Driver Version: 440.59       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K2000        Off  | 00000000:05:00.0 Off |                  N/A |
| 30%   43C    P0    N/A /  N/A |      0MiB /  1997MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

So it works!

# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title

More Complex Markdown Help

Formatting Help

almost 5 years ago by jordimassaguerpla | Reply

As a result, I updated the docs: https://github.com/jordimassaguerpla/SUSEhackweek18/commit/5fca6c12034b4df34c403f14276be754e809b086#diff-2df0241dfedf44f37dcafae751ab29ae

almost 5 years ago by jordimassaguerpla | Reply

The previous link got broken ... damn markdown ;) docs

almost 5 years ago by jordimassaguerpla | Reply

Upstream (NVIDIA) uses Dockerfiles to build the packages for the other distros.

Here a small experiment of building the SUSE Leap RPM with a Dockerfile within OBS:

https://build.opensuse.org/package/show/home:jordimassaguerpla:branches:openSUSE:Templates:Images:15.1/libnvidia-containers

almost 5 years ago by jordimassaguerpla | Reply

Result of the experiment. Using a Dockerfile works very good because you can develop and debug using "docker build" and then you can commit that to obs to have a build on a central location, store the sources, etc. etc.

The issue is that the result is an image, it can't be the RPM. There is no "-v" option to mount a volume during the build. Thus, even you can build the image in obs, then you have to run the image to extract the RPM.

obs=build.opensuse.org.

Similar Projects

machinelearning

FamilyTrip Planner: A Personalized Travel Planning Platform for Families by pherranz

Description

FamilyTrip Planner is an innovative travel planning application designed to optimize travel experiences for families with children. By integrating APIs for flights, accommodations, and local activities, the app generates complete itineraries tailored to each family’s unique interests and needs. Recommendations are based on customizable parameters such as destination, trip duration, children’s ages, and personal preferences. FamilyTrip Planner not only simplifies the travel planning process but also offers a comprehensive, personalized experience for families.

Goals

This project aims to: - Create a user-friendly platform that assists families in planning complete trips, from flight and accommodation options to recommended family-friendly activities. - Provide intelligent, personalized travel itineraries using artificial intelligence to enhance travel enjoyment and minimize time and cost. - Serve as an educational project for exploring Go programming and artificial intelligence, with the goal of building proficiency in both.

Resources

To develop FamilyTrip Planner, the project will leverage: - APIs such as Skyscanner, Google Places, and TripAdvisor to source real-time information on flights, accommodations, and activities. - Go programming language to manage data integration, API connections, and backend development. - Basic machine learning libraries to implement AI-driven itinerary suggestions tailored to family needs and preferences.

containers

Enable the containerized Uyuni server to run on different host OS by j_renner

Description

The Uyuni server is provided as a container, but we still require it to run on Leap Micro? This is not how people expect to use containerized applications, so it would be great if we tested other host OSs and enabled them by providing builds of necessary tools for (e.g. mgradm). Interesting candidates should be:

openSUSE Leap
Cent OS 7
Ubuntu
???

Goals

Make it really easy for anyone to run the Uyuni containerized server on whatever OS they want (with support for containers of course).

Improve Development Environment on Uyuni by mbussolotto

Description

Currently create a dev environment on Uyuni might be complicated. The steps are:

add the correct repo
download packages
configure your IDE (checkstyle, format rules, sonarlint....)
setup debug environment
...

The current doc can be improved: some information are hard to be find out, some others are completely missing.

Dev Container might solve this situation.

Goals

Uyuni development in no time:

using VSCode:
- setting.json should contains all settings (for all languages in Uyuni, with all checkstyle rules etc...)
- dev container should contains all dependencies
- setup debug environment
implement a GitHub Workspace solution
re-write documentation

Lots of pieces are already implemented: we need to connect them in a consistent solution.

Resources

https://github.com/uyuni-project/uyuni/wiki

SUSE AI Meets the Game Board by moio

Use tabletopgames.ai’s open source TAG and PyTAG frameworks to apply Statistical Forward Planning and Deep Reinforcement Learning to two board games of our own design. On an all-green, all-open source, all-AWS stack!

Results: Infrastructure Achievements

We successfully built and automated a containerized stack to support our AI experiments. This included:

a Fully-Automated, One-Command, GPU-accelerated Kubernetes setup: we created an OpenTofu based script, tofu-tag, to deploy SUSE's RKE2 Kubernetes running on CUDA-enabled nodes in AWS, powered by openSUSE with GPU drivers and gpu-operator
Containerization of the TAG and PyTAG frameworks: TAG (Tabletop AI Games) and PyTAG were patched for seamless deployment in containerized environments. We automated the container image creation process with GitHub Actions. Our forks (PRs upstream upcoming):
- https://github.com/moio/TabletopGames/tree/hackweek24
- https://github.com/moio/PyTAG/tree/hackweek24

./deploy.sh and voilà - Kubernetes running PyTAG (k9s, above) with GPU acceleration (nvtop, below)

Results: Game Design Insights

Our project focused on modeling and analyzing two card games of our own design within the TAG framework:

Game Modeling: We implemented models for Dario's "Bamboo" and Silvio's "Totoro" and "R3" games, enabling AI agents to play thousands of games ...in minutes!
AI-driven optimization: By analyzing statistical data on moves, strategies, and outcomes, we iteratively tweaked the game mechanics and rules to achieve better balance and player engagement.
Advanced analytics: Leveraging AI agents with Monte Carlo Tree Search (MCTS) and random action selection, we compared performance metrics to identify optimal strategies and uncover opportunities for game refinement .
- more about Bamboo on Dario's site
- more about R3 on Silvio's site (italian, translation coming)
- more about Totoro on Silvio's site

A family picture of our card games in progress. From the top: Bamboo, Totoro, R3

Results: Learning, Collaboration, and Innovation

Beyond technical accomplishments, the project showcased innovative approaches to coding, learning, and teamwork:

"Trio programming" with AI assistance: Our "trio programming" approach—two developers and GitHub Copilot—was a standout success, especially in handling slightly-repetitive but not-quite-exactly-copypaste tasks. Java as a language tends to be verbose and we found it to be fitting particularly well.
AI tools for reporting and documentation: We extensively used AI chatbots to streamline writing and reporting. (Including writing this report! ...but this note was added manually during edit!)
GPU compute expertise: Overcoming challenges with CUDA drivers and cloud infrastructure deepened our understanding of GPU-accelerated workloads in the open-source ecosystem.
Game design as a learning platform: By blending AI techniques with creative game design, we learned not only about AI strategies but also about making games fun, engaging, and balanced.

Last but not least we had a lot of fun! ...and this was definitely not a chatbot generated line!

The Context: AI + Board Games

Port the classic browser game HackTheNet to PHP 8 by dgedon

Description

The classic browser game HackTheNet from 2004 still runs on PHP 4/5 and MySQL 5 and needs a port to PHP 8 and e.g. MariaDB.

Goals

Port the game to PHP 8 and MariaDB 11
Create a container where the game server can simply be started/stopped

Resources

https://github.com/nodeg/hackthenet

ClusterOps - Easily install and manage your personal kubernetes cluster by andreabenini

Description

ClusterOps is a Kubernetes installer and operator designed to streamline the initial configuration and ongoing maintenance of kubernetes clusters. The focus of this project is primarily on personal or local installations. However, the goal is to expand its use to encompass all installations of Kubernetes for local development purposes.
It simplifies cluster management by automating tasks and providing just one user-friendly YAML-based configuration config.yml.

Overview

Simplified Configuration: Define your desired cluster state in a simple YAML file, and ClusterOps will handle the rest.
Automated Setup: Automates initial cluster configuration, including network settings, storage provisioning, special requirements (for example GPUs) and essential components installation.
Ongoing Maintenance: Performs routine maintenance tasks such as upgrades, security updates, and resource monitoring.
Extensibility: Easily extend functionality with custom plugins and configurations.
Self-Healing: Detects and recovers from common cluster issues, ensuring stability, idempotence and reliability. Same operation can be performed multiple times without changing the result.
Discreet: It works only on what it knows, if you are manually configuring parts of your kubernetes and this configuration does not interfere with it you can happily continue to work on several parts and use this tool only for what is needed.

Features

distribution and engine independence. Install your favorite kubernetes engine with your package manager, execute one script and you'll have a complete working environment at your disposal.
Basic config approach. One single config.yml file with configuration requirements (add/remove features): human readable, plain and simple. All fancy configs managed automatically (ingress, balancers, services, proxy, ...).
Local Builtin ContainerHub. The default installation provides a fully configured ContainerHub available locally along with the kubernetes installation. This configuration allows the user to build, upload and deploy custom container images as they were provided from external sources. Internet public sources are still available but local development can be kept in this localhost server. Builtin ClusterOps operator will be fetched from this ContainerHub registry too.
Kubernetes official dashboard installed as a plugin, others planned too (k9s for example).
Kubevirt plugin installed and properly configured. Unleash the power of classic virtualization (KVM+QEMU) on top of Kubernetes and manage your entire system from there, libvirtd and virsh libs are required.
One operator to rule them all. The installation script configures your machine automatically during installation and adds one kubernetes operator to manage your local cluster. From there the operator takes care of the cluster on your behalf.
Clean installation and removal. Just test it, when you are done just use the same program to uninstall everything without leaving configs (or pods) behind.

Planned features (Wishlist / TODOs)

Containerized Data Importer (CDI). Persistent storage management add-on for Kubernetes to provide a declarative way of building and importing Virtual Machine Disks on PVCs for