Uyuni read-only replica by cbosdonnat

Description

For now, there is no possible HA setup for Uyuni. The idea is to explore setting up a read-only shadow instance of an Uyuni and make it as useful as possible.

Possible things to look at:

• live sync of the database, probably using the WAL. Some of the tables may have to be skipped or some features disabled on the RO instance (taskomatic, PXT sessions…)
• Can we use a load balancer that routes read-only queries to either instance and the other to the RW one? For example, packages or PXE data can be served by both, the API GET requests too. The rest would be RW.

Goals

• Prepare a document explaining how to do it.
• PR with the needed code changes to support it

---

Enable more features in mcp-server-uyuni by j_renner

Description

I would like to contribute to mcp-server-uyuni, the MCP server for Uyuni / Multi-Linux Manager) exposing additional features as tools. There is lots of relevant features to be found throughout the API, for example:

• System operations and infos
• System groups
• Maintenance windows
• Ansible
• Reporting
• ...

At the end of the week I managed to enable basic system group operations:

• List all system groups visible to the user
• Create new system groups
• List systems assigned to a group
• Add and remove systems from groups

Goals

• Set up test environment locally with the MCP server and client + a recent MLM server [DONE]
• Identify features and use cases offering a benefit with limited effort required for enablement [DONE]
• Create a PR to the repo [DONE]

Resources

• MCP Server Uyuni
• MLM API Docs

---

Uyuni Saltboot rework by oholecek

Description

When Uyuni switched over to the containerized proxies we had to abandon salt based saltboot infrastructure we had before. Uyuni already had integration with a Cobbler provisioning server and saltboot infra was re-implemented on top of this Cobbler integration.

What was not obvious from the start was that Cobbler, having all it's features, woefully slow when dealing with saltboot size environments. We did some improvements in performance, introduced transactions, and generally tried to make this setup usable. However the underlying slowness remained.

Goals

This project is not something trying to invent new things, it is just finally implementing saltboot infrastructure directly with the Uyuni server core.

Instead of generating grub and pxelinux configurations by Cobbler for all thousands of systems and branches, we will provide a GET access point to retrieve grub or pxelinux file during the boot:

/saltboot/group/grub/$fqdn and similar for systems /saltboot/system/grub/$mac

Next we adapt our tftpd translator to query these points when asked for default or mac based config.

Lastly similar thing needs to be done on our apache server when HTTP UEFI boot is used.

Resources

---

Ansible to Salt integration by vizhestkov

Description

We already have initial integration of Ansible in Salt with the possibility to run playbooks from the salt-master on the salt-minion used as an Ansible Control node.

In this project I want to check if it possible to make Ansible working on the transport of Salt. Basically run playbooks with Ansible through existing established Salt (ZeroMQ) transport and not using ssh at all.

It could be a good solution for the end users to reuse Ansible playbooks or run Ansible modules they got used to with no effort of complex configuration with existing Salt (or Uyuni/SUSE Multi Linux Manager) infrastructure.

Goals

• [v] Prepare the testing environment with Salt and Ansible installed
• [v] Discover Ansible codebase to figure out possible ways of integration
• [v] Create Salt/Uyuni inventory module
• [v] Make basic modules to work with no using separate ssh connection, but reusing existing Salt connection
• [v] Test some most basic playbooks

Resources

GitHub page

Video of the demo

---

Uyuni Health-check Grafana AI Troubleshooter by ygutierrez

Description

This project explores the feasibility of using the open-source Grafana LLM plugin to enhance the Uyuni Health-check tool with LLM capabilities. The idea is to integrate a chat-based "AI Troubleshooter" directly into existing dashboards, allowing users to ask natural-language questions about errors, anomalies, or performance issues.

Goals

• Investigate if and how the grafana-llm-app plug-in can be used within the Uyuni Health-check tool.
• Investigate if this plug-in can be used to query LLMs for troubleshooting scenarios.
• Evaluate support for local LLMs and external APIs through the plugin.
• Evaluate if and how the Uyuni MCP server could be integrated as another source of information.

Resources

Grafana LMM plug-in

Uyuni Health-check

---

Extended private brain - RAG my own scripts and data into offline LLM AI by tjyrinki_suse

Description

For purely studying purposes, I'd like to find out if I could teach an LLM some of my own accumulated knowledge, to use it as a sort of extended brain.

I might use qwen3-coder or something similar as a starting point.

Everything would be done 100% offline without network available to the container, since I prefer to see when network is needed, and make it so it's never needed (other than initial downloads).

Goals

1. Learn something about RAG, LLM, AI.
2. Find out if everything works offline as intended.
3. As an end result have a new way to access my own existing know-how, but so that I can query the wisdom in them.
4. Be flexible to pivot in any direction, as long as there are new things learned.

Resources

To be found on the fly.

Timeline

Day 1 (of 4)

• Tried out a RAG demo, expanded on feeding it my own data
• Experimented with qwen3-coder to add a persistent chat functionality, and keeping vectors in a pickle file
• Optimizations to keep everything within context window
• Learn and add a bit of PyTest

Day 2

• More experimenting and more data
• Study ChromaDB
• Add a Web UI that works from another computer even though the container sees network is down

Day 3

• The above RAG is working well enough for demonstration purposes.
• Pivot to trying out OpenCode, configuring local Ollama qwen3-coder there, to analyze the RAG demo.
• Figured out how to configure Ollama template to be usable under OpenCode. OpenCode locally is super slow to just running qwen3-coder alone.

Day 4 (final day)

• Battle with OpenCode that was both slow and kept on piling up broken things.
• Call it success as after all the agentic AI was working locally.
• Clean up the mess left behind a bit.

Blog Post

Summarized the findings at blog post.

---

Explore LLM evaluation metrics by thbertoldi

Description

Learn the best practices for evaluating LLM performance with an open-source framework such as DeepEval.

Goals

Curate the knowledge learned during practice and present it to colleagues.

-> Maybe publish a blog post on SUSE's blog?

Resources

https://deepeval.com

https://docs.pactflow.io/docs/bi-directional-contract-testing

---

Bugzilla goes AI - Phase 1 by nwalter

Description

This project, Bugzilla goes AI, aims to boost developer productivity by creating an autonomous AI bug agent during Hackweek. The primary goal is to reduce the time employees spend triaging bugs by integrating Ollama to summarize issues, recommend next steps, and push focused daily reports to a Web Interface.

Goals

To reduce employee time spent on Bugzilla by implementing an AI tool that triages and summarizes bug reports, providing actionable recommendations to the team via Web Interface.

Project Charter

Bugzilla goes AI Phase 1

Description

Project Achievements during Hackweek

In this file you can read about what we achieved during Hackweek.

Project Achievements

---

"what is it" file and directory analysis via MCP and local LLM, for console and KDE by rsimai

Description

Users sometimes wonder what files or directories they find on their local PC are good for. If they can't determine from the filename or metadata, there should an easy way to quickly analyze the content and at least guess the meaning. An LLM could help with that, through the use of a filesystem MCP and to-text-converters for typical file types. Ideally this is integrated into the desktop environment but works as well from a console. All data is processed locally or "on premise", no artifacts remain or leave the system.

Goals

• The user can run a command from the console, to check on a file or directory
• The filemanager contains the "analyze" feature within the context menu
• The local LLM could serve for other use cases where privacy matters

TBD

• Find or write capable one-shot and interactive MCP client
• Find or write simple+secure file access MCP server
• Create local LLM service with appropriate footprint, containerized
• Shell command with options
• KDE integration (Dolphin)
• Package
• Document

Resources

---

Song Search with CLAP by gcolangiuli

Description

Contrastive Language-Audio Pretraining (CLAP) is an open-source library that enables the training of a neural network on both Audio and Text descriptions, making it possible to search for Audio using a Text input. Several pre-trained models for song search are already available on huggingface

Goals

Evaluate how CLAP can be used for song searching and determine which types of queries yield the best results by developing a Minimum Viable Product (MVP) in Python. Based on the results of this MVP, future steps could include:

• Music Tagging;
• Free text search;
• Integration with an LLM (for example, with MCP or the OpenAI API) for music suggestions based on your own library.

The code for this project will be entirely written using AI to better explore and demonstrate AI capabilities.

Result

In this MVP we implemented:

• Async Song Analysis with Clap model
• Free Text Search of the songs
• Similar song search based on vector representation
• Containerised version with web interface

We also documented what went well and what can be improved in the use of AI.

You can have a look at the result here:

• Result Presentation
• hw25-song-search Github repo

Future implementation can be related to performance improvement and stability of the analysis.

References

• CLAP: The main model being researched;
• huggingface: Pre-trained models for CLAP;
• Free Music Archive: Creative Commons songs that can be used for testing;

---

Try out Neovim Plugins supporting AI Providers by enavarro_suse

Description

Experiment with several Neovim plugins that integrate AI model providers such as Gemini and Ollama.

Goals

Evaluate how these plugins enhance the development workflow, how they differ in capabilities, and how smoothly they integrate into Neovim for day-to-day coding tasks.

Resources

• Neovim 0.11.5
• AI-enabled Neovim plugins:
  • avante.nvim: https://github.com/yetone/avante.nvim
  • Gp.nvim: https://github.com/Robitx/gp.nvim
  • parrot.nvim: https://github.com/frankroeder/parrot.nvim
  • gemini.nvim: https://dotfyle.com/plugins/kiddos/gemini.nvim
  • ...
• Accounts or API keys for AI model providers.
• Local model serving setup (e.g., Ollama)
• Test projects or codebases for practical evaluation:
  • OBS: https://build.opensuse.org/
  • OBS blog and landing page: https://openbuildservice.org/
  • ...

---

Bring to Cockpit + System Roles capabilities from YAST by miguelpc

Bring to Cockpit + System Roles features from YAST

Cockpit and System Roles have been added to SLES 16 There are several capabilities in YAST that are not yet present in Cockpit and System Roles We will follow the principle of "automate first, UI later" being System Roles the automation component and Cockpit the UI one.

Goals

The idea is to implement service configuration in System Roles and then add an UI to manage these in Cockpit. For some capabilities it will be required to have an specific Cockpit Module as they will interact with a reasource already configured.

Resources

A plan on capabilities missing and suggested implementation is available here: https://docs.google.com/spreadsheets/d/1ZhX-Ip9MKJNeKSYV3bSZG4Qc5giuY7XSV0U61Ecu9lo/edit

Linux System Roles:

• https://linux-system-roles.github.io/
• https://build.opensuse.org/package/show/openSUSE:Factory/ansible-linux-system-roles Package on sle16 ansible-linux-system-roles

First meeting Hackweek catchup

• Monday, December 1 · 11:00 – 12:00
• Time zone: Europe/Madrid
• Google Meet link: https://meet.google.com/rrc-kqch-hca

---

Song Search with CLAP by gcolangiuli

Description

Contrastive Language-Audio Pretraining (CLAP) is an open-source library that enables the training of a neural network on both Audio and Text descriptions, making it possible to search for Audio using a Text input. Several pre-trained models for song search are already available on huggingface

Goals

Evaluate how CLAP can be used for song searching and determine which types of queries yield the best results by developing a Minimum Viable Product (MVP) in Python. Based on the results of this MVP, future steps could include:

• Music Tagging;
• Free text search;
• Integration with an LLM (for example, with MCP or the OpenAI API) for music suggestions based on your own library.

The code for this project will be entirely written using AI to better explore and demonstrate AI capabilities.

Result

In this MVP we implemented:

• Async Song Analysis with Clap model
• Free Text Search of the songs
• Similar song search based on vector representation
• Containerised version with web interface

We also documented what went well and what can be improved in the use of AI.

You can have a look at the result here:

• Result Presentation
• hw25-song-search Github repo

Future implementation can be related to performance improvement and stability of the analysis.

References

• CLAP: The main model being researched;
• huggingface: Pre-trained models for CLAP;
• Free Music Archive: Creative Commons songs that can be used for testing;

---

Testing and adding GNU/Linux distributions on Uyuni by juliogonzalezgil

Join the Gitter channel! https://gitter.im/uyuni-project/hackweek

Uyuni is a configuration and infrastructure management tool that saves you time and headaches when you have to manage and update tens, hundreds or even thousands of machines. It also manages configuration, can run audits, build image containers, monitor and much more!

Currently there are a few distributions that are completely untested on Uyuni or SUSE Manager (AFAIK) or just not tested since a long time, and could be interesting knowing how hard would be working with them and, if possible, fix whatever is broken.

For newcomers, the easiest distributions are those based on DEB or RPM packages. Distributions with other package formats are doable, but will require adapting the Python and Java code to be able to sync and analyze such packages (and if salt does not support those packages, it will need changes as well). So if you want a distribution with other packages, make sure you are comfortable handling such changes.

No developer experience? No worries! We had non-developers contributors in the past, and we are ready to help as long as you are willing to learn. If you don't want to code at all, you can also help us preparing the documentation after someone else has the initial code ready, or you could also help with testing :-)

The idea is testing Salt (including bootstrapping with bootstrap script) and Salt-ssh clients

To consider that a distribution has basic support, we should cover at least (points 3-6 are to be tested for both salt minions and salt ssh minions):

1. Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file)
2. Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
3. Package management (install, remove, update...)
4. Patching
5. Applying any basic salt state (including a formula)
6. Salt remote commands
7. Bonus point: Java part for product identification, and monitoring enablement
8. Bonus point: sumaform enablement (https://github.com/uyuni-project/sumaform)
9. Bonus point: Documentation (https://github.com/uyuni-project/uyuni-docs)
10. Bonus point: testsuite enablement (https://github.com/uyuni-project/uyuni/tree/master/testsuite)

If something is breaking: we can try to fix it, but the main idea is research how supported it is right now. Beyond that it's up to each project member how much to hack :-)

• If you don't have knowledge about some of the steps: ask the team
• If you still don't know what to do: switch to another distribution and keep testing.

This card is for EVERYONE, not just developers. Seriously! We had people from other teams helping that were not developers, and added support for Debian and new SUSE Linux Enterprise and openSUSE Leap versions :-)

In progress/done for Hack Week 25

Guide

We started writin a Guide: Adding a new client GNU Linux distribution to Uyuni at https://github.com/uyuni-project/uyuni/wiki/Guide:-Adding-a-new-client-GNU-Linux-distribution-to-Uyuni, to make things easier for everyone, specially those not too familiar wht Uyuni or not technical.

openSUSE Leap 16.0

The distribution will all love!

https://en.opensuse.org/openSUSE:Roadmap#DRAFTScheduleforLeap16.0

Curent Status We started last year, it's complete now for Hack Week 25! :-D

• [W] Reposync (this will require using spacewalk-common-channels and adding channels to the .ini file) NOTE: Done, client tools for SLMicro6 are using as those for SLE16.0/openSUSE Leap 16.0 are not available yet
• [W] Onboarding (salt minion from UI, salt minion from bootstrap scritp, and salt-ssh minion) (this will probably require adding OS to the bootstrap repository creator)
• [W] Package management (install, remove, update...). Works, even reboot requirement detection

---

Collection and organisation of information about Bulgarian schools by iivanov

Description

To achieve this it will be necessary:

• Collect/download raw data from various government and non-governmental organizations
• Clean up raw data and organise it in some kind database.
• Create tool to make queries easy.
• Or perhaps dump all data into AI and ask questions in natural language.

Goals

By selecting particular school information like this will be provided:

• School scores on national exams.
• School scores from the external evaluations exams.
• School town, municipality and region.
• Employment rate in a town or municipality.
• Average health of the population in the region.

Resources

Some of these are available only in bulgarian.

• https://danybon.com/klasazia
• https://nvoresults.com/index.html
• https://ri.mon.bg/active-institutions
• https://www.nsi.bg/nrnm/ekatte/archive

Results

• Information about all Bulgarian schools with their scores during recent years cleaned and organised into SQL tables
• Information about all Bulgarian villages, cities, municipalities and districts cleaned and organised into SQL tables
• Information about all Bulgarian villages and cities census since beginning of this century cleaned and organised into SQL tables.
• Information about all Bulgarian municipalities about religion, ethnicity cleaned and organised into SQL tables.
• Data successfully loaded to locally running Ollama with help to Vanna.AI
• Seems to be usable.

TODO

• Add more statistical information about municipalities and ....

Code and data

• Github

---

Enhance git-sha-verify: A tool to checkout validated git hashes by gpathak

Description

git-sha-verify is a simple shell utility to verify and checkout trusted git commits signed using GPG key. This tool helps ensure that only authorized or validated commit hashes are checked out from a git repository, supporting better code integrity and security within the workflow.

Supports:

• Verifying commit authenticity signed using gpg key
  
• Checking out trusted commits
  

Ideal for teams and projects where the integrity of git history is crucial.

Goals

A minimal python code of the shell script exists as a pull request.

The goal of this hackweek is to:

• DONE: Add more unit tests
  • New and more tests can be added later
    
• Partially DONE: Make the python code modular
  
• DONE: Add code coverage if possible
  

Resources

• Link to GitHub Repository: https://github.com/openSUSE/git-sha-verify

---

Flaky Tests AI Finder for Uyuni and MLM Test Suites by oscar-barrios

Description

Our current Grafana dashboards provide a great overview of test suite health, including a panel for "Top failed tests." However, identifying which of these failures are due to legitimate bugs versus intermittent "flaky tests" is a manual, time-consuming process. These flaky tests erode trust in our test suites and slow down development.

This project aims to build a simple but powerful Python script that automates flaky test detection. The script will directly query our Prometheus instance for the historical data of each failed test, using the jenkins_build_test_case_failure_age metric. It will then format this data and send it to the Gemini API with a carefully crafted prompt, asking it to identify which tests show a flaky pattern.

The final output will be a clean JSON list of the most probable flaky tests, which can then be used to populate a new "Top Flaky Tests" panel in our existing Grafana test suite dashboard.

Goals

By the end of Hack Week, we aim to have a single, working Python script that:

1. Connects to Prometheus and executes a query to fetch detailed test failure history.
2. Processes the raw data into a format suitable for the Gemini API.
3. Successfully calls the Gemini API with the data and a clear prompt.
4. Parses the AI's response to extract a simple list of flaky tests.
5. Saves the list to a JSON file that can be displayed in Grafana.
6. New panel in our Dashboard listing the Flaky tests

Resources

• Jenkins Prometheus Exporter: https://github.com/uyuni-project/jenkins-exporter/
• Data Source: Our internal Prometheus server.
• Key Metric: jenkins_build_test_case_failure_age{jobname, buildid, suite, case, status, failedsince}.
• Existing Query for Reference: count by (suite) (max_over_time(jenkins_build_test_case_failure_age{status=~"FAILED|REGRESSION", jobname="$jobname"}[$__range])).
• AI Model: The Google Gemini API.
• Example about how to interact with Gemini API: https://github.com/srbarrios/FailTale/
• Visualization: Our internal Grafana Dashboard.
• Internal IaC: https://gitlab.suse.de/galaxy/infrastructure/-/tree/master/srv/salt/monitoring

Outcome

• Jenkins Flaky Test Detector: https://github.com/srbarrios/jenkins-flaky-tests-detector and its container
• IaC on MLM Team: https://gitlab.suse.de/galaxy/infrastructure/-/tree/master/srv/salt/monitoring/jenkinsflakytestsdetector?reftype=heads, https://gitlab.suse.de/galaxy/infrastructure/-/blob/master/srv/salt/monitoring/grafana/dashboards/flaky-tests.json?ref_type=heads, and others.
• Grafana Dashboard: https://grafana.mgr.suse.de/d/flaky-tests/flaky-tests-detection @ @ text

---

Local AI assistant with optional integrations and mobile companion by livdywan

Description

Setup a local AI assistant for research, brainstorming and proof reading. Look into SurfSense, Open WebUI and possibly alternatives. Explore integration with services like openQA. There should be no cloud dependencies. Mobile phone support or an additional companion app would be a bonus. The goal is not to develop everything from scratch.

User Story

• Allison Average wants a one-click local AI assistent on their openSUSE laptop.
• Ash Awesome wants AI on their phone without an expensive subscription.

Goals

• Evaluate a local SurfSense setup for day to day productivity
• Test opencode for vibe coding and tool calling

Timeline

Day 1

• Took a look at SurfSense and started setting up a local instance.
• Unfortunately the container setup did not work well. Tho this was a great opportunity to learn some new podman commands and refresh my memory on how to recover a corrupted btrfs filesystem.

Day 2

• Due to its sheer size and complexity SurfSense seems to have triggered btrfs fragmentation. Naturally this was not visible in any podman-related errors or in the journal. So this took up much of my second day.

Day 3

• Trying out opencode with Qwen3-Coder and Qwen2.5-Coder.

Day 4

• Context size is a thing, and models are not equally usable for vibe coding.
• Through arduous browsing for ollama models I did find some like myaniu/qwen2.5-1m:7b with 1m but even then it is not obvious if they are meant for tool calls.

Day 5

• Whilst trying to make opencode usable I discovered ramalama which worked instantly and very well.

Outcomes

surfsense

I could not easily set this up completely. Maybe in part due to my filesystem issues. Was expecting this to be less of an effort.

opencode

Installing opencode and ollama in my distrobox container along with the following configs worked for me.

When preparing a new project from scratch it is a good idea to start out with a template.

opencode.json

``` {

---

Is SUSE Trending? Popularity and Developer Sentiment Insight Using Native AI Capabilities by terezacerna

Description

This project aims to explore the popularity and developer sentiment around SUSE and its technologies compared to Red Hat and their technologies. Using publicly available data sources, I will analyze search trends, developer preferences, repository activity, and media presence. The final outcome will be an interactive Power BI dashboard that provides insights into how SUSE is perceived and discussed across the web and among developers.

Goals

1. Assess the popularity of SUSE products and brand compared to Red Hat using Google Trends.
2. Analyze developer satisfaction and usage trends from the Stack Overflow Developer Survey.
3. Use the GitHub API to compare SUSE and Red Hat repositories in terms of stars, forks, contributors, and issue activity.
4. Perform sentiment analysis on GitHub issue comments to measure community tone and engagement using built-in Copilot capabilities.
5. Perform sentiment analysis on Reddit comments related to SUSE technologies using built-in Copilot capabilities.
6. Use Gnews.io to track and compare the volume of news articles mentioning SUSE and Red Hat technologies.
7. Test the integration of Copilot (AI) within Power BI for enhanced data analysis and visualization.
8. Deliver a comprehensive Power BI report summarizing findings and insights.
9. Test the full potential of Power BI, including its AI features and native language Q&A.

Resources

1. Google Trends: Web scraping for search popularity data
2. Stack Overflow Developer Survey: For technology popularity and satisfaction comparison
3. GitHub API: For repository data (stars, forks, contributors, issues, comments).
4. Gnews.io API: For article volume and mentions analysis.
5. Reddit: SUSE related topics with comments.

---

Song Search with CLAP by gcolangiuli

Description

Contrastive Language-Audio Pretraining (CLAP) is an open-source library that enables the training of a neural network on both Audio and Text descriptions, making it possible to search for Audio using a Text input. Several pre-trained models for song search are already available on huggingface

Goals

Evaluate how CLAP can be used for song searching and determine which types of queries yield the best results by developing a Minimum Viable Product (MVP) in Python. Based on the results of this MVP, future steps could include:

• Music Tagging;
• Free text search;
• Integration with an LLM (for example, with MCP or the OpenAI API) for music suggestions based on your own library.

The code for this project will be entirely written using AI to better explore and demonstrate AI capabilities.

Result

In this MVP we implemented:

• Async Song Analysis with Clap model
• Free Text Search of the songs
• Similar song search based on vector representation
• Containerised version with web interface

We also documented what went well and what can be improved in the use of AI.

You can have a look at the result here:

• Result Presentation
• hw25-song-search Github repo

Future implementation can be related to performance improvement and stability of the analysis.

References

• CLAP: The main model being researched;
• huggingface: Pre-trained models for CLAP;
• Free Music Archive: Creative Commons songs that can be used for testing;

---

Self-Scaling LLM Infrastructure Powered by Rancher by ademicev0

Self-Scaling LLM Infrastructure Powered by Rancher

---

Description

The Problem

Running LLMs can get expensive and complex pretty quickly.

Today there are typically two choices:

1. Use cloud APIs like OpenAI or Anthropic. Easy to start with, but costs add up at scale.
2. Self-host everything - set up Kubernetes, figure out GPU scheduling, handle scaling, manage model serving... it's a lot of work.

What if there was a middle ground?

What if infrastructure scaled itself instead of making you scale it?

Can we use existing Rancher capabilities like CAPI, autoscaling, and GitOps to make this simpler instead of building everything from scratch?

Project Repository: github.com/alexander-demicev/llmserverless

---

What This Project Does

A key feature is hybrid deployment: requests can be routed based on complexity or privacy needs. Simple or low-sensitivity queries can use public APIs (like OpenAI), while complex or private requests are handled in-house on local infrastructure. This flexibility allows balancing cost, privacy, and performance - using cloud for routine tasks and on-premises resources for sensitive or demanding workloads.

A complete, self-scaling LLM infrastructure that:

• Scales to zero when idle (no idle costs)
• Scales up automatically when requests come in
• Adds more nodes when needed, removes them when demand drops
• Runs on any infrastructure - laptop, bare metal, or cloud

Think of it as "serverless for LLMs" - focus on building, the infrastructure handles itself.

How It Works

A combination of open source tools working together:

Flow:

• Users interact with OpenWebUI (chat interface)
• Requests go to LiteLLM Gateway
• LiteLLM routes requests to:
  • Ollama (Knative) for local model inference (auto-scales pods)
  • Or cloud APIs for fallback

---