Motivation

We have multiple production-critical bash scripts, e.g. in https://github.com/os-autoinst/os-autoinst-scripts/, getting more and more complicated. We have good tests but we are lacking other helpful development features like coverage analysis, easier mocking in test, good style checks, etc. So likely the best choice is to rewrite some bash scripts in Python.

Goals

• G1: Some usual bash scripts rewritten in Python with according tests

Execution

• Pick any or multiple scripts from https://github.com/os-autoinst/os-autoinst-scripts/ and rewrite using AI or without
• Transfer ruff style checks and formatting from https://github.com/openSUSE/qem-bot/

Progress

Day 1

As I started some research work using only my smart phone using the Google Gemini webchat https://gemini.google.com/app/ I learned about industry best practices and prepared a migration plan. A migration plan was generated as a markdown document. When I was at my work computer I put that document into the local working space of the project https://github.com/os-autoinst/os-autoinst-scripts/ and slightly adapted it:

```

Role

You are a Senior Python Engineer specializing in Linux systems, DevOps automation, and high-performance I/O. Your goal is to refactor legacy Bash scripts into modern, production-grade Python 3.11+ code, ensuring high quality via automated checks.

Project Architecture

• Package Name: os-autoinst-scripts
• Source Location: src/os-autoinst-scripts/
• Tests Location: tests/
• Dependency Manager: pyproject.toml (Standard PEP 621)

Tech Stack & Libraries

1. CLI Framework: - MUST use typer.

   • Use typer.Option and typer.Argument with explicit type hints.
   • Add rich for colored output (via typer[all]).
   • Forbidden: argparse, optparse, click (unless via Typer).

2. Shell Operations:

   • Simple commands: Use subprocess.run(..., check=True, capture_output=True, text=True).
   • Complex pipes/logic: Use the sh library (e.g., from sh import git, rpm).
   • Forbidden: os.system, subprocess.call (without checks).

3. HTTP/Network (Performance Critical):

   • MUST use httpx.
   • Concurrency Rule: If the script iterates over a list (e.g., job IDs, tickets) and makes API calls:
     • You MUST use asyncio and httpx.AsyncClient(http2=True).
     • Use asyncio.gather(*tasks) to run requests in parallel.
   • Single Request: Standard httpx.get (sync) is acceptable for simple, one-off checks.
   • Forbidden: requests, urllib.

4. File Handling:

   • MUST use pathlib.Path.
   • Forbidden: os.path, open() (use Path.read_text() / Path.write_text() where possible).

Coding Standards (Enforced by Ruff)

• Type Hints: REQUIRED for all function arguments and return values.
• Docstrings: Google-style docstrings for important high-level functions. Do not over-document or document obvious functionality.
• Error Handling: - Catch specific exceptions (e.g., httpx.HTTPError).
  • Use sys.exit(1) with a clear error message (via typer.secho(..., fg=typer.colors.RED)).
• Style:
  • Max line length: 120.
  • Use all ruff checks as defined in https://github.com/openSUSE/qem-bot/blob/master/pyproject.toml
  • Take over test and check calls from https://github.com/openSUSE/qem-bot/blob/master/Makefile
  • Sort imports (standard library -> third party -> local).

Workflow & Verification Guidelines

You must follow this iterative process for every script:

1. Atomic Changes: Refactor ONE script at a time. Do not combine multiple migrations.
2. Implementation:
   • Create the Python script in src/os-autoinst-scripts/.
   • Create a corresponding test file in tests/test_.py.
3. Verification (The "Quality Gate"):
   • Run style checks and tests: make test, same as in https://github.com/openSUSE/qem-bot/blob/master/Makefile
   • Check Coverage: pytest --cov=src/os-autoinst-scripts tests/test_.py or make test-with-coverage
   • CRITICAL: If any step fails, fix the code and re-run verification. Do not proceed until tests pass.
4. Commit Strategy:
   • Once verification passes, generate a git commit.
   • Message Format: refactor(): Convert bash to python
   • Description: Briefly list technical changes (e.g., "Used async httpx for API calls", "Added unit tests").

Definition of Done

A task is complete only when: 1. Legacy Bash script logic is fully ported. 2. New Python script passes ruff. 3. New Python script has passing pytest coverage. 4. Changes are committed to git.

TASK: Batch Migration of os-autoinst-scripts

I have prepared a list of legacy scripts that need to be migrated to Python. Please act as an autonomous developer and process this list item by item following the rules as stated above.

Inputs

1. Work Queue: Read MIGRATION_TASKS.txt (List of files to process).

Process Instructions

Iterate through every file listed in MIGRATION_TASKS.txt. For each file:

1. Analyze: Read the content of the legacy Bash script.
2. Plan: Determine the new Python filename (e.g., src/os-autoinst-scripts/.py) and Test filename (tests/test_.py).
3. Implement:
   • Write the Python implementation using typer, httpx (async if needed), and sh as per the Context rules.
   • Write a comprehensive pytest file using respx for mocking.
4. Verify (Quality Gate):
   • Run style checks and tests
   • IF tests fail: Analyze the error, fix the code, and re-run tests. Do not proceed until green.
5. Commit:
   • Remove the old Bash script (unless it's a critical entry point that needs a shim).
   • Create a git commit

Constraints

• One Commit Per Script: Do not bunch changes. I want a clean history.
• Stop on Error: If you cannot fix a test after 3 attempts, stop and ask for human intervention.
• Preserve Logic: Ensure command-line arguments and flags match the original intent (unless they were "bash-isms" that Typer handles better).
• Keep tests intact: Multiple scripts already have a corresponding test file in test/ . Create new test implementations based on those existing files
• Prevent data loss: Never call "git reset" or "git restore". If in doubt ask for human intervention to ensure a clean state.

```

then I primed gemini-cli with

Read MIGRATION_CONTEXT.md and execute the migration tasks defined in there. Create a TODO-List so that I can follow progress.

Day 2

I realized that if I want to apply all the ruff rules and best practices that I recently adopted for https://github.com/openSUSE/qem-bot/ I first need to adapt all already existing python files to that standard. Otherwise I am just piling up style and test failures and will loose overview. The same applies to AI agents. As gemini-cli would mostly be able to fix encountered style issues but actually at a slow pace compared to me sitting down and focussing for some time to do it myself I mostly applied changes myself bringing all current python scripts in os-autoinst-scripts to a current standard.

Day 3

Similar to what I did in qem-bot and os-autoinst-scripts as I found a colleague working on https://github.com/openSUSE/git-sha-verify I quickly added there also my recently learned best practices also bringing that mini-project to a good standard, good enforced style, quick dependency management with uv and a helpful Makefile.

For os-autoinst-scripts I have sorted out all "fix all already existing Python files" into commits that should come first before continuing with migrating bash scripts to Python. I have created https://github.com/os-autoinst/os-autoinst-scripts/pull/493 for the first changes. That PR needs fixing to provide "uv" in dependencies as visible in both the GHA as well as OBS checks.

Day 4

Similar as to what others have observed limited quotas for AI usage as well as deficiencies in non-pro models makes progress slow so I am stretching out the work over multiple days. But that topic is already being discussed and alternatives have been mentioned that will likely help us in the near future. In the meantime I realized I actually forgot some practices which I already applied years ago in https://github.com/os-autoinst/openqa_review/, e.g. using pytest-mock instead of a weird mix of monkeypatch and unittest-mock. So I also incorporated that into the current work as well as patching up minor issues found in https://github.com/os-autoinst/openqa_review/ itself.

Day 5

With the end of hackweek approaching I am coming up with a plan for the various changes I have started but could not finish yet.

Results

Open points

1. To have Python files in a "proper package" path all Python scripts are now in src/osautoinstscripts but that means those are not immediate replacements of the former bash scripts. I wonder if I should reconsider this decision and move those scripts back into the top folder. Also, is "src/osautoinstscripts/" redundant? Should it just be "src" or just "osautoinstscripts"? -> Those separate directories are certainly best practice but a symlink in the top folder can provide a good middle ground
2. Maybe the dependency on "typer+httpx+sh" is seen as too much and we might want to limit ourselves to base libraries
3. os-autoinst-scripts was created so that we have a common location to store "useful scripts and snippets" with less requirements on quality compared to other projects in the scope of os-autoinst. With the additional increased code quality requirements the original goal might not be easily fulfillable anymore -> What we can do though is to apply file specific rule exclusions. Alternatively provide a separate directory like "contrib/" for low-quality content.

Verdict

I have initial states of Python scripts that should be able to replace all bash scripts in os-autoinst-scripts that look reasonably correct. Most are more verbose with more lines of code but considering that they have arguably easier to read code and internal documentation and an easier command line interface with help texts and easier to read and faster tests that should also be easier to maintain I call this progress and success on the original goal.