Comments

• 

  5 days ago by ancorgs | Reply

  Time for some reporting.

  Both @joseivanlopez and me have been doing experiments with AI and the unit tests of Agama's web interface (Javascript + React).

  Probably @joseivanlopez will write in more detail about his experience. But this report is about some common experiments we both did using different AI solutions. Let's start with some context.

  There is a branch api-v2 in the Agama repository that dramatically changes how the web UI interacts with the backend. The code already works but the javascript unit tests are not adapted accordingly yet. The main idea was to simplify the process of adapting those unit tests with the help of AI.

  @joseivanlopez did it using the company-provided Gemini, this pull request shows some partial results. Gemini was able to adapt several tests. Although it would be more accurate to say that it rewrote the tests. It feels like it ignored the current unit tests and wrote another ones from scratch. Those generated unit tests are indeed useful, they cover many scenarios and look quite sane, although some of them are not very semantic.

  Gemini was not blazing fast (it took 10+ minutes to adapt a single test) and sometimes it struggled to find its way (felt like a pure trial and error process). But the outcome is certainly useful. The experiment can be labeled as a relative success.

  But all that applies only to the gemini-pro model. Sadly it looks like the SUSE-provided license provides a very limited number of tokens to be spend on gemini-pro. After spending those in adapting 4 or 5 unit tests, everything fall backs to the useless gemini-flash model. That means only a few tests per developer can be adapted every day.

  In parallel I ran a very similar experiment but using Claude.ai, an AI solution that is not endorsed by SUSE, so we cannot use it for production code. I used the completely free version that only provides access to a web console so I had to upload many source-code files manually) and that only allows a few queries to their intermediate model (using it for longer or accessing the advanced model would have implied a fee).

  Even with all those limitations, I feel the experiment was clearly more successful than the Gemini one. You can see some partial results in this pull request.

  When asked to adapt existing unit tests, Claude really did all the necessary changes to get them running again, but without rewriting everything. Sometimes it added a missing scenario, but it respected the approach of the existing tests and scenarios. When asked to write a new test from scratch, it apparently produced a quite comprehensive and semantic unit test. Claude really felt like a tool that could potentially save a lot of manual work in a reliable way.

  Compared to Gemini, Claude was way faster and straight to the point. It was able to produce good results in seconds without really having access to my development environment. Gemini seemed to work a bit more by trial and error, with several iterations of adjusting things to then run the tests and adjust things again.

×
Edit Comment 11784
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
Time for some reporting. Both [[[[[[@joseivanlopez](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez) and me have been doing experiments with AI and the unit tests of Agama's web interface (Javascript + React). Probably [[[[[[@joseivanlopez](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez) will write in more detail about his experience. But this report is about some common experiments we both did using different AI solutions. Let's start with some context. There is a branch `api-v2` in the Agama repository that dramatically changes how the web UI interacts with the backend. The code already works but the javascript unit tests are not adapted accordingly yet. The main idea was to simplify the process of adapting those unit tests with the help of AI. [[[[[[@joseivanlopez](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez)](/users/joseivanlopez) did it using the company-provided Gemini, [this pull request](https://github.com/agama-project/agama/pull/2927) shows some partial results. Gemini was able to adapt several tests. Although it would be more accurate to say that it rewrote the tests. It feels like it ignored the current unit tests and wrote another ones from scratch. Those generated unit tests are indeed useful, they cover many scenarios and look quite sane, although some of them are not very semantic. Gemini was not blazing fast (it took 10+ minutes to adapt a single test) and sometimes it struggled to find its way (felt like a pure trial and error process). But the outcome is certainly useful. **The experiment can be labeled as a relative success**. But all that applies only to the `gemini-pro` model. Sadly it looks like the SUSE-provided license provides a very limited number of tokens to be spend on `gemini-pro`. After spending those in adapting 4 or 5 unit tests, everything fall backs to the useless `gemini-flash` model. That means only a few tests per developer can be adapted every day. In parallel I ran a very similar experiment but using Claude.ai, an AI solution that is not endorsed by SUSE, so we cannot use it for production code. I used the completely free version that only provides access to a web console so I had to upload many source-code files manually) and that only allows a few queries to their intermediate model (using it for longer or accessing the advanced model would have implied a fee). Even with all those limitations, I feel the experiment was **clearly more successful than the Gemini one**. You can see some partial results in [this pull request](https://github.com/agama-project/agama/pull/2922). When asked to adapt existing unit tests, Claude really did all the necessary changes to get them running again, but without rewriting everything. Sometimes it added a missing scenario, but it respected the approach of the existing tests and scenarios. When asked to write a new test from scratch, it apparently produced a quite comprehensive and semantic unit test. Claude really felt like a tool that could potentially save a lot of manual work in a reliable way. Compared to Gemini, Claude was way faster and straight to the point. It was able to produce good results in seconds without really having access to my development environment. Gemini seemed to work a bit more by trial and error, with several iterations of adjusting things to then run the tests and adjust things again.
 
×
Reply to ancorgs
Time for some reporting.
Both @joseivanlopez and me have been doing experiments with AI and the unit tests of Agama's web interface (Javascript + React).
Probably @joseivanlopez will write in more detail about his experience. But this report is about some common experiments we both did using different AI solutions. Let's start with some context.
There is a branch api-v2 in the Agama repository that dramatically changes how the web UI interacts with the backend. The code already works but the javascript unit tests are not adapted accordingly yet. The main idea was to simplify the process of adapting those unit tests with the help of AI.
@joseivanlopez did it using the company-provided Gemini, this pull request shows some partial results. Gemini was able to adapt several tests. Although it would be more accurate to say that it rewrote the tests. It feels like it ignored the current unit tests and wrote another ones from scratch. Those generated unit tests are indeed useful, they cover many scenarios and look quite sane, although some of them are not very semantic.
Gemini was not blazing fast (it took 10+ minutes to adapt a single test) and sometimes it struggled to find its way (felt like a pure trial and error process). But the outcome is certainly useful. The experiment can be labeled as a relative success.
But all that applies only to the gemini-pro model. Sadly it looks like the SUSE-provided license provides a very limited number of tokens to be spend on gemini-pro. After spending those in adapting 4 or 5 unit tests, everything fall backs to the useless gemini-flash model. That means only a few tests per developer can be adapted every day.
In parallel I ran a very similar experiment but using Claude.ai, an AI solution that is not endorsed by SUSE, so we cannot use it for production code. I used the completely free version that only provides access to a web console so I had to upload many source-code files manually) and that only allows a few queries to their intermediate model (using it for longer or accessing the advanced model would have implied a fee).
Even with all those limitations, I feel the experiment was clearly more successful than the Gemini one. You can see some partial results in this pull request.
When asked to adapt existing unit tests, Claude really did all the necessary changes to get them running again, but without rewriting everything. Sometimes it added a missing scenario, but it respected the approach of the existing tests and scenarios. When asked to write a new test from scratch, it apparently produced a quite comprehensive and semantic unit test. Claude really felt like a tool that could potentially save a lot of manual work in a reliable way.
Compared to Gemini, Claude was way faster and straight to the point. It was able to produce good results in seconds without really having access to my development environment. Gemini seemed to work a bit more by trial and error, with several iterations of adjusting things to then run the tests and adjust things again.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  4 days ago by joseivanlopez | Reply

  AI Experiment Report: Gemini-CLI for Agama Unit Test Automation

  This report summarizes the results of an experiment using the gemini-cli tool (powered by the Gemini Pro model) to automatically update outdated React unit tests in the Agama UI codebase.

  Scenario & Goal

  The Agama UI code was adapted to use a new HTTP API, leaving existing unit tests broken and outdated. The goal was to use gemini-cli to automatically fix and adapt these broken React unit tests.

  • Tool: gemini-cli
  • Model: Gemini Pro
  • Example Prompt: "Fix tests from [@src](/users/src)/components/storage/PartitionPage.test.tsx"

  Key Results and Observations

  Success and Capability

  • High Adaptation Rate: The AI demonstrated its capability to adapt a significant number of existing React tests to the new API structure and component logic. (See results: https://github.com/agama-project/agama/pull/2927)
  • Actionable Output: The output was often directly usable, requiring minimal manual cleanup or correction.

  Performance and Efficiency Challenges

  • Speed/Time: The process was very slow. Adapting a single test suite typically took around 15 minutes. This time investment sometimes approaches or exceeds the time a developer might take to fix the tests manually, impacting developer workflow adoption.
  • Reliability: The process was unstable and sometimes stalled completely. This requires developer intervention (canceling the request and resubmitting) to complete the task.
  • Strategy: The model appeared to operate in a "try/error" mode (iterative guessing based on error messages) rather than demonstrating a deep comprehension of the code. This trial-and-error approach contributes directly to the poor performance and high latency observed.

  Conclusion

  Based on the experiment's results, while the Gemini Pro model currently exhibits significant performance issues (slowness and stalling) that make large-scale, automated fixes impractical, it demonstrates core capabilities that point to its potential value in specific scenarios within the Agama project.

  Creating Tests From Scratch

  Gemini is highly useful for generating the initial boilerplate and structure for new unit tests. A developer shouldn't spend time setting up mocks, imports, and basic assertion structures for a new component. The AI can quickly create a functional test file based solely on the component's public interface. This dramatically lowers the barrier to writing new tests and speeds up the initial development phase, turning test creation from a chore into a rapid scaffolding process.

  Progressive and Incremental Adaptation

  The AI is valuable for progressive adaptations as code evolves. Instead of waiting for a massive refactor that breaks hundreds of tests (creating a daunting backlog), a developer should use the AI immediately after making small, targeted changes to a component's internal logic, API, or prop structure. This strategy ensures unit tests are fixed incrementally, preventing the large backlog of broken tests that often results from major refactoring efforts.

  Resource Constraint: Token Limits

  A critical limiting factor impacting the viability of extensive AI usage is the limited token quota provided by SUSE for the Gemini Pro model. Due to the model's observed "try/error" strategy and the resulting high number of queries needed to complete a task, the tokens are consumed rapidly, typically becoming exhausted after only about two hours of intensive usage.

  This severe constraint means that even if the performance were better, continuous, large-scale automation is not possible under the current resource allocation.

  In summary, given the constraints of high latency and limited token availability, we must pivot our strategy. We should shift the focus from using the AI as a brute-force bug-fixing tool to using it as a scaffolding and incremental maintenance assistant.

×
Edit Comment 11898
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
## AI Experiment Report: Gemini-CLI for Agama Unit Test Automation This report summarizes the results of an experiment using the `gemini-cli` tool (powered by the Gemini Pro model) to automatically update outdated React unit tests in the Agama UI codebase. ### Scenario & Goal The Agama UI code was adapted to use a new HTTP API, leaving existing unit tests broken and outdated. The goal was to use `gemini-cli` to automatically fix and adapt these broken React unit tests. * Tool: `gemini-cli` * Model: Gemini Pro * Example Prompt: `"Fix tests from [@src](/users/src)/components/storage/PartitionPage.test.tsx"` ### Key Results and Observations #### Success and Capability * High Adaptation Rate: The AI demonstrated its capability to adapt a significant number of existing React tests to the new API structure and component logic. *(See results: [https://github.com/agama-project/agama/pull/2927](https://github.com/agama-project/agama/pull/2927))* * Actionable Output: The output was often directly usable, requiring minimal manual cleanup or correction. #### Performance and Efficiency Challenges * Speed/Time: The process was **very slow**. Adapting a single test suite typically took around 15 minutes. This time investment sometimes approaches or exceeds the time a developer might take to fix the tests manually, impacting developer workflow adoption. * Reliability: The process was **unstable** and sometimes stalled completely. This requires developer intervention (canceling the request and resubmitting) to complete the task. * Strategy: The model appeared to operate in a **"try/error"** mode (iterative guessing based on error messages) rather than demonstrating a deep comprehension of the code. This trial-and-error approach contributes directly to the poor performance and high latency observed. ### Conclusion Based on the experiment's results, while the Gemini Pro model currently exhibits significant performance issues (slowness and stalling) that make large-scale, automated fixes impractical, it demonstrates core capabilities that point to its potential value in specific scenarios within the Agama project. #### Creating Tests From Scratch Gemini is highly useful for generating the initial boilerplate and structure for new unit tests. A developer shouldn't spend time setting up mocks, imports, and basic assertion structures for a new component. The AI can quickly create a functional test file based solely on the component's public interface. This dramatically lowers the barrier to writing new tests and speeds up the initial development phase, turning test creation from a chore into a rapid scaffolding process. #### Progressive and Incremental Adaptation The AI is valuable for progressive adaptations as code evolves. Instead of waiting for a massive refactor that breaks hundreds of tests (creating a daunting backlog), a developer should use the AI immediately after making small, targeted changes to a component's internal logic, API, or prop structure. This strategy ensures unit tests are fixed incrementally, preventing the large backlog of broken tests that often results from major refactoring efforts. #### Resource Constraint: Token Limits A critical limiting factor impacting the viability of extensive AI usage is the limited token quota provided by SUSE for the Gemini Pro model. Due to the model's observed "try/error" strategy and the resulting high number of queries needed to complete a task, the tokens are consumed rapidly, typically becoming exhausted after only about two hours of intensive usage. This severe constraint means that even if the performance were better, continuous, large-scale automation is not possible under the current resource allocation. In summary, given the constraints of high latency and limited token availability, we must pivot our strategy. We should shift the focus from using the AI as a brute-force bug-fixing tool to using it as a scaffolding and incremental maintenance assistant.
 
×
Reply to joseivanlopez
AI Experiment Report: Gemini-CLI for Agama Unit Test Automation
This report summarizes the results of an experiment using the gemini-cli tool (powered by the Gemini Pro model) to automatically update outdated React unit tests in the Agama UI codebase.
Scenario & Goal
The Agama UI code was adapted to use a new HTTP API, leaving existing unit tests broken and outdated. The goal was to use gemini-cli to automatically fix and adapt these broken React unit tests.
• Tool: gemini-cli
• Model: Gemini Pro
• Example Prompt: "Fix tests from [[@src](/users/src)](/users/src)/components/storage/PartitionPage.test.tsx"
Key Results and Observations
Success and Capability
• High Adaptation Rate: The AI demonstrated its capability to adapt a significant number of existing React tests to the new API structure and component logic. (See results: https://github.com/agama-project/agama/pull/2927)
• Actionable Output: The output was often directly usable, requiring minimal manual cleanup or correction.
Performance and Efficiency Challenges
• Speed/Time: The process was very slow. Adapting a single test suite typically took around 15 minutes. This time investment sometimes approaches or exceeds the time a developer might take to fix the tests manually, impacting developer workflow adoption.
• Reliability: The process was unstable and sometimes stalled completely. This requires developer intervention (canceling the request and resubmitting) to complete the task.
• Strategy: The model appeared to operate in a "try/error" mode (iterative guessing based on error messages) rather than demonstrating a deep comprehension of the code. This trial-and-error approach contributes directly to the poor performance and high latency observed.
Conclusion
Based on the experiment's results, while the Gemini Pro model currently exhibits significant performance issues (slowness and stalling) that make large-scale, automated fixes impractical, it demonstrates core capabilities that point to its potential value in specific scenarios within the Agama project.
Creating Tests From Scratch
Gemini is highly useful for generating the initial boilerplate and structure for new unit tests. A developer shouldn't spend time setting up mocks, imports, and basic assertion structures for a new component. The AI can quickly create a functional test file based solely on the component's public interface. This dramatically lowers the barrier to writing new tests and speeds up the initial development phase, turning test creation from a chore into a rapid scaffolding process.
Progressive and Incremental Adaptation
The AI is valuable for progressive adaptations as code evolves. Instead of waiting for a massive refactor that breaks hundreds of tests (creating a daunting backlog), a developer should use the AI immediately after making small, targeted changes to a component's internal logic, API, or prop structure. This strategy ensures unit tests are fixed incrementally, preventing the large backlog of broken tests that often results from major refactoring efforts.
Resource Constraint: Token Limits
A critical limiting factor impacting the viability of extensive AI usage is the limited token quota provided by SUSE for the Gemini Pro model. Due to the model's observed "try/error" strategy and the resulting high number of queries needed to complete a task, the tokens are consumed rapidly, typically becoming exhausted after only about two hours of intensive usage.
This severe constraint means that even if the performance were better, continuous, large-scale automation is not possible under the current resource allocation.
In summary, given the constraints of high latency and limited token availability, we must pivot our strategy. We should shift the focus from using the AI as a brute-force bug-fixing tool to using it as a scaffolding and incremental maintenance assistant.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  4 days ago by ancorgs | Reply

  I was planning to also try copilot with some public LLM engine (eg. ChatGPT) only for completeness. Unfortunately, I got sick in Thursday and spent the whole Thursday's afternoon and bed. I don't expect it to get any better during Friday.

×
Edit Comment 12015
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
I was planning to also try copilot with some public LLM engine (eg. ChatGPT) only for completeness. Unfortunately, I got sick in Thursday and spent the whole Thursday's afternoon and bed. I don't expect it to get any better during Friday.
 
×
Reply to ancorgs
I was planning to also try copilot with some public LLM engine (eg. ChatGPT) only for completeness. Unfortunately, I got sick in Thursday and spent the whole Thursday's afternoon and bed. I don't expect it to get any better during Friday.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  4 days ago by joseivanlopez | Reply

  I also experimented with other command-line interface tools, specifically cline. The tool performed exceptionally well, offering the key advantage of enabling concurrent execution of different AI models. This allows for testing free models available through platforms like Ollama (e.g., gpt-oss or deepseek-r1). I utilized it successfully with the cloude-soonet model. However, the severe limitations of the free usage tier ultimately prevented me from conducting any meaningful or conclusive tests.

×
Edit Comment 12021
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
I also experimented with other command-line interface tools, specifically [cline](https://cline.bot). The tool performed exceptionally well, offering the key advantage of enabling concurrent execution of different AI models. This allows for testing free models available through platforms like Ollama (e.g., gpt-oss or deepseek-r1). I utilized it successfully with the cloude-soonet model. However, the severe limitations of the free usage tier ultimately prevented me from conducting any meaningful or conclusive tests.
 
×
Reply to joseivanlopez
I also experimented with other command-line interface tools, specifically cline. The tool performed exceptionally well, offering the key advantage of enabling concurrent execution of different AI models. This allows for testing free models available through platforms like Ollama (e.g., gpt-oss or deepseek-r1). I utilized it successfully with the cloude-soonet model. However, the severe limitations of the free usage tier ultimately prevented me from conducting any meaningful or conclusive tests.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  3 days ago by ancorgs | Reply

  I ran an extra experiment. Not about unit tests but about code refactoring. TBH, I didn't have time yet to analyze the result. But some of the unit tests are still green (not all of them). See this pull request

×
Edit Comment 12264
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
I ran an extra experiment. Not about unit tests but about code refactoring. TBH, I didn't have time yet to analyze the result. But some of the unit tests are still green (not all of them). See [this pull request](https://github.com/agama-project/agama/pull/2932)
 
×
Reply to ancorgs
I ran an extra experiment. Not about unit tests but about code refactoring. TBH, I didn't have time yet to analyze the result. But some of the unit tests are still green (not all of them). See this pull request
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

- Use hyphens
- for unordereed
- lists

This is an [link to example.com](http://example.com/)

This is an image ![an openSUSE geeko icon](https://en.opensuse.org/images/d/d0/Icon-distribution.png)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview