Rancher AI screen assistant
an invention by mfukumoto
Updated 2 days ago. No love. 1 follower.

Description

The best way of using AI is to have the proper prompt to be used. Images are usually something that can be easily understandable by the AI and improve the answer. Images of an application is common as well and it makes easier for the AI to give a good answer. Creating an AI Screen assistant, either as an extension or direct into the code. In the end it works as a shortcut to take screenshot and send to AI. The prompt will be the screenshot + a text provided, the answer should appear on the screen.

Goals

  • Create the feature in a way that can be used in Rancher specially focused for internal usage.

Resources

Some tutorial for Vue.js screenshot: https://www.digitalocean.com/community/tutorials/vuejs-screenshot-ui OpenAI API: https://platform.openai.com/docs/api-reference/introduction Gemini API: https://ai.google.dev/gemini-api/docs

Looking for hackers with the skills:

Nothing? Add some keywords!

This project is part of:

Hack Week 25

Activity

  • 4 days ago: mfukumoto started this project.
  • 10 days ago: mfukumoto originated this project.


    • Comments

    • mfukumoto
      4 days ago by mfukumoto | Reply

      Tried some interesting ways of sending the information to the AI. I could use the API Key from Gemini but I don't have access to create one. Another option is using API Key from OpenAI. But because I wanted it for a way that I can use it on the application right now, I tried to copy to the Clipboard. It copied to it with TEXT + IMAGE. Trying to paste it to Gemini it did not work. It worked on ChatGPT.

      To make in a way that can use Gemini as well I decided to merge the PROMPT together with the image, so now it copies only an IMAGE to the Clipboard, containing the PROMPT embedded to it. Using that we can easily copy the prompt to Gemini and get the specific required information.

      I created 2 prompts right now. One that is an OPS Expert to help using Rancher, another one is an UI/UX Designer to give some insights on the UI. To make it more interesting, I gave an option to take many screenshots and merge into one image only, together with the prompt as well. With that it is possible to make a continuous flow.

      So, what is done: - Take partial area screenshot - Take full screenshot - Take more than one full screenshot and embed on the same image - Add a prompt to the image that can be: - Ops Expert - UI/UX Designer - Copy it to Clipboard to be used on the AI

      Next steps would be add more prompt options, and improve the options as well. I want to add an option to write the user owns prompt, and no prompt at all for clean image. (would be good to save the option to the user)

      I added the code to use the API directly, but because of the cost, I would believe is not a good approach. If "free" or monthly cost, than I can add it back.

    • mfukumoto
      4 days ago by mfukumoto | Reply

      I added some new things. I added a shortcut to take screenshot, it should be: Ctrl + Shift + A for Partial,, and Ctrl + Shift + S for Full screen.

      Another feature is to add an option to add the user own Prompt, and can save on LocalStorage to use later. Add an option for no text as well.

      Added 3 new personas. Security Analyst Architect Developer

      Trying to get different perspectives.

      As my normal usage it would be to analyze if there are missing options or if there is correct coverage for the options. Second usage would be if the UI/UX can have some improvements or anything is missing.

    • mfukumoto
      2 days ago by mfukumoto | Reply

      Here are example of prompts, the prompts are created using the AI as well and requesting for interesting prompts.

      const OPS_PROMPT = `This screenshot belongs to Rancher. Please analyze this screen at an expert Kubernetes and Rancher Ops level. I want the best possible explanation and guidance based only on what appears in the screenshot.

      For every field, section, tab, option, toggle, or message in the screenshot, please:

      Identify what it is and its purpose inside Rancher Explain what inputs are typically expected Provide Rancher best practices for production clusters Provide Kubernetes or RKE2 expert-level recommendations Explain hidden behaviors or implications of each field Suggest secure defaults, performance optimizations, and anti-patterns to avoid If relevant, provide examples for cloud providers, on-prem deployments, node pools, registries, networking, security context, schedulers, or storage settings Identify any warnings, errors, or risk areas shown on the screenshot Describe what an Ops engineer should verify before creating or updating the cluster Give next-step actions the user should take Focus on clarity, precision, security, and real-world production operations guidance. Assume I am creating or modifying a Rancher-managed Kubernetes cluster and want the best expert explanation possible.`;

      const UX_PROMPT = `This screenshot belongs to Rancher. Please analyze the UI from the perspective of an expert product designer specializing in enterprise cloud platforms, Kubernetes dashboards, and DevOps tools. I want the best possible design, UX, and interaction analysis based only on what is visible in the screenshot.

      For every visible element, component, state, or pattern in the screenshot, please:

      Identify the UI components and their purpose within Rancher

      Identify the UI components and their purpose within Rancher Describe the visual hierarchy and whether it helps or harms clarity Analyze spacing, alignment, typography, color usage, and iconography Evaluate consistency with common Rancher design patterns Evaluate consistency with Kubernetes UI conventions Suggest improvements for readability and cognitive load Suggest layout optimizations for complex Ops workflows Identify accessibility issues (contrast, keyboard navigation, focus management, screen readers, ARIA roles) Give UX heuristics feedback (Nielsen, Shneiderman, progressive disclosure, affordance, feedback) Analyze the clarity of actions (Are the buttons, states, and labels self-explanatory?) Suggest alternative UI patterns when appropriate Highlight potential confusion for operators, SREs, or cluster admins Propose improvements for both beginner and expert workflows Suggest microcopy improvements where labels or titles are ambiguous Document any visual or interaction anti-patterns Explain how this design impacts large-scale multi-cluster operations Recommend best-in-class enterprise console design practices that Rancher could adopt Provide both quick wins and long-term UI/UX refactors Focus on enterprise Kubernetes usability and operational clarity Tailor the entire analysis toward the needs of DevOps, SREs, and cloud platform teams.

      Please be precise, clear, and deeply analytical. Assume I am evaluating the Rancher interface to improve usabillity, navigation, visual hierarchy, and operational UX for Kubernetes users.`;

      const SECURITY_PROMPT = `This screenshot belongs to Rancher. Please analyze this screen from the perspective of an expert Security Analyst focused on hardening Kubernetes environments. I want the best possible security analysis and recommendations based only on what is visible in the screenshot.

      For every field, section, tab, option, toggle, or message in the screenshot, please:

      • Identify its security implications and potential risks.
      • Recommend secure defaults and best practices (e.g., RBAC, Pod Security Policies, Network Policies, CIS benchmarks).
      • Analyze how secrets, credentials, and sensitive information are handled or exposed.
      • Assess compliance with security standards like CIS, PCI-DSS, or HIPAA if applicable from the context.
      • Highlight any UI elements that could lead a user to make an insecure choice.
      • Suggest ways to reduce the attack surface.
      • Point out any missing security features or information.
      • Describe what a security engineer should verify before applying these settings.

      Focus on identifying vulnerabilities, misconfigurations, and areas for security hardening. Assume I am responsible for the security and compliance of this Rancher-managed cluster.`;

      const DEV_PROMPT = `This screenshot belongs to Rancher. Please analyze this screen from the perspective of an Application Developer who uses Rancher to deploy and manage their applications. The developer may not be a Kubernetes expert. The goal is to evaluate the developer experience (DevEx).

      For every visible UI element, please:

      • Evaluate the simplicity and clarity of the workflow for a developer.
      • Assess the cognitive load. Is the terminology accessible, or is it heavy on infrastructure jargon?
      • Identify how the UI helps or hinders common developer tasks (e.g., deploying an app, viewing logs, scaling, setting environment variables).
      • Suggest abstractions or simplifications that would improve the developer experience.
      • Point out areas of potential confusion or friction for someone not deeply familiar with Kubernetes internals.
      • Analyze the discoverability of features a developer would need.
      • Recommend changes to make the UI more intuitive and self-service for development teams.

      Focus on developer productivity, ease of use, and reducing the need for deep infrastructure knowledge. Assume I am a developer trying to get my application running on Kubernetes using Rancher.`;

      const ARCHITECT_PROMPT = `This screenshot belongs to Rancher. Please analyze this screen from a high-level, Solutions Architect perspective. Focus on how the features shown fit into a larger enterprise ecosystem and solve broader business or architectural problems.

      For the features and configurations visible, please:

      • Analyze how they integrate with other systems (e.g., CI/CD, GitOps, monitoring, identity providers, cloud services).
      • Evaluate the strategic value of the feature for an organization. Does it enable key business outcomes?
      • Discuss the scalability, availability, and reliability implications of the choices presented.
      • Compare the functionality shown to offerings from major cloud providers (AWS, Azure, GCP) or other competitors in the ecosystem.
      • Identify key use cases, architectural patterns, and anti-patterns.
      • Discuss the total cost of ownership (TCO) or operational cost implications.
      • Explain how this part of Rancher contributes to a comprehensive platform strategy.

      Focus on the "big picture," strategic fit, and architectural best practices. Assume I am an architect designing a complete cloud-native platform for an enterprise.`;

    • mfukumoto
      2 days ago by mfukumoto | Reply

      Here is the branch of the development at my fork of rancher/dashboard. https://github.com/marcelofukumoto/dashboard/tree/hack-week-2025-screenshot

      I created a PR with the information about usage, goals, screenshots and other things: https://github.com/marcelofukumoto/dashboard/pull/3

    Similar Projects

    This project is one of its kind!