Implement a "Fuel Gauge" for Harvester
a project by mrussell
Updated almost 2 years ago. 1 hackers ♥️. 2 followers.

Project Description

As an attempt to stick with the theme of this Hack Week - the idea is that we implement a way to funnel out information (probably on an out of the box Grafana chart or something) about:

  • power draw (best effort) + maybe some averages
  • geographic location based information of what the "producers" of the power are (IE: wind, solar, natural gas, hydro-electric, etc.) while being "air-gapped" friendly

Goal for this Hackweek

To have a working Proof-of-Concept that can display information about power-related things, thus being somewhat of a "fuel gauge" for Harvester. The tie into the "hacking for humanity" aspect is that it's just another tool that offers information for the end user about power usage and what resources make up their power.
Stretch:

  • make it international, right now "a" resource for the US to fetch power information would be: https://www.epa.gov/egrid/power-profiler#/ - but international extensibility is something that would need a bit of investigation

Resources

  • leverage existing Harvester Grafana Dashboard based content, extending that
  • skills that I definitely currently lack:
    • familiarity with how to stand up Harvester customizations
    • familiarity with crafting Grafana dashboards
    • familiarity with Go (though I have a bit)

Join this project

Looking for hackers with the skills:

Nothing? Add some keywords!

This project is part of:

Hack Week 21

Activity

  • almost 2 years ago: lizhang started this project.
  • almost 2 years ago: lizhang liked this project.
  • almost 2 years ago: mrussell originated this project.


    • Comments

    • lizhang
      almost 2 years ago by lizhang | Reply

      It's interesting. But I am not familiar with Harvester or Go. I would like to try it.

    • mrussell
      almost 2 years ago by mrussell | Reply

      Hi @lizhang !

      So far I've been doing a bit of research into this, trying to understand the scope and how there might be an implementation.

      But as a primer, with Harvester to get up and running with a 'test' instance:

      Once you're logged in via navigating to the https://harvester_management_url setting a password for admin - you can under Cluster Metrics on the Harvester -> Dashboard click the link of Grafana to take you out into the front-end for the Grafana service running in the cattle-monitoring-system namespace.

      Following this:

      https://rancher.com/docs/rancher/v2.6/en/monitoring-alerting/guides/customize-grafana/

      You can log with:

      • username: admin
      • password: prom-operator

      From the harvester-installer I was able to see that there are ConfigMaps that get built for Grafana/Prometheus Dashboards here: https://github.com/harvester/harvester-installer/blob/master/pkg/config/templates/rancherd-12-monitoring-dashboard.yaml

      Now I've just been reading up a bit on Grafana/Prometheus because I'm pretty unfamiliar with it add-emoji

      I know there are a handful of different ways to get power information:

      • upower
      • powertop
      • cat /sys/class/power_supply/BAT0/voltage_now
      • or other ways to fetch information add-emoji

      I'm really curious if there are some built-ins with Grafana / Prometheus to audit power usage.

      If there was something that could be hooked into that would be great. If not this may be requiring to customize the harvester-installer not necessarily just the dashboard yaml but other places to introduce maybe an open-source power auditing tool that can funnel data to Grafana/Prometheus. Then testing those edits via building the .iso from the harvester-installer repository directly.

      Any info, insight, thoughts would be awesome add-emoji ! Let me know too if you have some questions or run into issues with getting Harvester running as a VM and accessing the Grafana/Prometheus dashboard - I'll see if I can help add-emoji

    • lizhang
      almost 2 years ago by lizhang | Reply

      Hi, @mrussell

      Thanks a lot for your suggestions. I can install harvester successfully. From my investigation, I didn't see any information about power. We need add the power metrics. It may be like the metrics in this project, https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14.

      • mrussell
        almost 2 years ago by mrussell | Reply

        add-emoji That's awesome to hear that you got it installed successfully! I'll dig into that project, thank you for providing that link add-emoji !

    • mrussell
      almost 2 years ago by mrussell | Reply

      Started to dig into a few things a bit more. Got this: https://github.com/hubblo-org/scaphandre Running locally. It's a Rust based binary that could dive & scrape for metrics and push them to Prometheus. We can run it containerized. Yet ssh'ing into the Harvester instance with rancher@ip_address and the password set up during Harvester install - I found the quickstart guide: https://hubblo-org.github.io/scaphandre-documentation/tutorials/quickstart.html with the docker run, including the --vm flag, seemed to have issues surrounding it getting up and running due to possibly linux-kerenel-module extras not existing or something for the Harvester 5.3.18.* version with uname -r - yet because Harvester's file system is read only, figuring out what package it needs in order for Scaphandre to run - we'd need to then modify the base image for Harvester: https://github.com/harvester/os2/blob/dd83d7fa88ac95ec649a17c3e26e2c00cecb7361/Dockerfile#L95-L107 Although I'm not sure with OpenSUSE / zypper how to get what Scaphandre is missing. Assuming that the missing packages (whatever those might be) needed for Scaphandre to work can be installed then we could. Then we could maybe add additionally a spec for running Scaphandre containerized in the harvester-installer here: https://github.com/harvester/harvester-installer/blob/256ecdd8385908e871484a5484f32b338020eead/pkg/config/templates/rancherd-13-monitoring.yaml

      Still wrapping my head around a few things since power metrics aren't currently working by default with things.

      • lizhang
        almost 2 years ago by lizhang | Reply

        It's cool. I will take a look at it. :)

      • lizhang
        almost 2 years ago by lizhang | Reply

        I saw the same problem with you, and I found there is a bug which is still open which looks like the same issue. https://github.com/hubblo-org/scaphandre/issues/131 . I tried to run docker on my host with SLE15SP3, there is no problem.

        • mrussell
          almost 2 years ago by mrussell | Reply

          Hi Li! I ended up digging into this a bit more today some too.

          I was able to figure out how to attach additional base packages via zypper to Harvester.

          It involves juggling a locally built Go binary of Luet, Harvester-Installer and Harvester/os2:

          For Luet: https://github.com/irishgordo/luet/tree/hack-week-power-consumption-harvester (checkout hack-week-power-consuption-harvester branch), then run make

          For Harvester/os2: https://github.com/irishgordo/os2/tree/hackweek (checkout hackweek branch, accidentally committed on dev but it's nothing permanent) Then run ./scripts/build - to build it Copy what it gets tagged as when it's done: suse-workstation-team-harvester➜ os2 : dev ✘ :✹ ᐅ docker images | grep -ie 'harvester'

          For Harvester-Installer: https://github.com/irishgordo/harvester-installer/tree/hackweek-power-consumption (checkout hackweek-power-consumption) then copy the luet binary that was built directly to the directory. Modify the BASEOSIMAGE variable's value to be what the harvester-os locally was tagged as. Then run make and it will build everything. It should build the .iso file.

          Then that .iso could be used to spin up a VM and it would have the additional packages of these for instance: https://github.com/irishgordo/os2/blob/hackweek/Dockerfile#L108-L114

          But I'm thinking it's not so much a 'package' issue with Scaphandre, but rather that the VM or at least the one I'm running with Virtual Machine Manager, isn't able to see 'host' power information.

          Locally, from the host I can see:

          suse-workstation-team-harvester➜  harvester-installer : hackweek-power-consumption ✘ :✹✭ ᐅ  tree /sys/class/powercap 
/sys/class/powercap
├── dtpm -> ../../devices/virtual/powercap/dtpm
├── intel-rapl -> ../../devices/virtual/powercap/intel-rapl
├── intel-rapl:0 -> ../../devices/virtual/powercap/intel-rapl/intel-rapl:0
├── intel-rapl:0:0 -> ../../devices/virtual/powercap/intel-rapl/intel-rapl:0/intel-rapl:0:0
├── intel-rapl:0:1 -> ../../devices/virtual/powercap/intel-rapl/intel-rapl:0/intel-rapl:0:1
├── intel-rapl:1 -> ../../devices/virtual/powercap/intel-rapl/intel-rapl:1
├── intel-rapl-mmio -> ../../devices/virtual/powercap/intel-rapl-mmio
└── intel-rapl-mmio:0 -> ../../devices/virtual/powercap/intel-rapl-mmio/intel-rapl-mmio:0

8 directories, 0 files
suse-workstation-team-harvester➜  harvester-installer : hackweek-power-consumption ✘ :✹✭ ᐅ  tree /sys/class/power_supply 
/sys/class/power_supply
├── AC -> ../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:1e/PNP0C09:00/ACPI0003:00/power_supply/AC
└── BAT0 -> ../../devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:1e/PNP0C09:00/PNP0C0A:00/power_supply/BAT0

2 directories, 0 files

          Or for instance on the Host this:

          suse-workstation-team-harvester➜  harvester-installer : hackweek-power-consumption ✘ :✹✭ ᐅ  sudo lshw -c power
[sudo] password for mike: 
  *-battery                 
       product: 5B10W13959
       vendor: Celxpert
       physical id: 1
       slot: Front
       capacity: 94000mWh
       configuration: voltage=11.5V

          But within the VM:

          rancher@testvmharvester:/tmp> sudo lshw -c power
rancher@testvmharvester:/tmp>

          I'm not seeing anything, so I'm guessing that because the VM doesn't see 'power' information from the host there might be issues. But I'm not too sure.

          • mrussell
            almost 2 years ago by mrussell | Reply

            Note: - add-emoji - the way I implemented the addition of the extra packages is far-from the "best-way" haha, of doing it, but, it is a way add-emoji haha

          • lizhang
            almost 2 years ago by lizhang | Reply

            I think this is the issue with KVM. A document stated from AWS ec2 that:
            There are different possible options [15] to read RAPL values, but from our different attempts on a c5.xlarge it seems that some MSRs are not accessible anymore on modern EC2 instances that use newer hypervisors (KVM). We didn’t spend too much time on it and directly tried our luck on bare metal instances. https://medium.com/teads-engineering/estimating-aws-ec2-instances-power-consumption-c9745e347959#8e6b It's not easy to implement it. AWS has done some experimental works according to the document. But as you mentioned, we can use powertop command. This command works in a VM. I think we can use this command to collect the power information. I am not sure if it is accurate to show the power consuming.

            • lizhang
              almost 2 years ago by lizhang | Reply

              Ah, I see someone is working on this to support RAPL for KVM. https://www.spinics.net/lists/kernel/msg4200035.html. If it works, we can try it.

        • mrussell
          almost 2 years ago by mrussell | Reply

          Sadly didn't finish all the way but we got pretty close add-emoji

          https://github.com/irishgordo/harvester-fuel-gauge-mindmap

          TY again for all the help and extra info! add-emoji

          • lizhang
            almost 2 years ago by lizhang | Reply

            We can continue to do it when we have time in the future. :)

    • lizhang
      almost 2 years ago by lizhang | Reply

      Hi Mike, @mrussell I take a look at QEMU/KVM support for Scaphandre.
      https://hubblo-org.github.io/scaphandre-documentation/how-to_guides/propagate-metrics-hypervisor-to-vm_qemu-kvm.html The VM's metrics could be read from host by using virtiofs, which is a shared filesystem between the host and the VM. It needs to reconfigure the VM, I change the VM xml: <memoryBacking> <source type='memfd'/> <access mode='shared'/> </memoryBacking>

      <filesystem type='mount' accessmode='passthrough'> <driver type='virtiofs'/> <source dir='/var/lib/libvirt/scaphandre/opensuse'/> <target dir='scaphandre'/> </filesystem>

      As it said, it needs to run "scaphandre qemu" as the agent, and run "scaphandre --vm prometheus" in the VM to export the metrics. In the VM, I think it will read the power information from the shared filesystem. On the host, the directory is in /var/lib/libvirt/scaphandre/DOMAIN_NAME, on the VM, the directory is mounted on /var/scaphandre/, I can see the file: /var/scaphandre/intel-rapl:0. I think it will read this file to get the information. But "scaphandre --vm prometheus" still has problem when I tried to run it. There should be some problem in the code. I think it's better to run a harvester server on a bare metal, and read the power information with shared filesystem from bare metal in a VM.

    Similar Projects

    This project is one of its kind!