Monasca is a powerful tool in the OpenStack arsenal. It brings metrics and alarms to the operator. It even pairs grafana with horizon. It currently monitors Swift, but just the processes. But there is so much more we can grab.
Swift actually generates heaps of metrics. For one It has supported sending metrics via statsd for years. All the internal daemons send statsd. Further, built into each server is a reconnaissance API (recon), that you can hit over REST or by using the swift-recon tool. Swift operators have been using these for years.
The Monaca agent can act as a statsd server, so really this is the perfect partnership. Question is, can I just point my swift daemons at the monasca agent statsd endpoint? Will all the stats be available in grafana via the monasca datasource?
There are also other metrics you can gather from Swift, and the monasca agent is extendable via check plugins. So how about writing a swift_recon plugin that will hit this recon api and send extra metrics back.
Finally, there is a really useful representation that a SwiftStack Swift core shared and even provided a gist of a script to the community that is close to what is needed. It's a great way of seeing the health of the cluster and when to rebalance based off monitoring how many handoff partitions storage nodes are currently holding. Let's see if we can get something similar this into monasca as well. So we can all get it for free on OpenStack deployments.
Plan, Step 1:
- setup devstack and a SAIO (a seperate SAIO as it is a fully functional swift deployment and I know it really well).
- Install the monasca agent onto the SAIO.
- Point the Swift daemons to the statsd endpoint of the monasca agent.
- See what happens.
Step 2:
- Write a swift_recon check plugin for monasca agent.
- Some useful recon metrics are timestamps, is there a way to send and just see them in monasca and grafana?
Step 3:
- Write a swift_handoffs check plugin
Looking for hackers with the skills:
Nothing? Add some keywords!
This project is part of:
Hack Week 17
Activity
Comments
-
over 6 years ago by mattoliverau | Reply
I will be proposing this hackweek project as an OpenStack summit proposal.
-
over 6 years ago by mattoliverau | Reply
I managed to complete what I set out here. Well mostly, now just need some cleanups and to push the plugins up to Monasca. I've also submitted a talk for the Berlin OpenStack summit based on this work. I've just quickly written up a blog post about it over on my blog: https://oliver.net.au/?p=358
Similar Projects
This project is one of its kind!