Abstract:

• Cloud hosted applications are now composed of many microservices often deployed in containers. When things go wrong, are slow, or just not understood, where can we turn?

  Jaeger https://github.com/jaegertracing/jaeger is one such tool but requires instrumentation code to be placed in the source code prior to deployment.

  Using Linux bcc/BPF tracing tools https://github.com/iovisor/bcc and Salt, instrumentation code can be remotely injected anywhere in the cloud "on-the-fly" without requiring prior source code modification.

  Salt https://github.com/saltstack/salt is used in both the "tracer injection" and "tracer data collection" phases to facilitate Jaeger code flow tracing.

Slide Deck Presentation:

• https://docs.google.com/presentation/d/1iJqfq9_W2cBTKTTpDtgkXZqEJZgw6m355jqTfpEUetE/edit?usp=sharing

Todo:

• sluicegate --> sluicebox implemented as an actual salt returner
• Jaeger “tag” support
• regex style event filtering by sluice_gate for traces
• Implement a direct python debugger (pdb) “sluice-style” tracer
• Span, tag, reporting and collation by sluice_box
• Optimize the salt client for passing sluicegate events to sluicebox
• more, better, “jaeger friendly” BPF tracing tools