Minimum Viable Instrumentation

Welcome to Edition #51 of the newsletter!

I was lucky enough to be at this week’s European Observability Summit in Prague. Unsurprisingly, AI was the dominant theme but another important thread that ran through the day was that of community. It reflects something very real about observability – even though there are many competing vendors, there is also a tremendous culture of solidarity and collaboration.

Nowhere is this more evident than in the OpenTelemetry project, where thousands of specialists across different vendors and domains are working together. As a number of speakers reiterated, you don’t need to be a Go guru to contribute, and you will find plenty of mentorship and support. Check out this video on getting started by Abhimanyu Narwal and this upcoming talk by Diana Todea on how you can get involved even if you are not a developer.

Feedback

Get in touch at:

[email protected]

BlueSky

NEWS

A $4m seed round to tend the OllyGarden

Over the past year OllyGarden have blazed a trail in their mission to tackle bad telemetry and provide tooling for building high-quality instrumentation. The announcement of $4m in seed funding means that we can look forward to the garden continuing to bloom.

As well as the funding boost, the company’s press release also unveiled their vision for Minimum Viable Instrumentation (MVI). This is an extension of the company’s Rose product, which examines your codebase, fixes misconfigured instrumentation and injects new instrumentation where necessary. The essence of MVI is that it is both economical and intelligent. It is not about plastering boilerplate code into your projects.

Goodbye Chronosphere, hello Cortex

One of the big stories of last year was the acquisition of Chronosphere by Palo Alto. Things have been pretty quiet since the acquisition, but a recent press release has unveiled a new name and a new focus for the company.

Cortex XCOR may sound like some kind of killer robot from a Marvel movie, but the new branding is designed to reflect the company’s shift to “AI-driven observability”. XCOR is not just a marketing term, it also reflects a new multi-layered platform architecture that includes XCOR Agents, XCOR Fabric (a context layer) and XCOR Data Lake.

Control Theory Launch Dstl8

We covered Control Theory back in issue 33 of the newsletter, when they emerged as a player in the burgeoning observability pipeline sector. Like other vendors in that niche, they are harnessing AI to redefine their position and have now launched Dstl8 – which they describe as a “Runtime Feedback Loop for AI-Speed Engineering”.

As this description suggests, it is a tool designed to support engineering teams working in the new normal of AI-driven development, where increasing velocity requires that feedback occurs faster and earlier in the SDLC.

One of the great challenges of this kind of tooling is identifying which signals actually signify a meaningful regression. Dstl8 tackles this problem by evaluating telemetry not just against anomalies and patterns but also against “sentiment”. Hit the link below to get the full picture.

The Observability Cosmos is Expanding

Earlier in the year we launched the Observability Cosmos – our unique mapping of the observability space. Categorising and mapping such a multi-faceted and dynamic space as observability is an ongoing endeavour and one that entails a degree of subjectivity. Boundaries constantly shift and many products do not drop easily into a single category - and, of course, new vendors are continually entering the space whilst existing vendors diversify and pivot their products.

The latest version of the Cosmos is out now, with new entries in AI SRE, Application Observability and Observability Pipelines categories; You can get all the details in this article on the Observability 360 web site.

Observe roll out Agent Observability

If you are familiar with Observe (now branded as Observe by Snowflake), you will know that AI SRE has been part of their product for quite some time. They will now be complementing this with a dedicated tool for agent observability, called… Agent Observability – yep, the branding team worked overtime on that one.

Features such as tracing agent calls and tracking latency, errors and costs are now pretty much table stakes for AI observability tooling. Agent Observability covers all of these bases but also ships with some other notable capabilities. Firstly, there is the Agent Explorer, which enables searching and debugging across conversations, and secondly there is LLM-as-judge evaluations, which tracks non-deterministic signals such as accuracy, relevance and hallucination rate. The product goes live on October 22nd and if you want to see it in action you can register to join the launch event.

From the Blogosphere

A journey into the life of a metric

A lot of conversations about metrics tend to take a widescreen view – dealing with issues of cardinality explosions and hyperscale ingestion. This article by Jesús Espino at VictoriaMetrics takes a more atomic view, looking at the anatomy and lifecycle of a single metric as it is created in memory, travels over the wire and journeys through an observability backend.

This may sound dry and technical, but it unfolds almost like a nature documentary - all that’s missing is a David Attenborough voiceover as our humble metric metamorphoses and migrates across habitats. This is an article that really sheds light on the intricacies of metrics processing and the engineering that goes into maximising performance and optimising storage efficiency at a low level.

The Prometheus Exporters you need in your metrics toolkit

Exporters are one of the pillars of the Prometheus ecosystem, taking raw metrics data and translating it for compliance with the Prometheus protocol. At present, there are over 800 community exporters covering every kind of hardware and infrastructure resource. In this Medium article,

Neel Shah, developer advocate at StackGen, zooms in on seven of the most important exporters which will cover the majority of day-to-day use cases. This is a concise guide that explains the purpose of each exporter, how you can install it and also covers some of the key metrics emitted.

Atlassian’s mass migration to oTel

This is an article from the CNCF blog that, although relatively brief, tells the story of an epic IT project. Atlassian are a huge presence in the DevOps space and they operate a correspondingly large-scale metrics infrastructure – 100k hosts running across 14 regions. The task facing the engineering team was to migrate the entire system from their legacy gostatsd (no - that’s not a spelling mistake :-)) implementation to OpenTelemetry.

With thousands of services running across multiple different platforms, the option of simply ripping out gostatsd and switching to OpenTelemetry in one big bang project was not on the table. As the authors note, “the obvious plan (tear out the old pipeline, get every team to re-instrument on the oTel SDK, flip the switch) is a pipe dream”.

This is a really illuminating article that records the engineering decisions that the team made to ensure the successful delivery of a major project.

Prometheus and OpenTelemetry - nudging closer

The OpenTelemetry project has recently published a follow-up to the 2024 survey on the state of interoperability with Prometheus. Whilst there has been a significant improvement in compatibility, the latest survey shows engineering teams still experience frictions due to differences in areas such as labelling and naming conventions.

On the whole, the outlook is positive, with both camps working hard to iron out incompatibilities. The article also sheds light on cultural differences in the choice of metrics platforms. Infrastructure teams lean towards Prometheus, whilst OpenTelemetry seems to be the preferred option for application observability.

Measuring the OpenTelemetry overhead

Although OpenTelemetry has successfully established itself as the industry standard, it still involves trade-offs in comparison to other protocols. This is not surprising - a generic library is always going to carry more baggage than one dedicated to a specific platform.

The question is – how big is that overhead? Well, this analysis in the Coroot docs attempts to provide some numbers to answer that question in relation to distributed tracing. The benchmarks cover eight different languages and frameworks including Go, Java, Python and .NET. They also cover a range of scenarios and different sampling levels. What is the difference between sampling at 0% and not loading the SDK at all? Check out the page on the Coroot docs to find out.

AI

A causal graph to supercharge your agents

Causely occupies something of a unique position in the observability space, being one of the few platforms that genuinely applies the principles of causal AI (as opposed to correlation) to root cause analysis. Over the years, the product has evolved significantly and now positions itself as the deterministic reasoning engine that sits between raw observability data and autonomous AI workflows.

The premise is that, armed with causal context, agents can reason more quickly and efficiently. This article on the Causely blog compares the performance of agents attempting to identify the cause of a regression in a simulated microservice environment. According to the results, the agent using Causely’s MCP server was faster, made fewer tool calls and burnt fewer tokens. Both the simulation and the evaluation harness are available on the Causely GitHub repo.

Honeycomb’s AI Ecosystem - fleet management for your agents

In edition 48 of the newsletter, we covered Honeycomb’s Agent Timeline. This is a feature that tackles the problem of bringing together all of the disparate calls made by an agent into a single ConversationId. Honeycomb have now followed this up with AI Ecosystem, which zooms out and, rather than focusing on a single conversation, provides insights across your entire agent fleet.

This is another really valuable tool, which enables teams to view issues such as cost spikes or failures at a high level and then drill down into the causes. The tool is currently available as a closed beta.

Social

Michele Mancioppi’s Observability Concepts series

One of the frustrations of social media is that feeds can cycle through posts too quickly – meaning that you can miss out on really useful content. Over the past few weeks, Michele Mancioppi of Dash0 has created a series of expert summaries of OpenTelemetry concepts. In case the algorithm didn’t drop them into your timeline, you can find some links below:

Trace context

Signal correlation

Trace sampling

Log bridges

That’s all for this edition!

If you have friends or colleagues who may be interested in subscribing to the newsletter, then please share this link!

This edition’s quote is from the mathematician and physicist James Clerk Maxwell

About Observability 360

Hi! I’m John Hayes. As well as publishing the Observability 360 newsletter, I am also an Observability Advocate at SquaredUp.

The Observability 360 newsletter is entirely autonomous. All opinions expressed in the newsletter are my own.