Observability starts in the IDE

Welcome to Edition #48 of the newsletter!

Welcome to the latest edition of the newsletter. If there is any particular motif that might emerge from this edition, it may simply be an appreciation of the broad and all-encompassing nature of observability.

It is not a reactive reflex, instead it is a process that embeds itself deeply into all aspects of the SDLC. Observability doesn’t begin when code is shipped to production, observability starts in the IDE.

Feedback

We love to hear your feedback. Let us know how we are doing at:

[email protected]

BlueSky

NEWS

Datadog are in the (Click)House!

One of the more surprising stories of the past few weeks was a ClickHouse blog article announcing a “partnership” with Datadog. They may seem like unlikely bedfellows, with ClickHouse having regularly panned Datadog as a “legacy” provider unable to the demands of large-scale observability.

The backdrop to the story is the release of Datadog’s Observability Pipelines feature – announced at their recent Dash bash. This enables users to ship their logs to ClickHouse and then access them either from Datadog or ClickStack. As the article says – this gives users the best of both worlds - Datadog’s UX combined with the power of the ClickHouse engine.

Although this has been described as a partnership, it is not actually an exclusive arrangement. Datadog’s log federation feature also integrates with providers such as Databricks – and one would imagine that other providers also come on stream further down the line.

Industry’s observability revolution

Although we tend to focus on observability as applied to software and computer networks, it has obvious applications for other contexts. This paper by Lukasz Ciukaj is an attempt to map out the current state and challenges of observability in industrial settings. The pain points and frictions that Lukasz enumerates will have an all-too-familiar ring to them – a huge number of different device types all emitting telemetry in proprietary protocols.

In the industrial context though, the challenges are rather more intractable and the risks are more daunting. Engineers need to eke telemetry out of legacy machines, hard-wired embedded systems and mission-critical black boxes. The way forward may have a familiar ring to it – a strategy involving Prometheus, OpenTelemetry, semantic conventions and open standards.   

Visual debugging with Snapshot Diffs

The latest innovation to roll off the Sentry production line is a really smart aid to testing and debugging for unintended UI changes. With Snapshop Diffs, users can generate snapshots with each commit to a codebase. Sentry will then block the PR if any of the snapshots differ from a baseline image. As the article argues, this is especially useful in an era when so much code is AI-generated and it is much easier for unintended UI changes to sneak in under the radar.

What is also interesting is that as well as being used to monitor agent changes, Diffs can also actually be used by agents themselves. By adding metadata to your snapshots you give agents additional structured context.

The article is refreshingly honest about the fact that Snapshot Diffs are not a magic wand and that this is an evolving feature. However, it has been heavily dog-fooded and has been stress-tested by OpenAI – who used it whilst building both ChatGPT and Codex.

Products

Telflo - fleet management without tears

Even if you are not operating at large scale, it is likely that you will be operating more than one oTel Collector. As the number of collectors increases, the need for effective Fleet Management becomes an increasingly important concern for observability teams. If you are in the market for fleet management tooling then Telflo should be on your shortlist. It is an open source project designed to help you build, configure, test and deploy OpenTelemetry Collectors.

It is built with usability in mind so that you can create sophisticated configurations using a beautiful and intuitive UI rather than having to hand-craft thousands of lines of YAML – although viewing and editing in YAML is also fully supported.

The editor is a real pleasure to work with – you can drag and drop components onto a canvas and then make further refinements in a properties window. The application also ships with an AI assistant so you can also build your configurations using natural language commands.

There are other really powerful features such as a visual builder for OTTL functions and a testing framework that means you can verify that your telemetry transformations are working before you build and deploy your fleet.

Bronto - Bring your own UI

Bronto is a product whose name may hark back to the Jurassic era, but technically they are definitely not a dinosaur. The company itself was founded in 2024 and released the first version of their product last year. The founders have a background in building log management systems – and that is one of the platform’s strong suits – but Bronto also provides full support for metrics and traces.

As a newcomer, they are unencumbered by legacy technologies and boast a high performance columnar db backend that is capable of petabyte-scale ingestion and sub-second querying. Naturally, the system has been built from the ground up with AI in mind. This means that as well as the usual support for MCP servers and agents, the product exposes all of its backend functionality via its API. This means that you can completely replace the standard UI – indeed they even give you guidance in this blog article.

The system has a simple pricing model with three tiers and you can also take it for a test-drive with a 14 day trial.

From the Blogosphere

Do you really need that Kafka instance?

As telemetry volumes continue to spiral, more and more teams have turned to using Kafka as a means to regulate the processing burden. Whilst this can be an option that provides robustness and reliability – it does come with its downsides. Kafka is itself a highly complex system and also has a large footprint. Glassflow is a product positioning itself as a more streamlined option for telemetry pipelines. Whereas Kafka is a general-purpose messaging platform, Glassflow is dedicated to telemetry ingestion and ClickHouse streaming.

This article on the Glassflow blog goes into considerable depth in exploring pipeline architectures, the use-cases for Kafka vs Glassflow and the trade-offs involved.

How GitHub built an internal analytics agent

Pretty much every observability stack now ships with its own AI agents and there is a large and growing AI SRE sector. At the same time, there are plenty of use cases for observability teams to build their own agent stacks – especially in instances where data sovereignty and privacy are paramount.

This article on the GitHub blog describes how they built Qubot, an internal data analytics agent used by teams across their organisation to enable self-service data analytics. If you have ever been involved in building an internal data mart, you will know that these projects that always look good on paper but can be fiendishly difficult to build in practice. For anybody interested in going down this road, you will find solid advice and insights on data processing, architecture and UX

OpenTelemetry

Bringing Apache Arrow into the oTel pipeline

Way back in November 2023 in Edition 5 of the newsletter, we reported on the integration of Apache Arrow as a transport protocol within the OpenTelemetry stack. This delivered a huge performance boost thanks to massively improved compression ratios.

The Arrow integration has now entered its second phase. This time, the aim is to harness the efficiency of Arrow’s columnar storage to bring efficiencies to operations within the telemetry pipeline itself. This article on the OpenTelemetry blog discusses the technical benefits of migrating to OTAP (OpenTelemetry Arrow Protocol) as well as providing a detailed analysis of the results of benchmark comparisons between OTLP and OTAP under specific loads.

SRE

Google’s AI SRE strategy - you don’t have to work at Google scale to adopt it

We are probably at the point where most companies are using AI in their observability practice – either through third party AI SREs or through their own in-house agentic AIs. Obviously, few of us work at Google scale but they have recently released a white paper on harnessing AI in SRE and its fundamental principles actually have wide applicability.

The overall thesis is that AI is not just a productivity accelerator, instead it is disrupting every part of the SDLC and changing the very nature of the SRE role – a trend which we also highlighted in our recent Observability 360 piece on the future of SRE.
If you haven’t got time to digest the full white paper, you can find the executive summary here.

AI

Honeycomb Agent Timeline

AI has unleashed an extraordinary wave of innovation not only in observability but in software production in general. Selecting just one or two to include in the newsletter is not easy but one of the recent standouts is Honeycomb’s Agent Timeline feature.
Agentic operations do not run as a single self-contained thread. Instead, they spawn multiple loosely coupled sub-processes.

Just as distributed tracing emerged as the solution to tracking requests through distributed microservices, Agent Timelines are a solution which manages to bring these disparate threads together into a single hyper-connected fabric. Honeycomb refer to all of these hand-offs, tool calls and processes as a conversation and Timelines binds all of these disparate strands together using a logical conversation_id. As long as your instrumentation emits spans with OpenTelemetry GenAI attributes, conversations will be automatically surfaced in the Honeycomb UI.

Observability starts in the IDE

One of the impacts of AI is that it redefines observability as a holistic and overarching practice. Observability is not a brain in a jar that gets deployed when code hits production. As soon as you start coding with an AI agent, you are burning tokens, running up costs and potentially interacting with third-party APIs.

This is a two-part series on AI observability on the SquaredUp blog. The stack it uses is Claude, OpenTelemetry and Azure but the principles can be applied to other coding assistants and backends. It provides an end-to-end guide to instrumenting Claude Code, configuring your oTel collector and visualising analytics in SquaredUp. The first article covers key metrics such as token usage and costs. The second takes a deeper dive into areas such as skills usage, distributed tracing and tracking exceptions.

How OpenSearch snap out of AI hallucinations

Hallucinations are one of the major risks involved in developing LLM-based applications. However, as this talk by Laysa Uchoa suggests, hallucinations are not simply a quirk of the model architecture. They can also be seen as an attempt by LLMs to fill in the blanks in their worldview – and are therefore also fuelled by sparseness and incompleteness in the datasets they work with.

This has major implications for chunking and retrieval strategies in RAG applications. With the right approach to your RAG configuration, hallucinations can be reduced by up to 80%.

You can find an overview of Laysa’s approach in this article and get a full breakdown in this YouTube video of the talk.

Events

Signals Conference, Berlin

There are an enormous number of events on the observability calendar, but this one has really caught our eye. Unlike some events which are glorified corporate sales pitches, Signals has more of the feel of a symposium. The focus of the event is the AI/observability interface, and the aim is to explore the most important questions facing practitioners in the space rather than being an opportunity for vendor self-promotion.

The committee for the event is drawn from representatives of many of the leading players in the domain – Dash0, Honeycomb, RunWhen, Causely, OllyGarden, Tsuga – to name but a few. What is noteworthy about this particular event is its ethos “to create a high-quality conversation among people who are actively shaping this field.” Capacity is limited to 250 attendees – so if this takes your interest, book now - this is going to be a great gig.

Podcasts

eBPF - changing everything without breaking anything

eBPF has possibly been at the same time one of the most radical but also one of the quietest tech revolutions we have ever seen. The kernel has been opened up but rather than unleashing a firestorm, we have seen engineering teams adopting the technology to achieve enormous gains in security, network management and observability. The fact that eBPF just works and gets on with the job perhaps means that its achievements can pass under the radar.

This podcast with Daniel Finneran and Olimpiu Pop on How eBPF empowers Developers is a great reminder of how far we have come in a relatively short period of time. This is a fascinating and wide-ranging discussion that takes in the eBPF verifier, eBPF on Windows, Tetragon, Auto-Instrumentation, Cluster security and even Self-Healing Infrastructure.

That’s all for this edition!

If you have friends or colleagues who may be interested in subscribing to the newsletter, then please share this link!

This edition’s quote is a beautifully simple observation from programming guru Rob Pike

About Observability 360

Hi! I’m John Hayes. As well as publishing the Observability 360 newsletter, I am also an Observability Advocate at SquaredUp.

The Observability 360 newsletter is entirely autonomous. All opinions expressed in the newsletter are my own.