50 Not Out!

Welcome to Edition #50 of the newsletter!

It’s our half-century! The newsletter kicked off roughly three years ago, and since that first edition much has changed.

In just our second edition, we were reporting on the acquisition of Splunk by Cisco for an eye-watering $29bn – a deal which truly underscored the growing financial and business importance of observability. Since then we have seen the rise and rise of OpenTelemetry, the mainstreaming of eBPF, the victory of columnar storage, and, of course, the small matter of the AI revolution.

Almost since we started, there has been talk of the “observability bubble” bursting. Instead, the sector has gone from strength to strength. In the AI era its strategic importance is greater then ever. Assuming we don’t get replaced by AI bots, we look forward to ringing in issue 100.

Feedback

Don’t hold back - give it to us with both barrels at:

[email protected]

BlueSky

NEWS

SigNoz - Humans eclipsed by AI agents.

Commentators have been remarking for some time that, in the post AI world, observability systems will mostly be consumed by agents rather than human visitors – and now SigNoz have come up with numbers to show that this threshold has been passed for usage of their platform.

In a post on LinkedIn, SigNoz co-founder Pranay Pratek noted that, in July of this year, agents clocked up 674k transactions on their platform, as opposed to 478k human actions. The speed of the turnaround is remarkable – coming just three months after the company released their MCP server. The fact that the next visitor to your platform is more than likely going to be an agent naturally has significant implications for platform design, and this article on the SigNoz blog digs into these issues in more depth.

Datadog unveil new pricing maze model for metric cardinality

High cardinality metrics have long been a financial pain point for Datadog customers, with many legendary stories of bill shock. With fierce competition from vendors such as ClickHouse and Honeycomb offering near-unlimited cardinality, the company has begun to turn the tanker around.

An article on the company’s blog has heralded the rollout of Infinite Cardinality Metrics. According to the article, metrics will now be charged by volume rather than cardinality. That is the theory. However, digging into the detail on the pricing page reveals a rather more complex picture. In fact, you are propelled into a byzantine maze of Indexed Series, Indexed Points charges, host allotments and other financial exotica.  The bottom line is that where you have sparse cardinality you will potentially see cost savings. Unfortunately we can’t tell you how much because the fundamental unit cost is not published.

Our opinion. We had to get Claude Code to build a pricing calculator for us and even then we only came up with an educated guess. A simpler pricing model with more transparency is needed.

Arize and shine! Dynatrace snap up AI observability platform

AI observability has rapidly shifted from being a niche concern to being table stakes for full-stack observability vendors. Coding assistants, agentic AI and LLMs have thrown up a new genre of concerns that vendors are rushing to address.

Dynatrace have responded to this growing need with the acquisition of Arize – one of the leading AI observability platforms on the market. This is not a side project – Dynatrace have invested $915m in the acquisition. In return they have acquired a platform that is used by some of the biggest companies in the world and boasts cutting edge AI observability capabilities. What this truly underlines is that in the post-AI world, observability is deeply embedded in the development process itself.

Dash0 Snap Up Polar Signals

Dash0 are a company in a hurry. Having achieved a funding round that values the company at $1bn, they have further expanded their functional reach with the acquisition of Polar Signals. Profiling is an important tool for debugging application performance issues. However, doing so without placing excessive burden on the host system can be a difficult trick to pull off.

With the acquisition of Polar Signals, Dash0 can now offer a profiling function that is at least the equal of any other on the market. The clincher for Dash0 may be that Polar Signals goes beyond competitors, with the ability to continuously profile CUDA processes on NVIDIA GPUs - providing valuable insights into the performance of AI workloads. 

Products

Ottermon - the baseline is the metric

AI is not just an accelerator or a force multiplier. It also creates possibilities for tackling technical challenges in new and creative ways. Ottermon is a product harnessing AI to take a new slant on enterprise observability.

One of the principal differentiators of the system is its approach to understanding incidents and system health. Instead of reacting to ad hoc outages or individual metrics, it learns about the patterns and shape of your system and establishes baselines. It can then intelligently evaluate whether an error is a critical failure or just background noise.

Like a number of modern systems, Ottermon also attempts to build up the full context of your business operations rather than just relying on the traditional observability signals.

Reiver - AI Observability Control Plane

AI-driven development has fundamentally altered the surface area of governance and observability. Interaction with external LLMs will occur throughout the development lifecycle and across all of your environments. Dedicated LLM gateways are an option that is gaining traction for organisations that want greater oversight and control of traffic to and from LLM providers.

Reiver is a platform that not only provides an LLM gateway, it positions itself as a full AI observability control plane. It provides tracing, model routing and agent SDKs. Onboarding does not even require a human in the loop. You just point your agent at the docs.Step away from the keyboard, human, we’ve got this covered.

From the Blogosphere

ClickHouse - are they winning the “observability war”?

ClickHouse has experienced a remarkable rise over the past few years. The speed and scalability of its database engine positioned the product perfectly for the explosive growth in telemetry volumes.

The performance of the ClickHouse platform is the culmination of a number of engineering and design decisions but author Mat Duggan argues that the most significant of these is the decision to use a columnar db architecture. For Mat, this is the reason that ClickHouse is “winning” the “observability wars”.

However, the article is not about cheerleading, it is a really excellent dissection of the architectural differences between ClickHouse and other vendors such as Grafana, Datadog and Elastic. The point that Mat really brings out is how performance degrades as workloads scale so that a system that wins at ingesting 5GB a day can grind and thrash once the volume is amped up to 10TB.

Telemetry Pipelines – the new brains of observability systems?

Over the past few years, telemetry pipelines have played an increasingly prominent role in observability architectures – particularly in filtering telemetry streams and routing workloads to specific backends. This Linkedin article by BindPlane CEO Ryan Goines raises the stakes and argues that, in the AI era, pipelines are not merely gateways for telemetry processing but are now becoming the “operational backbone of modern infrastructure”.

The thrust of the argument is that the pipeline is not just a gateway, it is now the gatekeeper, making critical decisions about which telemetry gets passed to the backend platform. There are obvious counter-arguments to this, but it is a thought-provoking and highly readable piece.

OpenObserve roll out their vision for Observability 3.0

OpenObserve was founded in 2022 and has established itself as a leading open source observability platform. Earlier this year the company secured $10m in Series A funding to "reimagine observability for the AI era". If you’d like to know what that means in practice then check out the latest lead article on the Observability 360 web site.

Although the platform is capable of petabyte scale ingestion, it can also run as a single binary. In the article we dig deeper into the platform’s architecture as well as exploring the features behind the company's vision for Observability 3.0.

OpenTelemetry

Honey – I shrank the tail sample

For most of us sampling is a fact of life. Sending 100% of your telemetry to an observability vendor is a luxury that many of us can’t afford – especially when 90% of it may be redundant.

Developing sampling rules, though ,is not just about turning a dial, especially in the case of distributed tracing.  For maximum robustness, tail sampling is the best option, but it also carries a major logistical overhead. Kudos then to Honeycomb, who have now donated their adaptive tail sampling processor to the OpenTelemetry project.

The problem with most sampling implementations is that they are rule-based – which makes them rigid and unable to adapt to changing traffic patterns. Adaptive sampling, on the other hand, takes a very much more nuanced approach. Rather than applying a filter across the board, it analyses traces according to their fingerprint and ensures that it captures the spectrum of behaviour within each of those fingerprint categories.

This is a great dive into the intricacies and tradeoffs of backend observability engineering.

AI

Who judges the LLM judge?

One of the great Catch 22s of using the LLM-as-a-judge pattern for evaluating LLM outputs is that you are still using one LLM to judge another. This might mitigate some errors, but it is still fundamentally error prone.

The guiding principle that the team at Netflix adopted was that humans must always be in the loop. However, this is not the inner loop but an outer loop that iterates continually using human-defined inputs and evaluations

The name they have given to their approach is "Reasoning-Aligned Rubric Tuning". This is an exhaustive and iterative quality cycle that involves creating test data, testing the judge, deployment and then monitoring drift. 

OpenAI Astra - did it steal your audit trail?

OpenAI’s Astra has been hailed as a breakthrough model, achieving blistering scores across a range of benchmarks including 100% on ExploitBench and 98% on FrontierMath Tier 4. There have even been some (rather overheated) claims that it heralds the arrival of AGI. 

Whilst it may represent a leap in capabilities, Astra also uses a new method of “reasoning” called “recurrent depth”, which can result in a major loss of traceability. The model no longer exposes its Chain of Thought - the notes that an LLM makes to describe its workings at each intermediate step. Some people have described this as representing a monitorability crisis.  

In this article Tanmay Bansal dives into the new model and assesses the implications for transparency and observability, concluding with some advice for how you may need to rethink your instrumentation strategy for calls made to Astra.

Books

Observability in the AI-Native Era

The title of this book may be "Observability in the AI-Native era", but this is not specifically about AI observability. It is about harnessing AI tooling to work alongside and within your stack to gain the richest insights into your applications and systems.

This is clearly a work based on extensive experience in the trenches of building sophisticated observability solutions to manage large and complex applications. There is a wealth of tactical detail that goes into considerable depth.

Taking a slightly similar approach to the way in which the Phoenix Project anchored the DevOps philosophy in the context of a fictional engineering team, this book illustrates the value of applying AI-driven observability principles to "Acme Financial Services" and charts the experiences of a team of observability engineers over successive iterations.

This is not just a book to read and file away. It is also a manual to keep by your side for building and refining your observability strategy.

That’s all for this edition!

If you have friends or colleagues who may be interested in subscribing to the newsletter, then please share this link!

This edition’s quote is a little bit different. It is an example of the dialects used by AI agents when they speak to each other. You can find out more in this Guardian article. If you think you know what it means, drop us a line.

About Observability 360

Hi! I’m John Hayes. As well as publishing the Observability 360 newsletter, I am also an Observability Advocate at SquaredUp.

The Observability 360 newsletter is entirely autonomous. All opinions expressed in the newsletter are my own.