Signal Brief · the newsletter this call came out of
Five new AI tools every weekday morning, for B2B SaaS product managers.
One line on what each tool does, one line on why it matters for a PM, and the link. Give us your email and this morning’s issue — all five tools — lands in your inbox. No card, no password, nothing to install.
$10/month or $96/year after 7 free days, cancel in one click
Do-not-adopt call · 4 August 2026· free to read, no signup
Should we buy an LLM observability platform? Self-host first, then decide
You shipped one AI feature and you cannot tell why it answered badly on Tuesday. That is a real problem with a real fix, and the fix does not have to start with a platform contract priced per seat and per trace.
~6 min read · reviewed 2026-08-10· no sponsorships, no affiliate links
The call, up front
Do not adopt yet. Instrument with OpenTelemetry, self-host an MIT-licensed collector, and buy a platform when eval workflows — not tracing — become the bottleneck.
Do not adopt — LLM observability
A commercial LLM observability platform, for a team with one or two AI features
Why not. At one or two shipped features the job is tracing plus a way to read failures, and the open-source tools do that at parity: Langfuse's core is MIT-licensed and self-hostable, Arize Phoenix is open source, and both speak OpenTelemetry so instrumentation is portable rather than vendor-shaped. Two things make paying early a bad trade. First, prompts and completions contain your customers' data — often the most sensitive text your product touches — so the default posture should be keeping traces inside your own infrastructure until there is a reason not to. Second, if you already pay for a general APM, LLM observability may be inside a contract you have; buying a second observability vendor before checking is how one system's traces end up in two places and one team debugs across both.
Instead. Instrument with OpenTelemetry so the exporter is a configuration change rather than a rewrite, run Langfuse or Phoenix self-hosted for a fortnight, and check what your existing APM contract already includes for LLM traces. Write down the trigger for buying: a trace volume you no longer want to operate, or an evaluation workflow (labelled datasets, regression gating in CI, reviewer queues) that a self-hosted deployment cannot sustain. Buy against that sentence — it is usually eval and dataset management that justifies the contract, not tracing.
Spend avoided: Low five figures a year on a platform contract at this stage, plus the vendor review; unclear beyond that, because per-seat-plus-usage pricing scales with traffic nobody can forecast pre-launch
Written against this stack: One customer-facing AI feature in a Next.js app, models called via API, Postgres, an existing APM contract (Datadog or equivalent), a four-engineer team owning the feature.
A library that says no to everything is as useless as one that says yes to everything. These are the conditions under which this purchase is right — and if one of them describes you, stop reading and go and buy it.
Evaluation is your bottleneck, not tracing: you need labelled datasets, reviewer queues, prompt versioning and regression gates in CI, maintained by people who are not the four engineers who built the feature. That is what the commercial tiers are genuinely good at.
You have a policy against self-hosting stateful infrastructure, and the engineering time to run it is more expensive than the contract. A correct decision for many teams — just make it explicitly rather than by drift.
Trace volume has grown past what you want to operate. Retention, storage and query performance at high volume are real operational work, and paying someone else to do it is a defensible trade once you know your volume.
01— Detail
Instrument in a way you can walk away from
The decision that matters at this stage is not which vendor — it is whether your instrumentation is portable. Emit traces through OpenTelemetry and the choice of backend becomes an exporter configuration: self-hosted today, commercial next quarter, your APM the quarter after, with no code change and no re-instrumentation project. Instrument against one vendor's proprietary SDK and you have made a decision you will pay to reverse, before you had the data to make it well.
That framing also removes the false urgency. You do not need to pick a platform to start seeing your failures; you need spans around the retrieval step, the model call and the post-processing, with prompt, completion, token counts, latency and an identifier that ties a trace back to the conversation a customer complained about. Anything that reads OTel traces will then show you what happened on Tuesday.
Do the data-handling design at the same time, because it is cheap now and expensive later: decide what gets redacted before it leaves the process, how long traces are retained, and who can read them. Prompts and completions frequently contain exactly the customer text your DPA is most specific about, and the answer “it is in a vendor's trace store, indefinitely” is one you do not want to discover during a security review.
02— Detail
What actually justifies the contract
The commercial tiers in this category earn their money on evaluation, not observation. Labelled datasets that grow from production failures, prompt and model versioning tied to those datasets, automated scoring, human review queues, and a regression gate that fails a deploy when quality drops — that is a genuine product, it is a lot of work to build, and it is worth paying for when you have enough traffic and enough people that a spreadsheet no longer holds the process.
Tracing, by contrast, is commoditised. Every tool ingests OTel spans and renders a waterfall with prompts and completions attached, including the free self-hosted ones. If a vendor's pitch to you is mostly tracing, you are being sold the commoditised half at platform prices.
So write the trigger down before you shop: “we buy when we have N labelled examples we need to regression-test on every deploy, and a non-engineer needs to run that process.” That sentence is checkable, it prevents both premature purchase and the opposite failure of running a self-hosted deployment two years past the point it made sense, and it makes the eventual renewal conversation short.
Emit OpenTelemetry spans, whichever backend you choose. Portability is the whole game at this stage.
Check your existing APM contract for LLM observability before adding a second observability vendor.
Buy for evals, datasets and regression gating — not for trace rendering, which is free at parity.
The cheaper path
What to do with the money instead.
Every entry is a real product or a real capability with a link to the vendor’s own page. Where a vendor publishes a price we say so and date it; where they do not, we describe the posture rather than invent a number.
An LLM engineering platform — tracing, prompt management, evals — whose core is MIT-licensed and can run in your own infrastructure.
Why it covers the job. Covers the tracing job at parity while keeping prompts and completions inside your perimeter, and it speaks OpenTelemetry so moving to a paid tier later is a configuration change.
Cost: Free to self-host under the MIT-licensed core (you pay for the infrastructure and the operating time); managed cloud tiers are published on their site. · langfuse.com
An open-source tracing and evaluation tool for LLM and agent applications, runnable locally or in your own environment.
Why it covers the job. A second credible open-source option, useful if you want evaluation tooling with a more ML-experiment flavour before committing to any vendor.
Cost: Open source and free to run yourself; Arize's commercial platform is priced separately. · phoenix.arize.com
LLM tracing and monitoring inside the general-purpose observability platform many orgs already pay for.
Why it covers the job. Keeps AI traces next to the application traces they are causally connected to, and may already be inside a contract you signed — check before adding a vendor.
Cost: Part of your existing observability contract or a priced add-on to it; ask your account team what your current plan includes. · datadoghq.com
Keeping this honest
What would change our mind.
If one of these has happened since 2026-08-10, this page is out of date — tell us and we will change it. Pricing and plan contents rot fastest, so check the vendor’s current page before you budget either way.
A commercial platform whose evaluation tooling is so far ahead that it changes quality outcomes measurably in the first month, not the first year. That is the claim worth paying for and the one nobody demonstrates with data.
Self-hosting becoming genuinely expensive to operate at small scale — a heavy dependency footprint or storage requirements that make a fortnight's trial a project. That would flip the recommendation for small teams immediately.
Your own trigger firing: enough labelled examples and a non-engineer owning the review process. That is not us changing our mind, it is the rule working.
Straight answers
The questions people ask next.
Is Langfuse good enough instead of LangSmith?
For tracing, prompt inspection and basic evaluation on one or two features, yes — the core is MIT-licensed and self-hostable, so your prompts and completions stay in your infrastructure. The gap appears in mature evaluation workflows at scale: managed datasets, reviewer queues and regression gating that a non-engineer operates. Buy when that is your bottleneck, not before.
Do we need a separate LLM observability tool if we already use Datadog?
Check your contract first — LLM observability exists inside the major APM platforms, and keeping AI traces beside application traces has real debugging value. If your existing plan covers it, adding a second observability vendor duplicates spend and splits the investigation across two systems.
What should we instrument first?
OpenTelemetry spans around retrieval, the model call and post-processing, carrying prompt, completion, token counts, latency and an id that ties the trace to the specific customer interaction. Redact before export, set retention deliberately, and keep the exporter configurable — then any backend, free or paid, will answer the question you actually have.
How this was written
Written at the Signal Brief desk against the same hard rules the teardown generator follows. Licence and self-hosting claims were checked on 10 August 2026 against Langfuse's own repository and site and Arize Phoenix's site; LangSmith's commercial posture against LangChain's pricing page; and the existence of LLM observability inside a general APM against Datadog's own product page. No per-seat or per-trace figures are quoted because this category reprices frequently — read the current pricing page before you budget.
No sponsorships, no affiliate links and no paid placements anywhere on this site. That is a business-model decision rather than a virtue: a page that names a product and says don’t buy it cannot exist if a vendor is funding it. Corrections to [email protected].
Tomorrow morning, five tools you haven’t heard of yet.
Do-not-adopt calls like this one come out of reading every new AI tool that ships. Signal Brief is that reading, five tools a morning: what each does, why it matters for a PM, and the link. Start with this morning’s issue, free.