Trace and score an agent with Langfuse

Why an agent is a black box

A prompt is one call: input, output, easy to read. An agent is a graph — as I tell it in from a prompt to agentic: it gathers context, decides, calls one or more models, reviews itself, loops. When the output is bad, the real question is: which step went wrong?

The vocabulary, in 4 words

Before instrumenting, four notions are enough to understand everything.

Instrument in three lines

Install the Langfuse SDK and its LangChain integration, then set the keys — those of your instance.

$ npm i langfuse langfuse-langchain @langchain/langgraph @langchain/openai

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_BASEURL=https://langfuse.mondomaine.com   # ton instance auto-hebergee

Instrumentation is one object: the langfuse-langchain CallbackHandler passed to graph.invoke links every graph node — and every model call — to a single trace. No decorator to sprinkle everywhere.

// app/api/agent/route.ts
import { CallbackHandler } from "langfuse-langchain";
// un handler par run — relie chaque noeud LangGraph a une trace
const handler = new CallbackHandler({ sessionId });  // sessionId : optionnel
const result = await graph.invoke(
  { brief },
  { callbacks: [handler] },   // gather -> clarify -> draft -> review -> apply
);
const traceId = handler.getTraceId();   // pour y accrocher des scores ensuite

Every node that calls a ChatOpenAI then shows up as a generation — prompt, response, tokens, cost — with nothing extra.

import { ChatOpenAI } from "@langchain/openai";
const model = new ChatOpenAI({ model: "gpt-4o" });
// un noeud du graphe : l'appel ChatOpenAI est trace comme "generation"
// (prompt, reponse, tokens, cout) sans une ligne de plus
async function draft(state: AgentState) {
  const res = await model.invoke(buildPrompt(state));
  return { draft: res.content };
}

Scoring: three levels

A trace tells you what happened. A score tells you whether it was good. Three sources, from least to most automatic.

Business rule and user feedback first — simple, reliable, free.

import { Langfuse } from "langfuse";
const langfuse = new Langfuse();
// 1. score programmatique (regle metier)
langfuse.score({
  traceId,
  name: "roadmap_valide",
  value: plan.milestones.length ? 1 : 0,
  comment: "au moins un jalon produit",
});
// 2. feedback utilisateur (pouce) — depuis une route API,
//    avec le traceId qu'on a renvoye au front
langfuse.score({
  traceId,
  name: "user_feedback",
  value: 1,               // 1 = pouce haut, 0 = pouce bas
  dataType: "BOOLEAN",
});

LLM-as-a-judge

For subjective quality (faithfulness, tone, relevance), an LLM-as-a-judge scores another’s output. Trace it too — a judge that drifts is debugged like everything else.

import { ChatOpenAI } from "@langchain/openai";
import { z } from "zod";
// un petit modele, en sortie structuree
const judge = new ChatOpenAI({ model: "gpt-4o-mini" }).withStructuredOutput(
  z.object({ score: z.number(), reason: z.string() }),
);
async function scoreOutput(question: string, answer: string, traceId: string) {
  const v = await judge.invoke(
    `Note la reponse de 1 a 5 sur sa fidelite au contexte.
Q: ${question}
R: ${answer}`,
  );
  langfuse.score({ traceId, name: "fidelite", value: v.score, comment: v.reason });
  return v;
}

Close the loop: datasets & experiments

The real win comes when you replay a frozen set of cases on every change, via Langfuse datasets & experiments. Freeze ~30 representative briefs, and before each deploy re-run the agent on them, scored by the judge. A regression shows up before production, not after.

// rejouer un jeu de cas fige AVANT de deployer -> pas de regression
const dataset = await langfuse.getDataset("briefs-de-reference");
for (const item of dataset.items) {
  const handler = new CallbackHandler();
  const output = await graph.invoke(
    { brief: item.input },
    { callbacks: [handler] },
  );
  await item.link(handler, "v2-review-loop");   // rattache le cas au run
  await scoreOutput(item.input, output, handler.getTraceId());
}
await langfuse.flushAsync();   // vide le buffer avant la fin du process
// -> compare v1 vs v2 dans l'onglet Experiments

In production — the gotchas

Anchor traces in reality: sessions, users, versions

An isolated trace answers “did this run work?”. But in production the real question is “did this run work *for this user*, in *this conversation*, on *this version* of the agent?”. Three attributes are enough to go from a heap of anonymous runs to something you can filter and aggregate.

With the CallbackHandler there is nothing new to install: you pass these attributes in the invocation `metadata`, and Langfuse lifts them to the trace level. Sessions accept any US-ASCII string under 200 characters as an identifier.

const handler = new CallbackHandler();
await graph.invoke(
  { brief },
  {
    callbacks: [handler],
    metadata: {
      langfuseSessionId: conversationId, // regroupe la conversation / groups the conversation
      langfuseUserId: user.id,           // segmente par user / segments by user
      langfuseTags: ["prod", "agent-v3"],
    },
  },
);

The prompt as a versioned object, not a hardcoded string

As long as your prompt lives in the code, changing an instruction means redeploying. Langfuse prompt management takes it out of the code: each edit creates an immutable version (1, 2, 3…), and a movable label (`production`, `staging`) points at the active version. You roll back by reassigning the `production` label to the older version, no code change. The SDK caches the prompt client-side, so it adds zero latency.

Scaling up: OTel, sampling, alerts

Three problems show up as traffic grows: fitting into an existing observability stack, not blowing up on volume, and getting alerted when things go sideways without watching the dashboard all day.

OpenTelemetry: Langfuse as an OTLP backend

If you already run an OpenTelemetry collector, you do not have to choose. Langfuse is OTel-native: it exposes an OTLP endpoint at `/api/public/otel/v1/traces` (HTTP, JSON or protobuf — no gRPC yet). You add Langfuse as an *exporter* on your collector, and spans emitted by any OTel-instrumented library nest automatically into the right trace.

Sampling: keep 20% instead of everything

At high volume, tracing everything costs storage and drowns the signal. Sampling is handled client-side: you set `LANGFUSE_SAMPLE_RATE` (or `sampleRate` in the constructor) between 0 and 1. The decision is made *at the trace level* — if a trace is kept, all its observations and scores are; otherwise nothing ships. No half-sent traces.

# Garde 20% des traces en prod haut-volume / keep 20% of traces at high volume
LANGFUSE_SAMPLE_RATE=0.2

Alerts: stop staring at the dashboard

Nobody watches a dashboard at 3 a.m. Monitors & alerts fire on a threshold — average cost, p95 latency, or a quality aggregate — and notify via Slack, webhook or a GitHub Actions trigger. “p95 latency > 8s” or “average cost per trace > $0.05” ping you before the user complains.

Sources & further reading