How to Trace LLMs with LangSmith in 2026

Introduction

LangSmith is LangChain's observability platform, essential for debugging, testing, and monitoring LLM-based applications. In 2026, as LLM chains grow more complex, tracing every API call, input/output, and latency is key to optimizing costs and performance. Imagine debugging a RAG chain like a Python profiler: visualize flows in real-time through an intuitive interface.

This beginner tutorial takes you from account creation to automated evaluations. You'll integrate native tracing, explore traces in the web UI, and build datasets for reproducible tests. By the end, you'll master LangSmith for rapid iteration on AI agents, slashing errors by 50% on average per LangChain benchmarks. Ready to make your LLMs transparent? (128 words)

Prerequisites

Python 3.10+
Free account on LangSmith (create via Google/GitHub)
OpenAI API key (optional for advanced examples)
pip installed
Basic Python and LLM prompting knowledge

Install the Dependencies

terminal

pip install langchain langsmith langchain-openai python-dotenv

This command installs LangChain for LLM chains, LangSmith for tracing, and OpenAI integration. Use a virtualenv for isolation: python -m venv env && source env/bin/activate. Avoid conflicts by pinning versions if needed: pip install langchain==0.3.0.

Get and Configure Your LangSmith API Key

Log in to smith.langchain.com, go to Settings > API Keys, and generate a LANGCHAIN_API_KEY. Also create a project via Projects > New Project (name it my-first-project).

Store it in a .env file for security: it will automatically trace to this project. Think of it like an AWS key—never hardcode it in your code.

Environment Configuration

.env

LANGCHAIN_API_KEY=lsv2_your_api_key_here
LANGCHAIN_PROJECT=mon-premier-projet
OPENAI_API_KEY=sk-your_openai_key_here

This .env file enables automatic tracing and sets up OpenAI. Load it with dotenv in your scripts. Pitfall: if not auto-detected, add LANGCHAIN_TRACING_V2=true; use export LANGCHAIN_TRACING_V2=true in the terminal.

Load Environment and Test

test_env.py

import os
from dotenv import load_dotenv

load_dotenv()

print("API Key loaded:", "OK" if os.getenv("LANGCHAIN_API_KEY") else "KO")
print("Project:", os.getenv("LANGCHAIN_PROJECT"))

# Test basic tracing
os.environ["LANGCHAIN_TRACING_V2"] = "true"
print("Tracing enabled!")

This script verifies your config and explicitly enables V2 tracing (recommended in 2026). Run python test_env.py. Value: catches errors early. Avoid print(os.getenv('key')) in production for security.

Create Your First Chain with Automatic Tracing

Run this code and check smith.langchain.com > your project. You'll see an interactive trace: inputs, outputs, latencies. Click the trace to zoom into each step, like a visual debugger.

First Traced LangChain Chain

simple_chain.py

import os
from dotenv import load_dotenv
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

load_dotenv()

# LLM model
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# Prompt template
prompt = ChatPromptTemplate.from_template("Explique {topic} en 3 points simples.")

# Chain
chain = prompt | llm

# Auto-traced execution
result = chain.invoke({"topic": "LangSmith"})
print(result.content)

This full chain pipes a prompt to GPT-4o-mini, automatically traced via env vars. Copy-paste and run: python simple_chain.py. See results in console + web trace. Pitfall: without temperature=0, outputs vary, making tests flaky.

Explore Traces in the LangSmith Interface

Go to smith.langchain.com > my-first-project > Runs.
Click a trace: view Inputs/Outputs, Token usage, Latency (e.g., 2s for LLM call).
Use Filters to tag by user_id or errors.

Analogy: like Chrome DevTools for JS, but for LLM chains.

Advanced Chain with Parsing and Tags

advanced_chain.py

import os
from dotenv import load_dotenv
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI

load_dotenv()

llm = ChatOpenAI(model="gpt-4o-mini")
prompt = ChatPromptTemplate.from_template(
    "Liste 3 faits sur {topic}. Réponds en JSON: {{'faits': ['f1', 'f2', 'f3']}}"
)
parser = StrOutputParser()

chain = prompt | llm | parser

# Run with metadata for filtering
result = chain.invoke({"topic": "Python"}, config={"tags": ["test", "beginner"]})
print(result)

Adds JSON-like parsing and tags for filtering traces (e.g., view 'test' only). Run it and check the UI: tags are visible. Improves debugging by grouping runs. Avoid parsers without JSON instructions to prevent malformed outputs.

Create a Dataset and Run Automated Evaluation

dataset_eval.py

import os
from dotenv import load_dotenv
from langsmith import Client
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

load_dotenv()
client = Client()

# Create dataset
dataset_name = "test-llm"
data = [
    {"input": "topic", "output": "Explication en 3 points."},
    {"input": "Python", "output": "Langage interprété."}
]
client.create_dataset(dataset_name=dataset_name)
for item in data:
    client.create_example(inputs={"topic": item["input"]}, outputs=item["output"], dataset_id=dataset_name)

# Evaluate
llm = ChatOpenAI(model="gpt-4o-mini")
prompt = ChatPromptTemplate.from_template("Explique {topic} en 3 points.")
chain = prompt | llm
results = client.run_on_dataset(
    dataset_name=dataset_name,
    llm_or_chain_factory=lambda: chain,
    evaluation="LangSmith built-in evaluators"
)
print(results)

Creates a dataset with 2 examples and runs auto-evaluation (BLEU/ROUGE scores). Aim for >0.8 scores. Check Evaluations in the UI. Great for A/B testing models. Pitfall: empty datasets crash—always use 2+ examples.

Best Practices

Always enable V2 tracing: os.environ["LANGCHAIN_TRACING_V2"] = "true" for maximum granularity.
Tag your runs: config={"tags": ["prod", "user_id:123"]}, filter by cohorts.
Use datasets for testing: Aim for 80% coverage before production, integrate with CI/CD.
Monitor costs: UI > Analytics > Token usage, set budget alerts.
Secure keys: .env + gitignore, rotate monthly.

Common Errors to Avoid

Missing API key: empty traces; check with langsmith list CLI.
No project specified: runs go to 'default'; set LANGCHAIN_PROJECT.
No dotenv: keys not loaded; call load_dotenv() first.
Over-prompting: chains >5 steps slow down; profile latencies in UI.

Next Steps

Official docs: LangSmith Quickstart
Advanced CLI: pip install -U langsmith && langsmith login
Integrate with Streamlit/Gradio for full-stack apps.

Check out our Learni AI trainings to master LangGraph and autonomous agents.

How to Get Started with LangSmith to Trace Your LLMs in 2026