How-to guides

Step-by-step guides that cover key tasks and operations in LangSmith.

Setup

See the following guides to set up your LangSmith account.

• Create an account and API key
• Create an organization
• Set up billing
• Set up a workspace
• Set up access control (enterprise only)
  • Create a role
  • Assign a role to a user

Tracing

Get started with LangSmith's tracing features to start adding observability to your LLM applications.

• Annotate code for tracing
  • Use @traceable/traceable
  • Wrap the OpenAI API client
  • Use the RunTree API
  • Use the trace context manager (Python only)
• Toggle tracing on and off
• Log traces to specific project
  • Set the destination project statically
  • Set the destination project dynamically
• Set a sampling rate for traces
• Add metadata and tags to traces
• Implement distributed tracing
• Access the current span within a traced function
• Log multimodal traces
• Log retriever traces
• Log custom LLM traces
  • Chat-style models
  • Specify model name
  • Stream outputs
  • Manually provide token counts
  • Instruct-style models
• Prevent logging of sensitive data in traces
  • Example masking UUIDs and emails in inputs and outputs
• Export traces
  • Use filter arguments
  • Use filter query language
• Share or unshare a trace publicly
• Trace generator functions
• Trace with LangChain
  • Installation
  • Quick start
  • Trace selectively
  • Log to specific project
  • Add metadata and tags to traces
  • Customize run name
  • Access run (span) ID for LangChain invocations
  • Ensure all traces are submitted before exiting
  • Trace withouth setting environment variables
• Trace with Instructor (Python only)
• Trace without setting environment variables

Datasets

Manage datasets in LangSmith to evaluate and improve your LLM applications.

• Manage datasets in the application
  • Create a new dataset and add examples manually
  • Add inputs and outputs from traces to datasets
  • Upload a CSV file to create a dataset
  • Export a dataset
  • Create and manage dataset splits
• Manage datasets programmatically
  • Create a dataset from list of values
  • Create a dataset from traces
  • Create a dataset from a CSV file
  • Create a dataset from a pandas DataFrame
  • Fetch datasets
  • Fetch examples
• Version datasets
  • Create a new version of a dataset
  • Tag a version
• Share or unshare a dataset publicly

Evaluation

Evaluate your LLM applications to measure their performance over time.

• Evaluate an LLM application
  • Run an evaluation
  • Use custom evaluators
  • Evaluate on a particular version of a dataset
  • Evaluate on a subset of a dataset
  • Evaluate on a dataset split
  • Evaluate on a dataset with repetitions
  • Use a summary evaluator
  • Evaluate a LangChain runnable
• Bind and evaluator to a dataset in the UI
• Run an evaluation from the prompt playground
• Evaluate on intermediate steps
• Use LangChain off-the-shelf evaluators (Python only)
  • Use question and answer (correctness) evaluators
  • Use criteria evaluators
  • Use labeled criteria evaluators
  • Use string or embedding distance metrics
  • Use a custom LLM in off-the-shelf evaluators
  • Handle multiple input or output fields
• Compare experiment results
  • Open the comparison view
  • View regressions and improvements
  • Filter on regressions or improvements
  • Update baseline experiment
  • Select feedback key
  • Open a trace
  • Expand detailed view
  • Update display settings
• Evaluate an existing experiment
• Unit test LLM applications (Python only)
• Run pairwise evaluations
  • Use the evaluate_comparative function
  • Configure inputs and outputs for pairwise evaluators
  • Compare two experiments with LLM-based pairwise evaluators
  • View pairwise experiments

Human feedback

Collect human feedback to improve your LLM applications.

• Capture user feedback from your application to traces
• Set up a new feedback criteria
• Annotate traces inline
• Use annotation queues
  • Create an annotation queue
  • Assign runs to an annotation queue
  • Review runs in an annotation queue

Monitoring and automations

Leverage LangSmith's powerful monitoring and automations features to make sense of your production data.

• Filter traces in the application
  • Create a filter
  • Filter for intermediate runs (spans)
  • Advanced: filter for intermediate runs (spans) on properties of the root
  • Advanced: filter for runs (spans) whose child runs have some attribute
  • Copy the filter
  • Manually specify a raw query in LangSmith query language
  • Use an AI Query to auto-generate a query
• Use monitoring charts
  • Change the time period
  • Slice data by metadata or tag
  • Drill down into specific subsets
• Set up automation rules
  • Create a rule
  • View logs for your automations
• Set up online evaluations
  • Configure online evaluations
  • Set API keys
• Set up webhook notifications for rules (beta)
  • Webhook payload
  • Example with Modal
• Set up threads
  • Group traces into threads
  • View threads

Prompts

Organize and manage prompts in LangSmith to streamline your LLM development workflow.

• Create a prompt
  • Name your new prompt
  • Pick a prompt type
  • Compose your prompt
  • Save your prompt
  • View your prompts
• Update a prompt
  • Update metadata
  • Update the prompt content
  • Version a prompt
• Pull/push a prompt
  • Install packages
  • Configure environment variables
  • Pull a prompt and use it
  • Push a prompt to your personal organization
• LangChain Hub