Get in Touch
 Duration 35 hours

Course Outline

Introduction and Diagnostic Foundations

  • Overview of LLM system failure modes and common Ollama-specific challenges
  • Establishing reproducible experiments and controlled testing environments
  • Debugging toolkit: local logs, request/response captures, and sandboxing techniques

Reproducing and Isolating Failures

  • Techniques for crafting minimal failing examples and test seeds
  • Stateful vs. stateless interactions: isolating context-dependent bugs
  • Managing determinism, randomness, and controlling nondeterministic behavior

Behavioral Evaluation and Metrics

  • Quantitative metrics: accuracy, ROUGE/BLEU variants, calibration, and perplexity proxies
  • Qualitative evaluations: human-in-the-loop scoring and rubric design
  • Task-specific fidelity checks and defining acceptance criteria

Automated Testing and Regression

  • Unit tests for prompts and components, along with scenario and end-to-end tests
  • Building regression suites and establishing golden example baselines
  • CI/CD integration for Ollama model updates and automated validation gates

Observability and Monitoring

  • Structured logging, distributed tracing, and correlation ID management
  • Key operational metrics: latency, token usage, error rates, and quality signals
  • Alerting, dashboards, and SLIs/SLOs for model-backed services

Advanced Root Cause Analysis

  • Tracing through graphed prompts, tool calls, and multi-turn conversation flows
  • Comparative A/B diagnosis and conducting ablation studies
  • Data provenance, dataset debugging, and addressing failures induced by data

Safety, Robustness, and Remediation Strategies

  • Mitigation strategies: filtering, grounding, retrieval augmentation, and prompt scaffolding
  • Rollback, canary, and phased rollout patterns for model updates
  • Post-mortem analysis, lessons learned, and continuous improvement loops

Summary and Next Steps

Requirements

  • Extensive experience in developing and deploying LLM applications
  • Proficiency with Ollama workflows and model hosting mechanisms
  • Confidence in working with Python, Docker, and fundamental observability tools

Intended Audience

  • AI engineers
  • ML Ops professionals
  • QA teams accountable for production LLM systems

Custom Corporate Training

Training solutions designed exclusively for businesses.

  • Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
  • Flexible Schedule: Dates and times adapted to your team's agenda.
  • Format: Online (live), In-company (at your offices), or Hybrid.
Investment

Price per private group, online live training, starting from 7250 € + VAT*

Contact us for an exact quote and to hear our latest promotions

Upcoming Courses

Related Categories