Get in Touch
 Duration 21 hours

Course Outline

Introduction to Ollama Scaling

  • Overview of Ollama’s architecture and key scaling factors
  • Identifying common bottlenecks in multi-user setups
  • Best practices for ensuring infrastructure readiness

Resource Allocation and GPU Optimization

  • Strategies for maximizing CPU and GPU efficiency
  • Considerations regarding memory and bandwidth usage
  • Applying resource constraints at the container level

Deployment via Containers and Kubernetes

  • Containerizing Ollama using Docker
  • Executing Ollama within Kubernetes clusters
  • Implementing load balancing and service discovery

Autoscaling and Batching

  • Formulating autoscaling policies specific to Ollama
  • Utilizing batch inference techniques to enhance throughput
  • Balancing latency against throughput performance

Latency Optimization

  • Profiling inference performance metrics
  • Employing caching strategies and model warm-up techniques
  • Minimizing I/O and communication overhead

Monitoring and Observability

  • Integrating Prometheus for metric collection
  • Developing dashboards using Grafana
  • Establishing alerting mechanisms and incident response protocols for Ollama infrastructure

Cost Management and Scaling Strategies

  • Implementing cost-aware GPU allocation
  • Evaluating cloud versus on-premises deployment models
  • Adopting strategies for sustainable scaling

Summary and Next Steps

Requirements

  • Proficiency in Linux system administration
  • Conceptual understanding of containerization and orchestration
  • Background in deploying machine learning models

Target Audience

  • DevOps Engineers
  • ML Infrastructure Teams
  • Site Reliability Engineers

Custom Corporate Training

Training solutions designed exclusively for businesses.

  • Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
  • Flexible Schedule: Dates and times adapted to your team's agenda.
  • Format: Online (live), In-company (at your offices), or Hybrid.
Investment

Price per private group, online live training, starting from 4350 € + VAT*

Contact us for an exact quote and to hear our latest promotions

Upcoming Courses

Related Categories