The explosion of interest in Large Language Models (LLMs) and other Foundation Models (FMs) over the past two years has meant that more organizations want to know what it takes to run solutions using these models in production. In this talk, Andy will share his learnings on how to organise your people and technology to take full advantage of these capabilities and run production grade solutions at scale. He will discuss common architectures, emerging team structures and how evaluation and monitoring practices need to be upgraded for the GenAI era. This talk is useful for anyone involved in the deployment and operations of systems where the latest wave of machine learning models play a role.

In the case study I provide a deep dive into the area of food production (bakery goods) in europe´s largest bakery factory. The case study is about how we successfully implemented Computer Vision to monitor our production process to evaluate 3.5 mio. bakery goods every day. I will show how this leads to a reduction of food waste during production and how this is aligned with environmental/sustainability goals in food production. I will motivate why AI on the edge was identified as being a crucial success factor and which considerations we took as Schwarz group to get started in this area 3 years ago. I will discuss – based on the case study – how we selected and benchmarked hardware chips (Nvidia, Intel, GPU vs CPU…), how we needed to change software engineering to accommodate for Edge AI, how we integrated edge AI models into our cloud-based infrastructure, how we do the deployment and how/why we still use cloud-based GPUs for model training.