ADaSci Banner 2024

Exploring the Essence of Pipelines and Monitoring in AI

Delve into the latest in generative AI, uncovering essential insights on pipelines and monitoring strategies
Pipelines

The Machine Learning Developers Summit (MLDS) 2024, held in Bengaluru in February, emerged as a pivotal gathering for innovators, thought leaders, and enthusiasts in the field of Generative AI. Among the various luminaries who graced the event, Siddharth Sahani’s talk on “What can go wrong with Unmonitored Models & Pipelines” stood out, shedding light on the intricacies and challenges in deploying machine learning models in real-world scenarios. Siddharth Sahani, a seasoned expert in the realm of machine learning, opened the floor with a glimpse into his rich background as a consultant, developer, and academician. His extensive experience formed the backdrop for his insightful discourse on the pitfalls of neglecting model monitoring and the reverberations this neglect can have across different domains.

Real-world Shifts and Model Inputs

In his talk, Sahani emphasized the common threads that connect the worlds of consulting and academia, notably the tendency to treat model deployment as the conclusion of the development journey. Sahani contended that this perception, while prevalent, is fundamentally flawed, as the story doesn’t end with shipping a model. He invoked historical examples, such as Amazon’s biased hiring model, to underscore the need for continuous monitoring, not just for accuracy but for societal implications.

One of Sahani’s key points touched upon the dynamic nature of real-world data and the potential for unforeseen shifts. Drawing on practical examples like the introduction of face masks during the pandemic, he illustrated how a sudden paradigm shift can render existing models ineffective. Sahani delved into the communication gap between developers and data producers, emphasizing the critical role of effective collaboration to avoid accuracy-killing bottlenecks.

The talk further navigated the treacherous waters of unintended bugs and unexpected policy changes, using instances like Apple’s introduction of privacy features to elucidate the impact on model features. Sahani warned against complacency in handling shifts in data characteristics, advocating for proactive strategies to avoid model accuracy degradation.

Influence Beyond Accuracy

Sahani’s discourse extended beyond the technical nuances, delving into how models can inadvertently shape markets and societal norms. Using the analogy of bidding strategies in the advertising ecosystem, he highlighted how aggressive model deployment can create a distorted market perception, a phenomenon with far-reaching consequences. This perspective invited contemplation on the broader responsibility of AI practitioners in shaping ethical and fair ecosystems.

Active Learning for Continuous Learning

Transitioning seamlessly, Sahani introduced the concept of active learning as a solution for minimizing data needs and ensuring models adapt to evolving scenarios. This innovative approach, he argued, enables the model to self-identify instances where it lacks confidence, facilitating ongoing learning and refinement.

Demystifying Model Monitoring

The latter part of Sahani’s talk navigated the intricate landscape of model monitoring, differentiating it from traditional DevOps monitoring. He emphasized that the stochastic and probabilistic nature of machine learning models makes monitoring a distinct challenge compared to the deterministic metrics commonly used in DevOps.

Sahani elucidated the critical components of model monitoring, from input and output distributions to the importance of understanding the decision-making process. The talk underscored that effective model monitoring is a holistic, user-driven, and UX-centric process, requiring not just technical prowess but also clear communication to stakeholders.

Making Model Monitoring a Priority

In the final segment, Sahani advocated for the integration of model monitoring within the purview of data science teams. Quoting the ‘Well-Architected Framework’ from Amazon, he stressed that the responsibility of monitoring should not be shirked but rather embraced by data science teams to ensure the longevity and impact of machine learning models.

Conclusion

Siddharth Sahani’s talk at MLDS 2024 was a profound exploration of the challenges and responsibilities inherent in deploying machine learning models. As we step into the future of AI, Sahani’s insights serve as a compass, guiding practitioners towards a more holistic approach to development, monitoring, and societal impact. The Machine Learning Developers Summit proved to be a melting pot of such invaluable insights, positioning Bengaluru as a hub for innovation and discourse in the ever-evolving landscape of generative AI.

Picture of Shreepradha Hegde

Shreepradha Hegde

Shreepradha is an accomplished Associate Lead Consultant at AIM, showcasing expertise in AI and data science, specifically Generative AI. With a wealth of experience, she has consistently demonstrated exceptional skills in leveraging advanced technologies to drive innovation and insightful solutions. Shreepradha's dedication and strategic mindset have made her a valuable asset in the ever-evolving landscape of artificial intelligence and data science.

The Chartered Data Scientist Designation

Achieve the highest distinction in the data science profession.

Elevate Your Team's AI Skills with our Proven Training Programs

Strengthen Critical AI Skills with Trusted Generative AI Training by Association of Data Scientists.

Our Accreditations

Get global recognition for AI skills

Chartered Data Scientist (CDS™)

The highest distinction in the data science profession. Not just earn a charter, but use it as a designation.

Certified Data Scientist - Associate Level

Global recognition of data science skills at the beginner level.

Certified Generative AI Engineer

An upskilling-linked certification initiative designed to recognize talent in generative AI and large language models

Join thousands of members and receive all benefits.

Become Our Member

We offer both Individual & Institutional Membership.