Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP
From AI Engineer · · 52 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.
If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.
Description
Coding agents fan out, retry and burst, and if they hit an LLM you host yourself, your inference layer will happily accept every request and bury the ones you needed first. Harshul Jain and Parv Khatri make the case for admission control: deciding what to refuse before it reaches the GPU. Drawing on a review of 55 systems papers and eight real production incidents, they share five questions to ask any admission policy before you deploy it, and a checklist you can use right away.
Harshul Jain, Audible: https://x.com/hj1393 | https://www.audible.com Parv Khatri, SAP: https://www.sap.com