AI features can drift in ways traditional software can't — a provider quietly raises latency, a model regresses on a class of question, costs creep on a metered API. AI Watchdog is the surface where someone responsible for the deployment can see what the AI is actually doing across the organisation.
Every AI call is logged with prompt text, response text, function name, provider, model, input/output tokens, duration, status, error message, and cost in USD. The log is filterable by user, instance, function, status, provider, and date range — so investigations narrow quickly when something looks off.
- Cost transparency. Per-call cost, per-user totals, per-instance totals, per-function totals. Bring-your-own-key customers see what their provider would have charged; RaulWalter-supplied-key customers see exactly what we did charge.
- Ludwig effectiveness. Retry-rate detection flags conversations where the user rephrased the same question within five minutes — a high retry rate means Ludwig didn't satisfy the first time. Implemented with Jaccard similarity over content tokens (multilingual stop-words removed), so two genuinely different questions don't score as retries.
- Question-category mix. Every Ludwig conversation is bucketed by the tools invoked: risks, controls, policies, assets, processes, incidents, tasks, audit, locator, actions. You see what your team is asking the AI for, not just how often.
- Error visibility. Failed calls surface with their error message, provider, and model — fast feedback when a provider outage or quota cap hits.
Listed in the catalogue so the feature is discoverable, but never billed separately — every Cybsis instance carries it. System-admin scope: the people accountable for the AI deployment see the AI deployment.