Case Study on the Production Deployment of an AI Powered Data Warehouse Management Service

This paper assesses the performance of a big language model (LLM) driven agent for the data warehouse (DW) when used during production for five months. The service translates users' conversational requests to typed operation primalies, and deals with their execution via semantic constraints, static verification, cost estimation and staged execution, as well as automatic rollback. It performed 3, 870 operations, including 1, 260 pipeline creations (32.6%) and 2, 610 other management tasks (67.4%). Logistic regression revealed that for every added planning step, the odds of task success decreased (OR = 0.81: 95% CI: 0.76–0.87); and the 0.89 point estimate for each additional schema table and the 0.42 estimate for schema evolution vs. schema pipeline creation. The median was 6.4 minutes for agent tasks, and 2.1 business days for work orders handled by the engineer (412), but undocumented matching and time boundaries make it non-sensical to make a causal comparison. A total of 42 rollbacks were made, with 3.2-minutes average recovery time with no manual intervention. The actual resource saved from a downgrade or rejection of a plan is based on the number of slot-hours that are estimated to be used, which came to an estimated 210, 000 slot-hours. Of these, the completion rate was 87.1% with validation and 79.6% without and 23 (4.8%) were identified as possible unsafe operations in this offline 480 work order ablation study. The results can then be used for auditable execution controls, and schema evolution will still have to be audited manually.