Your AI bill is a data engineering problem

Does every question need the largest model?

Most organisations that start using language models end up with the same default. Every request, whatever its content, goes to the largest model on the market, hosted by a vendor abroad and billed by the token. It is the easiest setup, and in a small company it is often how things begin, with staff pasting documents into a public chat tool. For a life-science company this default carries three costs that only show up later: a monthly bill nobody can forecast, an energy footprint that grows with every query, and the occasional document that should never have left the building. The three have a common cause. Nobody decided, before the request was sent, how much model it needed and where it was allowed to run.

Most requests are small

A working day in an R&D team produces a mixed bag of requests. Minutes from a meeting recording. An SOP reformatted to the house template. A slide deck condensed for the board. Then, less often, a question that requires reasoning across sources: comparing yields across a series of purification runs, or checking whether a construct overlaps a competitor’s patent claims. The first group is routine text work that a small model handles well. The second is where a large model earns its cost. Sending both to the same endpoint means paying the reasoning price for reformatting.

This is not a new observation. Ong and colleagues trained routers that send each query to a strong or a weak model depending on its predicted difficulty, and report cost reductions of more than half on standard benchmarks with no measurable loss in answer quality [1]. The mechanism is unglamorous. A lightweight classifier, or even a set of rules based on request type and source system, assigns each request to a tier before any model is called.

Send less in the first place

The second lever is what goes into the prompt. Instrument exports, electronic lab notebook entries and vendor proposals arrive as long PDFs and spreadsheets in which most of the content is boilerplate. Parsing and filtering them locally, before anything reaches a model, cuts the token count directly. Extracting the ten fields that matter from a batch record is cheaper and usually more reliable than sending the whole record and asking the model to find them. Work on prompt compression from Microsoft Research shows that prompts can be shortened by a large factor with little effect on output quality [2], and simple structured extraction achieves much of the same benefit without any special tooling.

Fewer tokens means lower cost and faster answers. It also means less compute. Luccioni and colleagues measured the energy of inference across model types and found that general-purpose generative models cost orders of magnitude more per query than models built for a single task [3]. More recent benchmarks show that energy per query varies widely with model size even within the generative family [4]. Routing and pre-filtering act on both of these directly.

Decide where it runs

The third lever is independent of difficulty. Some requests should not leave Swiss or European infrastructure, whatever model they need. Unpublished sequences, CMC data, and any clinical or HR record containing personal information fall into this category. For personal data the revised Federal Act on Data Protection, in force since September 2023, sets the conditions for transfer abroad [5]. For intellectual property the constraint is contractual rather than statutory, but the consequence of a leak is the same.

The same layer that classifies a request by difficulty can classify it by sensitivity. Sensitive requests go to a private instance on Swiss or EU hosting, or to a model running on the company’s own hardware; everything else can use a standard endpoint. Once that layer exists, it is also the natural place to meter consumption per team and set hard budget caps, so that token spend appears on a dashboard rather than on the invoice.

Take-home message

The largest model is not the safe default, only the expensive one. A thin layer between users and models, which filters inputs, routes by difficulty and by sensitivity, and meters what is spent, turns cost, energy and confidentiality from three separate worries into one design decision. Most of that layer is data engineering rather than AI research, which is why it tends to be the first thing worth building.

Milad Adibi

References

1. Ong, I. et al. RouteLLM: Learning to Route LLMs from Preference Data. International Conference on Learning Representations (2025). arXiv:2406.18665.
2. Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y. & Qiu, L. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. Proc. EMNLP 2023, 13358–13376 (2023).
3. Luccioni, A. S., Jernite, Y. & Strubell, E. Power Hungry Processing: Watts Driving the Cost of AI Deployment? Proc. ACM FAccT ’24, 85–99 (2024).
4. Jegham, N. et al. How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference. arXiv:2505.09598 (2025).
5. Federal Act on Data Protection of 25 September 2020 (FADP, SR 235.1), in force 1 September 2023.