Inference is the act of running a trained AI model to produce an output, as opposed to training it; every prompt answered is one inference call.
Category
AI
Also known as
inference call; model inference
Training happens before the model is used; inference happens every time it is used, and that is what you pay for and wait on. Latency and cost per call are set at inference by the model's size, the hardware it runs on, and how many tokens go in and out.
In practice
Real-time personalization has an inference budget: a model that takes two seconds cannot sit between a page load and a banner. Pre-compute what can be pre-computed and reserve live inference for what changes per session.
← Previous
In-Session Prediction (Bloomreach Engagement)
Next →
Instance vs Project (Bloomreach Engagement)
Something out of date?