Description
Large-language-model inference can be served with higher throughput and memory efficiency. This engine helps developers run model serving, batching, and inference workflows for LLM-powered applications.
It can expose a server and process prompts, generated text, model files, and GPU memory. Restrict access and protect prompt data, because served models may handle sensitive or untrusted content.