Sample question 1 — AI Integration
A real-time translation feature is being integrated into a video conferencing app. The generative model currently takes 800 ms per sentence, causing noticeable lag. The team wants to reduce latency without retraining the model. Which integration technique is most effective?
- A. Use dynamic batching and model quantization to reduce computation and memory bandwidth during inference.
- B. Increase the model's context window so it can process more of the conversation at once, reducing the number of calls.
- C. Switch to a larger, more accurate model to reduce the need for post-processing and correction steps.
- D. Cache the model's outputs for common phrases and reuse them for similar sentences in the conversation.
Answer: A. Option A is correct because dynamic batching groups requests to improve throughput and quantization reduces precision to speed up inference, both without retraining. Option B is incorrect because a larger context window increases computation per call and does not directly reduce per-sentence latency. Option C is incorrect because a larger model typically increases latency, not decreases it. Option D is incorrect because caching helps only for repeated phrases and does not address the general per-sentence latency for novel inputs.