AWS Machine Learning Engineer - Associate Practice Questions

Sample question 1 — ML Solution Monitoring, Maintenance, and Security

A recommendation team has trained a new ranking model and wants to learn how it performs against the model currently serving live traffic. Leaders do not want to switch all users at once, and they want the comparison based on real requests rather than historical logs. The team needs a way to expose a small slice of production traffic to the new model while the current one keeps serving everyone else. Which approach should the team use?

1. A. Replay last month's request logs through the new model and compare the offline scores.
2. B. Add a second production variant to the endpoint and give it a small share of live requests.
3. C. Replace the current model with the new one and monitor the click-through rate for a week.
4. D. Duplicate the endpoint in a staging account and run synthetic traffic against both models.

Answer: B. Option B is correct because adding a second variant to the live endpoint and routing a small share of requests to it yields a real-world comparison without exposing the whole audience to an unproven ranker. Replaying last month's logs, option A, cannot show how the new ranking alters what users actually see. Wholesale replacement, option C, discards the baseline the comparison needs and risks the experience. Mirroring the endpoint in a separate account, option D, exercises synthetic rather than genuine production traffic.