Start by clarifying the input shape and the need to compute speed from consecutive waypoints using the 10 Hz sampling rate. Then implement two reward variants: a count-based penalty for any waypoint exceeding the limit, and a magnitude-based penalty proportional to the speed violation. Discuss trade-offs such as sparsity vs. smoothness and potential normalization.
Pro tip: Mention that in real-world RL for autonomous driving, reward shaping must balance safety and comfort; a pure count penalty can be too sparse, while magnitude penalty might encourage slight overspeed. Consider using a smooth penalty like squared violation to avoid abrupt gradients.
Confirm the trajectory shape [batch, num_waypoint, 2] and that it's sampled at 10 Hz. Compute speed between consecutive waypoints as Euclidean distance divided by 0.1 seconds.
Assume a speed limit (scalar or per-waypoint). Create a boolean mask where computed speeds exceed the limit.
Sum the boolean mask (or count exceeding waypoints) and multiply by a penalty coefficient to get the reward component.
Compute the difference between speed and limit for violating waypoints, then sum or average these violations (possibly squared) and multiply by a penalty coefficient.
Compare the two variants: count is sparse and binary, magnitude provides dense feedback but may need scaling. Discuss how to combine with other reward terms and handle edge cases like zero speed.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Acknowledge that speed limits are dynamic, time-varying constraints that require the policy to adapt in real-time. Explain that the policy should treat the speed limit as part of the observation and learn a reactive control strategy, while the reward should penalize violations based on the current limit. Emphasize that simply recomputing reward with state-dependent limits is insufficient because the policy must anticipate and smoothly transition between limits.
Pro tip: Frame the problem as a Partially Observable Markov Decision Process (POMDP) where the speed limit is a latent variable that can change; the policy should infer it from observations and act accordingly. Highlight the importance of temporal abstraction and smooth control to avoid abrupt braking or acceleration.
Restate the scenario: speed limit changes from 50 mph to 30 mph after 2 seconds. Discuss the need for the policy to handle such transitions safely and efficiently, considering vehicle dynamics and passenger comfort.
Include the current speed limit as an observation, along with time since last change or a history of limits. This allows the policy to anticipate changes and adjust proactively.
Use a reward that penalizes exceeding the current speed limit, but also includes terms for smoothness (e.g., jerk) and progress. The reward should be computed based on the current limit at each timestep, but the policy must learn to predict and react to changes.
Use a recurrent policy (e.g., LSTM) or transformer to process sequences of observations, enabling the policy to remember past limits and anticipate future ones. Alternatively, use a hierarchical policy with a high-level planner that sets target speeds based on predicted limits.
Randomize speed limit changes during training to improve generalization. Evaluate the policy on scenarios with different change timings and magnitudes to ensure robust performance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.