Autonomous Vehicle RL — DDPG on CARLA

Overview

The core question this project starts from is deceptively simple: can a neural network learn to drive a car without being told explicitly what to do? The answer turns out to be yes, but arriving at that answer requires solving a surprisingly large set of interconnected engineering problems, and the choices made along the way reveal a great deal about how reinforcement learning actually behaves in a continuous, physically simulated environment. The simulator used here is CARLA 0.9.10, built on Unreal Engine 4. CARLA is not a game engine dressed up as a research tool. It is a purpose-built autonomous driving simulator that provides accurate vehicle physics, realistic sensor models, configurable traffic and weather, and a Python API that gives full programmatic control over every aspect of the environment. Choosing CARLA over a simpler grid world or toy environment is a deliberate decision. If the goal is to understand autonomous driving, then the simulation environment needs to approximate the complexity of the real problem, not reduce it to something easier to train on. The reinforcement learning algorithm at the center of this project is Deep Deterministic Policy Gradient, commonly called DDPG. The reason DDPG is the right choice here, rather than a simpler algorithm like DQN, comes down to the nature of the action space. Driving a vehicle requires controlling steering angle and throttle as continuous real-valued outputs. A discrete action algorithm would need to quantize those outputs into a finite set of choices, which introduces approximation errors and produces jerky, unnatural control. DDPG handles continuous action spaces natively, which means the agent learns to output smooth, physically plausible commands directly. DDPG is an actor-critic algorithm, which means it maintains two separate neural networks with different responsibilities. The actor network takes the current state of the vehicle as input and outputs an action, specifically a steering value and a throttle value. The critic network takes both the current state and the action as input and outputs a scalar estimate of how good that action is in that state, which is called the Q value. During training, the actor is updated to maximize the Q values estimated by the critic, while the critic is updated to make its Q value estimates more accurate. The two networks improve each other iteratively across thousands of training episodes. The state representation fed into the actor is composed of two components. The first is a set of local waypoints, which are relative path coordinates describing the road ahead of the vehicle in the vehicle's own coordinate frame. These give the agent the information it needs to understand where the road goes and how it should position itself within it. The second component is the vehicle's current kinematics, specifically its velocity and heading angle. Together these inputs give the agent both the spatial context of the road and the dynamic context of its own motion. The reward function is where the desired behavior of the agent is encoded, and designing it well is one of the most consequential parts of building any reinforcement learning system. The reward structure here reflects the actual goals of driving rather than a simple proxy. The agent receives a positive reward proportional to its velocity multiplied by how well it is aligned with the road direction, which encourages it to drive fast when it is going in the right direction and discourages it from accumulating reward by driving fast in the wrong one. A centrality penalty subtracts from the reward when the vehicle drifts from the center of the lane, which encourages it to hold its position within the lane boundaries rather than wandering. A collision with any object terminates the episode immediately and assigns a reward of negative two hundred, which is a strong signal that collisions are categorically unacceptable. A lane departure triggers the same termination and the same penalty. Successfully reaching the destination yields a reward of positive one hundred, which gives the agent a clear terminal objective to optimize toward. The learning process unfolds across 8,000 episodes. In the early episodes, the agent behaves almost randomly. It explores the action space without any learned policy to guide it, which means it collides frequently, departs its lane constantly, and rarely accumulates meaningful reward. This exploration is not a failure condition. It is how the replay buffer is populated with experience. Every interaction between the agent and the environment, described as a tuple containing the state, the action taken, the reward received, and the next state, is stored in the replay buffer regardless of how good or bad the interaction was. Training proceeds by sampling random batches from the replay buffer rather than learning from experiences in the sequential order they were collected. This is a critical design decision. Sequential learning would expose the networks to highly correlated experiences, which would destabilize training by causing the network to overfit to recent interactions and forget earlier ones. Random sampling from a large replay buffer breaks that correlation and produces more stable gradient updates. The target network mechanism addresses a different stability problem. Without target networks, the Q value estimates that the critic is trained to match would shift every time the critic's weights were updated, creating a moving target problem where the loss function is chasing an objective that changes with every gradient step. DDPG addresses this by maintaining a separate copy of both the actor and critic, called target networks, whose weights are updated much more slowly by taking a weighted average of the main network weights and the target network weights at each step. This keeps the training targets stable enough for the critic to learn from them effectively. Over thousands of episodes, the behavior that emerges from this process transitions from erratic exploration to something recognizable as driving. The agent learns to stay in its lane, to align itself with the road ahead, to accelerate on straight sections and slow down for curves, and to avoid the collisions and lane departures that terminate episodes with large negative rewards. None of this is programmed as explicit rules. It all emerges from the gradient signal propagated backward through the network based on the reward function. This project was the first substantial layer of a longer research direction. Training an agent to control a vehicle reliably in simulation raised a set of questions that turned out to be more interesting than the original one. The agent learns a policy, but it cannot articulate that policy. It cannot explain why it turned left, what it expected to happen, or whether its internal reasoning is consistent with its observable behavior. In a system where reasoning, language, and action are all supposed to work together, that gap between what a model does and what it can be said to understand became the central question of the work that followed.