README.md
4.3 KB · 158 lines · markdown Raw
1 ---
2 library_name: stable-baselines3
3 pipeline_tag: reinforcement-learning
4 tags:
5 - stable-baselines3
6 - reinforcement-learning
7 - deep-reinforcement-learning
8 - ppo
9 - LunarLander-v3
10 - custom-reward
11 - reward-shaping
12 - actor-critic
13 ---
14
15 # PPO LunarLander flip, recover and land agent
16
17 This repository contains a Stable-Baselines3 PPO
18 actor-critic agent trained on a customised
19 `LunarLander-v3` environment.
20
21 ## Learned task
22
23 The curriculum teaches one policy to:
24
25 1. complete a full rotation in a fixed direction;
26 2. recover upright and arrest angular motion;
27 3. enter the landing zone;
28 4. reduce descent speed and land safely.
29
30 Reward configuration version: `v4-soft-touchdown`.
31
32 ## Changes in this upload
33
34 1. Continued from the selected Phase C checkpoint.
35 2. Added an altitude-dependent vertical-speed target.
36 3. Added a near-ground quadratic descent-overspeed penalty.
37 4. Added a dedicated in-zone crash penalty.
38 5. Reduced risky late horizontal corrections using a deadband.
39
40 ## Reward design
41
42 The shaped reward includes:
43
44 - one-off rotation-progress and flip-completion rewards;
45 - an upright post-flip recovery reward;
46 - horizontal guidance towards the landing zone;
47 - an altitude-dependent vertical-speed target;
48 - attitude and angular-speed control;
49 - a near-ground descent-overspeed penalty;
50 - distinct penalties for off-zone landings and in-zone crashes.
51
52 | Parameter | Value |
53 |---|---:|
54 | `required_rotations` | 1 |
55 | `rotation_direction` | 1 |
56 | `upright_tolerance_radians` | 0.3 |
57 | `recovery_angular_velocity_tolerance` | 0.5 |
58 | `pre_flip_original_reward_weight` | 0.15 |
59 | `post_flip_original_reward_weight` | 3 |
60 | `rotation_progress_bonus` | 300 |
61 | `flip_completion_bonus` | 500 |
62 | `recovery_bonus` | 250 |
63 | `flip_landing_bonus` | 2000 |
64 | `landing_without_flip_penalty` | 300 |
65 | `no_flip_terminal_penalty` | 300 |
66 | `failed_landing_penalty` | 800 |
67 | `outside_zone_landing_penalty` | 500 |
68 | `in_zone_crash_penalty` | 1200 |
69 | `post_flip_shaping_weight` | 2 |
70 | `post_flip_shaping_gamma` | 0.999 |
71 | `post_flip_shaping_clip` | 30 |
72 | `post_flip_center_weight` | 80 |
73 | `post_flip_horizontal_speed_weight` | 50 |
74 | `post_flip_vertical_speed_weight` | 100 |
75 | `post_flip_angle_weight` | 70 |
76 | `post_flip_angular_speed_weight` | 40 |
77 | `post_flip_leg_contact_weight` | 30 |
78 | `landing_zone_half_width` | 0.2 |
79 | `post_flip_zone_excess_weight` | 120 |
80 | `post_flip_target_vx_gain` | 0.5 |
81 | `post_flip_max_target_vx` | 0.35 |
82 | `post_flip_horizontal_deadband` | 0.08 |
83 | `post_flip_target_vy_high` | -0.45 |
84 | `post_flip_target_vy_near_ground` | -0.12 |
85 | `near_ground_height` | 0.6 |
86 | `safe_touchdown_vertical_speed` | 0.18 |
87 | `near_ground_overspeed_weight` | 120 |
88
89 ## Evaluation
90
91 Deterministic evaluation over
92 100 fixed-seed episodes:
93
94 | Metric | Value |
95 |---|---:|
96 | Mean shaped reward | -325.25 |
97 | Mean original reward | -706.46 |
98 | Full-rotation rate | 36.0% |
99 | Recovery rate | 12.0% |
100 | Recovery given a flip | 33.3% |
101 | Safe-landing rate | 0.0% |
102 | Flip-and-land rate | 0.0% |
103 | Terminal in-zone rate | 20.0% |
104 | In-zone crash rate | 8.0% |
105
106 ## Architecture
107
108 - Algorithm: PPO
109 - Policy: MLP actor-critic
110 - Actor hidden layers: `[128, 128]`
111 - Critic hidden layers: `[128, 128]`
112 - Observation dimensions: `11`
113 - Discrete actions: `4`
114
115 ## Training configuration
116
117 | Parameter | Value |
118 |---|---:|
119 | Phase timesteps | 2000000 |
120 | Parallel environments | 16 |
121 | Learning rate | 5e-05 |
122 | Rollout steps per environment | 1024 |
123 | Batch size | 64 |
124 | Optimisation epochs | 4 |
125 | Gamma | 0.999 |
126 | GAE lambda | 0.98 |
127 | Entropy coefficient | 0.005 |
128 | PPO clip range | 0.1 |
129 | Training seed | 43 |
130
131 ## Replay
132
133 - Seed: `20016`
134 - Original reward: `-679.11`
135 - Shaped reward: `-88.08`
136 - Rotations completed: `1.03`
137 - Flip completed: `True`
138 - Recovery completed:
139 `True`
140 - Landed safely: `False`
141 - Outcome: `flip_but_failed_landing`
142
143 [![Flip, recovery and landing replay](replay.gif)](replay.mp4)
144
145 [Open the full MP4 replay](replay.mp4)
146
147 ## Repository files
148
149 - `ppo-LunarLander-v3-flip-128x128.zip`: selected PPO model
150 - `flip_landing_reward_wrapper.py`: custom environment wrapper
151 - `training_config.json`: PPO training settings
152 - `reward_config.json`: reward configuration
153 - `episode_results.csv`: fixed-seed evaluation episodes
154 - `results.json`: machine-readable evaluation summary
155 - `config.json`: compact model metadata
156 - `replay.gif`: model-card preview
157 - `replay.mp4`: full replay
158