Chapter 14 Reinforcement Learning based Orbit Station-Keeping

Chapter 14 Reinforcement Learning based Orbit Station-Keeping#

Project Overview#

In this project you will train an agent with Proximal Policy Optimization (PPO) to perform station-keeping in a realistic orbital environment.

The satellite must counteract small perturbations (e.g. drag) while staying within tight position constraints and minimizing fuel consumption (Δv).

The custom Gymnasium environment, reward function, and helper utilities are provided in the station_model package.

Check out the current project leaderboard to see how your model compares to others! 🚀

Project Files (all in project_handout/)#

  • RL_Station_Keeping_Handout.ipynb → Main exercises / your working notebook

  • coordinate system sketch.ipynb → Reference for coordinate system visualization

  • RL_Station_Sub_Problem_1.ipynb → Special notebook for Sub-Problem 1 (data exploration)

  • RL_Station_Sub_Problem_2.ipynb → Special notebook for Sub-Problem 2 (attitude control)

  • station_model/ → Python package (core code):

    • _environment.py → The main model code

    • _sub_problem.py → Helper functions for sub-problems

    • _hidden_functions.py → Simulated data for noisy thrusters and drag

    • __init__.py

  • constrained_ppo_orbit.zip, my_fancy_system_name_improved.zip → Pre-generated data/models (will be unzipped automatically)

How to Run#

Option 2: Locally / Offline#

In Terminal / PowerShell:#

cd path/to/comp_eng/project_handout
conda activate compeng
jupyter lab

No extra installation steps needed if you already followed the course environment setup.

How to Submit Solutions:#

After completing the exercises in RL_Station_Keeping_Handout.ipynb, save your work in a notebook, score prints and your trained model (.zip file). If you are using Colab make sure that you download your finished file (click the folder in the left pane). Your notebooks must be a clean version of the changes that you made (if you use new libraries include a cell to install it). Upload an additional .ipynb file for every subproblem you completed. You must ensure that the notebook runs in the default course environment without errors or it will be disqualified.

You may submit any number of solutions, and your best score will count towards your final grade. All files must be submitted in the standard e-mail (sent to s.endres@iwt.uni-bremen.de) using the template below. Finish all the square brackets with your own data and indicate whether you completed each subproblem (put n/a if you did not attempt it). You may also upload a failed attempt toward the subproblem for consideration towards your grade. Attach all relevant files to the e-mail, incomplete submissions will not be graded.

Subject: "CompEng Project Submission: [Your Name], [Your Student ID]"
Body:

Model name (anonymously name to be added to leaderboard): [e.g. Best_cruiser_model.zip]
Days in orbit: [e.g. 63.5 days]
Fuel mass remaining: [e.g. 397.65295973265404 kg]
Score = [e.g. 174232.19139185583]
Notes (Optional): [Here you may give hints about your approach and methods used e.g. A2C with custom reward shaping etc. for your fellow students]

Main notebook: RL_Station_Keeping_Handout.ipynb
Sub-problem notebooks:
- RL_Station_Sub_Problem_1.ipynb : [success, fail or n/a]
- RL_Station_Sub_Problem_2.ipynb : [success, fail or n/a]
- RL_Station_Sub_Problem_3.ipynb : [success, fail or n/a]
- RL_Station_Sub_Problem_4.ipynb : [success, fail or n/a]
- RL_Station_Sub_Problem_5.ipynb : [success, fail or n/a]

NOTE 1: In the handout you are only given starting code for Sub-Problems 1 and 2. The other subproblems are left to you to implement from scratch.

NOTE 2: Please stick to this template as close as possible (avoid salutations etc.) to make it easier to process your submission.

Contents