Setting Up a Training Slurm Cluster on AWS

How to setup a minimal Slurm cluster

Material covering how to setup a Slurm cluster for trainees to use during the courses in this theme. Currently uses AWS and Flight Solo.


[HT25] Introduction to the Message Passing Interface

Part of a series of courses in Software Engineering delivered by OxRSE

Introduction to Software Engineering (Schmidt)

This workshop will provide an introduction to [PyBaMM](https://pybamm.org/) (Python Battery Mathematical Modelling), an open-source battery simulation package written in Python. It will cover the basic functionality to solve and customise battery models.

# Preparation for PyBaMM workshop

You will be working through the practical exercises using your own laptops. You will need to setup your laptop with a few things, which you may already have. These are:

## Command Line Tool

You will need a command line tool (shell/console) in order to install PyBaMM and run Python scripts. You should have a command line tool already installed on your laptop:
 
On Windows, you can use cmd, Powershell or Git Bash (which is included in Git For Windows package - https://gitforwindows.org/).
On macOS and Linux, you will already have a command line tool available on your system. You can use a command line tool such as Bash (https://www.gnu.org/software/bash/), or any other command line tool that has similar syntax to Bash (e.g. Zsh). 
 
## Python Distribution
 
The material has been developed using the standard Python distribution version 3.8 – 3.10 and is using venv for virtual environments and pip for package management. The material has not been extensively tested with other Python distributions and package managers, but most sections are expected to work with some modifications. For example, package installation and virtual environments would need to be managed differently, but Python script invocations should remain the same regardless of the Python distribution used.
 
To download a Python distribution for your operating system, please head to Python.org (https://www.python.org/downloads/) .
 
## venv and pip
 
If you are using a Python 3 distribution from Python.org, venv and pip will be automatically installed for you. If not, please make sure you have these two tools (that correspond to your Python distribution) installed on your machine.
 
## VSCode or other IDE
 
The course does not assume you are using any particular IDE, and you could use any text editor to edit your Python scripts and run them via the command line. However, we do recommend the use of an IDE when using PyBaMM. For example, you can use Microsoft’s free Visual Studio Code (VSCode) tool.
 
You can download and install VSCode here (https://code.visualstudio.com/). Make sure you install the Python Extension via the “Extension” button on the LHS sidebar (search for “python” in the search bar and install the extension titled “Python” by Microsoft)
 
## PyBaMM

Once you have the above, you can create a virtual environment by typing the following into your command-line.

```bash
python -m venv env
```

This will create a new virtual environment in the subdirectory env. Note that `python` is the name of the python interpreter that you have installed. In some systems you might need to use `python3` instead. If you are using **Windows**, you can then activate your virtual environment by using the following command:

```bash
./env/Scripts/activate
```

If you are using Powershell you might need to capitalise `Activate`.

For **Mac or Linux** systems you can use

```bash
source env/bin/activate
```

Once you have activated your virtual environment, you can install PyBaMM into it using the Python package manager `pip`

```bash
pip install pybamm
```

For more installation information on PyBaMM, or for more options, please consult the PyBaMM documentation here: ​​https://docs.pybamm.org/en/latest/source/user_guide/installation/index.html

You can test your PyBaMM installation by creating a new file `test.py` with the following contents:

```python
import pybamm
import matplotlib.pyplot as plt
model = pybamm.lithium_ion.DFN()
sim = pybamm.Simulation(model)
sim.solve([0, 3600])
fig = sim.plot()
plt.savefig("test.png")
```

Run this script using 

```bash
python test.py
```

And it should generate an image file `test.png`, which contains the result of simulating a DFN model at time $t = 0$



Faraday PyBaMM Training Workshop

An introduction to PyBaMM (Python Battery Mathematical Modelling)

## Timetable

This teacher-led in-person course is 3 hours long and contains lecture and practical class elements.

| Date          | Time  | Event         |
| ------------- | ----- | ------------- |
| Tue 28-Jan-25 | 14:00 | Course starts |
| Tue 28-Jan-25 | 17:00 | Course ends   |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

This course introduces the concept of continuous integration and how to set it up for a Python project using GitHub Actions.

## Objectives

- Describe and build a basic GitHub Actions workflow
- Create a workflow for static analysis, unit tests, and code coverage
- Diagnose workflow issues and parameterise across operating systems
- Apply and automate code coverage tools to assess test coverage
- Highlight the importance of good documentation and use Sphinx and Read the Docs for project documentation


## What you will need

You should bring your own laptop.
Please follow our [standard setup instructions](https://train.rse.ox.ac.uk/event/37#What-you-will-need) to make sure you have everything ready to go when the course starts.

If you intend to follow the C++ material, please additionally ensure you have set up a suitable environment, including having a C++ compiler installed.

---


[HT25] Continuous Integration

---

## Timetable

The course takes place over five Mondays: 13 May, 20 May, 03 June, 10 June, 17 June.

| Time            | Format            | Location       |
|-----------------|-------------------|----------------|
| 08:00-09:30 UTC+1 | Online (Zoom)   | Singapore      |
| 09:30-10:30 UTC+1 | In person | Oxford         |
| 14:00-15:00 UTC+1 | In person | Oxford         |
| 16:00-17:30 UTC+1 | Online (Zoom)   | North America  |

Note that all times are UTC+1 (British Summer Time):
 - 08:00-09:30 is 15:00-16:30 UTC+8 in Singapore
 - 16:00-17:30 is 11:00-12:30 UTC-4 (EDT) or 08:00-09:30 UTC-7 (PDT) in North America

The in-person sessions will be at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/) except on the 17th June which will be in the [Department of Statistics](https://www.stats.ox.ac.uk/).

---


## Preparation for the course

You will be working through the practical exercises using your own laptops. You will need to set up your laptop with a few things, which you may already have. This course requires that you have installed:

- Python 3
- The `venv` module for Python 3
- The `git` version control package
- A `bash` command-line shell
- VSCode Interactive Development Environment (IDE)
- Docker

If you already have these then great, otherwise please read on for instructions on how to install them on your machine. The instructions below are different depending on the operating system you have installed on your laptop.

### Microsoft Windows

All the material assumes you have a unix-like environment, and the easiest way to get this on a Windows laptop is to install Windows Subsystem for Linux (WSL).

You can install WSL using the instructions here:

https://learn.microsoft.com/en-us/windows/wsl/install

Once you have an Ubuntu 22.04 distribution installed, open the Ubuntu terminal window and use this command to install all the necessary packages (Python, venv and git):

```bash
sudo apt install -y python3-pip python3-venv python3-dev build-essential git
```

We will be using the VSCode IDE for the course. Make sure to install VSCode using the *Windows* installer here: https://code.visualstudio.com/download

Once VSCode has installed, run it and install the "Python (Microsoft)" and the "WSL (Microsoft)" extensions.

Finally, install Docker. If you are unfamiliar with Docker, the easiest way is probably to install [Docker Desktop](https://www.docker.com/products/docker-desktop/).


### macOS

If you have a Mac then we recommend using homebrew to install the dependencies, you can find installation instructions here: https://brew.sh/. Then you can use brew to install everything you need, like so:

```bash
brew install python3 git
brew install --cask visual-studio-code
```

Once VSCode has installed, run it and install the "Python (Microsoft)" extension.

Finally, install Docker. If you are unfamiliar with Docker, the easiest way is probably to install [Docker Desktop](https://www.docker.com/products/docker-desktop/).


### Ubuntu 22.04

The material has been written using Ubuntu 22.04, but if you are running another Linux distribution then you should be able to follow along with minimal changes. If you wish to install Ubuntu 22.04 then you can from here:

https://ubuntu.com/download/desktop

Once Ubuntu is installed, log in and open a command-line terminal. Install the required software via apt by typing in the following command:

```bash
sudo apt install -y python3-pip python3-venv python3-dev build-essential git
```

Then install VSCode using snap:

```bash
sudo snap install --classic code
```

Once VSCode has installed, run it and install the "Python (Microsoft)" extension.

Finally, install Docker. If you are unfamiliar with Docker, the easiest way is probably to install [Docker Desktop](https://www.docker.com/products/docker-desktop/).


## Prerequisites

You will need to have some familiarity with the bash shell, and Python. If you are new to either of those, we recommend making your way through our introductory material:

- https://train.oxrse.uk/material/HPCu/technology_and_tooling/bash_shell
- https://train.oxrse.uk/material/HPCu/introductory_courses/python


## HPC

One session during the course will familiarise you with using a High Performance Computing (HPC) system.
You should try and create an account on your home institution's HPC resource.

**You should do this early as it may take some time to create an account and successfully log in.**

For instance, if you are a fellow at Oxford, you should create an account on [ARC](https://www.arc.ox.ac.uk/home) using the project `dtce-schmidt`. Fellows at other universities should consult their local HPC service.
Please get in touch with us if you cannot create an account on your local HPC service.


Schmidt Sciences Software Engineering

## Timetable

This teacher-led in-person course is 3 hours long and contains lecture and practical class elements.

| Date          | Time  | Event         |
| ------------- | ----- | ------------- |
| Tue 11-Feb-25 | 14:00 | Course starts |
| Tue 11-Feb-25 | 17:00 | Course ends   |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

This course introduces Snakemake, a Python-based system for creating scalable, automated workflows. Attendees will learn to define workflows, manage dependencies, and leverage distributed computing, all within a flexible, general-purpose framework.


## Objectives

- Set up and configure environments for running Snakemake workflows
- Build basic workflows with Snakemake, defining rules and automating dependencies
- Scale workflows with advanced features like parallelisation and wildcards
- Integrate external tools, scripts, and data formats in workflows
- Use Snakemake to automate data analysis and reproducibility in research projects


## What you will need

You should bring your own laptop.
Please follow our [standard setup instructions](https://train.rse.ox.ac.uk/event/37#What-you-will-need) to make sure you have everything ready to go when the course starts.

---


[HT25] Workflows with Snakemake

This course provides an introduction to software engineering for 1st year DTC students. It covers the main software architecture paradigms: procedural, object-orientated and functional programming, version control with Git, testing and continuous integration, project packaging and containerisation, an intro to using HPC clusters and computational workflows with snakemake.

It is assumed that students have previous programming experience. The course is taught in Python, but can be taken by students coming from a different language as an introduction to Python is built into the material.

### Week 1 (9/10/2023 - 13/10/2023)
  - Monday: Procedural and Object-orientated programming
  - Tuesday: Functional programming
  - Wednesday: Version control with Git and Github
  - Thursday: Software testing and continuous integration
  - Friday: Python packaging and containerisation with docker

### Week 2 (16/10/2023 - 20/10/2023)
  - Monday: Introduction to HPC clusters
  - Tuesday: Workflows with snakemake
  - Wednesday - Friday: Group projects

### Week 3 (23/10/2023 - 27/10/2023)
  - Monday - Friday: SABS students will start year-long software engineering projects, NERC/DTP students will continue group projects

DTC Software Engineering Course

An introduction to software engineering for 1st year DTC students

## Timetable

This teacher-led in-person course is 3 hours long and contains lecture and practical class elements.

| Date          | Time  | Event         |
| ------------- | ----- | ------------- |
| Tue 11-Feb-25 | 09:30 | Course starts |
| Tue 11-Feb-25 | 12:30 | Course ends   |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

An introduction to high-performance computing (HPC), covering connecting to HPC resources and the slurm job scheduler.
You will gain hands-on experience using Oxford's [ARC](https://www.arc.ox.ac.uk/home) system, and a temporary account will be provided for you for the duration of the course.


## Objectives

- Understand HPC systems, configure access, and connect
- Survey resources and manage filesystems on the cluster
- Submit, monitor, and inspect job execution
- Manage data and transfer files to/from the cluster
- Load software packages and install dependencies
- Prepare and launch parallel jobs, analyse performance
- Follow best practices for shared system usage and data backup


## What you will need

You should bring your own laptop.
Please follow our [standard setup instructions](https://train.rse.ox.ac.uk/event/37#What-you-will-need) to make sure you have everything ready to go when the course starts.

---


[HT25] Introduction to HPC

## Why Maths?

Maths is the language we use to quantitatively describe the world.

You are going into research, and you

- **may** need to be proficient in maths as a tool for describing what you work on
- **will** have to read and understand papers that use maths

Some of you **may** feel you don't need to know any maths, but this course will be useful to you even if you never have to write down a system of differential equations yourself.

**Data analysis:**

- Interpretation and inference
- Identify patterns, trends, relationships
- Deal robustly with uncertainty and variation

**Describe the behaviour of systems:**

- Remove ambiguity: explicit assumptions
- Quantitative hypotheses
- Make predictions, through simulation and analysis
- "If I make this intervention, I expect to see that change"
- Explain **why** something is observed


**Vital for dynamic and nonlinear systems**

- Simple intuition breaks down
- Most of biology is dynamic and nonlinear!


## Course aims

- Develop confidence in your mathematical abilities
    - Extensive practice
- Become able to communicate effectively with mathematical collaborators
- Ensure you can read and understand mathematical papers in your field
- Build on your ability to apply computational tools from **Python** to solve problems

## Topics covered

- Graphs, and basic tools such as logs
- Calculus: differentiation & integration
- Complex numbers
- Ordinary differential equations
- Linear algebra (matrices)
- Coupled systems of ordinary differential equations
- Use of Python

DTC Essential Maths Course 2023

An introduction to essential mathematics, for 1st year DTC students

slides are [here](https://docs.google.com/presentation/d/1valsEPTBRXl1-QiQRkMnNdwStJ48q7qE36HtvcL6Vk0/edit?usp=sharing)

Version Control with Git

This couse gives an introduction to Git, a popular version control systems used extensivly in academia and industry. 

## Timetable

This teacher-led in-person course is 6 hours long and contains lecture and practical class elements.

| Date          | Time  | Event                       |
| ------------- | ----- | --------------------------- |
| Tue 18-Feb-25 | 09:30 | Course starts               |
| Tue 18-Feb-25 | 12:30 | Course breaks for lunch     |
| Tue 18-Feb-25 | 14:00 | Course restarts after lunch |
| Tue 18-Feb-25 | 17:00 | Course ends                 |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

This course introduces OpenMP for parallel programming in C, covering threading, synchronization, and techniques for efficient shared-memory computation.
It equips learners to implement parallel applications effectively on multicore systems.

## Objectives

- Understand the basics of OpenMP and its use in parallel programming with C
- Implement threading to enhance performance on shared-memory systems
- Apply synchronisation techniques to manage data consistency and avoid race conditions
- Develop efficient parallel applications optimised for multicore processors
- Analyse and improve code performance using OpenMP directives and pragmas


## What you will need

You should bring your own laptop.
Ideally you should have the ability to install additional software.
At a minimum, we assume that all participants have an environment and software set up as described below.

The course materials assume that you are using a computer with a typical Linux-like environment, but it should be straightforward to set this up on a Linux, Windows or macOS operating system.
There are instructions below for the following operating systems:

- Ubuntu 24.04 LTS
- Windows (using Windows Subsystem for Linux)
- macOS

The following software is necessary, and will be installed by following any of the instructions below:

- [A C compiler compatible with OpenMP](https://www.openmp.org/resources/openmp-compilers-tools/)
- [Git](https://git-scm.com/)

In addition, we recommend you set up your preferred IDE and development tools.
If in doubt, we recommend [Visual Studio Code](https://code.visualstudio.com/) with the following extensions:

- [C/C++](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools)
- [omp-pragma](https://marketplace.visualstudio.com/items?itemName=idma88.omp-pragma)


### Ubuntu 24.04 LTS

1. Open a command-line terminal. Install the required software via apt by running the following commands:

    ```bash
    sudo apt update
    sudo apt install -y build-essential git
    ```

1. Install Visual Studio Code using snap (if you don't wish to use snap you can install via the website instructions <https://code.visualstudio.com/download>):

    ```bash
    sudo snap install --classic code 
    ```

1. Once Visual Studio Code has installed, run it and install the following extensions:
    - C/C++
    - opm-pragma


### Windows (using Windows Subsystem for Linux)

1. You can install WSL using the instructions here: <https://learn.microsoft.com/en-us/windows/wsl/install>
1. Install “Ubuntu 24.04.1 LTS” from the Microsoft Store, and then open the Ubuntu 24.04.1 LTS app and follow the instructions to create an account.
1. Install the required software via apt by running the following commands:

    ```bash
    sudo apt update
    sudo apt install -y build-essential git
    ```

1. Install Visual Studio Code directly on Windows from the following link: <https://code.visualstudio.com/download>
1. Run Visual Studio Code from Linux **by typing `code` into the Ubuntu command line** and install the following extensions:
    - C/C++
    - opm-pragma


### macOS

1. We recommend using homebrew which you can install following the instructions here: <https://brew.sh/>.
1. Use brew to install everything you need by running the following commands from a terminal window:

    ```bash
    brew install gcc make git
    brew install --cask visual-studio-code
    ```

1. Once Visual Studio Code has installed, run it and install the following extensions:
    - C/C++
    - opm-pragma

---


[HT25] Parallel Programming with OpenMP

PyBaMM Developer Training

# Preparation for the course

You will be working through the practical exercises using your own laptops. You will need to setup your laptop with a few things, which you may already have. This course requires that you have installed:

- Python 3
- The `venv` module for Python 3
- The `git` version control package
- A `bash` command-line shell
- VSCode Interactive Development Environment (IDE)

If you already have these then great, otherwise please read on for instructions on how to install them on your machine. The instructions below are different depending on the OS you have installed on your laptop.

## MS Windows

All the material assumes you have a unix-like environment, and the easiest way to get this on a Windows laptop is to install Windows Subsystem (WSL) for Linux.

You can install WSL using the instructions here:

https://learn.microsoft.com/en-us/windows/wsl/install

Once you have an Ubuntu 22.04 distribution installed use the ubuntu terminal and use this command to install all the necessary packages (Python, venv and git):

```bash
sudo apt install -y python3-pip python3-venv python3-dev build-essential git
```

We will be using the VSCode IDE for the course. Make sure to install VSCode using the *Windows* installer here: https://code.visualstudio.com/download

Once VSCode has installed, run it and install the "Python (Microsoft)" and the "WSL (Microsoft)" extensions.


## MacOS

If you have a Mac then we recommend using homebrew to install the dependencies, you can find installation instructions here: https://brew.sh/. Then you can use brew to install everything you need, like so:

```bash
brew install python3 git
brew install --cask visual-studio-code
```

Once VSCode has installed, run it and install the "Python (Microsoft)" extension.


## Ubuntu 22.04

The material has been written using Ubuntu 22.04, but if you are running another Linux distribution then you should be able to follow along with minimal changes. If you wish to install Ubuntu 22.04 then you can from here:

https://ubuntu.com/download/desktop

Once Ubuntu is installed, log in and open a command-line terminal. Install the required software via apt by typing in the following command:

```bash
sudo apt install -y python3-pip python3-venv python3-dev build-essential git
```

Then install VSCode using snap:

```bash
sudo snap install --classic code
```

Once VSCode has installed, run it and install the "Python (Microsoft)" extension.


Software Engineering for bioscientists (DTP)

This course attempts to give an introduction to a useful set of algorithms for scientific computing, along with practical skills in implementation of these algorithms in Python using linear algebra and other scientific libraries. The topics covered are

**Week 1**
- *Monday*: Linear algebra direct solvers and matrix decompositions
- *Tuesday*: Sparse matrices and iterative solvers
- *Thursday*: Solution of Ordinary Differential Equations
- *Friday*: Non-linear Optimisation

**Week 2**
- *Monday*: Bayesian Inference and MCMC sampling
- *Tuesday - Friday*: Individual projects

On the software side, we will focus exclusively on implementation in Python, using the Numpy library for multidimensional arrays, and the Scipy library for both sparse matrices and many, many other general purpose scientific computing algorithms.

Scientific Computing

This course gives an introduction to a useful set of algorithms for scientific computing, including: Linear algebra direct solvers and matrix decompositions; Sparse matrices and iterative solvers; Solution of Ordinary Differential Equations; Non-linear Optimisation; Bayesian Inference and MCMC sampling

This course provides an introduction to software engineering for 1st year DTC students. It covers the main software architecture paradigms: procedural, object-orientated and functional programming, version control with Git, testing and continuous integration, project packaging and containerisation, an intro to using HPC clusters and computational workflows with snakemake.

It is assumed that students have previous programming experience. The course is taught in Python, but can be taken by students coming from a different language as an introduction to Python is built into the material.

### Week 1 (14/10/2024 - 18/10/2024)
  - Monday: Procedural Programming 
  - Tuesday: Object-orientated programming and Functional programming
  - Wednesday: Version control with Git and Github
  - Thursday: Software testing and continuous integration
  - Friday: Python packaging and containerisation with docker

### Week 2 (21/10/2024 - 25/10/2024)
  - Monday: Introduction to HPC clusters
  - Tuesday: Workflows with snakemake
  - Wednesday - Friday: Group projects

### Week 3 (28/10/2024 - 1/11/2024)
  - Monday - Friday: SABS students will start year-long software engineering projects, NERC/DTP students will continue group projects

---

## Timetable

| Date           | Time           | Location       |
|----------------|----------------|----------------|
| Thu 4th July   | 09:30-12:30    | DTC 4/5/6/7    |
| Thu 4th July   | 14:00-17:00    | DTC 4/5/6/7    |
| Wed 10th July  | 14:00-17:00    | DTC 4/5/6/7    |
| Wed 17th July  | 14:00-17:00    | DTC 4/5/6/7    |
| Wed 24th July  | 14:00-17:00    | DTC 4/5/6/7    |

All sessions will be at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/)

---


UNIQ+ Python

Intelligent Earth: pre-course training

Pre-course training material for Intelligent Earth CDT students.

Software engineering training for the Healthcare Data Science CDT

Healthcare Data Science CDT: Software Engineering

## Timetable

Fundamentals of Software Engineering for Research consists of the following 10 half-day courses:

| Date          | Time          | Course                                                                     |
| ------------- | ------------- | -------------------------------------------------------------------------- |
| Tue 14-Jan-25 | 09:30 - 12:30 | [Object-oriented programming](https://train.rse.ox.ac.uk/event/20)         |
| Tue 14-Jan-25 | 14:00 - 17:00 | [Functional Programming](https://train.rse.ox.ac.uk/event/21)              |
| Tue 21-Jan-25 | 09:30 - 12:30 | [Version control with Git](https://train.rse.ox.ac.uk/event/22)            |
| Tue 21-Jan-25 | 14:00 - 17:00 | [Collaboration with GitHub](https://train.rse.ox.ac.uk/event/23)           |
| Tue 28-Jan-25 | 09:30 - 12:30 | [Software Testing](https://train.rse.ox.ac.uk/event/27)                    |
| Tue 28-Jan-25 | 14:00 - 17:00 | [Continuous Integration](https://train.rse.ox.ac.uk/event/26)              |
| Tue 04-Feb-25 | 09:30 - 12:30 | [Packaging and Dependency Management](https://train.rse.ox.ac.uk/event/28) |
| Tue 04-Feb-25 | 14:00 - 17:00 | [Containerization with Docker](https://train.rse.ox.ac.uk/event/29)        |
| Tue 11-Feb-25 | 09:30 - 12:30 | [Introduction to HPC](https://train.rse.ox.ac.uk/event/30)                 |
| Tue 11-Feb-25 | 14:00 - 17:00 | [Workflows with Snakemake](https://train.rse.ox.ac.uk/event/31)            |


## Description

This course covers a breadth of topics that are fundamental to writing reliable software in a research context.
This course should be accessible to almost any researcher, including research students, working in any field that involves writing code.


## Objectives

Familiarity with each of the 10 core topics covered in the course.


## Prerequisites

While a basic proficiency in Python programming and a familiarity with a Unix-like shell environment are required, it would be sufficient to have worked through the following:

- [Introduction to the Bash shell](https://train.rse.ox.ac.uk/material/HPCu/technology_and_tooling/bash_shell)
- [Procedural programming in Python](https://train.rse.ox.ac.uk/material/HPCu/software_architecture_and_design/procedural)


## Location

All courses are held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## What you will need


### GitHub account

You will need a free [GitHub](https://github.com/) account both to enrol on this course, and for much of the training material.


### Laptop

You should bring your own laptop.
Ideally you should have the ability to install additional software.
At a minimum, we assume that all participants have an environment and software set up as described below.

The course materials assume that you are using a computer with a typical Linux-like environment, but it should be straightforward to set this up on a Linux, Windows or macOS operating system.
There are instructions below for the following operating systems:

- Ubuntu 24.04 LTS
- Windows (using Windows Subsystem for Linux)
- macOS

The following software is necessary, and will be installed by following any of the instructions below:

- [Python 3.10 or later](https://www.python.org/)
- [Git](https://git-scm.com/)
- [Visual Studio Code](https://code.visualstudio.com/)


### Ubuntu 24.04 LTS

1. Open a command-line terminal. Install the required software via apt by running the following commands:

    ```bash
    sudo apt update
    sudo apt install -y python3-pip python3-venv python3-dev build-essential git
    ```

1. Install Visual Studio Code using snap (if you don't wish to use snap you can install via the website instructions <https://code.visualstudio.com/download>):

    ```bash
    sudo snap install --classic code 
    ```

1. Once Visual Studio Code has installed, run it and install the following extensions:
    - Python (Microsoft)
    - GitLens


### Windows (using Windows Subsystem for Linux)

1. You can install WSL using the instructions here: <https://learn.microsoft.com/en-us/windows/wsl/install>
1. Install “Ubuntu 24.04.1 LTS” from the Microsoft Store, and then open the Ubuntu 24.04.1 LTS app and follow the instructions to create an account.
1. Install the required software via apt by running the following commands:

    ```bash
    sudo apt update
    sudo apt install -y python3-pip python3-venv python3-dev build-essential git
    ```

1. Install Visual Studio Code directly on Windows from the following link: <https://code.visualstudio.com/download>
1. Run Visual Studio Code from Linux **by typing `code` into the Ubuntu command line** and install the following extensions:
    - Python (Microsoft)
    - GitLens


### macOS

1. We recommend using homebrew which you can install following the instructions here: <https://brew.sh/>.
1. Use brew to install everything you need by running the following commands from a terminal window:

    ```bash
    brew install python3 git
    brew install --cask visual-studio-code
    ```

1. Once Visual Studio Code has installed, run it and install the following extensions:
    - Python (Microsoft)
    - GitLens

---


[HT25] Fundamentals of Software Engineering for Research

## Timetable

This teacher-led in-person course is 3 hours long and contains lecture and practical class elements.

| Date          | Time  | Event         |
| ------------- | ----- | ------------- |
| Tue 14-Jan-25 | 09:30 | Course starts |
| Tue 14-Jan-25 | 12:30 | Course ends   |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

The Object-Oriented Paradigm expands on the Procedural Paradigm, organising code around data.
This course will introduce you to the basics of Object-Oriented Programming in Python.
Practical exercises can be undertaken in Python or C++.


## Objectives

- Understand the object-oriented programming paradigm
- Define and use classes to encapsulate data
- Explain composition and inheritance in class design
- Differentiate between composition and inheritance
- Define and apply polymorphism principles


## What you will need

You should bring your own laptop.
Please follow our [standard setup instructions](https://train.rse.ox.ac.uk/event/37#What-you-will-need) to make sure you have everything ready to go when the course starts.

If you intend to follow the C++ material, please additionally ensure you have set up a suitable environment, including having a C++ compiler installed.

---


[HT25] Object-Oriented Programming

## Timetable

This teacher-led in-person course is 3 hours long and contains lecture and practical class elements.

| Date          | Time  | Event         |
| ------------- | ----- | ------------- |
| Tue 14-Jan-25 | 14:00 | Course starts |
| Tue 14-Jan-25 | 17:00 | Course ends   |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

Functional Programming is based around the idea that programs are constructed by applying and composing or chaining **functions**.
This course will introduce you to the basics of functional programming in Python.
Practical exercises can be undertaken in Python or C++.


## Objectives

- Understand program state
- Define side effects and pure functions
- Understand the benefits and drawbacks of functional code
- Write a recursive function
- Define first-class, lambda, and higher-order functions


## What you will need

You should bring your own laptop.
Please follow our [standard setup instructions](https://train.rse.ox.ac.uk/event/37#What-you-will-need) to make sure you have everything ready to go when the course starts.

If you intend to follow the C++ material, please additionally ensure you have set up a suitable environment, including having a C++ compiler installed.

---


[HT25] Functional Programming

## Timetable

This teacher-led in-person course is 3 hours long and contains lecture and practical class elements.

| Date          | Time  | Event         |
| ------------- | ----- | ------------- |
| Tue 04-Feb-25 | 09:30 | Course starts |
| Tue 04-Feb-25 | 12:30 | Course ends   |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

This course introduces the basics of packaging and dependency management in Python.
We introduce `venv` for virtual environments, and `pip` for package management and how to structure a modern Python package and publish it to PyPI.


## Objectives

- Organise Python code into packages and manage dependencies
- Use virtual environments to isolate project dependencies
- Package and distribute Python projects using modern tooling
- Reuse and share code across projects
- Automate testing, installation, and publishing to PyPI
- Apply best practices for Python packaging


## What you will need

You should bring your own laptop.
Please follow our [standard setup instructions](https://train.rse.ox.ac.uk/event/37#What-you-will-need) to make sure you have everything ready to go when the course starts.

---


[HT25] Packaging and Dependency Management

## Timetable

This teacher-led in-person course is 3 hours long and contains lecture and practical class elements.

| Date          | Time  | Event         |
| ------------- | ----- | ------------- |
| Tue 21-Jan-25 | 14:00 | Course starts |
| Tue 21-Jan-25 | 17:00 | Course ends   |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

This course will show you how to use Git for effective collaboration with others using the GitHub platform.
After completion of the course, you will be able to contribute to projects on GitHub, including opening issues and pull requests.


## Objectives

- Describe common code review techniques
- Explain how code reviews and pull requests improve team communication and code quality
- Raise and review pull requests on GitHub
- Identify characteristics of a good code review process
- Understand and create GitHub issues
- Classify and refer to issues appropriately


## What you will need

You should bring your own laptop.
Please follow our [standard setup instructions](https://train.rse.ox.ac.uk/event/37#What-you-will-need) to make sure you have everything ready to go when the course starts.

If you intend to follow the C++ material, please additionally ensure you have set up a suitable environment, including having a C++ compiler installed.

---


[HT25] Collaboration with GitHub

## Timetable

This teacher-led in-person course is 3 hours long and contains lecture and practical class elements.

| Date          | Time  | Event         |
| ------------- | ----- | ------------- |
| Tue 04-Feb-25 | 14:00 | Course starts |
| Tue 04-Feb-25 | 17:00 | Course ends   |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

This course introduces Docker, focusing on creating reproducible computational environments for research.
Participants will learn to use, build, and manage Docker containers, with a focus on ensuring software reproducibility across different systems, including laptops, cloud, and HPC environments.


## Objectives

- Understand the basics of Docker and containerisation
- Build and manage Docker containers for research applications
- Ensure software reproducibility across different environments, including HPC, cloud, and local systems
- Use Docker to create portable and consistent computational environments
- Implement best practices for managing and distributing containerised applications


## What you will need

You should bring your own laptop.
Please follow our [standard setup instructions](https://train.rse.ox.ac.uk/event/37#What-you-will-need) to make sure you have everything ready to go when the course starts.

You should have [Docker](https://www.docker.com/) installed.
If you are unfamiliar with Docker, we recommend installing [Docker Desktop](https://www.docker.com/products/docker-desktop/) which will get your system set up with everything you need.

Finally, you should have an account on [Docker Hub](https://hub.docker.com/).

---


[HT25] Containerization with Docker

## Timetable

| Date       | Time        | Content                             |
| ---------- | ----------- | ----------------------------------- |
| Mon 04-Nov | 09:30-12:30 | Graphs                              |
| Mon 04-Nov | 14:00-17:00 | Indices, logs, and exponentials     |
| Tue 05-Nov | 09:30-12:30 | Differentiation 1                   |
| Tue 05-Nov | 14:00-17:00 | Differentiation 2                   |
| Thu 07-Nov | 09:30-12:30 | Differentiation 3                   |
| Thu 07-Nov | 14:00-17:00 | Integration 1                       |
| Fri 08-Nov | 09:30-12:30 | Integration 2                       |
| Fri 08-Nov | 14:00-17:00 | Complex numbers                     |
| Mon 11-Nov | 09:30-12:30 | Differential equations 1            |
| Mon 11-Nov | 14:00-17:00 | Differential equations 2            |
| Tue 12-Nov | 09:30-12:30 | Differential equations 3            |
| Tue 12-Nov | 14:00-17:00 | Linear algebra 1                    |
| Thu 14-Nov | 09:30-12:30 | Linear algebra 2                    |
| Thu 14-Nov | 14:00-17:00 | Systems of differential equations 1 |
| Fri 15-Nov | 09:30-12:30 | Systems of differential equations 2 |
| Fri 15-Nov | 14:00-17:00 | Systems of differential equations 3 |


## Why Maths?

Maths is the language we use to quantitatively describe the world.

You are going into research, and you

- **may** need to be proficient in maths as a tool for describing what you work on
- **will** have to read and understand papers that use maths

Some of you **may** feel you don't need to know any maths, but this course will be useful to you even if you never have to write down a system of differential equations yourself.


### Data analysis

- Interpretation and inference
- Identify patterns, trends, relationships
- Deal robustly with uncertainty and variation


### Describe the behaviour of systems

- Remove ambiguity: explicit assumptions
- Quantitative hypotheses
- Make predictions, through simulation and analysis
- "If I make this intervention, I expect to see that change"
- Explain **why** something is observed


### Vital for dynamic and nonlinear systems

- Simple intuition breaks down
- Most of biology is dynamic and nonlinear!


## Course aims

- Develop confidence in your mathematical abilities
    - Extensive practice
- Become able to communicate effectively with mathematical collaborators
- Ensure you can read and understand mathematical papers in your field
- Build on your ability to apply computational tools from **Python** to solve problems


## Topics covered

- Graphs, and basic tools such as logs
- Calculus: differentiation & integration
- Complex numbers
- Ordinary differential equations
- Linear algebra (matrices)
- Coupled systems of ordinary differential equations
- Use of Python


2024 DTC Essential Maths

Material for the 2024 edition of the Essential Maths module at the Doctoral Training Centre, University of Oxford

# Preparation for PyBaMM conference training day

You will be working through the practical exercises using your own laptops. You will need to setup your laptop with a few things, most of which you should already have installed if you use PyBaMM. If you have any issues with the installation, please let us know in advance so we can help you.

## Command Line Tool

You will need a command line tool (shell/console) in order to install PyBaMM and run Python scripts. You should have a command line tool already installed on your laptop:
 
On Windows, you can use cmd, Powershell or Git Bash (which is included in Git For Windows package - https://gitforwindows.org/).
On macOS and Linux, you will already have a command line tool available on your system. You can use a command line tool such as Bash (https://www.gnu.org/software/bash/), or any other command line tool that has similar syntax to Bash (e.g. Zsh). 
 
## Python Distribution
 
The material has been developed using the standard Python distribution version 3.8 – 3.10 and is using venv for virtual environments and pip for package management. The material has not been extensively tested with other Python distributions and package managers, but most sections are expected to work with some modifications. For example, package installation and virtual environments would need to be managed differently, but Python script invocations should remain the same regardless of the Python distribution used.
 
To download a Python distribution for your operating system, please head to Python.org (https://www.python.org/downloads/) .
 
## venv and pip
 
If you are using a Python 3 distribution from Python.org, venv and pip will be automatically installed for you. If not, please make sure you have these two tools (that correspond to your Python distribution) installed on your machine.
 
## VSCode or other IDE
 
The course does not assume you are using any particular IDE, and you could use any text editor to edit your Python scripts and run them via the command line. However, we do recommend the use of an IDE when using PyBaMM. For example, you can use Microsoft’s free Visual Studio Code (VSCode) tool.
 
You can download and install VSCode here (https://code.visualstudio.com/). Make sure you install the Python Extension via the “Extension” button on the LHS sidebar (search for “python” in the search bar and install the extension titled “Python” by Microsoft)
 
## PyBaMM

Once you have the above, you can create a virtual environment by typing the following into your command-line.

```bash
python -m venv env
```

This will create a new virtual environment in the subdirectory env. Note that `python` is the name of the python interpreter that you have installed. In some systems you might need to use `python3` instead. If you are using **Windows**, you can then activate your virtual environment by using the following command:

```bash
./env/Scripts/activate
```

If you are using Powershell you might need to capitalise `Activate`.

For **Mac or Linux** systems you can use

```bash
source env/bin/activate
```

Once you have activated your virtual environment, you can install PyBaMM into it using the Python package manager `pip`

```bash
pip install pybamm
```

For more installation information on PyBaMM, or for more options, please consult the PyBaMM documentation here: ​​https://docs.pybamm.org/en/latest/source/user_guide/installation/index.html

You can test your PyBaMM installation by creating a new file `test.py` with the following contents:

```python
import pybamm
import matplotlib.pyplot as plt
model = pybamm.lithium_ion.DFN()
sim = pybamm.Simulation(model)
sim.solve([0, 3600])
fig = sim.plot()
plt.savefig("test.png")
```

Run this script using 

```bash
python test.py
```

And it should generate an image file `test.png`, which contains the result of simulating a DFN model at time $t = 0$

PyBaMM Conference: Developer Training

This course attempts to give an introduction to a useful set of algorithms for scientific computing, along with practical skills in implementation of these algorithms in Python using linear algebra and other scientific libraries. The topics covered are

- *Monday*: Linear algebra direct solvers and matrix decompositions
- *Tuesday*: Sparse matrices and iterative solvers
- *Thursday*: Solution of Ordinary Differential Equations
- *Friday*: Non-linear Optimisation

On the software side, we will focus exclusively on implementation in Python, using the Numpy library for multidimensional arrays, and the Scipy library for both sparse matrices and many, many other general purpose scientific computing algorithms.

This course gives an introduction to a useful set of algorithms for scientific computing, including: Linear algebra direct solvers and matrix decompositions; Sparse matrices and iterative solvers; Solution of Ordinary Differential Equations; and Non-linear Optimisation.

## Timetable

This teacher-led in-person course is 3 hours long and contains lecture and practical class elements.

| Date          | Time  | Event         |
| ------------- | ----- | ------------- |
| Tue 21-Jan-25 | 09:30 | Course starts |
| Tue 21-Jan-25 | 12:30 | Course ends   |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

This course introduces the basics of version control using the Git version control system.
You will learn how to set up Git and use it to track changes in your code.


## Objectives

- Understand the benefits and basics of automated version control
- Configure Git and manage global settings
- Set up SSH keys and GitHub integration
- Create, clone, and work with repositories
- Perform the modify-add-commit cycle and track changes
- Compare, restore, and manage file versions
- Use Git revision numbers and handle conflicts
- Apply `git push`, `git pull`, and branching with `git branch` and `git merge`


## What you will need

You should bring your own laptop.
Please follow our [standard setup instructions](https://train.rse.ox.ac.uk/event/37#What-you-will-need) to make sure you have everything ready to go when the course starts.

---


[HT25] Version control with Git

## Timetable

This teacher-led in-person course is 3 hours long and contains lecture and practical class elements.

| Date          | Time  | Event         |
| ------------- | ----- | ------------- |
| Tue 28-Jan-25 | 09:30 | Course starts |
| Tue 28-Jan-25 | 12:30 | Course ends   |

It is held at the [Doctoral Training Centre](https://www.dtc.ox.ac.uk/), 1-4 Keble Road.


## Description

This course introduces the basics of automated testing and debugging in Python, using the [pytest](https://docs.pytest.org/) framework.


## Objectives


- Explain the importance of testing and the main types of tests
- Implement unit tests, use a debugger, and handle edge cases
- Apply error handling, defensive programming, and linting in CI
- Use parameterisation, code coverage, test fixtures, and mocks


## What you will need

You should bring your own laptop.
Please follow our [standard setup instructions](https://train.rse.ox.ac.uk/event/37#What-you-will-need) to make sure you have everything ready to go when the course starts.

---


[HT25] Software Testing

test

Procedural Programming

Variables

Functions

Containers

Arrays

Procedural Programming is based around the idea that code should be structured
into a sequence of **procedures** that operate on data. This course will
introduce you to the basics of procedural programming in either Python or C++.


Object-Orientated Programming

Classes

Inheritance and Composition

Polymorphism

The Object Oriented Paradigm builds upon the Procedural Paradigm, but builds code around data.
This course will introduce you to the basics of Object Oriented Programming in either Python or C++.


Functional Programming

State and Side Effects

Recursion

Higher-Order Functions

Side Effects

Functional Programming is based around the idea that programs are constructed
by applying and composing/chaining **functions**. This course will introduce
you to the basics of functional programming in either Python or C++.


Software Architecture and Design

Programming design concepts and patterns, such as procedural, object-orientated, functional, generic. Covers data structures, memory allocation, ownership.


The Bash shell

Introducing the Shell

Files and Directories

Creating Things

Pipes and Filters

Shell Scripts

Loops

The Bash shell is a command-line interface used in Unix-based operating systems
such as Linux and macOS. This course will introduce you to the basics of using
the Bash shell.


Version Control

Automated Version Control

Setting Up Git

Creating a Repository

Tracking Changes

Exploring History

Remote Repositories

Branches

Ignoring Things

This course introduces the basics of version control using the Git version control system.
We will learn how to setup Git, and use it to track changes in our code.


Packaging and Dependency Management

Packaging Python Projects

Making Packages

Reusing Packages

Sharing Packages

Virtual Environments

Introduction to CMake

This course introduces the basics of packaging and dependency management in Python and C++.
For Python, we introduce `venv` for virtual envronments, and `pip` for package management and how to structure a modern Python package and publish it to PyPI.
For C++, we introduce the CMake build system and how use it to manage dependencies and the build process.


Best Practices

Code Style

Linters

This course covers how to style your Python code, and use linters to enforce a consistant style and highlight any code that can lead to commonly encountered bugs or problems.


IDEs

VSCode

Integrated Development Environments (IDEs)provide programmers with a complete development environment to
write, edit, debug, and deploy their code. This course introduces the popular VSCode IDE, both for Python
and C++ development.


Testing

Automated Testing

Diagnosing Issues

Scaling Up Unit Testing

Testable Code and Fixtures

Using Mocks in Tests

This course introduces the basics of automated testing and debugging in Python, using the Pytest framework.


Containerisation with Docker

Introduction

Docker Command Line

Cleaning Up Containers

Running-containers

Finding Containers on Docker Hub

Creating Containers

More Complex Containers

Examples

Using Docker with Github Actions

Using Docker with Jekyll

Reproducibility and Granularity

Setup

This course aims to introduce the use of Docker containers with the goal of
using them to effect reproducible computational environments.


Snakemake

Basics: An example workflow

Advanced: Decorating

Additional features

Short tutorial

This course introduces the text-based workflow system Snakemake. Snakemake is a general-purpose workflow management system for any discipline.


Technology and Tooling

Basic tools like bash, version control, containers, IDEs, and snakemake.


Collaboration on GitHub

Git Refresher

Collaborative Worklow

Issue Management

This course will show you how to use Git for effective collaboration with others
using the GitHub platform. After completion of the course, you will be able to
contribute to projects on GitHub, including opening issues and pull requests.


Continuous Integration

GitHub Actions

Code Coverage

Documentation

This course introduces the concept of continuous integration and how to set it up for a Python project using GitHub Actions.


Software Project Management

Courses on software project management such as on collaborative programming and continuous integration.


Introduction to High Performance Computing

Why Use a Cluster?

Connecting to Remote HPC systems

Exploring Remote Resources

Introduction to Job Scheduling

Accessing Software via Modules

Transferring Files with Remote Computers

Running a parallel job

Using Resources Effectively

Using Shared Resources Responsibly

An introduction to high-performance computing (HPC), covering connecting to HPC resources and the slurm job scheduler.


Introduction to Parallelism

This session introduces the basics of parallelisation and parallel programming, covering shared and distributed memory, different parallel paradigms, and the distinctions between data parallelism and message passing.


HPC Scalability Profiling

Understanding Code Scalability

Scalability Profiling

This session introduces the concept of scalability and how to profile the scalability of your code with an increasing number of cores.


Introduction to the Message Passing Interface

Communicating Data in MPI

Point-to-Point Communication

Collective Communication

Non-blocking Communication

Derived Data Types

Porting Serial Code to MPI

Optimising MPI Applications

Common Communication Patterns

Advanced Communication Techniques

This session introduces the Message Passing Interface (MPI), teaching trainees how to use the MPI API for compiling and running applications across multiple processes. It covers point-to-point and collective communication, non-blocking functions, and handling non-contiguous memory with derived data types. Attendees will practice converting serial code to parallel, understanding scaling performance, and using profilers to optimise MPI applications.


Parallel Programming with OpenMP

Introduction to OpenMP

Parallelisation with OpenMP

Synchronisation and Race Conditions

Introduction to Hybrid Parallelism

This session covers the fundamentals of OpenMP, including its API, compilation, and execution. Participants will learn to parallelise their programs using OpenMP's pragma directives, manage threads, and control variable scoping and loop scheduling. The session also addresses thread synchronization, race conditions, and the use of hybrid parallelism with OpenMP and MPI.


High Performance Computing

An introduction to high-performance computing (HPC), covering connecting to HPC resources, job schedulers (also known as batch systems), and techniques for determining the scalability of code on HPC resources.


Intro to Python

Running Python

Variables and Types

Writing and running Python from an IDE

Built-in Functions and Help

Libraries

Analyzing and Visualizing Data

This course introduces the basics of programming in Python.
We will learn how to run Python code, and how to use variables, functions, and control flow.
We will also learn how to use Python for data analysis and visualization.


Introductory Courses

Introductory courses on programming languages.
