---
title: Connecting to the Lab Linux Machine via SSH From a Windows Computer
tags:
  - Software
description: Quick how-to on SSHing into Dr. Bittle's Linux Machine
date_created: 01/10/23
date_modified: 01/12/23
publish: true
type: note
---

SSH (Secure Shell) is a protocol for securely accessing remote servers. It will allow you to completely control the remote server using your shell (terminal, command line, whatever your call it)

## Basic Info

`ip address: <ask max if you need to know>`

If you are on a MacOS or Linux, the instructions below are largely the same, only easier. Just open a terminal and type `ssh …`

## Prerequisites

Before you begin, you'll need to make sure that the Linux machine you want to connect to has SSH enabled. This is typically the case by default, but you can check by running the following command on the Linux machine:

`$ systemctl status ssh`

You should see a message that says "Active: active (running)" if SSH is running on the machine.

**The machine that you are ssh’ing from has to be on the campus network, either via VPN or being on-premise**

## Installing an SSH Client on Windows

On Windows, you can use the built-in Windows PowerShell or the third-party software PuTTY to connect to the Linux machine via SSH.

### Option 1: Windows PowerShell

If you're running Windows 10 version 1803 or later, you already have the OpenSSH client installed on your machine. You can start an SSH session by opening a PowerShell window and running the following command:

`ssh user@hostname`

Replace "user" with the username you want to use to log in to the Linux machine, and replace "hostname" with the hostname or IP address of the Linux machine. As an example, Max’s is :

`ssh max@10.115.104.235`

### Option 2: PuTTY

If you prefer to use a third-party client or are your windows version is older, you can download and install PuTTY from the official website. Once you've installed it, open PuTTY and enter the hostname or IP address of the Linux machine in the "Host Name (or IP address)" field. Make sure the "Connection type" is set to SSH, and then click the "Open" button to start the connection.

## Connecting to the Linux Machine

Once you've started an SSH session, you'll be prompted to enter your username and password for the Linux machine. If you're using Windows PowerShell, you'll see the command prompt change to indicate that you're logged in to the Linux machine. If you're using PuTTY, you'll see a terminal window open and display a login message.

**That's it! You're now connected to the Linux machine via SSH and can run commands, transfer files, and more.**

## Extras

### TMUX

The command you run in an SSH shell will be killed when you disconnect from the machine.

You can get around that issue by running the command inside of a `tmux` session, among other ways (which I will let you Google)

### Using SSH Keys

Using SSH keys is a more secure way to connect to a Linux machine than using a password. It also saves time as you don’t have to enter your password & is the more accepted practice if you are connecting

#### Generating an SSH Key Pair on Windows

1. Open the PowerShell or command prompt and run the following command to generate an SSH key pair:

Copy code

`ssh-keygen -t rsa`

You will be prompted to specify a file to save the key and to enter a passphrase. You can press enter to leave the passphrase blank, but this is not recommended from a security standpoint.

2. The key pair will be saved to the location specified in the previous step. The private key should be kept on the Windows machine and the public key should be added to the Linux machine.

#### Adding the Public Key to the Linux Machine

1. To add the public key to the Linux machine, copy the contents of the public key file (e.g., `~/.ssh/id_rsa.pub`) to your clipboard.
1. On the Linux machine, open the `~/.ssh/authorized_keys` file. If the file doesn't exist, create it.
1. Append the contents of your clipboard to the end of the authorized_keys file and save it.

#### Using the Private Key to Connect

Once the public key has been added to the Linux machine, you can use the private key to connect to the Linux machine without entering a password.

* On Windows PowerShell, you will use the ssh command like this:

Copy code

`ssh -i path-to-private-key user@hostname`

If you add the private key folder to your path, then you don’t need to specify the `-i <path-to-private-key>`

* On PuTTY you will load the private key in to the application and connect as usual.


Quick how-to on SSHing into Dr. Bittle's Linux Machine

Connecting to the Lab Linux Machine via SSH From a Windows Computer

---
tags: null
publish: true
title: Sobol Sensitivity Analysis
description: For my own understanding
date_created: 10/01/22
date_modified: 10/05/22
---

## Basics

Let’s say that we want to analyze how the output, $y$ of the function $f$ below varies with respect to the input parameters, $\mathbf{X}$ :

$$y = f(\mathbf{X})$$

$$\mathbf{X} = \left\{ x_1, x_2, …, x_k \right\}$$

*How do we separate out the contribution of each of the parameters?*

### The One-at-a-Time Approach

Traditional *Sensitivity Analysis* approach, where one input variable $x_i$ is moved while keeping the others are their baseline. The sensitivity of the model output to a change in the input $x_i$ is commonly represented as

$$\left\vert \frac{\partial y }{\partial x_i } \right \vert_{x_i^*} = \frac{f(x_1^*, x_2^*, …, x_i^* + \Delta x_i, …, x_k^*) - f(x_1^*, x_2^*, …, x_k^*)} {\Delta x_i}$$

where $x_i^*$ represents a baseline value. However, the OAT approach has several limitations:

1. OAT not fully explore the input space, and the unexplored space grows [super-exponentially](https://en.wikipedia.org/wiki/Sensitivity_analysis#Variogram_analysis_of_response_surfaces_(VARS):~:text=OAT%20approach%20grows-,superexponentially,-with%20the%20number)
1. OAT does not consider parameters interactions
1. OAT does not handle nonlinearity

There are, however, modifications to the OAT approach that extend extend it to be *global*, including the [Morris method](https://en.wikipedia.org/wiki/Morris_method) , but they still face limitations that have been discussed in literature.

## Variance Based Approach (Sobol)

Sobol’s *Sensitivity Analysis* is a form of variance-based sensitivity analysis, that ranks parameters based on the variance they contribute to the output. Here we are referring to variance as the **expected value of the squared deviation from the mean**. For a parameter $y$, it’s commonly written as:

$$V(y) = \mathbf{E}\left[ \left( y - \mu \right)^2\right]$$

where $\mu$ is the expected value of $y$, $E[y]$. [Expanding variance](https://en.wikipedia.org/wiki/Variance#:~:text=.%20The%20expression%20for%20the%20variance%20can%20be%20expanded%20as%20follows) results in the following equivalent expression

$$V(y) = \mathbf{E}\left[y^2\right] - \mu^2$$

which will be critical to understanding Sobol’s method.

To understand Sobol’s method, we have to first start with *functional decomposition* or ANOVA decomposition, which stated that the function $y = f(\mathbf{X})$ can be represented as:

$$f(\mathbf{X}) = f_0 + \sum_i^k\sum_{j<i}^k f_{i..j}(x_i, …, x_j)$$

written more explicitly it is:

$$f(\mathbf{X}) = f_0 + \sum_i^k f_{i}(x_i) + \sum_j^k f_{i, j}(x_i, x_j) + … f_{1,2,…,k}(x_1, x_2, …, x_k)$$

The number of terms in total is $2^k$, including $f_0$. It as this point that we make a couple assumptions:

1. Each term is [square integrable](https://en.wikipedia.org/wiki/Square-integrable_function#Non-examples)
1. The functions are orthogonal, or satisfy the unicity condition:
   1. $\int_0^1 f_{i..j} (x_i, …, x_k) dx_s = 0$ for $s = i, …, k$
   1. [Ge and Mendez](https://doi.org/10.1016/j.apm.2021.04.012) call this the ANOVA-decomposition terms having zero mean

With the assumptions in mind, the functions $f_0, f_i, f_{ij}$ are represented as: $$f_0 = E\left[ Y\right] = \int f(\mathbf{X})d\mathbf{X}$$$$f_i = E_{\mathbf{X_{-i}}}\left[ Y \vert x_i\right] - E\left[Y\right]$$$$f_{i, j} = E_{\mathbf{X_{-i,j}}}\left[ Y \vert x_i, x_j\right] - f_i - f_j - E\left[Y\right]$$

\*\*Note I removed $p(x)$ from the below equations, which is the probability function of $x$. This is allowed in this context because we are assuming independent input samples.

Variance can similarly be decomposed as:

$$V(y) = \sum_i^k V_i + \sum_i^k \sum_{j<i}^k V_{ij} + ….$$

where $V(y)$ represents the total output variance.

### First-Order Effects

$V_i$ represents the output variance that is attributed to parameter $x_i$ and $V_{i,j}$ is the variance caused by the interactions of parameters $x_i$ and $x_j$:

$$V_i = V(f_i) = V_{x_i}\left(E_{\mathbf{X}_i}\left[ Y \vert x_i\right]\right)$$

$$V_{i,j} = V\left(E\left[ Y \vert x_i, x_j\right]\right) -V_i - V_j$$

Re-arranging the above equations we also have:

$$V(y) = V\left(E\left[ Y \vert x_i \right]\right) + E\left[V\left( Y \vert x_i \right)\right]$$

$$V\left(E\left[ Y \vert x_i \right]\right) = V(y) - E\left[V\left( Y \vert x_i \right)\right]$$

which shows that the total variance in $y$ is equivalent to the variance in the expected value of $y$ given $x_i$ plus the expected variance in $y$ given $x_i$. Using the definition of variance as $\mathbf{E}\left[y^2\right] - \mu^2$, the equation for $V(y)$ above can be re-written in integral form as:

$$ V(y) = \int \int … \int f^{2}(x_{1}, …, x_{k})\Pi_{i=1}^{k} dx_{i} - E^{2}[y] $$

Re-writing the equation for $V_i$ at one point $x_i = \hat{x}_i$, we have the following:

$$V(y \vert x_i = \hat{x}_i) = \int \int .. \int f^2(x_1, x_2, …, \hat{x}_i, …, x_k) \Pi_{xj\neq i}^k dx_j - E^2[y \vert x_i = \hat{x}_i]$$

For a global sensitivity analysis, we don’t want the variance at one point, but instead we want to attribute the variance caused by $x_i$ over all values of $\hat{x}_i$ . So, we add an additional integral, over $x_i$. This turns above into an equation for the **expected** value of variance over all $x_i$

$$E[V(y \vert x_i)] = \int \int .. \int f^2(x_1, x_2, …, x_i, …, x_k) \Pi_{j = 1}^k dx_{j} - \int E^2[y \vert x_i = \hat{x}_i] d\hat{x}_i$$

Using the equality from above, we see that the first term in the above equation is the same as the first term in the integral form of $V(y)$ . Subtracting above from $V(y)$, we are left with

$$V(y) - E[V(y \vert x_{i})]= \int E^{2} [y \vert x_i = \hat{x}_i] d\hat{x}_{i} - E^2[y]$$

The left side of this equation is equivalent to $V\left(E\left[ Y \vert x_i \right]\right)$ through the equation that we explored above. So, we have simplified the equation to only the square of the expected outcome of the model, which is quite easy to compute, and a double integral. Using the shortened notation of $V_i$, the equation becomes

$$V_i = \int E^{2} [y \vert x_i = \hat{x}_i] d\hat{x}_{i}^{2 } - f_{0}^2$$

Remember that $E[y]$ is the same as $\int f(X)dx$. We re-introduce the notation of $\mathbf{X}_{-i}$, which means for all values of $\mathbf{X}$ except for the $ith$ column. With this notation, the first part of the above equation becomes:

$$\int E^{2} [y \vert x_i = \hat{x}_i] d\hat{x}_{j}^{2 } = \int \mathbf{E}_{\mathbf{X_{-i}}} \left[ y \vert x_i \right] \cdot \mathbf{E}_{\mathbf{X_{-i}}} \left[ y \vert x_i \right] dx_i = \int \int f(x_1, …, \mathbf{x_i}, … x_k) f(x_1', …, \mathbf{x_i}, … x_k')d\mathbf{X}d\mathbf{X}_{-i}'$$

The interpretation is that there are two loops, one to compute the integral over $d\mathbf{X}$ and the second to compute the integral over $d\mathbf{X}_i$. It’s here that a second matrix $\mathbf{X}'$ is introduced. This matrix is an independent sample of the input space and replaces the inner for loop that would need to occur for the double integral (Sobol has a proof of the validity of this method).

If you look closely at the function, you will also notice that $x_i$ is the same in both evaluations of $f$. It is formalized using three matrices: $\mathbf{A}$, $\mathbf{B}$ , and $\mathbf{A}_{\mathbf{B}}$. The construction of the matrices is a simple process. Two independent samples of the input are made with dimensions $N \times k$ . Then, a third matrix, $\mathbf{A}_{\mathbf{B}}$, is composed by taking $\mathbf{A}$ and replacing the $ith$ column with the corresponding column of $\mathbf{B}$. The result is a 3 dimensional matrix, $N \times k \times k$.
![Pasted image 20221006070003.png](resources/pasted-image-20221006070003.png)

From this representation, it is easy to derive the number of simulations needed for the sensitivity analysis. We simply “stack” the matrices to arrive at $N (k + 2)$. Using the sample matrices, the above equation is greatly simplified in the Monte Carlo framework. Only one integral remains, with the equation being written as:

$$V_i = \frac{1}{N} \sum^{N}_{j=1} f(\mathbf{B})_{j} \left (f \left(\mathbf{A}_{\mathbf{B}}^{(i)}\right)_{j} - f\left(\mathbf{A})_{j}\right)\right)$$

#### Higher-Order Effects

So far we have been focused on $V_i$, (really $V_{x_i}\left(E_{\mathbf{X}_i}\left[ Y \vert x_i\right]\right)$), which is also referred to as the _*first-order variance_*. It is the variance that $x_i$ alone causes in the output. Yet, the power of variance-based or global sensitivity methods is that they can tell us about the interactions between parameters. Unless the model of interest is linear, there will be interaction effects, and it’s likely that we care about them. These further-order effects are usually summarized by two metrics in literature. with one investigating the total variance that $x_i$ causes and another investigating the variance due to two parameters, $x_i$ and $x_j$.

The total variance is a rather simple modification to the first order equations. Basically, instead of computing the variance induced by a parameter $x_i$, we want to find **all the other** first order variance and subtract that variance from the total. Its written as

$$V_{T_{i}}= V(y) - V_{\mathbf{X}_{-i}}(\mathbf{E}[y \vert \mathbf{X}_{-i}]) = \mathbf{E}_{\mathbf{X}_{-i}}[V_{x_i}(y \vert \mathbf{X}_{~ i})]$$

The integral form is

$$V_{\mathbf{X}_{-i}}(\mathbf{E}[y \vert \mathbf{X}_{-i}]) = \frac{1}{2} \int \left( f(x_1, …, \mathbf{x_i}, … x_k) - f(x_1, …, \mathbf{x_i'}, … x_k) \right )^{2} d\mathbf{X} dx_{i}'$$

and the Monte Carlo version

$$\mathbf{E}_{\mathbf{X}_{-i}}[V_{x_i}(y \vert \mathbf{X}_{~ i})] = \frac{1}{2N} \sum^{N}_{j=1} \left( f(\mathbf{A})_{j} - f \left(\mathbf{A}_{\mathbf{B}}^{(i)}\right)_{j} \right)^2 $$

As compared to $V_i$, this equation uses $\mathbf{A}$ and $\mathbf{A}_B$, which have all columns in common, **except** for the $ith$. Remaining are the **second order** sensitivity indicies:

$$\mathbf{E}_{\mathbf{X}_{-i,j}}[V_{x_i,x_j}(y \vert \mathbf{X}_{~ i,j})] = \frac{1}{2N} \sum^{N}_{j=1} \left( f(\mathbf{A})_{j} - f \left(\mathbf{A}_{\mathbf{B}}^{(i)}\right)_{j} \right)^2 $$

### Re-Implementing in *Rust*

[GitHub - mschrader15/sobol-sensitivity-rs: Basic Implementation of Sobol's SA in Rust](https://github.com/mschrader15/sobol-sensitivity-rs)

````rust
fn calc_s1(slices: &ArraySlices, y: &Array1<f64>, res: &mut SARes) -> () {
	// calculate the first order sensitivity indices
	// inner = (AB - A)
	let mut inner = slices.ab.to_owned().sub(&slices.a);
	// innner = B (1d) * (inner (2 dimensional) ), iterate columns
	inner.columns_mut().into_iter().map(|x| x.mul(&slices.b));
	// 1/N SUM (inner)
	let vi = inner.mean_axis(Axis(0)).unwrap();
	// Si = Vi / V
	res.set_s1(vi / y.std(0.0));
}
````

### References

1. https://doi.org/10.1016/j.cpc.2009.09.018
1. [Global sensitivity indices for nonlinear mathematical models…](https://reader.elsevier.com/reader/sd/pii/S0378475400002706?token=F67BCBFB33343244FE7F20A2CC73EB36F821494D70846B028C89C0B91F92454723C4FF3A68C8192CD0FA474A8038D50C&originRegion=us-east-1&originCreation=20221003135531)
1. [Variance based sensitivity analysis of model output…](https://reader.elsevier.com/reader/sd/pii/S0010465509003087?token=94872EBE7C12175F5CE947637D82C504A66110E766C7D624B7CD9FE47EEEAF8EFB1522FB675B090FD8952E9E55B6661B&originRegion=us-east-1&originCreation=20221003155441)


Sobol Sensitivity Analysis

---
publish: true
title: Reinforcement Learning and SUMO
description: Combining SUMO and Reinforcement Learning to improve system energy consumption
tags:
  - SUMO
  - Traffic Simulation
date_created: 04/30/21
date_modified: 01/15/22
type: note
---

## Background

The DOE project is a joint venture between the University of Alabama (UA) and Oak Ridge National Laboratory, with the aim being to improve network-wide fuel economy. Initial findings from a test corridor in Tuscaloosa show that improved light timings correspond to a 2-12% reduction in fuel consumption. As part of the project, a partnership with the German Aerospace Center (DLR), and specifically their SUMO traffic simulation software, has been created giving UA the tools to perform microscopic traffic simulations of the Tuscaloosa region and to test potential light timing improvements. SUMO has a built-in vehicle model, but the ability of that model to accurately capture vehicle dynamics is under question. To get a better estimate of energy efficiency improvements, Oak Ridge and UA are partnering to develop more detailed vehicle models.

The optimization of traffic signal timing has been researched extensively in the past, but this research typically focuses on increasing vehicle average velocity, maximizing vehicle flow, or decreasing waiting time. Most of the research in combining reinforcement learning (RL) and traffic simulation has been in influencing connected vehicle and autonomous vehicle (AV) control, with one of the classic examples being RL controlled AV’s dissipating traffic waves and bottleneck decongestion.

[Reinforcement Learning and SUMO](reinforcement-learning-and-sumo) focuses on combining SUMO micro-simulation vehicle emissions models and traffic signal optimization, using RL to potentially achieve significant improvements in overall system energy efficiency.

The environment model is a SUMO model of three intersections in Northport, Alabama. The three intersections serve industrial, commercial (Walmart, Lowes) and residential areas, respectively. The traffic in the simulation has been carefully calibrated to match real-world traffic as close as possible. Calibration is done by aggregated detector counts (stored in a SQL server) and deriving traffic routes that satisfy the counts most closely. To benchmark the RL-controlled traffic signals, the simulation will be run with the lights behaving exactly the way that they did on any given day.

## Procedure

The process of training a reinforcement learning (RL) agent to control three traffic signals can be divided into four major parts: **creating a SUMO network**, **generating traffic demand** and **following traffic signal states,** **creating an environment for the RL algorithm,** and **training the RL algorithm**. The first two were completed prior to the start of this project, but a brief overview will be presented below.

### SUMO Network

Traffic simulation starts with the creation of a network. In SUMO, the network file contains the definition of the road geometry, lane layout, intersection type, and traffic signal parameters. Figure 1 shows the completed network model, with the intersections labeled TL1-3, from west to east. It is important to note that this traffic network lies to the north of Tuscaloosa, Alabama, and represents a portion of US82, which connects Tuscaloosa to smaller towns to the west. ![](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fmax-schrader%2F8w3oD_-ZKv.png?alt=media&token=e7259e34-68c9-4b21-ac4c-ea5ccad55275)

### Demand Generation + Traffic Signal Following

Next in the traffic simulation workflow comes demand generation. Effectively comparing the RL-controlled traffic signals to those of the real world requires traffic demand to be as close to reality as possible, as intersection performance metrics are based on vehicle-based metrics. To create realistic traffic demand, real-world detector counts on a representative day (February 24th, 2020) were aggregated and corresponding route definitions were generated using SUMO’s [routeSampler](https://sumo.dlr.de/docs/Tools/Turns.html#routesamplerpy) tool. (Route definition is a SUMO term corresponding to the streets a vehicle traverses and the time it enters the network). It is not a one-shot process, and many iterations were required to meet USDOT calibration metrics.

Lumped in with demand generation is writing code to copy the light states of the target simulation day. The same database that stores the detector events also saves a history of traffic signal actions to a precision of 100ms, and thus it is possible to accurately copy the actions of the traffic signals. The ability to replay both the traffic signals and the traffic flows creates a crucial baseline to which the RL controlled traffic signals can be compared.

All demand generation and traffic signal coding occurred prior to the start of this project, so further explanation will be excluded. The source code is available in the [airport-harper-sumo](https://github.com/UnivOfAlabama-BittleResearchGroup/airport-harper-sumo) repository on GitHub.

### The Environment

This project picked up on the third part of the procedure: generating an environment for the RL algorithm to plug in to.

The original plan for this project was to utilize [Flow](https://github.com/flow-project/flow), but further analysis into its codebase made it clear that it would be hard to use with the airport-harper-sumo repository. Flow abstracts all the SUMO XML inputs, which the airport-harper-sumo repository uses almost exclusively. Not wanting to rewrite route generation and traffic-light-following code (and knowing that SUMO runs faster using XML inputs), the decision was made to write a new environment.

The environment borrowed from the Flow project in some areas, but it is essentially a custom wrapper around the [OpenAI Gym](https://gym.openai.com/). It has an observation space, an action space and a reward function. The Gym also dictates what happens when the RL algorithm takes a “step” as well as “resetting” between training iterations. Of interest for the report are the observation space, action space, and reward function, but all of the code can be found in the [reinforcement-learning-sumo](https://github.com/mschrader15/reinforcement-learning-sumo) repository on GitHub. The code is thread-safe and can be parallelized, as well as supporting policy replay with visualization and customizable outputs.

#### Observation Space

The observation space is a combination (OpenAI Tuple) of four arrays. The first array represents the number of cars in each lane approaching the three intersections in the network and **roughly** emulates a camera sitting in the center of the intersection. In practice, a circle with a radius of 100 meters is drawn from the center of each intersection and any vehicle inside that circle is added to its respective lane count. The next array reports each traffic signal’s state, which is enumerated as an integer pointing to the corresponding traffic signal state in a [SUMO traffic signal description file](https://sumo.dlr.de/docs/Simulation/Traffic_Lights.html#defining_new_tls-programs). The third array reports an enumerated value of the traffic signal’s current color and the fourth is the time that each traffic signal has been in a green phase.

#### Action Space

The goal of the action space was to emulate the way that an implemented artificial neural network (ANN) would interact with the lights. In the real world, the traffic signals cannot simply switch when requested, instead they must wait until internal count-down timers clear. Therefore, to place a restriction on the RL actor, the environment’s actor class implements the countdown timers. The RL agent can take up to eight actions at the three traffic signals, each corresponding to a movement of the traffic signal, for example serving phases 2 & 6. When the RL algorithm requests a phase transition that is not allowed or before the count-down timers clear, the request is ignored.

#### Reward

The reward function for this project was the Fuel Consumption Intersection Control Performance Index (FCIC - PI). Specifically, the FCIC-PI is a metric proposed by Stevanovic, Shayeb, & Patra in the aptly named paper [Fuel Consumption Intersection Control Performance Index](https://journals.sagepub.com/doi/abs/10.1177/03611981211004181) to incorporate fuel consumption into the common Performance Index. Stevanovic, Shayeb, & Patra present empirically calculated $K$ values for a road network in Florida, and these same values were used in the reward function implemented for this project. Vehicle delay is calculated by comparing each vehicle’s actual speed to the allowed speed of the edge. Further information can be found in the [SUMO documentation](https://sumo.dlr.de/docs/TraCI/Interfacing_TraCI_from_Python.html#retrieve_the_timeloss_for_all_vehicles_currently_in_the_network). The number of stopped vehicles is the count of all vehicles in the network with a speed less than 0.3 m/s.

To introduce stability, the reward at a given simulation step is calculated as the moving average of the past 10 simulation seconds of rewards. The choice of 10 seconds is completely arbitrary.  Going forward the role that the moving average plays in the result should be investigated.

### RL Algorithm + Training

Training the RL agent was the real focus of this project, but CPU time aside, it was the least time-consuming part. Reinforcement learning algorithms were tested on the environment using the [ray RLlib library](https://docs.ray.io/en/master/rllib.html). Ray is a distributed computing library that handles the overhead required for parallelizing training rollouts and optimizing policies. RLlib has many different reinforcement learning algorithms but, according to results in [the Flow benchmark](https://flow-project.github.io/papers/vinitsky18a.pdf) paper, only evolutionary strategies (ES) and proximal policy optimization (PPO) were trialed for this project. The authors found that gradient-free methods, specifically ES, perform the best for traffic signal optimization. PPO was applied to the environment to compare ES to a more classical RL method.

#### **Evolutionary Strategies**

ES is a black box optimization algorithm method that is not based on the Markov Decision Process (MDP). [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://arxiv.org/pdf/1703.03864.pdf) lays out several benefits for ES over MDP-based RL algorithms, but the fact that it performs well in environments with long time-horizons and complicated reward structures is important to traffic simulation, specifically traffic signal optimization. As presented in the previous section, the reward used for this project is a rolling average derived from vehicle behavior. The traffic signals certainly influence vehicle behavior, but so do interactions between other vehicles. The nature of traffic complexities makes the reward inherently complicated and, as the results section will show, ES proved the only method that converged on a solution.

## Results

The results presented in this section represent the findings from applying the ES and PPO RL algorithms to the [airport-harper-sumo](https://github.com/UnivOfAlabama-BittleResearchGroup/airport-harper-sumo) SUMO model. Traffic was simulated starting at 6:30 AM on the target day (Feb. 24th, 2020), then the network underwent an hour-long warmup period until the RL algorithm took over at 7:30 AM. The RL algorithm was then trained over 15 minutes of simulation, until 7:45 AM. All training was done on a Linux machine running Ubuntu 18.04 with two AMD 32-core processors.

### Algorithm Comparison

Both ES and PPO were used to train an artificial neural network with three hidden layers (100, 50, 25). The training hyper parameters were taken from [the Flow benchmarks.](https://flow-project.github.io/papers/vinitsky18a.pdf) They were trained for 500 iterations with 64 rollouts per iteration.

The figure below shows the average episode reward for the two different algorithms considered. It clearly shows that PPO does not converge to a solution, whereas the ES algorithm does more than 80% of its learning in the first 10% of the training period and converges. A video of the ES training iterations plotted below has been uploaded to [YouTube](https://youtu.be/wDe6mTLmpL4).

![](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fmax-schrader%2F4_odcf9vT1.png?alt=media&token=fe2c0a84-e5de-406c-8425-2af9cb7b1498)

### RL Vs Real-World

After training the RL agent, various comparisons were done between the real-world traffic signals and the RL-controlled ones, specifically the ES algorithm from above, as the PPO algorithm did not converge to a reasonable policy. Before the analysis begins, it is important to note that the real-world traffic signals function in simulation like static traffic lights, meaning they switch according to a time (corresponding to the date and time they switched in the real world) and not in reaction to simulation traffic. While the simulated traffic is calibrated to match the actual traffic as best as possible, the fact that the that the signals are not reactive puts them at a disadvantage to the RL trained traffic signals. An interesting research opportunity is to pair [software-in-loop traffic controllers](https://youtu.be/28x2Iye3FRM) with this calibrated simulation and conduct a more realistic comparison of RL vs. real-world.

The figure below shows the rolling reward over 15 minutes of simulation time. The RL agent stabilizes the reward more effectively than the real-world traffic signals, and clearly is not subject to the same dips that the real-world traffic signals cause. These dips correspond to periods when the side streets are being served by the real-world traffic signals.

![](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fmax-schrader%2F9MCP-7isqO.png?alt=media&token=df904bd3-e941-4cd3-aad5-a1fc41c99a4b)

A more detailed analysis of the simulation reveals why the reward function of the RL-controlled network is better than the real-world reward. The figure below illustrates some of the metrics influencing the reward function.

![](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fmax-schrader%2Fg0bsQ-CUPL.png?alt=media&token=0f6a7680-8a0b-4a59-983d-8ddf0dd1908c)

The RL algorithm does a better job at smoothing all network metrics and scores better on average. It most drastically cuts down the network waiting time, which is likely due to the reward function penalizing the number of stopped vehicles with the $K$ factor. The average speed in the network is almost uniformly higher. The average MPG for vehicles in the network is higher as well, however only slightly.

The next two figures highlight the difference between the real world and RL controlled traffic signals. In these figures, dark green corresponds with a green phase, while light green is a yielding or flashing left turn. In the real world traffic signal scenario, the traffic signal pattern is easy to spot: the traffic lights are coordinated, and they have a fixed cycle length.

On the other hand,  in the RL-controlled traffic signal scenario, there is no pattern. **The RL-agent is reacting to traffic flows and making decisions to maximize reward. It is interesting that it can switch so rapidly between phases and still maintain high average network speeds, but the figure above shows that it is successful.**

#### Real World Traffic Signals During Simulation Period

![](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fmax-schrader%2FbGeusZyZMr.png?alt=media&token=2fce5398-1604-4d4c-af74-8ffbd1c07c98)

#### **RL-Controlled Traffic Signals During Simulation Period**

![](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fmax-schrader%2FyAe6vO9VhL.png?alt=media&token=d05283d2-253c-45f3-a912-fd5c34147db8)

## Conclusions

Creating [reinforcement-learning-sumo](https://github.com/mschrader15/reinforcement-learning-sumo) was a lot of fun! Working on this project created an extensible codebase and baseline to carry over into the DOE Project. It will be interesting to see how the different reward functions change the trained outcomes going forward, and what those outcomes look like.

One clear conclusion of this project was that ES is at the very least a more forgiving reinforcement-learning framework. By simply copying the hyperparameters used in the Flow benchmarks, a solution was converged upon. With PPO, there was no such luck. That being said, more RL algorithms should be trialed with various hyper parameters before concluding that ES is the definitive way forward.

Not presented in the results was an attempt at transfer learning, where ANN trained in Figure 2 over the 7:30 – 7:45 time interval was transferred to 17:30 - 17:45 and trained for another 500 iterations. The final solution was converged upon in very few iterations and mirrored the results from training the RL agent on the 17:30 - 17:45 interval from scratch. It would be interesting to investigate transfer learning on the network further, as it could potentially save considerable computation time vs. training on longer time intervals. SUMO slows down significantly when traffic congestion increases, which happens in early RL training iterations. Longer simulations mean more congestion, so if transfer learning could be used instead, that would likely be more computationally efficient.


Combining SUMO and Reinforcement Learning to improve system energy consumption

Reinforcement Learning and SUMO

---
publish: true
title: GLE Data Tracking App
description: ''
type: project
date_modified: 01/08/22
date_created: 01/04/22
---

* ![](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fmax-schrader%2FLhsEjWjeTZ.png?alt=media&token=9449f2fb-ab37-44e6-b948-0c8c07332122)


GLE Data Tracking App

---
publish: true
title: High Conviction Bets
description: A note to myself on bet-sizing
tags:
  - Investing
type: note
date_modified: 01/08/22
date_created: 01/04/22
---

There have been times in my life where I have **known** with high conviction that a certain scenario will play out. They don’t happen often (once every couple of years?), but when the broader market doesn’t reflect this sentiment, I need to bet the house.

Below are personal notes on scenarios where I had high-conviction, and what I did and should have done differently.

## Coronavirus

By virtue of the people I follow on *Twitter*, I was aware of *COVID-19* in January 2020. Several of the *Complexity Science* professors / writers that I follow had written articles on how the transmission would play out, and so I had no doubt that it would spread out of China and to the US rapidly. Looking back, this sounds like a no-brainer, but at the time I was very early on this opinion.

Around late-January 2020, I also read on *Twitter* that surgical mask stocks ($APT) had surged during previous airborne epidemics. The market was not reflecting what I saw as a reality, and so I had two advantages:

1. N95 Mask Companies were trading near their 30 day trailing average
1. I had high conviction that *COVID-19* would impact the stock market drastically in the coming months

At the time, I had a Robinhood account, so I bought $APT at $5 and sold on the way up, selling some shares at a high of $36 in late February. That’s pretty good, but what could I have done better?

1. At the time, I didn’t understand or really know about options. Looking back, I could have bout call options. I don’t have the ability to look at their historic prices, but I am assuming that like the stock, they didn’t have the upcoming volatility priced in.
1. I also should have invested more $ into $APT. I was in my first semester of grad school, so not flush with cash by any means, but I was very conservative with my bet sizing.
1. Nearly all gains that I made from $APT, I subsequently lost trying to trade VIX with $UVXY. The lesson here is that is okay to sit on the sidelines. I had made 100+% return on my knowledge advantage with $APT in less than a month. I should have taken my earnings and stayed out of the highly uncertain market in March.

## Spotify

I first used *Spotify* in 2012 or 2013? I think it was still a relatively small Swedish startup at that time. I knew from the beginning that it was worlds better than any other streaming service. At the time I was 15 years old, and didn’t have access to invest.

There’s not much I could have done differently at the time, but looking towards the future, when I come across a startup that I feel with high-conviction that is transformative, I need to figure out how to invest.


High Conviction Bets

---
publish: true
relativeSavePath: /Library/Software
date_modified: 01/08/22
date_created: 01/04/22
title: SUMO Build Instructions
description: Covers the process of building SUMO from source on Windows
tags:
  - SUMO
type: note
---

## Overview

This document will walk through what I have found to be the best way to build *SUMO* as well as contributing to the project. In general, the workflow looks like:

1. Fork [SUMO](https://github.com/eclipse/sumo)
   1. Max has already forked to [BittleResearchGroup](https://github.com/UnivOfAlabama-BittleResearchGroup/sumo?organization=UnivOfAlabama-BittleResearchGroup&organization=UnivOfAlabama-BittleResearchGroup)
1. Clone the [SUMO fork](https://github.com/UnivOfAlabama-BittleResearchGroup/sumo?organization=UnivOfAlabama-BittleResearchGroup&organization=UnivOfAlabama-BittleResearchGroup) to your local computer
   1. Good practice to always check the upstream branch ([SUMO](https://github.com/eclipse/sumo)) for commits that might impact what you are working on, as it is under active development
      1. If a change has been made, merge the upstream branch into your development branch and continue
1. Create a branch for your development & switch to that branch
1. Build SUMO in it’s entirety once
1. Development Loop
   1. Make changes to SUMO’s internals
   1. Build (just `sumo` or `sumo-gui`)
   1. Test
1. Ready for contribution?
   1. [Follow the instructions for contributing](https://sumo.dlr.de/docs/FAQ.html#how_do_code_contributions_work)
   1. Create a pull request on Github

## Detailed Instructions

### Cloning the Repository

Cloning the repository should be the same on all platforms. If you haven’t used Github since August of 2021, you may have to create a personal access token. This [StackOverflow](https://stackoverflow.com/a/68781050) answer walks you through the process concisely.

Then, assuming that you are using the [BittleResearchGroup](https://github.com/UnivOfAlabama-BittleResearchGroup/sumo?organization=UnivOfAlabama-BittleResearchGroup&organization=UnivOfAlabama-BittleResearchGroup) SUMO branch, you would clone it using

````
git clone https://github.com/UnivOfAlabama-BittleResearchGroup/sumo.git
````

or if you setup an SSH key:

````
git clone git@github.com:UnivOfAlabama-BittleResearchGroup/sumo.git
````

### Prepping the Repository up for Development

First, make sure that your `git config` is setup properly, so that you can contribute to SUMO if the time comes. See [instructions for contributing](https://sumo.dlr.de/docs/FAQ.html#how_do_code_contributions_work).

You should then create a branch to track your development

````
git branch max-dev
git switch max-dev
````

If you want to add the ability to track and merge SUMO’s main repository:

````
git remote add upstream https://github.com/eclipse/sumo.git
git fetch --all
````

Then, if you ever want to merge  [SUMO’s main branch](https://github.com/eclipse/sumo) into your development branch:

````
git fetch --all
git merge upstream/main
````

### Prepping VSCode for Development (on Windows)

I have tried both [VSCode](https://code.visualstudio.com/) and [CLion](https://www.jetbrains.com/clion/) and come to the conclusion that VSCode (with add-ons) is the better *IDE*.

#### Installing the Extensions & Necessary C++ Build Tools

The first step will be adding the necessary VSCode extensions, which the [C/C++ Extension Pack](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools-extension-pack) should cover.

I like to use package managers (homebrew on MacOS and apt-get on Linux), so I installed [Chocolatey](https://docs.chocolatey.org/en-us/) on Windows. You can install Chocolatey by following the [instructions here](https://docs.chocolatey.org/en-us/choco/setup).

Once [Chocolatey](https://docs.chocolatey.org/en-us/) is installed, you can install `cmake` with (**Make sure that you are inside of an administrative shell**)

````
choco install cmake
````

#### Installing Necessary C++ Libraries

SUMO provides a bundle of necessary C++ Libraries for Windows development. You can clone the libraries from [this repository](https://github.com/DLR-TS/SUMOLibraries/). You should clone the library in the `sumo` folder per their instructions.

````
git clone --recursive https://github.com/DLR-TS/SUMOLibraries
````

I found that I also need to set the `SUMO_LIBRARIES` environmental variable. If that is the case for you, I would suggest cloning the repository outside of `sumo` folder - as it will make `git` more complicated.

You will likely have to close and re-open VSCode if you set the environmental variables.

#### Configuring CMAKE

After installing the extensions and libraries. Open the `sumo` folder in VSCode. The following notification should pop up in the bottom right corner:

![Pasted image 20211220142923.png](resources/pasted-image-20211220142923.png)

Select “Yes” to have CMAKE configure the project according to  `CMakeLists.txt`

Once the configuration finishes, the bottom bar should look something like below:

![vscode_footer.png](resources/vscode_footer.png)

* The **CMake button** allows you to select the build variant.
  * I have not strayed from `Debug + FMI + Python + noJava`
* The **Active Kit Button** is for choosing the CMake Kit
  * On UNIX, this is clang or gcc
  * On Windows, this is Visual Studio Community 2019 Releases
    * You have to have the [Visual Compiler Tools](https://visualstudio.microsoft.com/downloads/) installed
* The bracketed **\[ALL_BUILD\]** button allows you to select the *program* to build.
  * This allows you to build **only** the executable that you want to debug.
  * Initially I would recommend building all the sources, and then switching to just **\[sumo-gui\]**  of **\[sumo\]** which saves compile time while debugging.

### Building SUMO

Its possible that your selected build target from above already built. If not, click on the ⚙️ Build button and CMake will start building the target.

* You should first build all of the sources and then change to just `[sumo]` or `[sumo-gui]`
* ==Reach out to Max if errors occur during the build process==

### Debugging SUMO

Debugging in VSCode is reliant on the `launch.json` file. To create it, go to the “Run and Debug” tab on the left sidebar and select **create a launch.json file**. The auto-configuration doesn’t matter as you are going to copy the file below.

The following should be a working example of a `launch.json` file that runs the NEMA test files from SUMO. It is platform specific, so if you are on something other than Window, reach out to Max.

````json
{
 // Use IntelliSense to learn about possible attributes.
 // Hover to view descriptions of existing attributes.
 // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
 "version": "0.2.0",
 "configurations": [
 {
	 "name": "SUMO NEMA Debug",
	 "type": "cppvsdbg",
	 "request": "launch",
	 // from https://github.com/microsoft/vscode-cmake-tools/blob/main/docs/debug-launch.md
	 "program": "${command:cmake.launchTargetPath}",
	 // These args come from ${workspaceFolder}/tests/sumo/basic/tls/NEMA/custom_detector
	 "args": [
		 "--net-file=net.net.xml",
		 "--routes=input_routes.rou.xml",
		 "-a",
		 "input_additional.add.xml,input_additional2.add.xml",
		 "-t",
		 "--no-step-log"
 	],
	 "stopAtEntry": false,
	 "cwd": "${workspaceFolder}/tests/sumo/basic/tls/NEMA/custom_detector",
	 "environment": [],
	 "console": "externalTerminal",
 }
 ]
}
````

The launch task configured in `launch.json` does a couple of things automatically:

1. It builds the target specified in the bottom tool bar (**\[sumo-gui\]**)
1. It changes the directory of the subsequent task to what’s specified in `“cwd”`
1. It launches the built target with the arguments specified in `“args”`in an external terminal
1. It attaches a debugger to target

#### Attaching to an Existing SUMO Instance

If you are following along in *Single Intersection Airport Harper*, you were probably routed here.

To debug a “running” simulation, we need to attach to the process. The following addition to the `launch.json` will allow us to do just that

````json
        {
            "name": "(cppvsdbg) Attach",
            "type": "cppvsdbg",
            "request": "attach",
            "program": "${workspaceRoot}/bin/sumo-guiD",
            "processId": "${command:pickProcess}",
        },
````

Then run the “(cppvsdbg) Attach” in the debug window and it should open a dialog box for you to search for the process ID. Just simply search “sumo” and you should find the process. This will attach the debugger and you can step through the simulation!

### Running a Simulation with the Debugger

Because this document was created to help extend *NEMA dual ring* controller, we will start by debugging its entry point: `src\microsim\traffic_lights\NEMAController.cpp`.

The bulk of the logic occurs inside the [NEMALogic](https://github.com/UnivOfAlabama-BittleResearchGroup/sumo/blob/99315ba0f2ed1f4a9411520633dbcab8bfeb8bcd/src/microsim/traffic_lights/NEMAController.cpp#L593) function.

You can place a break point on any line in this code and start debugging.

[The annotated PDF](https://github.com/mschrader15/notes-public/blob/main/Library/Software/NEMA_flow_correct.pdf) provides a graphical view of the NEMALogic function.

**More will be added to this document as necessary…**


Covers the process of building SUMO from source on Windows

The Garden

Connecting to the Lab Linux Machine via SSH From a Windows Computer

Sobol Sensitivity Analysis

Reinforcement Learning and SUMO

GLE Data Tracking App

High Conviction Bets

SUMO Build Instructions