
We've done [Pip-Boy discovery and relay](/posts/fallout-4-service-discovery-and-relay/) and
[created new libraries](/posts/decoding-the-pipboy-database-with-vault-community).
Now let's put them to good use. The first half of this tutorial does _not_ require having
a copy of Fallout 4 because we'll start by simulating data.

Things we'll cover:

- Simulating data
- Observables and RxJS
- Bundling code for the browser via `browserify` and `babel`
- Drawing on canvas

![reactive map](https://cloud.githubusercontent.com/assets/836375/11960060/5b49ac9c-a896-11e5-8e50-da9ce4330e1d.gif)

## Tools required

You need [node 5+ and npm 3](https://nodejs.org/en/) to follow this
tutorial. If you're gripping tightly to an older node version, try out
[nvm](https://github.com/creationix/nvm) or [n](https://github.com/tj/n) for all
your node environment switching needs.

## Generating Random Players

Set up a new project for this, by creating a new directory, changing to that
directory, and running `npm init`. You can accept the defaults if you like, or
mix it up.

```bash
mkdir multi-pip-fun
cd multi-pip-fun
npm init
```

We're making a map of players, so let's start by generating some
random players. We're going to get the list of player names that [codsworth can
speak](http://www.themarysue.com/fallout-4-names/) from.

For this tutorial, I'll use the package `clean-codsworth-names`. You can install
it too:

```bash
npm install --save clean-codsworth-names
```

You can also use the raw list, which has plenty of words to make the juvenile in
us giggle.

```bash
npm install --save codsworth-names
```

Now, create `fakes.js`, setting up new players and where they are on the map.

```javascript
const codsworthNames = require("clean-codsworth-names");
const mapSize = 2048;

// newPlayer creates a new player with a name
// and some random coordinates
function newPlayer(name) {
  return {
    name: name,
    x: Math.round(Math.random() * (mapSize - 1)),
    y: Math.round(Math.random() * (mapSize - 1)),
  };
}

// create a set of players with coordinates
const players = codsworthNames.map(newPlayer);

// Just to try this out, print out a few of these
console.log(players.slice(0, 4));
```

```bash
$ node fakes.js
[ { name: 'Aaliyah', x: 533, y: 81 },
  { name: 'Aaron', x: 612, y: 1091 },
  { name: 'Abigail', x: 422, y: 1918 },
  { name: 'Abram', x: 210, y: 541 } ]
```

Give each of these players a unique ID and provide
an adjective for their name (for example, Crazy Dave) by relying on
two more packages: `uuid` (for unique IDs) and `adjectives` (like it says on the tin)

```bash
npm install --save uuid adjectives
```

Now we'll `require` these libraries and modify `newPlayer`:

```javascript
const uuid = require("uuid");
const adjectives = require("adjectives");

// Upper case the names
const Adjectives = adjectives.map((adj) => {
  return (
    adj.charAt(0).toUpperCase() + adj.slice(1)
  );
});

// Simple grabber of a random element from an array
const choice = (arr) =>
  arr[Math.floor(Math.random() * arr.length)];

function newPlayer(name) {
  const adj = choice(Adjectives);

  return {
    name: `${adj} ${name}`,
    id: uuid.v4(),
    x: Math.round(Math.random() * (mapSize - 1)),
    y: Math.round(Math.random() * (mapSize - 1)),
  };
}
```

We've got some interesting characters already

```javascript
$ node fakes.js
[ { name: 'Exultant Aaliyah', x: 113, y: 403 },
  { name: 'Responsible Aaron', x: 1840, y: 1228 },
  { name: 'Impartial Abigail', x: 704, y: 1093 },
  { name: 'Small Abram', x: 223, y: 1163 } ]
```

## Go on a random walk

How can we simulate players moving?

By taking them on a [random walk](https://en.wikipedia.org/wiki/Random_walk).
On each iteration of the simulation, we'll add 1, 0, or -1 to the player's x
and y positions.

![let's go on a stroll](/assets/blog/lets-build-a-reactive-map/strolling-on-the-map.png)

There are a few ways to do this and we'll rely on some mathematical properties to
make this fast.

We can randomly acquire 0 and 1 by using `Math.round(Math.random())`. To get -1,
use the mathematical property that `cos(π) = -1` and `cos(0) = 1`. Multiply
that by `Math.round(Math.random())` and we have a nice little formula for generating
-1, 0, and 1.

```javascript
const change =
  Math.cos(Math.PI * Math.round(Math.random())) * // -1 or 1
  Math.round(Math.random()); // 0 or 1
```

Now we just have to apply it to a given point

```javascript
// Random walk -1, 0, 1
function walk(pt) {
  const change =
    // -1 or 1
    Math.cos(Math.PI * Math.round(Math.random())) *
    Math.round(Math.random()); // 0 or 1
  const newPt = pt + change;
  if (newPt < 0 || newPt >= mapSize) {
    return pt;
  }
  return newPt;
}
```

Try it out on one of the characters by appending this to the end of fakes.js:

```javascript
const player = players[0];
console.log(player);
player.x = walk(player.x);
player.y = walk(player.y);
console.log(player);
```

```
$ node fakes.js
{ name: 'Mountainous Aaliyah', x: 312, y: 1506 }
{ name: 'Mountainous Aaliyah', x: 311, y: 1506 }
```

## Keep those players moving!

To generate a continuous stream of player data, we're going to create and learn
about Observables through RxJS.

```
npm install --save @reactivex/rxjs
```

### Down the observable rabbit hole

![Join the rabbit](/assets/blog/lets-build-a-reactive-map/rabbit-hole.png)

If you've worked with JavaScript in the front end (or the back end for that matter),
you've gotten used to these core Objects:

<table>
   <th></th><th>Single return value</th><th>Multiple return values</th>
   <tr>
      <td>Pull / Synchronous <br /> Interactive</td>
      <td>Object</td>
      <td>Iterables like Array,  Set,  Map, Object</td>
   </tr>
   <tr>
      <td>Push / Asynchronous <br /> Reactive</td>
      <td>Promise</td>
      <td>????</td>
   </tr>
</table>

One of my favorite additions to JavaScript is the `Promise`. It makes a lot of
asynchronous code really clean, especially with `Promise` chains.

```javascript
fetch("/players.json")
  .then((resp) => response.json())
  .then(action)
  .catch(whoa);
```

What about cases where we want a promise to yield multiple values? Do we wait
for all the values to get computed, relying on `Promise.all()`? Not if we want
the intermediate values. In the end we'd have to fall back on `createEvent`/`EventEmitter`.

On top of that, we usually want to perform filtering and routing based on those
messages for separate consumers of the data. What we really need is a stream
of messages that we can operate on like an `Array`, similar to how we
created new players:

```
const players = codsworthNames.map(newPlayer)
```

There just _has_ to be something to fill that missing piece of the table. Turns out,
there's one more data type to come to a JavaScript near you: `Observable`

<table>
   <th></th><th>Single return value</th><th>Multiple return values</th>
   <tr>
      <td>Pull / Synchronous <br /> Interactive</td>
      <td>Object</td>
      <td>Iterables like Array,  Set,  Map, Object</td>
   </tr>
   <tr>
      <td>Push / Asynchronous <br /> Reactive</td>
      <td>Promise</td>
      <td>Observable</td>
   </tr>
</table>

Observables are asynchronous data streams, [_from the future_](https://tc39.es/proposal-observable/).

Technically, this pattern has been around for a while. People have been using Observables
[in Ruby](http://ruby-doc.org/stdlib-1.9.3/libdoc/observer/rdoc/Observable.html), across
languages through [Reactive Extensions](https://github.com/ReactiveX),
libraries like [Bacon](https://baconjs.github.io/), and many more. Where it
_really_ shines in Javascript is with RxJS.

Both node's EventEmitter and RxJS's Observable are implementations of the
Observer design pattern. The one big difference you'll see is how you can operate
on Observables like core primitives. While working on [pipboylib](https://github.com/RobCoIndustries/pipboylib),
several contributors highlighted how wonderful it would be if our interface was
based on RxJS.

Here's an example of what it looks like to get the player position continuously:

```javascript
pipboydb
  .map((x) => x.Map.World.Player)
  .map((x) => ({
    x: x.X,
    y: x.Y,
    deg: x.Rotation,
  }))
  .distinctUntilChanged()
  .subscribe((x) => {
    console.log("Player Position:", x);
  });
```

As the game updates, we see the change come into our subscription callback and
display our position over time. You can even use this to write out your
[localmap](https://github.com/RobCoIndustries/pipboylib/blob/master/examples/localmap.js).

![](https://cloud.githubusercontent.com/assets/836375/11534784/277a8f3c-98d6-11e5-8d80-a1213dd3adf3.png)

In [pipboy](https://github.com/RobCoIndustries/pipboy) we use this to display
your local map continuously like some sort of crazy sonar.

![localmap cray cray](https://cloud.githubusercontent.com/assets/2737108/11907104/8dd9d7da-a5d1-11e5-91de-400307f54a53.gif)

Let's learn more about Observables so you can do real-time apps like this too.

### Learn some Rx

Let's learn how to work with Observables. Pop open a
node terminal and `require('rx')`:

```javascript
> var Rx = require('rx')
```

Start with a simple `Observable` that emits a new value on each interval.

```javascript
> var obs = Rx.Observable.interval(10)
```

To subscribe to the event stream from this `Observable`, we use `subscribe`.
To make sure that we don't get overwhelmed with the stream, we'll chain with the
`take` operator, making our resulting subscription only get a few values.

```javascript
> var o = obs.take(4).subscribe(console.log)
0
1
2
3
```

We can operate on this stream, too, mapping the values. This interval `Observable`
is giving us an incrementing counter over time. Let's map the values
to something new that we'll subscribe to.

```javascript
> o = obs.map(x => x*10)
         .take(4)
         .subscribe(console.log)
0
10
20
30
```

Since we use `take`, the stream completes after 4 elements are emitted, which
allows us to perform a reduction:

```javascript
> var o = (
  obs.take(4)
     .reduce((x,y) => x + y, 0)
    .subscribe(console.log)
  )
6
> // Result comes from 0 + 1 + 2 + 3
```

How could we operate on an _infinite_ stream? For this, Rx provides the `scan`
operator to emit intermediates.

```javascript
> var o = (
  obs.scan((x,y) => x + y, 0)
     .take(4)
     .subscribe(console.log)
  )
0
1
3
6
```

This is enough tooling for us to start generating simulated player data. We'll
also use this to great effect when aggregating multiplayer data. For a more
thorough introduction to Reactive Programming, check out the
[Introduction to RP you've been missing](https://gist.github.com/staltz/868e7e9bc2a7b8c1f754).

## Generate player data with RxJS

Let's start by creating a single live player who updates themselves on
an interval. Add this to `fakes.js`:

```javascript
function livePlayer(player, period) {
  if (!player) {
    throw new Error("need a player");
  }
  if (!period) {
    period = 500;
  }
  return Rx.Observable.interval(period).scan(
    (p) => {
      return {
        name: p.name,
        id: p.id,
        x: walk(p.x),
        y: walk(p.y),
      };
    },
    player
  );
}
```

To create this for all the players, we'll use the `merge` operator to bring
lots of live players together. `merge` takes N many Observables and combines
them into one Observable stream.

```javascript
// create a set of players with coordinates
const defaultPlayers =
  codsworthNames.map(newPlayer);

function livePlayers(players, period) {
  if (!players) {
    players = defaultPlayers;
  }
  if (!period) {
    period = 500;
  }

  return Rx.Observable.merge(
    ...players.map((p) => livePlayer(p, period))
  );
}
```

Let's export this function for use by node. Try this out by appending this to the bottom of `fakes.js`:

```javascript
livePlayers(players)
  .take(4)
  .subscribe((x) => console.log(x));
```

Then, run `fakes.js` to get output like the following:

```javascript
{ name: 'Robust Aaliyah',
  id: 'dbfb8fdc-6930-4c59-a849-144f47a5fd93',
  x: 900,
  y: 1198 }
{ name: 'Alleged Aaron',
  id: '8bb2734c-8699-493a-bf57-fecf1a135077',
  x: 1648,
  y: 484 }
{ name: 'Tremendous Abigail',
  id: '749744d2-5e8e-4737-bd57-2aef0c1dae23',
  x: 829,
  y: 450 }
{ name: 'Womanly Abram',
  id: '7e6b1614-efc9-4549-8673-4ddc0ea62531',
  x: 920,
  y: 131 }
```

Now our players are being generated. It's time to put them on the screen.

![browserify logo](/assets/blog/lets-build-a-reactive-map/browserify.png)

To convert our command line scripts to something we can put up on the
web for everyone, we're going to use a bundler called `browserify` to package
all the JavaScript goodness into one file, and then ensure that all of the
JavaScript works on all the various browsers by using `babel` via `babelify`.

```bash
npm install --save-dev browserify babelify babel-preset-es2015
```

In your `package.json`, add `build` to your `scripts` section and create a
`babel` section:

```json
"scripts": {
  "build": "browserify index.js -t babelify -o bundle.js"
},
"babel": { "presets": ["es2015"] },
```

Next, write an `index.html`:

```html
<html>
  <head>
    <meta charset="utf-8" />
  </head>
  <body>
    <script
      type="text/javascript"
      src="bundle.js"
      charset="utf-8"
    ></script>
  </body>
</html>
```

This sets up loading the bundle within the main page. For the `index.js` file, start with something fairly simple:

```javascript
const fakes = require("./fakes");

fakes
  .livePlayers()
  .take(1)
  .subscribe((player) => {
    console.log(player);
    document.write(JSON.stringify(player));
  });
```

Now run `npm run build`. That should build your sources. To see your work in action,
open up `index.html` in the browser. You should see a generated player.

```json
{
  "name": "Robust Aaliyah",
  "id": "dbfb8fdc-6930-4c59-a849-144f47a5fd93",
  "x": 900,
  "y": 1198
}
```

Great! Now let's build that map.

## Build the map

The steps, roughly:

- Create a canvas
- Paint a background image
- Plot our points

### Create a canvas

We're going to start small and simply put the canvas right in the `index.html` file.

```html
<html>
  <head>
    <meta charset="utf-8" />
  </head>
  <body>
    <canvas id="map" width="2048" height="2048" />
    <script
      type="text/javascript"
      src="bundle.js"
      charset="utf-8"
    ></script>
  </body>
</html>
```

### Paint an image onto the canvas

Let's create a file called `mapping.js`. We'll start it off by drawing an image
onto a provided canvas:

```javascript
const MAP_SIZE = 2048;

function paint(canvas, image, players) {
  const ctx = canvas.getContext("2d");
  ctx.save();

  if (image) {
    ctx.globalCompositeOperation = "source-over";
    ctx.drawImage(image, 0, 0);
  } else {
    ctx.clearRect(0, 0, MAP_SIZE, MAP_SIZE);
  }

  // Note: we're not painting players yet

  ctx.restore();
}

module.exports = {
  paint,
};
```

Now call `paint` to display our image:

```javascript
const fakes = require("./fakes");
const paint = require("./mapping").paint;

const mapCanvas = document.getElementById("map");

var image;

const imageEl = new Image();
imageEl.src = "CompanionWorldMap.png";
imageEl.onload = () => {
  image = imageEl;
  paint(mapCanvas, image);
};
```

We'll need the world map for loading, so [download the companion world map](https://raw.githubusercontent.com/rgbkrk/lets-rxjs5/acebf65cb206258b0ec022c8287015eefb5728a6/CompanionWorldMap.png)
to your workspace.

Run `npm run build` again and open `index.html` again. You'll see that friendly
commonwealth.

![commonwealth](https://raw.githubusercontent.com/rgbkrk/lets-rxjs5/acebf65cb206258b0ec022c8287015eefb5728a6/CompanionWorldMap.png)

Now take all that player data and show it on screen. Within `index.js`,
after the setup for the image let's start taking that fake player data.

```javascript
fakes
  .livePlayers(fakes.defaultPlayers, 10)
  .scan((players, player) => {
    // collect the latest data for each player
    return players.set(player.id, player);
  }, new Map())
  // To render at approximately 60fps, we need to
  // throttle by ~16.66667 ms
  // 1 second = 1000 ms
  //   => 1000/60 = 16.6667 ms between frames
  .throttleTime(17)
  .subscribe((players) => {
    paint(mapCanvas, image, players);
  });
```

Now, the only thing we have to add to our `mapping.js` is plotting those players

```javascript
function paint(canvas, image, players) {
  const ctx = canvas.getContext("2d");
  ctx.save();

  if (image) {
    ctx.drawImage(image, 0, 0);
  } else {
    ctx.clearRect(0, 0, MAP_SIZE, MAP_SIZE);
  }

  // Each player gets plotted with a unique color
  for (var player of players.values()) {
    ctx.fillStyle = "#" + player.id.slice(0, 6);
    ctx.fillRect(player.x, player.y, 2, 2);
  }
  ctx.restore();
}
```

There you have it, a live updating map!

![animated dots](https://cloud.githubusercontent.com/assets/836375/11960060/5b49ac9c-a896-11e5-8e50-da9ce4330e1d.gif)

### Be friendly with the browser render cycle

Browsers provide a global function called `requestAnimationFrame` that lets you
call a function to update an animation before the next browser repaint. We need
to rely on this to trigger the actual call to `paint`.

```javascript
fakes
  .livePlayers(fakes.defaultPlayers, 10)
  .scan((players, player) => {
    // collect the latest data for each player
    return players.set(player.id, player);
  }, new Map())
  // We can lower our throttle since we're using rAF
  .throttleTime(10)
  .subscribe((players) => {
    window.requestAnimationFrame(
      paint.bind(null, mapCanvas, image, players)
    );
  });
```

## Summary

We've learned how to simulate data through the use of RxJS Observables, rely on
`browserify`, `babel`, and `npm` to package our app, and draw points on a canvas.
If you liked this, let me know if you want to learn how to build [the full
multiplayer map](https://github.com/rgbkrk/multipipboy).


Let's build a multiplayer Fallout 4 map using RxJS!


After making a post about [discovering running Fallout 4 servers and relaying them](/posts/fallout-4-service-discovery-and-relay/),
in only a few days, the full protocol from Fallout 4 was decoded. Several
packages for interacting with the Fallout 4 server have been posted:

- Node.js - [pipboylib](https://github.com/robcoindustries/pipboylib)
- Go - [pipboygo](https://github.com/nkatsaros/pipboygo)
- Ruby - [pippipe, pipparse](https://github.com/mattbaker/pipboy-explorations)
- Python - [pypipboy](https://github.com/matzman666/PyPipboy), [pipulator](https://github.com/Gavitron/pipulator)
- Java - [PipBoyClient](https://github.com/cpopp/PipBoyClient)
- .NET - [Fallout4-PipBoy](https://github.com/weberph/Fallout4-PipBoy)

Beyond bindings, there are a also several documentation efforts to bring
together information about the overall protocol:

- [nimvek/pipboy](https://github.com/NimVek/pipboy)
- [mattbaker/pipboyspec](https://github.com/mattbaker/pipboyspec)

And several UI efforts:

- Electron app - [pipboy](https://github.com/robcoindustries/pipboy)
- Node.js server - [pipboyclient](https://github.com/AlexanderDzhoganov/pipboyclient)
- Go server - [pipboygo](https://github.com/nkatsaros/pipboygo)

The [pipboylib](https://github.com/robcoindustries/pipboylib) bindings culminated
in the beginnings of the [Electron app, pipboy](https://github.com/robcoindustries/pipboy):

![](/assets/blog/decoding-the-pipboy-database/pipboyapp.gif)

The intention with the `pipboy` Electron app is to create a cross-platform
Pip-Boy UI for Fallout 4 that for running on a separate screen that is approachable
for others to add on to. [Electron](http://electron.atom.io/) forms the foundation for all
this by using standard web tech: HTML, CSS, and JavaScript with Chromium and Node.js.
If you're familiar with the Docker suite, [Kitematic](https://kitematic.com/) is
also an Electron app.

Having this basis makes it really easy to build things like a Pip-Boy database viewer:

![Database View](/assets/blog/decoding-the-pipboy-database/db-viewer.png)

![Deathclaw Search](/assets/blog/decoding-the-pipboy-database/deathclaw-search.png)

We'd welcome your contributions in pull requests, patches, and issues over at
[RobCoIndustries/pipboy](https://github.com/RobCoIndustries/pipboy).


Decoding Fallout 4's Pip-Boy database with a community of vault dwellers


Who's ready for some old school service discovery on Fallout 4?

I'll touch on investigating ports and protocols from an unknown server, using UDP broadcast to find services, running TCP and UDP relays, and digging into some binary streams.

Like many people this past week, I purchased a game I've been waiting a long
time for: Fallout 4. Like its predecessors, Fallout 4 has you leaving a Fallout
shelter called a vault with only your vault suit and a device on your arm called
a pip boy.

![put a pip in your step](/assets/blog/fallout-4-service-discovery-and-relay/pipboy-arm.jpg)

This time the developers, Bethesda, gave us a wonderful companion app
that gives you remote access to your pip boy. You can navigate the map, view
your inventory, change weapons, and fast travel. It's slick!

After discovering my running PS4, I immediately thought:

> "Hey, how are they discovering the game on my network?"

The first thing I did was scan my network to see where my PS4 was and what servers were listening, using `nmap`.

```
$ nmap 192.168.1.*
Nmap scan report for unknown0CFE45340EE1 (192.168.1.71)

Starting Nmap 6.47 ( http://nmap.org ) at 2015-11-19 22:20 CST
...
Host is up (0.00056s latency).
Not shown: 999 closed ports
PORT      STATE SERVICE
27000/tcp open  flexlm0
...
```

Armed with that, I could tell there's a TCP server running on 27000.

```
$ curl 192.168.1.71:27000
#{"lang":"en","version":"1.1.21.0"}
```

Whoa. HTTP worked. Sadly, without knowing about any REST endpoints, we have **no**
idea what else can be reached.

The pip boy app can't possibly know the IP address for my PS4 though, so it must be running another service to allow others to discover itself. Excited by the prospect of getting data out of this app, my fellow Jupyter developer [Jon Frederic](https://github.com/jdfreder) got some great packet captures that showed what the discovery protocol looks like as well as some of the TCP dump.

Running the scan for consoles from the mobile app while sniffing traffic was enough to see a UDP payload sent from the mobile phone to the broadcast address (255.255.255.255) on port 28000:

```json
{
  "cmd": "autodiscover"
}
```

as well as the server's response:

```json
{
  "IsBusy": false,
  "MachineType": "PS4"
}
```

Service discovery at its simplest.

If you install `socat`, you can do this yourself directly:

```bash
echo '{"cmd":"autodiscover"}' | \
  socat - UDP-DATAGRAM:255.255.255.255:28000,broadcast
```

Perfect, we now know the Fallout 4 server has at least two servers listening: UDP on port 28000 and TCP on port 27000. We don't know the protocol though, so we're going to need to see more legitimate traffic from the pip boy to the console. We can continue on doing more sniffing of traffic, poisoning ARP to do MITM, or we can write a relay. I opted for the last one which I'll explain here.

## Ripping data with a relay

The relay functions by establishing itself as another Fallout 4 server for
mobile apps to connect to, while actually sending packets on to the legitimate one.

![Close to Metal](/assets/blog/fallout-4-service-discovery-and-relay/close-to-metal.png)

You can do this with tools like socat, but I opted to do this in Node.js for
reasons that will become clear later. If you want to skip down to using the
relay, go to [the pipboy relay](#the-pip-boy-relay).

### UDP relay

The relay we're going to write functions by acting as an intermediary between
a real client (the pip boy app) and the Fallout server. To do this, every time a
client connects to the relay, we'll create our own fake client to connect to the
real Fallout server. As the client and server communicate, we'll pass messages to
each while making a copy of the messages they send.

The first thing we'll do is open up our server side socket:

```javascript
var upstreamInfo = {
  address: YOUR_CONSOLE_OR_PC_IP
  port: 28000 // The Fallout 4 UDP Port
}

var server = dgram.createSocket('udp4')
server.bind(upstreamInfo.port, '0.0.0.0')
```

For every message that comes in, we're going to set up another socket to act as
the fake client. We'll then forward messages between the _actual server_ and the pip
boy client.

```javascript
server.on("message", function (message, clientInfo) {
  var fakeClient = dgram.createSocket("udp4");
  // fakeClient.on('message', ...) // Defined below
  // fakeClient.bind(...)
});
```

The two parts commented out above are written out below with more detail. Each
time we get a message, we're going to copy it, note the telemetry data, send the
message on to the actual client, and provide the copied buffer and telemetry to
a callback.

```javascript
fakeClient.on("message", function (message, serverInfo) {
  var copiedBuffer = new Buffer(message.length);
  message.copy(copiedBuffer);

  // Now emulate our server
  server.send(
    message,
    0,
    message.length,
    clientInfo.port,
    clientInfo.address
  );
  var telemetry = {
    src: serverInfo,
    dst: clientInfo,
  };
  cb(copiedBuffer, telemetry);
});
```

Here's where we actually bind the client, do the same message copying, send the
message to the server, and propagate the data to the callback.

```javascript
// As soon as our client is ready, go ahead and send their message onward
fakeClient.bind(undefined, undefined, function () {
  var copiedBuffer = new Buffer(message.length);
  message.copy(copiedBuffer);

  fakeClient.send(
    message,
    0,
    message.length,
    upstreamInfo.port,
    upstreamInfo.address
  );
  var telemetry = {
    src: clientInfo,
    dst: upstreamInfo,
  };
  cb(copiedBuffer, telemetry);
});
```

### TCP relay

The setup for the TCP relay is similar. Here we'll create a `TCPRelay`
that has a `net.Server` underneath. The strongest difference here is that
we'll have to actually maintain all the fake clients that are being used

```javascript
/**
 * Create a TCP relay for an upstream server
 * @constructor
 */

/**
 * Listen for traffic to relay to/from an upstream server
 * @param {Object} upstreamInfo
 * @param {string} upstreamInfo.address
 * @param {number} upstreamInfo.port
 * @param {relayCallback} - callback that handles new data
 */
var TCPRelay = function TCPRelay() {
  this.server = net.createServer({ allowHalfOpen: true });
};
```

We'll slightly imitate `net.Server.listen` by providing `TCPRelay.listen`, taking
the net info structure that is used with the UDP layer.

```javascript
TCPRelay.prototype.listen = function listen(
  upstreamInfo,
  cb
) {
  this.server.on("connection", function (client) {
    // Now we create our fake client
    var fakeClient = new net.Socket();
    fakeClient.connect(
      upstreamInfo.port,
      upstreamInfo.address
    );

    // Get the actual client info for logging where the traffic really came from
    var actualClientInfo = {};
    actualClientInfo.address = client.remoteAddress;
    actualClientInfo.port = client.remotePort;
    actualClientInfo.family = client.remoteFamily;

    // Register handlers for the fakeClient
    // These each get explained below but should be inserted here

    // fakeClient.on('connect', function() { ... })
    // fakeClient.on('data', function() { ... })
    // fakeClient.on('close', function() { ... })
    // fakeClient.on('end', function() { ... })
  });
  this.server.listen({ port: upstreamInfo.port });
};
```

After the `fakeClient` is connected, we're ready to register the handler for data
from the `client`. We'll send data on from `client` to the `fakeClient`, which
sends it on to the Fallout 4 server.

```javascript
fakeClient.on("connect", function () {
  // Once we're connected, we can get each message from the client
  client.on("data", function (message) {
    var copiedBuffer = new Buffer(message.length);
    message.copy(copiedBuffer);

    // To the server
    fakeClient.write(message);

    var serverInfo = {};
    serverInfo.address = fakeClient.remoteAddress;
    serverInfo.port = fakeClient.remotePort;
    serverInfo.family = fakeClient.remoteFamily;

    var telemetry = {
      src: actualClientInfo,
      dst: serverInfo,
    };

    cb(copiedBuffer, telemetry);
  });
});
```

Every time the `fakeClient` receives data, it's coming from the Fallout 4 server
and we send it on to the real client.

```javascript
fakeClient.on("data", function (message) {
  var copiedBuffer = new Buffer(message.length);
  message.copy(copiedBuffer);

  var serverInfo = {};
  serverInfo.address = fakeClient.remoteAddress;
  serverInfo.port = fakeClient.remotePort;
  serverInfo.family = fakeClient.remoteFamily;

  var telemetry = {
    src: serverInfo,
    dst: actualClientInfo,
  };
  client.write(message);

  cb(copiedBuffer, telemetry);
});
```

We should also handle closing the `fakeClient` as well as the `client` on the
other side. Remembering that we need to mirror operations directly, we'll do
the same for ending the clients, too.

```javascript
fakeClient.on("close", function (hadError) {
  if (hadError) {
    console.log("closure error");
  }
  client.close();
});

fakeClient.on("end", function () {
  client.end();
});
```

## Putting the pieces together

Now we have a `TCPRelay` and a `UDPRelay`. Taking these and autodiscovering a
running server, we can make a nice little tool.

### Libraries

I've wrapped the relays and autodiscovery into a couple packages:

- [pipboylib](https://github.com/RobCoIndustries/pipboylib)
- [pipboyrelay](https://github.com/rgbkrk/pipboyrelay)

The second one, `pipboyrelay`, is a CLI tool that you can use to dump data
yourself that relies on `pipboylib`.

### The pip boy relay

You can run the relay directly yourself if you have a few things:

- Fallout 4 for the PC or PS4. (XBONE has not been diagnosed)
- The pip boy app for Android or iOS
- The pip boy app enabled in-game on Fallout 4
- Node.js and npm

With Node.js and `npm` set up you can install `pipboyrelay` as a CLI tool:

```
npm install -g pipboyrelay
```

Now run the `pipboyrelay` at the command line:

```
$ pipboyrelay
Discovered:  { IsBusy: false,
  MachineType: 'PS4',
  info: { address: '192.168.1.71', family: 'IPv4', port: 28000, size: 50 } }
```

It will autodiscover any games running with an active server on your local
network and create its own endpoints that the Android or iOS app will recognize.
Open up the mobile app and navigate to the connection settings screen.

![One of these things is not like the other](/assets/blog/fallout-4-service-discovery-and-relay/two-ps4s.png)

One of these things is not like the other! Connect to the address that was not
shown in the discovery. You should start seeing data fly by.

```
UDP and TCP Relay created for:  { address: '192.168.1.71', family: 'IPv4', port: 28000, size: 50 }
listening
[TCP Relay] 192.168.1.71:27000 -> ::ffff:192.168.1.67:55232
00000000: 2300 0000 017b 226c 616e 6722 3a22 656e  #....{"lang":"en
00000010: 222c 2276 6572 7369 6f6e 223a 2231 2e31  ","version":"1.1
00000020: 2e32 312e 3022 7d0a                      .21.0"}.

[TCP Relay] 192.168.1.71:27000 -> ::ffff:192.168.1.67:55232
00000000: 754e 0600 0306 5290 0f00 2447 656e 6572  uN....R...$Gener
00000010: 616c 0003 6290 0f00 1e00 0000 0661 900f  al..b........a..
00000020: 004c 6f63 6174 696f 6e73 2044 6973 636f  .Locations.Disco
...
[TCP Relay] 192.168.1.71:27000 -> ::ffff:192.168.1.67:55232
00000000: 7565 002d 910f 0074 6578 7400 2f91 0f00  ue.-...text./...
00000010: 7368 6f77 4966 5a65 726f 0000 0003 3291  showIfZero....2.
00000020: 0f00 0000 0000 0631 910f 0046 6974 7320  .......1...Fits.
...
```

Interesting! Look at byte 0x18 (0x1e), occurring right before
"Locations Discovered".

```
[TCP Relay] 192.168.1.71:27000 -> ::ffff:192.168.1.67:55232
00000000: 754e 0600 0306 5290 0f00 2447 656e 6572  uN....R...$Gener
00000010: 616c 0003 6290 0f00 1e00 0000 0661 900f  al..b........a..
------------------------------^^
00000020: 004c 6f63 6174 696f 6e73 2044 6973 636f  .Locations.Disco
```

That `0x1e` is the current number of locations discovered with the character I
used for this, 30.

It seems like the Fallout 4 format is a binary format and that's in plain text.
This is good news, because it will be easier to figure out the format and build
new things.

Run this relay while you explore the wasteland and you'll see your coordinates
scroll by in the relay output. Can you imagine mapping you and your friends
all together on one map?

## Vanilla stat dump

You don't really need all this code to get your current stats though. On OS X
and Linux, you can use `nc` (netcat) directly to get all your stats from your
console or PC.

```
$ nc 192.168.1.71 27000 | xxd | head -n 16
00000000: 2300 0000 017b 226c 616e 6722 3a22 656e  #....{"lang":"en
00000010: 222c 2276 6572 7369 6f6e 223a 2231 2e31  ","version":"1.1
00000020: 2e32 312e 3022 7d0a 59f0 0600 0306 81cc  .21.0"}.Y.......
00000030: 1e00 2447 656e 6572 616c 0003 91cc 1e00  ..$General......
00000040: 2200 0000 0690 cc1e 004c 6f63 6174 696f  "........Locatio
00000050: 6e73 2044 6973 636f 7665 7265 6400 0092  ns Discovered...
00000060: cc1e 0001 088f cc1e 0003 0091 cc1e 0076  ...............v
00000070: 616c 7565 0090 cc1e 0074 6578 7400 92cc  alue.....text...
00000080: 1e00 7368 6f77 4966 5a65 726f 0000 0003  ..showIfZero....
00000090: 95cc 1e00 0400 0000 0694 cc1e 004c 6f63  .............Loc
000000a0: 6174 696f 6e73 2043 6c65 6172 6564 0000  ations Cleared..
000000b0: 96cc 1e00 0108 93cc 1e00 0300 95cc 1e00  ................
000000c0: 7661 6c75 6500 94cc 1e00 7465 7874 0096  value.....text..
000000d0: cc1e 0073 686f 7749 665a 6572 6f00 0000  ...showIfZero...
000000e0: 0399 cc1e 000a 0000 0006 98cc 1e00 4461  ..............Da
000000f0: 7973 2050 6173 7365 6400 009a cc1e 0001  ys Passed.......
... more data ...
```

## Wrap up

We've done several things here.

- Found our running servers from Fallout 4
- Spoofed a Fallout 4 server, complete with being discoverable
- Dumped traffic from our relay

In a follow on post, I'll show you how to send your stats on to a remote server
for others to view. We can share our levels, our wasteland locations,
number of caps, and anything else we can decode.

Install `pipboyrelay` yourself or just use `nc` to explore. I'm really excited to
create mashups with this. Feel free to discuss your findings on the repository
for the [pip boy library](https://github.com/RobCoIndustries/pipboylib) or
the [pipboyrelay](https://github.com/rgbkrk/pipboyrelay).

If you're looking for libraries for your favorite language, check out the [follow
up post about the libraries that are available as well as some work in progresses](/posts/decoding-the-pipboy-database-with-vault-community/).



Fallout 4 Service Discovery and Relay


The IPython/Jupyter notebook is a wonderful environment for computations, prose, plots, and interactive widgets that you can share with collaborators. People use the notebook [all](http://blog.quantopian.com/quantopian-research-your-backtesting-data-meets-ipython-notebook/) [over](http://nbviewer.jupyter.org/github/GoogleCloudPlatform/ipython-soccer-predictions/blob/master/predict/wc-final.ipynb) [the](https://github.com/facebook/iTorch) [place](http://nbviewer.jupyter.org/url/norvig.com/ipython/TSPv3.ipynb) across [many varied languages](https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages). It gets used by [data scientists](http://nbviewer.jupyter.org/gist/wrobstory/1eb8cb704a52d18b9ee8/Up%20and%20Down%20PyData%202014.ipynb), researchers, analysts, developers, and people in between.

As I alluded to in a writeup on [Instant Temporary Notebooks](/posts/ipythonjupyter-tmpnb-debuts), we, a combination of IPython/Jupyter and Rackspace, were prepping for a big demo as part of a [Nature article on IPython Notebooks](https://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261) by [Helen Shen](https://twitter.com/HelenShenWrites). The impetus behind the demo was to show off the IPython notebook to readers in an interactive format. What better way than to [provide a live notebook server to readers on demand](https://www.nature.com/news/ipython-interactive-demo-7.21492)?

[![Screenshot of the Nature IPython notebook demo](https://d23f6h5jpj26xu.cloudfront.net/nvqcj7okftoqw_small.png)](https://img.svbtle.com/nvqcj7okftoqw.png)

To do this, we created a [temporary notebook service](https://tmpnb.org) in collaboration with the IPython/Jupyter team.

### How does this temporary notebook service work?

[tmpnb](https://github.com/jupyter/tmpnb) is a service that spawns new notebook servers, backed by Docker, for each user. Everyone gets their own sandbox to play in, assigned a unique path.

![tmpnb architecture](https://cloud.githubusercontent.com/assets/836375/5909206/f0b156da-a573-11e4-8e3f-f65dfe9d23b8.png)

When a user visits a tmpnb, they are actually hitting an [HTTP proxy](https://github.com/jupyter/configurable-http-proxy) which routes initial traffic to tmpnb's orchestrator. From here a new user container is set up and a new route, for example `/user/fX104pghHEha/tree`, is assigned on the proxy.

[![tmpnb setup animation](https://d23f6h5jpj26xu.cloudfront.net/z9gjan4yftabyq_small.gif)](https://img.svbtle.com/z9gjan4yftabyq.gif)

## Planning the Notebook Demo

When Brian Granger from Cal Poly and Richard Van Noorden from Nature asked for a demo, it was quite open what that could mean. Do we have people log in to a [JupyterHub](https://github.com/jupyter/jupyterhub) installation? Refer them to [Wakari](https://wakari.io/) or [SageMathCloud](https://cloud.sagemath.com/)?

The goal Richard stated was to provide at most 150 concurrent users. In the back of our minds, we in the IPython/Jupyter project knew the initial spike in traffic would be far greater and we should be able to handle the load.

I was incredibly lucky to teach at and attend the incredible and crazy event that is the Mozilla Festival. Richard Van Noorden from Nature was in attendance, as were my colleagues from the IPython community including [Matthias Bussonnier](https://twitter.com/Mbusson), [Aron Ahmadia](https://twitter.com/ahmadia), and [Jeramia Ory](https://twitter.com/DrLabRatOry). While we were all in one place we swarmed Richard with ideas about what should be in the notebook Nature readers got to interact with. After the iterations there, pretty soon we had more collaboration from [David Ketcheson, a researcher at KAUST](https://twitter.com/DavidKetch), [Stefan van der Walt, a scikit-image lead](https://twitter.com/stefanvdwalt), and others.

You can tell this is a community with a lot of passion and aligned around a common format that has helped propel their research.

The other crazy benefit about being in London was that I got to go to the Nature offices to talk about the architecture backing the demo and make plans for operations. [Chris Ryan](https://github.com/chris-creditdesign), art editor at Nature, would put an iframe as part of the article, expanding it into a lightbox for users when they clicked. For us, this just meant providing one URL they could rely on for content, as well as adjusting `CSP` or `X-Frame` restrictions.

## Kicking the Nature Notebooks into Operation

On the day of launch, we watched as the notebooks started getting gobbled up and recycled.

After some smooth sailing, we watched as it ticked toward our 512 user mark. After reading comments on various social media sites, we decided to kick it up a notch and allow for thousands of concurrent users while the demo was live.

This bit us in a couple ways. In order to scale across hosts we would need to put the proxy and tmpnb in front of multiple Docker hosts. This was before Docker [Swarm](https://github.com/docker/swarm). Trying to swap largely untested pieces out from underneath production while dealing with proxy issues did not sound ideal. Instead, [Min RK](https://github.com/minrk) quickly whipped up the tmpnb-redirector, which uses the `/stats` endpoint to redirect users to new servers. This made rotating old nodes out easy as well.

![Redirect tmpnb nodes](https://cloud.githubusercontent.com/assets/836375/5913707/3b282e20-a5ae-11e4-9768-c22089b70497.png)

## Closing Up

In the end we ended up serving more than 20,000 notebook servers and counting.

We love IPython notebooks, the overall architecture that has been built out here, and hope to keep supporting open-source projects doing interesting things on the internet in a way that benefits community, technology, and the whole ecosystem.


How the IPython and Jupyter community used tmpnb, Docker, and a proxy to serve more than 20,000 temporary notebook servers for Nature readers.

How did we serve more than 20,000 IPython notebooks for Nature readers?


For the past four months, the Jupyter developers -- in collaboration with [Rackspace](https://developer.rackspace.com), [Nature](http://www.nature.com/news/toolbox), and [O'Reilly Media](http://www.oreilly.com/) -- have been working on several experiments to give developers, data scientists, and researchers instant access to IPython Notebooks.

Our biggest experiment so far was a demo with Nature. The journal published [an article that featured IPython Notebooks](http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261) as a tool for scientists to share data and code. The Jupyter team wanted to show off the notebooks to readers in an interactive format. What better way to than to [provide a live notebook server to readers on demand](http://www.nature.com/news/ipython-interactive-demo-7.21492)?

[![The Nature Demo Had Interactive Widgets!](https://d23f6h5jpj26xu.cloudfront.net/nvqcj7okftoqw_small.png)](https://img.svbtle.com/nvqcj7okftoqw.png)

To do this, we created a [temporary notebook service](https://github.com/jupyter/tmpnb).

### How does the temporary notebook service work?

[tmpnb](https://github.com/jupyter/tmpnb) is a service that spawns new notebook servers, backed by Docker, for each user. Everyone gets their own sandbox to play in, assigned a unique path.

When a user visits a tmpnb, they're actually hitting an [http proxy](https://github.com/jupyter/configurable-http-proxy) which routes initial traffic to tmpnb's orchestrator. From here a new user container is set up and a new route (e.g. `/user/fX104pghHEha/tree`) is assigned on the proxy.

[![Users just drive by happily](https://d23f6h5jpj26xu.cloudfront.net/z9gjan4yftabyq_small.gif)](https://img.svbtle.com/z9gjan4yftabyq.gif)

Working our way up to the Nature demo, we had several live alpha prototypes around the same time at Strata NYC 2014:

- [Paco Nathan](https://twitter.com/pacoid)'s Just Enough Math tutorial at Strata, in coordination with O'Reilly Media/[Andrew Odewahn](https://twitter.com/odewahn)
- tmpnb announced and used during Fernando Perez' [PyData opening at Strata NYC 2014](http://strataconf.com/stratany2014/public/schedule/detail/37035)
- [Olivier Grisel](https://twitter.com/ogrisel) used it in his scikit learn tutorial

After all the above activity, we learned several things:

- Serve static assets via nginx instead of each user container
- Websockets and proxies gobble up ports
- People want to share notebooks
- Pool user containers ahead of time

### Static assets should be served via nginx instead of within each user container

nginx is really good at serving static resources. Let it handle the grunt work easily:

```
location ~ /(user[-/][a-zA-Z0-9]*)/static/(.*) {
    alias /srv/ipython/IPython/html/static/$2;
}
```

This is both for speed and to use fewer file descriptors across the system. We ended up pulling some neat tricks with our deployment setup to [mount a dummy user container's static files into an nginx container](https://github.com/jupyter/tmpnb-deploy/pull/3). However, if you don't do this, you get the neat side effect of never having caching problems when launching new versions of userland containers, and even means you could ship different ones to different users!

### Websockets and proxies gobble up ports

Each websocket opened by a user ends up opening several ports that will stay open. Since each notebook on a notebook server will open at least one websocket, the number of notebooks you have open directly correlates with the number of websockets.

Here's some trimmed and commented `lsof -i` output when a single user is accessing an IPython 3.x notebook server through tmpnb:

```
# The fixed pieces of networking for tmpnb itself
# Node proxy
node     2646 *:8000 (LISTEN)

# Orchestration layer of tmpnb
python   2669 *:9999 (LISTEN)
python   2669 *:9999 (LISTEN)
node     2646 ip6-localhost:8001 (LISTEN)

# Now for the pieces that each user will be "allocated"
# Established connection with client
node     2646 10.184.2.134:8000->10.223.242.4:53254 (ESTABLI)

# Docker exposes port for IPython notebook server
docker  18173 ip6-localhost:49216 (LISTEN)

# Proxy connects to docker container, keeping port allocated to websocket
node     2646 ip6-localhost:35831->ip6-localhost:49216 (ESTABLI)
docker  18173 ip6-localhost:49216->ip6-localhost:35831 (ESTABLI)

# Docker routes onward to the IPython Notebook
docker  18173 ip6-localhost:49216->ip6-localhost:35673 (ESTABLI)
python   2669 ip6-localhost:35673->ip6-localhost:49216 (ESTABLI)
```

This got really bad when the notebook server had 3 websockets per notebook and users were opening more than a few notebooks during a tutorial. Websockets have to stay established, by nature.

### People want to share notebooks

Honestly, I should know this from working on the [notebook viewer](http://nbviewer.jupyter.org/). On several occasions, people have passed me links on the demo version of tmpnb, hosted at tmpnb.org.

```
https://tmpnb.org/user/FKlVK3haOQRF/notebooks/SharingEffects.ipynb
```

That link _won't actually work_ for you, because the server is long gone. After a user has left their notebook server, it gets culled. Gone. Caput. Killed. Need to make room for new users and have plenty of fresh sandboxes.

While I would _love_ to enable people to do this, we probably need an alternate way to share these temporary calculations. I've used it myself and [wished for a way to get it directly on to nbviewer](https://twitter.com/rgbkrk/status/557942542063652864). Posting static content isn't the same as giving someone the ability to remix your code though. For now, this works and has no bearing on the Nature demo.

### Pool user containers ahead of time

Docker can only boot and route so many containers so quickly. To mitigate this, we created a pool of ready to go userland containers. [@smashwilson](https://github.com/smashwilson) came in and [added the spawn pools](https://github.com/jupyter/tmpnb/pull/69) after we dangled the problem in front of him.

[![The Notebook Grid in Action](https://d23f6h5jpj26xu.cloudfront.net/jlvadowzumttlg_small.gif)](https://img.svbtle.com/jlvadowzumttlg.gif)

This did introduce problems with initial spawns (Docker failing) and made our [introductory experience with a bit to be desired](https://github.com/jupyter/tmpnb/issues/87).

We learned this the hard way on boot up, having to code around responses from the API and making sure that we block on our own server instead of Docker.

## tmpnb in the wild

We didn't expect to build a system that would let people create user environments they could use for teaching and tutorials. I'm incredibly humbled that [Nikolay Koldunov](https://twitter.com/koldunovn) used it for [a tutorial at YaC 2014](http://koldunov.net/?p=950) while it was still in its infancy.

It's my hope that people will continue to launch temporary notebook servers, backed by their own tutorial/demo/class content.

- [tmpnb repository](https://github.com/jupyter/tmpnb) - Run your own, join us
- [Demo images used on tmpnb.org](https://github.com/jupyter/docker-demo-images) - Make suggestions about the running system


Ops Lessons on Instant Temporary IPython Notebooks


It's about time I write about [tmpnb](https://github.com/jupyter/tmpnb), a project the Jupyter/IPython developers have been working on. [Brian Granger](https://twitter.com/ellisonbg), one of the Jupyter cofounders, has been asking me over the past few months about enabling some more ambitious use cases of our Docker containers within [IPython](https://registry.hub.docker.com/u/ipython) and [Jupyter](https://registry.hub.docker.com/u/jupyter):

- Provisioned notebooks for usability studies
- Instant notebooks for demos
- JupyterHub for our team

[Helen Shen](https://twitter.com/HelenShenWrites), science writer, and [Richard Van Noorden](https://twitter.com/Richvn), [Nature](http://www.nature.com/) reporter, have been working on [an article on the IPython Notebook](http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261). They asked us if we could create a way for readers to try the notebook. After reflecting on the current JupyterHub code, I realized that with our Docker images and our [configurable http proxy](https://github.com/jupyter/configurable-http-proxy) (node-http-proxy with an admin API), we could easily setup paths and route users. Like any good (deranged) programmer, I first wrote this in bash:

```bash
#!/usr/bin/env bash
myrand=`head -c 30 /dev/urandom | xxd -p`
cont=$( docker run -it -d -P -e RAND_BASE=$myrand jupyter/demo )
port=$( docker port $cont 8888 | cut -d":" -f2 )

# proxy on         *:8000
# admin on 127.0.0.1:8001
curl -H "Authorization: token $CONFIGPROXY_AUTH_TOKEN" \
     -XPOST -d '{"target":"http://localhost:'$port'"}' \
     http://localhost:8001/api/routes/$myrand
curl -H "Authorization: token $CONFIGPROXY_AUTH_TOKEN" \
     http://localhost:8001/api/routes | python -m json.tool

# Spit out the URL for the new notebook server
echo `curl -s icanhazip.com`:8000/$myrand/
```

Yay! As soon as I had this working it was time to start on a Tornado app to do the same dynamically. I immediately told [Min RK](https://github.com/minrk) about it and he seemed to like it.

_"Nifty! Temporary notebook servers. That could be a thing."_

![](https://speakerd.s3.amazonaws.com/presentations/f5dca2a02e41013233bb1ac45923d988/slide_10.jpg?1412783050)

<h1 style="text-align: center;">↓</h1>

![](https://speakerd.s3.amazonaws.com/presentations/f5dca2a02e41013233bb1ac45923d988/slide_11.jpg?1412783050)

Roughly speaking, the tornado application does the above and just a bit more. When the user lands at a hosted tmpnb server:

- A Docker container gets started (the [jupyter/demo image](https://registry.hub.docker.com/r/jupyter/demo) by default)
- A random path is created for them, e.g. `/user-asdf/`
- The proxy is told to route `/user-asdf/` to the local Docker container running on some port
- The user is redirected to their fresh server.

_Whoa whoa whoa, are you saying this will run any Docker container with IPython notebook inside?_

Yes, with some caveats. tmpnb needs to be able to tell the underlying containers what path they're being served from so that instead of trying to load `/static/notebook.js`, it loads `/user-A/static/notebook.js`

We're working to make this compatible with any IPython notebook Docker container while allowing some configurability. [Ash Wilson](https://twitter.com/smashwilson) wants to go further, making this work with [any Docker container](https://github.com/jupyter/tmpnb/pull/69#discussion_r19055529). It's possible so long as all paths are relative (`../static/util.js`) or adjusted for their assigned base path (`/user-a/static/util.js`).

## Tutorials! Books! Learning!

![](https://d23f6h5jpj26xu.cloudfront.net/rujc2e9z7jada_small.gif)

[Andrew Odewahn](https://github.com/odewahn), the CTO of O'Reilly Media, and Paco Nathan, mad scientist, have been [driving efforts](https://github.com/odewahn/docker-at-oreilly) to make Dockerfiles & Docker images [paired with books](https://github.com/ceteri/jem-docker). They're doing fantastic work so that you can type:

```
docker run -it -p 8888:8888 ceteri/jem
```

to pull up an IPython notebook with content for a book and play along.

Just prior to the [public demo of tmpnb at PyTexas](https://speakerdeck.com/rgbkrk/tmpnb-at-pytexas), I called up Andrew to tell him about the tmpnb efforts. He was super excited, just as I was, to get something like this working for authored content.

Andrew: _"We're going to be running a tutorial at Strata using a Docker image we've been working on, do you think we could use this for that?"_

Kyle: _"Well, there's some engineering work to be done but we can try to make it work."_

Fast forward many PRs, some last minute coordinating of a trip to Strata, and we got our prototype pushed forward for the tutorial. Meanwhile, the IPython team at large is very excited about tmpnb. Notables: Min [added the culling](https://github.com/jupyter/tmpnb/pull/10) and wrote [a trie in Javascript](https://github.com/jupyter/configurable-http-proxy/pull/8) (key-value lookups are not O(1) in JavaScript??!?!?).

Once we got to a stable state and Andrew put up the Just Enough Math server to Strata, we did some load testing which was a combination of slimerjs/phantomjs and [Min's nbbot](https://github.com/minrk/nbbot) (that he wrote just for this). It was clear we were going to need spawn pools in the long run (containers ready to be routed) and that we had some socket and file descriptor issues.

![](https://d23f6h5jpj26xu.cloudfront.net/qvrhxbeay9zxpq_small.jpg)

The morning of PyData at Strata, Brian Granger comes over to me before his and Fernando's opening talks, "Is tmpnb ready?" I can only respond "Yes? Maybe. Sure, why not." As Brian walks back up to the stage, worry comes over me as I had not applied the updates to the currently deployed system. "Really hope this works." What preceded was 30 minutes of relative bliss as 5... 10... 20... 40... 80 users logged on to tmpnb.org. Everyone was able to follow along with the notebooks themselves, clicking and opening the notebooks that Fernando was running through. We got our first tweet as this was going on

<blockquote class="twitter-tweet" lang="en"><p>Attending <a href="https://twitter.com/hashtag/PyData?src=hash">#PyData</a> at <a href="https://twitter.com/hashtag/Strataconf?src=hash">#Strataconf</a> -- already impressed at the hosted <a href="http://t.co/2zUVPbfFAj">http://t.co/2zUVPbfFAj</a> notebooks.</p>&mdash; A.J. Rader (@indyrader) <a href="https://twitter.com/indyrader/status/522373218066505728">October 15, 2014</a></blockquote>

Woohoo! Eventually though, the dreaded hand raised:

_"Hey, tmpnb.org isn't working anymore. Can you share a link to the notebooks?"_

The proxy could no longer get file descriptors.

```
13:28:11.164 - Proxy error:  code=EMFILE, errno=EMFILE, syscall=connect
13:28:11.408 - Proxy error:  code=EMFILE, errno=EMFILE, syscall=connect
13:28:11.666 - Proxy error:  code=EMFILE, errno=EMFILE, syscall=connect
```

Ouch.

While checking on what was going on with the file descriptors, we noticed that `lsof` + Docker didn't play well since the filesystems aren't the same.

```
root@demo:~# lsof 2> /dev/null | grep python | cut -c 88- | sort | uniq -c | sort -nr | head
  25861 anon_inode
  13570 can't identify protocol
   7814 pipe
   3067 /dev/urandom
   2682 /home/jupyter/.ipython/profile_default/history.sqlite
   1833 /dev/null
   1637 /
   1587 /usr/lib/x86_64-linux-gnu/libzmq.so.3.1.0 (stat: No such file or directory)
   1587 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19 (path dev=8,1, inode=7783)
   1587 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6 (path dev=8,1, inode=7737)
```

Even with the troubles above, Olivier Grisel wanted to run one for his tutorial too. We quickly wrapped one up based on the `jupyter/demo` image within a few minutes and off he went.

```Dockerfile
FROM jupyter/demo

USER root
RUN rm -rf ipython_examples *.ipynb
ADD . /home/jupyter
RUN chown jupyter:jupyter . -R
RUN pip3 install psutil

USER jupyter
```

Apparently his ran without any hitches. He is one of my favorite people in the world. Note: he did not have nginx in front - it was straight to the configurable proxy AND the proxy was running in Docker instead of supervisor.

One optimization I took out of this (not clear in the above `anon_inode` references) was to pull the static content out of the container and serve via nginx. That really helped performance in addition to saving file descriptors.

### Just Enough Math

When it came to the Just Enough Math tutorial, I was able to get static content served on the nginx box quickly by pulling the right static content out of the docker images directly using `docker cp`. With that fix just in time for the tutorial, it was another chance to see tmpnb live (albeit on the strataconf.com domain).

<blockquote class="twitter-tweet" lang="en"><p>Watching <a href="https://twitter.com/pacoid">@pacoid</a> and <a href="https://twitter.com/allenday">@allenday</a> rocking the IPython notebook and the audience using tmpnb at <a href="https://twitter.com/hashtag/Strataconf?src=hash">#Strataconf</a> <a href="http://t.co/HwSyGn3RP2">pic.twitter.com/HwSyGn3RP2</a></p>&mdash; Kyle R Kelley (@rgbkrk) <a href="https://twitter.com/rgbkrk/status/522449621378154496">October 15, 2014</a></blockquote>

Great! It started off well and for most folks they got to just experience the content. No thought to the platform it was running on, they just got to focus on the IPython notebook and the tutorial content. Wonderful.

45 minutes in this time, with a bigger crowd (around 150) we hit our limits again. This time we ran out of actual socket descriptors for the proxy. Doing some digging live during the tutorial, we found out where the issues laid.

For each notebook of each user, 3 extra sockets were being opened - one for each of IPython's websockets for `shell`, `stdin`, and `iopub`. _That's on every notebook file._ This meant that the first notebook was "ok" but then new notebooks (Not servers, the actual files, e.g. Untitled0.ipynb, Untitled1.ipynb) would open 3 more socket descriptors. Because this was proxied from nginx -> node -> docker, you can double that number too.

With only two notebooks per user and 110 users, that's on the order of 1430 sockets. Node was running as nobody and could only open 1024 ports. Despite my (primitive) attempts to change ulimits on the fly, I couldn't get it stable right then and there.

### NYC Python Meetup

[James Powell](https://twitter.com/dontusethiscode), having seen [my talk at PyTexas on tmpnb](https://speakerdeck.com/rgbkrk/tmpnb-at-pytexas), asked if I could speak at the NYC Python meetup. Since I had the lightning talk ready and had a new system up with [Ash Wilson](https://twitter.com/smashwilson)'s spawn pool PR, I decided to give it a go.

<blockquote class="twitter-tweet" data-cards="hidden" lang="en"><p>Showed off <a href="https://twitter.com/smashwilson">@smashwilson</a>&#39;s tmpnb PR <a href="https://t.co/RWSj3Rd9ps">https://t.co/RWSj3Rd9ps</a> at <a href="https://twitter.com/nycpython">@nycpython</a> tonight. Flawless 0s nb demo! Thanks <a href="https://twitter.com/dontusethiscode">@dontusethiscode</a> and pals!</p>&mdash; Kyle R Kelley (@rgbkrk) <a href="https://twitter.com/rgbkrk/status/522965532770074624">October 17, 2014</a></blockquote>

Needless to say, that went flawlessly (besides my own laptop display issues) and the audience was wonderful.

## Put some Teeth on that Cloud

![](https://d23f6h5jpj26xu.cloudfront.net/j98s5ozwffseya_small.png)

Currently I've been deploying this on fairly hefty machines, subdividing the machines up to give individual users (roughly) the same amount of storage they would come to expect on a VPS (virtual private server). One of Rackspace's [OnMetal](http://www.rackspace.com/cloud/servers/onmetal/) Memory boxes has half a terabyte of RAM, so I figured I could subdivide that up into 512 MB chunks. With 1000 users, that gives us roughly 12 GB for the main OS.

The price of the OnMetal Memory machines is $1600, so the per user per month price is _$1.60_. Packing bare metal with containers looks very economical to me but I'm also subdividing 24 cores across 1000 users. What does a typical cheap VPS do for slicing cores (other than telling you that you're sharing a CPU)? I'd like to explore all this quite a bit more. If I'm off, please let me know.

## Thanks Rackspace, O'Reilly, Nature, Others

The temporary notebook service has been graciously hosted by [Rackspace](https://developer.rackspace.com) on their OnMetal servers. Andrew Odewahn & O'Reilly ran the Just Enough Math server at Strata. I am forever grateful to both companies for driving forward with such new technologies and letting me explore with them.

Thank you to Richard Van Noorden, Brian Granger, and Min RK for spawning the initial impetus and ideas behind tmpnb. Oh and Docker! :D

---

Comments on [Reddit via `/r/python`](http://www.reddit.com/r/Python/comments/2l3gk4/creating_an_instant_temporary_ipython_notebook/) or [on Twitter](https://twitter.com/rgbkrk/status/529005294064775168).

### Read this next

[Ops Lessons on Instant Temporary IPython Notebooks](/posts/ops-lessons-and-instant-temporary-ipython-notebooks)


Instant Temporary IPython Notebooks


Can we stop saying NoSQL? I'm really tired of describing things by what they're not.

I'd much rather describe them by what they are.

- "This is a JSON document store"
- "This is an object store"
- "This is a key-value store"
- "This is a graph database"
- "This is a distributed table"
- "This is a thinly veiled attempt at doing something that could have been implemented with Postgres"
- "This is me not caring about ACID guarantees and calling it statistical sampling"
- "My filesystem is a key-value store, get off my back"

```html
</rant>
```

In reality, I have little experience with most of the systems indirectly mentioned here. I'd like to find the right tool(s) for the job to make development easier, operations less painful, and user experience awesome. Is that so much to ask?

---

_For all my blog posts I've decided to hold discussion on Reddit, linking to the post. I didn't do that this time though. Today's post is a mini-post not really intended for anywhere._


No NoSQL


# Hijacking the IPython Notebook's WebSockets

**TL; DR** _On IPython ≤ 1.1, the Notebook server suffered from a flaw where it did not verify the origin of websocket requests. An attacker with knowledge of an active IPython kernel ID could run arbitrary code on a user's machine with the privileges of the user running the IPython kernel if the client visited a crafted malicious page. This was corrected upstream in the 1.2.0 and 2.0.0 releases._

## The IPython Notebook

For those that don't know, the IPython Notebook is an in-browser application for interactive computing where you can combine code, prose, mathematics, plots, and other rich media into a single document as well as [share with peers](https://nbviewer.jupyter.org/):

![](https://d23f6h5jpj26xu.cloudfront.net/rxluj5w6w1li9q_small.png)

The overall setup makes interactively working with code and data a breeze. Behind the scenes, the browser is communicating with IPython kernels (execution environments) over websockets.

![](https://d23f6h5jpj26xu.cloudfront.net/huzp0asjwourg_small.png)

These websockets make communication with the kernels very quick, but they have additional security concerns.

## Websockets and Cross Origin Restrictions

Modern browsers prevent JavaScript on one site from accessing content on another site. This prevents, for example, `twitter.com` from accessing your bank accounts on `mint.com`.

<script>
// Insert diagram of CORS
</script>

The story is different for websockets. Unlike builtin cross origin handling by modern browsers, the websocket specification has no such concept. The protection has to be done by the server. Some methods include:

- Authenticating requests
- Verifying Origin and Host
- Validating a `Sec-WebSocket-Key`

If these aren't setup for a given server, any remote site that you visit can access the websockets on that server. `example.com` can, with client side javascript, open a websocket to `yourbank.com`, from the comfort of JavaScript in _your_ browser.

## Cross Origin Websockets and IPython

Prior to [pull request #4845](https://github.com/ipython/ipython/pull/4845), the IPython Notebook did not perform origin verification. For notebook servers using authentication, this wasn't a problem (cookies are needed as part of authentication).

However, most users typically run the notebook locally with a default port (e.g. `127.0.0.1:8888`) and no authentication. It's only running on localhost not exposed to the web, what could go wrong? Well, an attacker could open `ws://127.0.0.1:8888/api/kernels/{some kernel id}` from their separate domain. Stated again, _any website could attempt to open up a websocket on your local machine with a little bit of JavaScript and knowledge of a kernel id_.

Simple example:

```JavaScript
// Create a basic Kernel object
var kernel = new IPython.Kernel();

// Use the default URL
kernel.ws_url = "ws://127.0.0.1:8888";
// Prior to this, acquire a kernel_id
kernel.kernel_url = "/kernels/" + kernel_id;
kernel.kernel_id = kernel_id;

kernel.start_channels();

// ...
// Trigger execution once you know the kernel is connected
// (there's an event for that):
kernel.execute('import os; os.mkdir("touchdown")');
```

If a site can use a kernel without your knowledge, they can run _anything_ they want _as you on your local, trusty machine_. [Remove all your files](/posts/rm-rf-remains), plop down some special purpose code, steal your data, and generally wreak havoc. Yikes.

### Demo

![](https://i.imgur.com/UscjI81.gif)

### Verifying Origin and Host

The vulnerability was fixed in time for the IPython 2.0 release (April 2014) and backported for the 1.x series (1.2 and above), as part of [#4845](https://github.com/ipython/ipython/pull/4845). We simply verify that the origin of the request matches the host. To make sure this light amount of protection happened by default I also submitted a [pull request to the tornado web framework](https://github.com/tornadoweb/tornado/pull/980).

This vulnerability is registered with
CVE ID [CVE-2014-3429](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-3429).

## Mitigations

This entire attack relies on knowledge of the id of the kernel. Considering this is a [UUID generated using uuid4](https://github.com/ipython/ipython/blob/rel-1.1.0/IPython/kernel/multikernelmanager.py#L105), this can't be brute forced in any reasonable amount of time. The more practical way is to use some amount of social engineering (see SciPy Gift Card demo below).

The only proactive mitigation you have if using an old version of the notebook is to set a password hash in your IPython Notebook configuration:

```python
c.NotebookApp.password = u'sha1:c43e9fe995cb73d73aca557de521d48...'
```

To create a password hash, use `IPython.lib.passwd()`:

```
In [1]: from IPython.lib import passwd
In [2]: passwd()
Enter password:
Verify password:
Out[2]: 'sha1:67c9e60bb8b6:9ffede0825894254b2e042ea597d771089e11aed'
```

If you're stuck on an old version and haven't upgraded, please do! You'll also get access to IPython widgets if you upgrade to 2.x.

## Lightning Talk at SciPy 2014

![](https://d23f6h5jpj26xu.cloudfront.net/vg8f9kjehosdcq_small.png)

The inimitable [Paul Ivanov](https://twitter.com/ivanov) let me use him on stage as part of the exploit's demonstration. Paul loves Philz Coffee (it is, in fact, incredible coffee), so I set up a phishing site just for him:

![](https://d23f6h5jpj26xu.cloudfront.net/wsy2vdai9lbbxg_small.png)

While I was able to show Paul his kernel IDs in an iframe, normal CORS protections prevented my javascript from grabbing the IDs directly. Paul had to enter the "gift card code" (a kernel id for a running kernel) manually.

Doing so ran some deviously playful AppleScript that [Min](https://github.com/minrk) wrote to open Photobooth and take Paul's picture on stage:

```JavaScript
var code = '%%bash\n' +
           'osascript -e \'tell application "Photo Booth" to activate\n' +
           'delay 6\n' +
           'tell application "Photo Booth"\n' +
           'activate\n' +
           'tell application "System Events" to keystroke return using {command down}\n' +
           'end tell\n\'';
```

<blockquote class="twitter-tweet" lang="en"><p>Remote execution confirmed! Snuck a picture of <a href="https://twitter.com/ivanov">@ivanov</a> during the &quot;One Weird Kernel Trick&quot; talk. <a href="http://t.co/IVSAesvRei">pic.twitter.com/IVSAesvRei</a></p>&mdash; Kyle R Kelley (@rgbkrk) <a href="https://twitter.com/rgbkrk/statuses/487363111897145345">July 10, 2014</a></blockquote>

Success

### Other References

- [Slides from Lightning Talk at SciPy 2014](https://speakerdeck.com/rgbkrk/one-weird-kernel-trick-hijacking-ipython-websockets)
- [Lightning Talk at SciPy 2014](http://youtu.be/ln4nE_EVDCg?t=44m30s), at 44:30.

There is an exploitation demo, but I've decided to tear that down and not link it because someone with a fortuitous network position (e.g. your router, your ISP, the person winking at you in the local coffee shop) could modify HTTP packets and run arbitrary code on your system. I'm willing to open source the demo though with enough pestering.

---

_For all my blog posts I've decided to hold discussion on Reddit, linking to the post. Today's post has been posted to [/r/python](http://www.reddit.com/r/Python/comments/2am4le/vulnerability_in_ipython_notebook_11_cross_origin/) as well as [/r/netsec](http://www.reddit.com/r/netsec/comments/2ao18a/cross_site_websocket_hijacking_of_localhost_via/). Feel free to cross-post it and [PM me](http://www.reddit.com/message/compose/?to=lambdaops) so I can link it here._


One Weird Kernel Trick


Just for fun, I decided to launch a new Linux server and run `rm -rf /` as root to see what remains. As I found out, `rm` lives in the future with idiots like me, so you have to specify `--no-preserve-root` to kick this exercise off.

    # rm -rf --no-preserve-root /

After committing this act of tomfoolery, great utilities like

- `/bin/ls`
- `/bin/cat`
- `/bin/chmod`
- `/usr/bin/file`

will all be gone! You should still have your connection over SSH as well as your existing bash session. This means you have all the bash builtins, like `echo`.

## Becoming Bash McGyver

```console
root@rmrf:/# ls
-bash: /bin/ls: No such file or directory
```

There is no `ls`, but `echo` and fileglobs are still around. What can we do with those?

```console
root@rmrf:/# echo *
dev proc run sys
# echo /dev/pts/*
/dev/pts/0 /dev/pts/3 /dev/pts/ptmx
```

Hey, we got to keep `/dev`, `/proc`, `/run`, and `/sys`. Now that we have `ls`, we might as well make it a little easier to read.

```console
root@rmrf:/# for file in /dev/pts/*; do echo $file; done
/dev/pts/0
/dev/pts/3
/dev/pts/ptmx
```

[Several](https://www.reddit.com/r/programming/comments/27j311/rm_rf_remains/ci1gc4i) [Redditors](https://www.reddit.com/r/programming/comments/27j311/rm_rf_remains/ci1gpli) pointed out that `printf` is available.

```console
root@rmrf:/# ls() { printf '%s\n' ${1:+${1%/}/}*; }
```

"_printf will [apply] the format string until it runs out of args._" - [camh-](https://www.reddit.com/user/camh-)

Since you can define functions in bash, we can create an `ls` utility, albeit very limited.

```console
root@rmrf:/# ls() { printf '%s\n' ${1:+${1%/}/}*; }
-bash: syntax error near unexpected token `('
```

What? That should be completely valid. Is `ls` already hashed or aliased?

```console
root@rmrf:/# type ls
ls is aliased to `ls --color=auto'
```

Ah, that gets expanded to `ls--color=auto () { printf '%s\n' ${1:+${1%/}/}*; }`. Yuck. Well, we can `unalias` that.

```console
root@rmrf:/# unalias ls
root@rmrf:/# ls() { printf '%s\n' ${1:+${1%/}/}*; }
root@rmrf:/# ls
/dev
/proc
/run
/sys
root@rmrf:/# ls /dev
/dev/pts
```

And save our work.

```console
root@rmrf:/# echo 'ls() { printf '%s\n' ${1:+${1%/}/}*; }' >> utils.sh
root@rmrf:/# source utils.sh
```

How about `cat`? The `read` builtin comes in handy along with pipes and redirection, so we can piece together a rudimentary `cat`.

```console
root@rmrf:/# (while read line; do echo "$line"; done) < utils.sh
ls() { printf '%s\n' ${1:+${1%/}/}*; }
```

With these abilities and the fact that we can write arbitrary bytes with `echo`, we could rebuild and then `curl` or `wget` the binaries we want directly. My first choice, [echoed by others](https://www.qfbox.info/bashcp), would be to get [busybox](https://www.busybox.net/about.html). Busybox is the Swiss Army Knife of Embedded Linux, with builtin versions of `wget`, `dd`, `tar`, and many others. [Eusebeîa goes into great detail about how to get a fully escaped version of busybox on your system](https://www.qfbox.info/bashcp), so I won't do that here.

There is a problem though.

Even if we `echo` all the bytes we need into creating entire binaries, those files won't be executable. No way to start busybox. The easiest workaround for this is to find something which is executable and overwrite it with `echo`. We've nuked all of `/usr` and `/bin` at this point though, so that's a bit tricky.

We can use shell globs and bash logic to find files with the executable bit set, making sure to ignore directories.

```console
executable () { if [[ ( ! -d $1 ) && -x $1 ]] ; then echo "$1"; fi }
```

### Find the executables!

```console
root@rmrf:/# for file in /*; do executable $file; done
root@rmrf:/# for file in /*/*; do executable $file; done
root@rmrf:/# for file in /*/*/*; do executable $file; done
/proc/1107/exe
/proc/1136/exe
/proc/1149/exe
/proc/1179/exe
/proc/1215/exe
/proc/1217/exe
/proc/1220/exe
/proc/1221/exe
/proc/1223/exe
/proc/1248/exe
/proc/1277/exe
/proc/1468/exe
/proc/1478/exe
/proc/1625/exe
/proc/1644/exe
/proc/1/exe
/proc/374/exe
/proc/378/exe
/proc/471/exe
/proc/616/exe
/proc/657/exe
/proc/self/exe
```

Great! Hold on a minute though, those are all symbolic links to executables that no longer exist on disk. We'll just update `executable()` to ignore symbolic links.

```console
root@rmrf:/# executable () { if [[ ( ! -d $1 ) && ( ! -h $1 ) && -x $1 ]] ; then echo "$1"; fi }
root@rmrf:/# for file in /*/*/*; do executable $file; done
root@rmrf:/# for file in /*/*/*/*; do executable $file; done
root@rmrf:/# for file in /*/*/*/*/*; do executable $file; done
root@rmrf:/# for file in /*/*/*/*/*/*; do executable $file; done
```

Well now, that's bad news bears. Perhaps there is something at the kernel level we can use. After all, we can restart the box using sysrq _magic_:

```console
root@rmrf:/# echo 1 > /proc/sys/kernel/sysrq
root@rmrf:/# echo "b" > /proc/sysrq-trigger
```

~~Now we're locked out and I should do something else on a Friday. Thanks for reading! Let me know if you figure out how to get an executable bit set.~~

UPDATE: Redditor throw_away5046 posted [a full, beautiful, solution to this](https://www.reddit.com/r/linux/comments/27is0x/rm_rf_remains/ci199bk).

#### From a non-hosed, same architecture box:

```console
$ mkdir $(xxd -p -l 16 /dev/urandom)
$ cd $_
$ apt-get download busybox-static
$ dpkg -x *.deb .
$ alias encode='{ tr -d \\n | sed "s#\$..\$#\\\\x\\1#g"; echo; }'
$ alias upload='{ xxd -p | encode | nc -q0 -lp 5050; }'
$ upload < bin/busybox
```

#### Back on the rmrf'ed machine

```console
# cd /
# alias decode='while read -ru9 line; do printf "$line"; done'
# alias download='( exec 9<>/dev/tcp/{IP OF NON HOSED BOX}/5050; decode )'
# download > busybox
```

#### Now to create a shared object that will change permissions on busybox

```console
$ cat > setx.c <<EOF
extern int chmod(const char *pathname, unsigned int mode);

int entry(void) {

        return !! chmod("busybox", 0700);
}
char *desc[] = {0};

struct quick_hack {

        char *name; int (*fn)(void); int on;
        char **long_doc, *short_doc, *other;

} setx_struct = { "setx", entry, 1, desc, "chmod 0700 busybox", 0 };
EOF
$ gcc -Wall -Wextra -pedantic -nostdlib -Os -fpic -shared setx.c -o setx
$ upload < setx
```

#### Time to `enable` setx as a built in and get busybox executable

```console
# ( download > setx; enable -f ./setx setx; setx; )
# /busybox mkdir .bin
# /busybox  --install -s .bin
# PATH=/.bin
```

In action:

![](/assets/blog/rm-rf-remains/busyboxrecovery.gif)

---

_For all my blog posts I've decided to hold discussion on Reddit, linking to the post. Today's post is on [/r/linux](https://www.reddit.com/r/linux/comments/27is0x/rm_rf_remains/) and on [/r/programming](https://www.reddit.com/r/programming/comments/27j311/rm_rf_remains/), but feel free to cross post it. PM me if you want me to link it here._

---

<a target="_blank" href="https://www.amazon.com/gp/product/0596003307?&_encoding=UTF8&tag=lambdaops02-20&linkCode=ur2&linkId=fc4ff3aebb455f4f223478d15f1aa330&camp=1789&creative=9325">Unix Power Tools</a>


Go forth and build your <a target="_blank" href="https://www.amazon.com/s/?_encoding=UTF8&camp=1789&creative=390957&keywords=bash&linkCode=ur2&qid=1402239800&rh=n%3A283155%2Ck%3Abash&sort=relevancerank&tag=lambdaops02-2&linkId=OXSOLI7W574KTESZ">bash knowledge</a>. Personally, I recommend <a href="https://www.amazon.com/gp/product/0596003307?&_encoding=UTF8&tag=lambdaops02-20&linkCode=ur2&linkId=fc4ff3aebb455f4f223478d15f1aa330&camp=1789&creative=9325">Unix Power Tools</a>.

<a href="https://www.amazon.com/gp/product/0596003307?&_encoding=UTF8&tag=lambdaops02-20&linkCode=ur2&linkId=fc4ff3aebb455f4f223478d15f1aa330&camp=1789&creative=9325"><img border="0" src="https://ws-na.amazon-adsystem.com/widgets/q?_encoding=UTF8&ASIN=0596003307&Format=_SL160_&ID=AsinImage&MarketPlace=US&ServiceVersion=20070822&WS=1&tag=lambdaops02-2" style="display:block; margin-left:auto; margin-right:auto"></a>

Through college and career, flipping through this book has always showed me something new.


rm -rf remains


Every time I need to share a credential (password, API Key, etc.) for a shared account I'm faced with a dilemma. How do I pass someone a secret, encrypted just for them? Public keys are an obvious choice. What if they're not using GPG though? What else does a typical developer have?

**SSH Keys.**

Armed with the knowledge that

- Just about everyone has SSH keys
- GitHub provides public SSH keys at `https://github.com/{user}.keys`
- `ssh-keygen` and `openssl` are typically available to handle crypt

It's pretty simple to do this.

```bash
# Get the user's keys, use the last key
wget https://github.com/$user.keys --quiet -qO- | tail -n 1 > $pubkey

# Convert to a pem file
ssh-keygen -f $pubkey -e -m PKCS8 > $pubkey.pem

# Encrypt some message
openssl rsautl -encrypt -pubin -inkey $pubkey.pem -ssl -in $infile -out $outfile
```

This relies on the fact that you can encrypt something small using a public key. Usually this "something small" is a key for a symmetric encryption algorithm (e.g. AES), which in turn is used to encrypt a much larger message body. Instead, we're encrypting our whole message (which is given some random padding by `openssl`).

There are some issues with doing this:

- It relies on a third party for key integrity, no fingerprints
- You can only encrypt up to a certain number of bytes

For a typical key size, it still gives you plenty more characters than Twitter does though.

| Key size (bits) | Maximum Message Size (bytes) |
| --------------- | ---------------------------- |
| 768             | 85                           |
| 1024            | 117                          |
| 2048            | 246                          |
| 4096            | 502                          |
| 8192            | 1018                         |

## Enter CryptHub ![](https://d23f6h5jpj26xu.cloudfront.net/knsubajidfhzqg_small.png)

Even with these concerns and a looming fear that there would be something else I'm missing, I wrote a gist and [shared it with the world](https://gist.github.com/rgbkrk/7827691).

<blockquote class="twitter-tweet" lang="en"><p>Hack of the day: Encrypting a file using a GitHub user&#39;s public key - <a href="https://gist.github.com/rgbkrk/7827691">https://gist.github.com/rgbkrk/7827691</a></p>&mdash; Kyle Ray Kelley (@rgbkrk) <a href="https://twitter.com/rgbkrk/statuses/409156138634973184">December 7, 2013</a></blockquote>

After a [good response on Twitter and the gist](https://gist.github.com/rgbkrk/7827691#comment-965445) I realized that, well, _maybe this could be a thing_. What if we build an ephemeral message service that relies on people's public keys from GitHub? (Or BitBucket, GitLab, etc.) Anyone with a GitHub account can post a message to another GitHub user, relying on the public keys the recipient has stored on GitHub. Authentication can be handled using GitHub's OAuth.

At the time I was thinking of this, Rackspace had just come out with [CloudQueues](http://www.rackspace.com/cloud/queues/) which is built from [OpenStack Marconi](https://wiki.openstack.org/wiki/Marconi). You can think of it like [AWS's SQS](https://aws.amazon.com/sqs/) or using RabbitMQ yourself. Each of these expire messages after a certain amount of time and consume short messages (for the astute reader, yes, I'm hand waving here).

Throw the encrypted messages on a queue per user, let the recipients download them later. Messages get deleted when the time is up or the recipient trashes them. The service can't see your keys or your plaintext messages.

### Informal Design

To get setup, a user

- goes to CryptHub.io, logs in using GitHub
- downloads a simple client application
- sets up GitHub OAuth keys with the client app

#### Sending a message

```
$ hubencrypt smashwilson -m "Drink more ovaltine"
Getting the key for smashwilson. ☁
Converting public key to a PEM PKCS8 public key. √
Encrypting message. √
Message sent!
```

The message gets placed on a message queue for a recipient to download.

#### Recipient

```
$ hubdecrypt -r
Downloading messages. ☁
Message from rgbkrk:
>> Drink more ovaltine
```

# Reservations and the Future

What are the security implications? Who would use it? Will this look poorly on me if this is Fundamentally a Bad Idea™?

This doesn't even solve the problem for people not on GitHub (unless they share public keys through other channels). If they're not even a techie, this is mostly useless for them. We'd have to build a native application for Windows that can generate the keys, etc. OK, maybe that isn't so bad.

Since this was just a side project I was thinking about with [Ash Wilson](https://github.com/smashwilson) while my wife and are currently chasing a 2 year old and an infant, I left this on the backburner. In the meantime, [Jan Schaumann](https://gist.github.com/jschauma) updated [jass](https://github.com/jschauma/jass) to use keys from Github, completing the client side portion.

Then [keybase](https://keybase.io/) came along and I thought _well there's one way to do it_. Not quite the way I'd want it, but pretty close. I won't lie that it made me kick myself in the butt for not just pushing out what we had so far in open source.

That's ok, there are more problems to be solved.

Still wondering though...

**Would you use it?**

- [Discussion on Reddit](http://www.reddit.com/r/programming/comments/25r5lf/the_encrypted_message_service_im_not_building/)


λ Blog.

Earnest posting amidst a sea of irony and generative content.