---
title: making a quick and dirty terminal emulator
ogTitle: making a quick and dirty terminal emulator
author: epilys
date: 2019-10-25 00:00:00
date_iso_8601: 2019-10-25T00:00:00+02:00
bees: What if bees were made of bigger bees but smaller?
---
<figure>
<img width="100%" height="auto" src="/images/hark_canine.png" alt="Nested meli instances" />
<figcaption>I gave up at 15 levels deep (though I had to stretch the height a lot to view them all)</figcaption>
</figure>

## Background
In order to be able to compose e-mail within meli and avoid writing an editor, making an embed terminal was obviously my only choice. I didn't bother to look if there are already libraries in Rust for this because as always my prime motivator is figuring things out, for fun.

If you have ever executed `tty` before you'd probably see a response like `/dev/pts/16`. `pts` stands for pseudoterminal slave: the process (such as a shell) running inside it thinks it's speaking directly to an actual terminal, i.e. the virtual console you see when you <kbd class="□">Ctrl</kbd> <kbd class="□">Alt</kbd> <kbd class="□">F1</kbd> etc. Terminal emulators such as `xterm` create pseudoterminals and communicate with them by reading and writing to the pts device and the pseudoterminal reads and writes to `/dev/ptmx`. The kernel handles the pairing of I/O between each master/slave. You'll read how to create one below, but for more information read `pty(7)` on Linux, `pty(4)` on *BSDs.

> A  pseudoterminal  (sometimes  abbreviated  "pty") is a pair of virtual
> character devices that provide a bidirectional  communication  channel.  One
> end  of  the channel is called the master; the other end is called the slave.
> The slave end of the pseudoterminal provides  an  interface that behaves
> exactly like a classical terminal.  A process that expects to be connected to
> a terminal, can open the slave end of a pseudoterminal  and  then  be  driven
> by a program that has opened the master end.  Anything that is written on the
> master end is provided to  the  process on the slave end as though it was
> input typed on a terminal.  For example, writing the interrupt character
> (usually control-C) to the  master device would cause an interrupt signal
> (SIGINT) to be generated for the foreground process group that is connected
> to the  slave.   Conversely, anything  that is written to the slave end of
> the pseudoterminal can be read by the process that is connected to the master
> end.   Pseudoterminals  are  used by applications such as network login
> services (ssh(1), rlogin(1), telnet(1)), terminal emulators such as xterm(1),
> script(1), screen(1), tmux(1), unbuffer(1), and expect(1).


## First steps

Let's begin with creating the pty:

```rust
use crate::terminal::position::Area;
use nix::fcntl::{open, OFlag};
use nix::{ioctl_none_bad, ioctl_write_ptr_bad};
use nix::pty::{grantpt, posix_openpt, ptsname, unlockpt, Winsize};
use nix::sys::stat::Mode;

use libc::TIOCSCTTY;
// ioctl request code to set window size of pty:
use libc::TIOCSWINSZ;
use std::path::Path;
use std::process::{Command, Stdio};

use std::os::unix::io::{FromRawFd, IntoRawFd};

// nix macro that generates an ioctl call to set window size of pty:
ioctl_write_ptr_bad!(set_window_size, TIOCSWINSZ, Winsize);

// request to "Make the given terminal the controlling terminal of the calling process"
ioctl_none_bad!(set_controlling_terminal, TIOCSCTTY);

pub fn create_pty() -> nix::Result<()> {
    /* Create a new master */
    let master_fd = posix_openpt(OFlag::O_RDWR)?;

    /* For some reason, you have to give permission to the master to have a
     * pty. What is it good for otherwise? */
    grantpt(&master_fd)?;
    unlockpt(&master_fd)?;

    /* Get the path of the slave */
    let slave_name = unsafe { ptsname(&master_fd) }?;

    /* Try to open the slave */
    let _slave_fd = open(Path::new(&slave_name), OFlag::O_RDWR, Mode::empty())?;

    /* Launch our child process. The main application loop can inspect and then
    pass the stdin data to it. */
    let _child_pid = match fork() {
        Ok(ForkResult::Child) => {
            /* Open slave end for pseudoterminal */
            let slave_fd = open(Path::new(&slave_name), OFlag::O_RDWR, stat::Mode::empty())?;

            // assign stdin, stdout, stderr to the tty
            dup2(slave_fd, STDIN_FILENO).unwrap();
            dup2(slave_fd, STDOUT_FILENO).unwrap();
            dup2(slave_fd, STDERR_FILENO).unwrap();
            /* Become session leader */
            nix::unistd::setsid().unwrap();
            unsafe { set_controlling_terminal(slave_fd) }.unwrap();
            nix::unistd::execv(
                &CString::new("/usr/bin/vim").unwrap(),
                Vec::new(),
                ).unwrap();
            /* This path shouldn't be executed. */
            std::process::exit(-1);
        }
        Ok(ForkResult::Parent { child }) => child,
        Err(e) => panic!(e),
    };

    /* Continue dealing with the pty in a thread so that the main application doesn't lock up */
    std::thread::Builder::new()
        .spawn(move || {
            let winsize = Winsize {
                ws_row: 25,
                ws_col: 80,
                ws_xpixel: 0,
                ws_ypixel: 0,
            };
            let master_fd = master_fd.into_raw_fd();
            /* Tell the master the size of the terminal */
            unsafe { set_window_size(master_fd, &winsize).unwrap() };
            let master_file = unsafe { std::fs::File::from_raw_fd(master_fd) };
            /* start the pty liaison */
            liaison(master_file);
        })
        .unwrap();
    Ok(())
}
```
So far graphics are not in the picture (heh). To draw directly on the screen we can print the bytes we read from the master without doing any sort of work in-between:

```rust
fn liaison(pty_fd: std::fs::File) {
    let stdout = std::io::stdout();
    let mut lock = stdout.lock();

    let mut bytes_iter = pty_fd.bytes();
    while let Some(Ok(byte)) = bytes_iter.next() {
      lock.write_all(&[byte]).unwrap();
    }
}
```


## Embedding graphics

A quick explanation on how meli graphics work: There's a two dimensional grid representing the state of the terminal at the current moment. Its elements are of type `Cell`:

```rust
struct Cell {
    ch: char,

    fg: Color,
    bg: Color,
    attrs: Attr,
}
```

To cache stuff I've made some UI components draw their stuff into their own grids and whenever I have to redraw, they copy the cache instead of recomputing data. To actually show changes on the screen each component pushes the area of the grid they have changed into the app state, and whenever it's time to draw the app displays only the dirty areas. On resize all of the components are marked dirty and redraw everything on the newly resized grid.

The embedded process can thus draw its output on such a grid and we can then draw it like a regular UI widget.

### How the terminal handles output

For terminal UI apps we're interested in the Alternative screen buffer mode. In this mode the application is responsible for handling the cursor. There is no scrollback, to emulate scrolling the app has to redraw the screen entirely or use a special scrolling functionality that is out of scope for this post.

To handle colours, cursor style changes, cursor position changes, the application sends *control sequences* to the terminal, which are sequences of bytes with designated prefixes and suffixes that correspond to a command. `xterm` control sequences are not standardised, but [here is one list](https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Functions-using-CSI-_-ordered-by-the-final-character_s_). A quick summary with examples can be found in the [Wikipedia Article](https://en.wikipedia.org/wiki/ANSI_escape_code). 

All sequences start with the Escape character, `0x1b`, and are sorted in separate categories. The basic one is `CSI` or *Command Sequence Introducer*, which is Escape followed by `[`.

Since we're pretending to be a terminal, we will encode the code sequence input's state in a state machine:

```rust
pub enum State {
    Normal, // Waiting for any kind of byte
    ExpectingControlChar, // We just got an ESC and expect a control sequence
    Csi, // ESC [ Control Sequence Introducer
    /* Multiparameter sequences are of the form KIND P_1 ; P_2 ; ... TERMINATION */
    Csi1(Vec<u8>), // CSI with one buffer (CSI <num>) waiting for new buffer digits, a second buffer or a termination character
    Csi2(Vec<u8>, Vec<u8>), // CSI with two buffers (CSI <num> ; <num>) as above
    Csi3(Vec<u8>, Vec<u8>, Vec<u8>), // CSI with three buffers
    CsiQ(Vec<u8>), // CSI followed by '?'
    Osc1(Vec<u8>), // ESC ] Operating System Command
    Osc2(Vec<u8>, Vec<u8>),
}
```

The embed grid will include all the simulated terminal's state:

```rust
pub struct EmbedGrid {
   /*
         1   2  ..
       ┏━━━┯━━━━━━━┅┅ x axis
     1 ┃1,1│
       ┠───┼┈┈
     2 ┃   ┊
       ┃
    .. ┃
y axis ┇
    */
    cursor: (usize, usize), // xterm cursor starts from (1, 1), but ours from (0, 0)

    pub grid: CellBuffer,
    pub state: State,
    pub stdin: std::fs::File,
    // Pid of the embed process
    pub child_pid: nix::unistd::Pid,
    // (width, height)
    pub terminal_size: (usize, usize),
    // Current colors
    fg_color: Color,
    bg_color: Color,

    show_cursor: bool,
    /* And others */
    ..
}
```

And here's the methods to control the terminal:

```rust
impl EmbedGrid {
    pub fn set_terminal_size(&mut self, new_val: (usize, usize)) {
        if new_val == self.terminal_size {
            return;
        }

        self.terminal_size = new_val;
        self.grid.resize(new_val.0, new_val.1, Cell::default());
        self.grid.clear(Cell::default());
        self.cursor = (0, 0);
        let winsize = Winsize {
            ws_row: <u16>::try_from(new_val.1).unwrap(),
            ws_col: <u16>::try_from(new_val.0).unwrap(),
            ws_xpixel: 0,
            ws_ypixel: 0,
        };

        let master_fd = self.stdin.as_raw_fd();
        /* This is the macrogenerated function we defined earlier, which calls
           ioctl underneath */
        unsafe { set_window_size(master_fd, &winsize).unwrap() };
        /* SIGWINCH informs the process the tty size has changed. TUI apps must
           handle this signal */
        nix::sys::signal::kill(self.child_pid, nix::sys::signal::SIGWINCH).unwrap();
    }

    pub fn wake_up(&self) {
        nix::sys::signal::kill(self.child_pid, nix::sys::signal::SIGCONT).unwrap();
    }

    pub fn stop(&self) {
        /* we can emulate Ctrl-z and suspend the child process from meli */
        nix::sys::signal::kill(self.child_pid, nix::sys::signal::SIGSTOP).unwrap();
    }

    pub fn process_byte(&mut self, byte: u8);
}
```

In the `process_byte` method we take input from the child process and we perform actions depending on the control sequence state:

- if it's `Normal` and we get a byte that does not start a sequence, we output it on the grid.
- if the state is in a sequence and we get a valid termination byte, we perform the sequence's action
- if the state is in a sequence and we get a valid continuation byte, we update the state
- else, the child process sent malformed output or we haven't implemented some sequence correctly (at this stage, probably the latter)

```rust
    pub fn process_byte(&mut self, byte: u8) {
        let EmbedGrid {
            ref mut cursor,
            ref terminal_size,
            ref mut grid,
            ref mut state,
            ref mut stdin,
            ref mut fg_color,
            ref mut bg_color,
            child_pid: _,
            ..
        } = self;

        macro_rules! increase_cursor_x {
            () => {
                if cursor.0 + 1 < terminal_size.0 {
                    cursor.0 += 1;
                }
            };
        }

        macro_rules! cursor_x {
            () => {{
                if cursor.0 >= terminal_size.0 {
                    cursor.0 = terminal_size.0.saturating_sub(1);
                }
                cursor.0
            }};
        }
        macro_rules! cursor_y {
            () => {
                std::cmp::min(
                    cursor.1 + scroll_region.top,
                    terminal_size.1.saturating_sub(1),
                )
            };
        }
        macro_rules! cursor_val {
            () => {
                (cursor_x!(), cursor_y!())
            };
        }

        let mut state = state;
        match (byte, &mut state) {
            (b'\x1b', State::Normal) => {
                *state = State::ExpectingControlChar;
            }
            (b']', State::ExpectingControlChar) => {
                let buf1 = Vec::new();
                *state = State::Osc1(buf1);
            }
            (b'[', State::ExpectingControlChar) => {
                *state = State::Csi;
            }
            /* 8<... */
            (b'H', State::Csi) => {
                /* move cursor to (1,1) */
                *cursor = (0, 0);
                *state = State::Normal;
            }
            /* 8<... */
            (b'\r', State::Normal) => {
                // carriage return x-> 0
                cursor.0 = 0;
            }
            (b'\n', State::Normal) => {
                // newline y -> y + 1
                if cursor.1 + 1 < terminal_size.1 {
                    cursor.1 += 1;
                }
            }
            /* replaced the actual ^G character here with '^' + 'G' for clarity: */
            (b'^G', State::Normal) => {
                // Visual bell ^G, ignoring
            }
            (0x08, State::Normal) => {
                /* Backspace */
                // x -> x - 1
                if cursor.0 > 0 {
                    cursor.0 -= 1;
                }
            }
            (c, State::Normal) => {
                /* Character to be printed. */

                /* We're UTF-8 bound, so check if byte starts a multi-byte
                codepoint and keep state on this in order to get the complete
                char, which will be 1-4 bytes long.
                Check UTF-8 spec to see how it's defined.
                */

                /* ...codepoint checks ...8< */
                grid[cursor_val!()].set_ch(c);
                grid[cursor_val!()].set_fg(*fg_color);
                grid[cursor_val!()].set_bg(*bg_color);
                increase_cursor_x!();
            }
            /* other various sequences: */
            (b'H', State::Csi2(ref y, ref x)) => {
                // Set Cursor Position [row;column] (default = [1,1])
                let orig_x = unsafe { std::str::from_utf8_unchecked(x) }
                    .parse::<usize>()
                    .unwrap_or(1);
                let orig_y = unsafe { std::str::from_utf8_unchecked(y) }
                    .parse::<usize>()
                    .unwrap_or(1);

                if orig_x - 1 <= terminal_size.0 && orig_y - 1 <= terminal_size.1 {
                    cursor.0 = orig_x - 1;
                    cursor.1 = orig_y - 1;
                } else {
                    eprintln!(
                        "[error] terminal_size = {:?}, cursor = {:?} but cursor set to  [{},{}]",
                        terminal_size, cursor, orig_x, orig_y
                    );
                }

                *state = State::Normal;
            }
            (b'm', State::Csi1(ref buf1)) => {
                // Set color
                match buf1.as_slice() {
                  /* See next sequence for 38 and 48 special meanings */
                    b"30" => *fg_color = Color::Black,
                    b"31" => *fg_color = Color::Red,
                    b"32" => *fg_color = Color::Green,
                    b"33" => *fg_color = Color::Yellow,
                    b"34" => *fg_color = Color::Blue,
                    b"35" => *fg_color = Color::Magenta,
                    b"36" => *fg_color = Color::Cyan,
                    b"37" => *fg_color = Color::White,

                    b"39" => *fg_color = Color::Default,
                    b"40" => *bg_color = Color::Black,
                    b"41" => *bg_color = Color::Red,
                    b"42" => *bg_color = Color::Green,
                    b"43" => *bg_color = Color::Yellow,
                    b"44" => *bg_color = Color::Blue,
                    b"45" => *bg_color = Color::Magenta,
                    b"46" => *bg_color = Color::Cyan,
                    b"47" => *bg_color = Color::White,

                    b"49" => *bg_color = Color::Default,
                    _ => {}
                }
                grid[cursor_val!()].set_fg(*fg_color);
                grid[cursor_val!()].set_bg(*bg_color);
                *state = State::Normal;
            }
            (b'm', State::Csi3(ref buf1, ref buf2, ref buf3)) if buf1 == b"38" && buf2 == b"5" => {
                /* ESC [ m 38 ; 5 ; fg_color_byte m */ 
                /* Set only foreground color */
                *fg_color = if let Ok(byte) =
                    u8::from_str_radix(unsafe { std::str::from_utf8_unchecked(buf3) }, 10)
                {
                    Color::Byte(byte)
                } else {
                    Color::Default
                };
                grid[cursor_val!()].set_fg(*fg_color);
                *state = State::Normal;
            }
            (b'm', State::Csi3(ref buf1, ref buf2, ref buf3)) if buf1 == b"48" && buf2 == b"5" => {
                /* ESC [ m 48 ; 5 ; fg_color_byte m */ 
                /* Set only background color */
                *bg_color = if let Ok(byte) =
                    u8::from_str_radix(unsafe { std::str::from_utf8_unchecked(buf3) }, 10)
                {
                    Color::Byte(byte)
                } else {
                    Color::Default
                };
                grid[cursor_val!()].set_bg(*bg_color);
                *state = State::Normal;
            }
            (b'D', State::Csi1(buf)) => {
                // ESC[{buf}D   CSI Cursor Backward {buf} Times
                let offset = unsafe { std::str::from_utf8_unchecked(buf) }
                    .parse::<usize>()
                    .unwrap();
                cursor.0 = cursor.0.saturating_sub(offset);
                *state = State::Normal;
            }
            /* and others */
        }
    }
```

A bit of UI glue code later:

<video src="/images/2019-11-04_13.33.15.mp4" controls loop="true" width="100%">
