# Eternal TF2 2Fort Bot Stream — headless setup

Everything needed to run TF2 bots on 2fort forever, headless, and stream it.
**Excludes the TF2 install itself** (~31 GB) — re-download with steamcmd (see Bootstrap).

Includes `assets/ctf_2fort.nav` — the generated nav mesh. **Copy it in and you skip
`nav_generate` entirely** (it is slow, and without it bots spawn but never move).

---

## Performance baseline — READ THIS BEFORE ASSUMING IT'S BROKEN

This was built on a **2 vCPU / 7.8 GB / no-GPU** box, where every layer was software:

```
TF2 (D3D9) → DXVK → Vulkan → lavapipe (SW) → llvmpipe (SW) → x264 (SW)
```

**It ran at about 3 fps.** That is the baseline to compare against — not zero. The pipeline was
correct and genuinely streaming to YouTube; it was purely CPU-starved.

Measured there:
- `tf_linux64` alone at **101–107% CPU** — a full core, just rasterizing 854x480
- load average **~4.7 on 2 cores**
- cutting bots 12 → 6 changed **nothing** → bottleneck is fill rate, not bot simulation
- fixing the encoder (native res, 15 fps, `-threads 1`, `nice`) got ffmpeg to ~16% CPU,
  but the game stayed pegged regardless

On the new box the two software layers should vanish (real Vulkan) and x264 can become NVENC, so
expect a different order of magnitude. **If it still crawls there, suspect config rather than the
approach** — verify you are NOT still on software rendering:

```bash
glxinfo | grep -i "renderer"      # must NOT say llvmpipe
vulkaninfo | grep -i deviceName   # must NOT say lavapipe
```

If those say llvmpipe/lavapipe, the GPU isn't being used and you'll get the same 3 fps.
Also drop `LIBGL_ALWAYS_SOFTWARE=1` and `GALLIUM_DRIVER=llvmpipe` from the launch env — they are
set in these scripts specifically to force software rendering and **must be removed on a GPU box**.

---

## Bootstrap on the new server

```bash
sudo apt update
sudo apt install -y xvfb x11-utils xdotool openbox ffmpeg curl tar bzip2 \
                    mesa-vulkan-drivers libvulkan1 vulkan-tools \
                    pulseaudio pulseaudio-utils lib32gcc-s1 libc6-i386

BASE=$HOME/tf2stream && mkdir -p $BASE && cd $BASE

# 1) steamcmd
mkdir -p steamcmd && cd steamcmd
curl -sSL https://steamcdn-a.akamaihd.net/client/installer/steamcmd_linux.tar.gz | tar xz
cd ..

# 2) TF2 client (app 440). F2P — ANY real Steam account works.
#    Anonymous login CANNOT download the client.
./steamcmd/steamcmd.sh +force_install_dir $BASE/tf2 +login YOUR_STEAM_LOGIN \
  +app_update 440 validate +quit

# 3) Steam Linux Runtime 3.0 "sniper" (app 1628350) — REQUIRED, anonymous is fine
./steamcmd/steamcmd.sh +force_install_dir $BASE/sniper +login anonymous \
  +app_update 1628350 validate +quit

# 4) Steam emulator so the client can init without a running Steam client
mkdir -p gbe && cd gbe
curl -sSL https://github.com/Detanup01/gbe_fork/releases/download/release-2025_01_09/emu-linux-release.tar.bz2 -o e.tar.bz2
tar -xjf e.tar.bz2 && rm e.tar.bz2 && cd ..
mkdir -p ~/.steam/sdk64
cp gbe/release/regular/x64/steamclient.so ~/.steam/sdk64/steamclient.so

# 5) drop in the nav mesh (saves a long nav_generate)
cp assets/ctf_2fort.nav tf2/tf/maps/ctf_2fort.nav

# 6) configs + blank the MOTD panels
cp cfg/autoexec.cfg cfg/server.cfg cfg/ctf_2fort.cfg tf2/tf/cfg/
: > tf2/tf/motd.txt; : > tf2/tf/motd_text.txt
: > tf2/tf/cfg/motd_default.txt; : > tf2/tf/cfg/motd_text_default.txt
```

Then put your YouTube key in `stream.env` and run `./go_live.sh`.

---

## Hard-won gotchas (read before debugging anything)

### Steam / launch
- **Sniper runtime is mandatory.** `tf.sh` reads `/etc/os-release` and aborts unless
  `VERSION_CODENAME=sniper`. Launch via `sniper/run-in-sniper -- ./tf.sh …`.
- **The client needs `SteamAPI_Init` to succeed.** Real `steamclient.so` is only a stub that
  IPCs to a *running* Steam client; with none, you get `create pipe failed` and an eternal
  black screen. Hence the emulator.
- **gbe_fork build choice is not arbitrary:**
  - Use **`regular`**, not `experimental` — experimental pulls X11 symbols and `dlopen` fails
    with `undefined symbol: XQueryPointer`.
  - Use **`release-2025_01_09`** (needs only GLIBC_2.29). The 2026 releases need **GLIBC_2.38**,
    which Ubuntu 22.04 (glibc 2.35) cannot load → `Sys_LoadModule failed`. Check with
    `objdump -T steamclient.so | grep -o 'GLIBC_[0-9.]*' | sort -V | tail -1`.
  - **Do NOT** overwrite the game's own `tf2/bin/linux64/libsteam_api.so` — that breaks startup
    with `Failed to load the launcher`. Only place the emu at `~/.steam/sdk64/steamclient.so`.
- `steam_appid.txt` containing `440` in the game root, plus `SteamAppId=440`.

### Rendering (headless)
- Modern TF2 Linux is **64-bit and renders D3D9 → DXVK → Vulkan**. Headless needs a **64-bit
  software Vulkan** driver: `mesa-vulkan-drivers` gives `lvp_icd.x86_64.json` (lavapipe).
  An i686-only ICD is NOT enough — you get
  `Failed to create SDL window: Installed Vulkan doesn't implement VK_KHR_surface`.
- **Resolution is fragile on Xvfb.** `854x480` fullscreen (Xvfb screen also 854x480) is the only
  combo verified stable. Observed failures:
  - `640x360` **fullscreen** → `SDL_GetClosestDisplayMode: Couldn't find any matching video modes`
    → `EnterFullscreenMode: Failed` → **SIGTRAP** (exit 133).
  - `640x360` **windowed** → **SIGFPE** (exit 136) right after `server.so loaded`.
  On a real GPU box this constraint mostly disappears; retest higher res there.
- Persist shader caches or every boot re-compiles: `DXVK_STATE_CACHE_PATH`, `MESA_SHADER_CACHE_DIR`.
- `FONTCONFIG_PATH=/etc/fonts` or the UI stalls on `Cannot load default config file`.

### Input automation (the part that wasted the most time)
- **A window manager is required.** With bare Xvfb and no WM, SDL never receives focus, so it
  **ignores all keyboard input** (mouse *motion* still works, which is misleading — buttons
  highlight on hover but clicks do nothing). Run `openbox`.
- **Use XTEST, not XSendEvent.** `xdotool key --window <id> …` sends synthetic events that SDL
  deliberately ignores. Drop `--window` and focus the window first:
  `xdotool windowactivate --sync $WID; xdotool key F8`.
- Find the window by class + geometry; **it has no name early on**:
  `xdotool search --class tf_linux64`, then match `getwindowgeometry` to `854x480`.

### Getting into spectator
- **`jointeam` and `spectate` do not exist in this build.** `find jointeam` returns nothing.
  The valid command is **`changeteam`** (e.g. `changeteam spectator`).
- Menu click path at 854x480 (each step needs several seconds):
  1. MOTD **CONTINUE** → `(738, 455)`
  2. Map briefing **CONTINUE** → `(738, 455)`
  3. "WANT COACHING?" **NO** → `(517, 304)`
  4. Team menu: press key **`2`** = the SPECTATE TV (doors are 1=RANDOM, 3=BLU, 4=RED)
- Blanking the MOTD files removes the *text* but the panel + CONTINUE still appear, so step 1
  is still required.
- `spec_autodirector 1` makes the camera auto-follow the action — ideal for an ambient stream.

### Server-side must-dos
- **Nav mesh or bots do nothing.** They spawn and stand still. `nav_generate` fixes it but is
  slow; ship `ctf_2fort.nav` instead (included).
- **Disable the AFK kick** or an idle spectator gets booted mid-stream:
  `mp_idledealmethod 0`, `mp_idlemaxtime 0`.
- **`mp_waitingforplayers_cancel 1`** — otherwise the round sits at "WAITING FOR PLAYERS".
- `tf_bot_add N`. `CreateFakeClient() returned NULL` just means slots are full — raise `maxplayers`.
- Cheat-flagged cvars (`mat_force_tonemap_scale`, `mat_hdr_tonemapscale`) need `sv_cheats 1`
  **and** must run *after* map load — put them in `cfg/<mapname>.cfg`, not `autoexec.cfg`.
  Note `cfg/<mapname>.cfg` runs in **server** context, so client commands like `changeteam`
  are "Unknown command" there — those must come from keybinds/console.
- The washed-out white screen people hit is usually just the **blurred main menu background**,
  not a render bug. Once actually in-game the exposure is correct.

### Audio
- No sound card → ALSA errors and silence. Create a PulseAudio null sink and point the game at it:
  ```bash
  pulseaudio --start --exit-idle-time=-1
  pactl load-module module-null-sink sink_name=tf2sink
  # game env: SDL_AUDIODRIVER=pulse PULSE_SINK=tf2sink
  # ffmpeg:   -f pulse -i tf2sink.monitor
  ```
- **Do not use `anullsrc`.** Pure digital silence encodes to ~0 kbps and YouTube warns
  "audio stream bit rate (0) is lower than recommended".

### Encoding
- **Never upscale.** Capturing 854x480 and scaling to 1280x720 makes x264 encode 2.25x the
  pixels and starves the game. Stream native.
- On a CPU-only box: `-preset ultrafast -threads 1` and `nice -n 15` the encoder so the game wins.
- **On the new GPU box:** drop all of that — use `h264_nvenc`, raise resolution and fps.

### Shell footgun
- `pkill -f <pattern>` matches **its own shell's command line** and kills your session
  (exit 144, empty output — looks exactly like "backgrounding doesn't work"). Use `pkill -x <name>`.

---

## Persistence

`run_in_background`-style tasks die with the session. For an actually-eternal stream use systemd
units (`systemd/` in this archive) — they restart on crash and survive logout:

```bash
mkdir -p ~/.config/systemd/user && cp systemd/*.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now tf2-xvfb tf2-openbox tf2-game
loginctl enable-linger $USER    # keep running after logout

journalctl --user -u tf2-game -f          # watch it
systemctl --user restart tf2-game         # kick it
```

Three units only — `tf2-game` runs `go_live.sh`, which itself starts the menu driver and the
ffmpeg push as children, so they restart together with the game.

On a GPU box, flip the game unit to `Environment=GPU=1` (drops llvmpipe forcing, switches the
encoder to NVENC).

## Files

| Path | What |
|---|---|
| `go_live.sh` | one-shot: display + WM + game + menu driver + stream |
| `run_game.sh` | just the game |
| `run_capture.sh` | just the ffmpeg push |
| `menu_driver.sh` | clicks MOTD → briefing → coaching → spectate |
| `cfg/autoexec.cfg` | client: low-end render settings + spectator binds |
| `cfg/server.cfg` | server: bots, never-ending match |
| `cfg/ctf_2fort.cfg` | per-map: exposure, idle-kick off, bots, cancel waiting |
| `assets/ctf_2fort.nav` | **generated nav mesh — copy to `tf2/tf/maps/`** |
| `systemd/` | units for run-forever |
| `stream.env` | resolution/bitrate + **YouTube key (fill in)** |