# source2rosetta

**Current CS2 and Dota 2 gamedata — re-derived from every Valve build, proven on a live server, published automatically.**

When Valve ships an engine update, every Metamod / CounterStrikeSharp plugin breaks until someone hand-reverse-engineers fresh gamedata: function signatures, vtable offsets, netvar layouts. That has historically taken days, sometimes weeks.

Here it takes **about half an hour, with nobody involved.** A timer notices the new build, re-derives the whole surface from the stripped `.so` libraries the dedicated server maps, launches its own vanilla server and *calls the functions* to prove they resolve, then publishes to a fixed URL. No one is paged and nothing is hand-checked — and if any stage fails, the run stops and the previous release stays up. What ships is never a guess.

```sh
# always the newest build
curl -fsSLO https://git.lo.sh/kamal/source2rosetta/releases/download/cs2-latest/gamedata-cs2.json
curl -fsSLO https://git.lo.sh/kamal/source2rosetta/releases/download/cs2-latest/netvars-cs2.json
```

The output is framework-neutral; `source2rosetta-gen` renders it into whatever your stack speaks. The deriver behind it is a standalone Rust tool — you only need that if you're self-hosting the pipeline or adding a game.

## Docs

- **[ATTRIBUTIONS.md](ATTRIBUTIONS.md) — start here.** This tool stands on a decade of community reverse-engineering, catalogues, dumpers, and research. The credits come first because the work does.
- **🎯 Render a release for your framework** → **[source2rosetta-gen](crates/source2rosetta-core/README.md)** — one command turns the JSON above into CounterStrikeSharp, Metamod/SourceMod, ModSharp, Swiftly, Plugify, or a typed C# SDK. No build, no corpus; what most people are here for.
- [CONTRIBUTING.md](CONTRIBUTING.md) — add or back-fill a gamedata entry.
- [LICENSE](LICENSE) — AGPL-3.0.

## Results

Ballpark from a recent build, on a 16-core desktop. These move build-to-build — treat them as orders of magnitude, not guarantees.

| | derived functions | typed schema | model | one-time distill |
|---|---|---|---|---|
| **CS2** | ~1,150 `core` + ~1,200 `high_confidence`, all live-validated, plus ~4,400 `experimental` name guesses | ~1,900 classes / ~12,300 fields | ~48 MB (a few MB gzipped) | ~15 min |
| **Dota 2** | ~1,900 `core` + ~1,000 `high_confidence`, plus ~6,100 `experimental` | ~2,960 classes / ~17,700 fields | ~570 MB | ~1 hr |

Both derive **0-dropped** — every offset and signature that ships passed live validation. Distilling the model is a one-time cost; after that each build's re-derive is minutes of compute, and the half hour in the headline is the whole loop: notice, update, derive, validate, publish.

---

## Staying current — the part with no human in it

Each game runs its own loop, independently:

1. A timer polls Steam every 15 minutes, comparing the installed build id against the live one.
2. On a change it updates the install and runs a single `produce`: derive → live-validate → typed netvars → roll the model forward.
3. It publishes an immutable `<game>-<buildid>-<patch>` snapshot, then moves `<game>-latest` onto it.

A CS2 update never rebuilds Dota, and vice versa. Two rules keep it honest: every stage **hard-fails rather than substituting** an older or on-disk input, and every entry in `core` / `high_confidence` is confirmed against the live process before it ships. A failed run publishes nothing and leaves the previous release standing.

| you want | use |
|---|---|
| the newest build, always | `…/releases/download/cs2-latest/gamedata-cs2.json` |
| a specific build, pinned | `…/releases/download/cs2-<buildid>-0/gamedata-cs2.json` |
| to know what you got | `manifest.json` — carries `version = <game>-<buildid>-<patch>` |

Follow `-latest` to adopt updates as they land, or pin a buildid tag to adopt them deliberately; old snapshots stay up either way. Whichever you choose, **check the manifest's build id against the server you're actually running** before loading — that is what stops stale offsets meeting a changed binary. (`patch` counts rebuilds on the same binary, e.g. a merged contribution.)

---

## How it works — read → derive → validate → emit

### 1. Read — the whole server, not just `libserver`

A dedicated server maps roughly 22 shared libraries, and `libserver.so` is only a fraction of the reachable engine surface. source2rosetta reads **all of them** (multilib): `libserver`, `libengine2`, `libtier0`, `libnetworksystem`, `libschemasystem`, and the rest — each locator it emits carries the library it belongs to. ELF parsing is done by hand; `.rela.dyn` relocations are resolved so that `.data.rel.ro` pointer slots (zero on disk) come back as their true as-loaded values.

### 2. Derive

**Offsets & netvars — near-deterministic.** Vtable offsets come from **Itanium C++ RTTI**: the type hierarchy and vtable layout are read straight out of the binary, so a method's slot index is a fact, not a guess. Field layouts come from **Valve's own SchemaSystem** reflection tables — the engine emits class/field metadata (name, type, offset) as static data for its own use, and source2rosetta reads it directly. No fingerprints, no guessing.

**Non-virtual signatures — located, then verified.** A stripped, non-virtual function has no slot and no symbol, so it has to be *found*. source2rosetta computes a **recompilation-invariant fingerprint** of each catalogued function — CFG shape, call-graph degree, mnemonic histograms, imported-symbol references. These are deliberately **abstracted statistics, never raw bytes**. Locating a function in a new build is **nearest-history under a plain, unweighted L1 distance** over those features, accepted only within a small fixed recompile threshold. To be exact about what this is *not*: there is **no trained model, no machine learning, no learned or weighted metric, no embedding network** — it's a deterministic nearest-neighbour lookup against recent history, and the per-game "model" it reads is a bundle of derived *facts*, not a network. Every located address is then **re-verified independently of the match**: a fresh byte-signature is regenerated at the predicted address and confirmed to be **unique** in the target library and on a **function prologue** — the same check a loader does. A match that doesn't verify isn't shipped.

**ABI-shape guard.** A byte-signature sees a function's *body* drift and re-derives it, but it can't see the *argument list* change while the prologue stays recognisable — the sig still resolves and points at real code, yet a caller using the old prototype passes the wrong registers. source2rosetta recovers each function's observable **SysV-AMD64 ABI shape** (which argument registers are live-in, plus the return class) via a bounded backward-liveness pass and diffs it across builds, flagging exactly those prototype changes and marking struct-by-value (sret) returns that are unsafe to blind-call.

### 3. Validate — against a live server, not a spec

This is what separates source2rosetta from a static dumper. `produce` and `integration-test` **launch their own** vanilla dedicated server for the game (bots on an empty deathmatch for pawn games; a pawn-less game like Dota waits on a `ready_class` proxy) — no Steam, no separate instance, no human. Then, reading the running process through `/proc/<pid>/mem` (read-only ptrace — no injection, no debugger), it checks against ground truth:

- every **offset** lands on a real vtable slot, and every **signature** on live executable code;
- a gamedata function is **actually called** via ptrace to prove it's the semantically right function, not a plausible byte-match (pawn games);
- derived probes are **fuzzed across changing game state** for many iterations;
- field **types** are read from the live process for the typed netvars — and fields that are non-null live but zero on disk (e.g. `m_pSchemaBinding`) confirm the reader is seeing real live state, not stale disk bytes.

The contract is blunt: **"degrades or stops loudly, never lies."** An entry live validation confidently rejects is dropped, not shipped under a banner claiming it resolves; if the oracle can't run, it fails loudly rather than emit an unverified result.

### 4. Confidence tiers — nothing vanishes silently

The output is a per-game **monolith** in which every catalogue entry is accounted for, sorted into four tiers:

| tier | meaning |
|---|---|
| `core` | derived and, in a full run, **live-validated** — the load-bearing gamedata |
| `high_confidence` | corroborated names folded in as verified offsets/sigs (dictionary-exact or macOS ground-truth transfer) |
| `experimental` | the least-filtered band — every graded name guess, each with a **resolvable locator** but an **unverified name** |
| `unresolved` | catalogued but not confidently produced this build, with a reason (`sig-drifted`, `offset-low-conf`, …) and no locator |

A function that couldn't be derived this build shows up as `unresolved` with a reason — it never just disappears.

### 5. Emit

The canonical model and every format emitter live in `crates/source2rosetta-core`; the deriver writes format-neutral JSON and the tiny `source2rosetta-gen` binary renders it into any framework's shape (see [Artifacts & formats](#artifacts-schemas--output-formats)).

Two things to keep straight. **Multi-game:** the ELF / RTTI / SysV / SchemaSystem machinery is engine-generic; game-specific knobs (library set, launch spec, schema-probe classes, pawn anchor) live on a `profile::GameProfile`, and `--game <cs2|dota2>` selects one — adding a game is a `const` plus a clap arm. **Naming is not derivation:** AI-assisted name extrapolation exists but is producer-side dev tooling, not part of the shipped deriver, and every proposed name is gated on self-naming or ground-truth corroboration and then live-validated. A wrong *name* only mislabels a real slot; it never touches the **offset**, which comes from RTTI.

---

## Install & CLI usage

The two binaries have different audiences. **`source2rosetta`** (the deriver) is only needed to run the pipeline yourself, fork it, or add a game. **`source2rosetta-gen`** is needed by anyone using a release — a release is framework-neutral JSON, so something has to render it into your stack's format — but you can download the prebuilt binary from a `gen-v*` release instead of building it, provided you're on Linux x86-64. Anywhere else, build it from source.

```sh
# The deriver (`source2rosetta`) — the root binary.
cargo build --release                        # → ./target/release/source2rosetta

# The renderer (`source2rosetta-gen`) lives in the core crate and is NOT built by the root
# build — build it explicitly (or use --workspace). See crates/source2rosetta-core/README.md.
cargo build --release -p source2rosetta-core # → ./target/release/source2rosetta-gen
```

`--game <cs2|dota2>` is a global flag (default `cs2`), given before the subcommand: `source2rosetta --game dota2 produce …`.

| command | one line |
|---|---|
| `produce` | The whole per-game build in one command: derive → fold → (with `--game-dir`) validate-live + typed netvars → roll the model forward, into `--out-dir`. **`--game-dir` present = full live-validated build; absent = fast offline build (gamedata + model only). That flag is the entire offline/full switch.** |
| `corpus-model` | Distill a corpus of past builds into one shippable model (vtable-alignment hops, reference fingerprints, slot timelines), so future derivation needs only the model + the target binary, not the corpus. |
| `fold-model` | Roll an existing model forward by ONE build (`model N + build → N+1`), reading only the model and that one binary — equal to a full re-distill. The production update path (also a sidecar inside `produce`). |
| `integration-test` | Stand-alone CI live oracle: launch a vanilla server, populate it, and verify derived gamedata against it — schema oracle, a semantic ptrace CALL on a live pawn, and (with `--gamedata`) a full validate-live plus optional live fuzzing. |
| `backfill` | Give an extrapolated name a real cross-build timeline — resolve its string anchor in every corpus build, or chain a vtable slot through the model — and report history depth + consistency (how a guess graduates to first-class). |
| `classify-change` | `--prev`/`--new` → `skip` / `normal` / `shift` + the exact % of function bodies that changed, comparing with position-dependent bytes masked so a pure layout shift reads as unchanged. Decides whether a build even warrants a re-derive. |
| `filter-corpus` | Collapse runs of code-identical builds to one representative, label each transition `normal`/`shift`, and segment the timeline into toolchain eras. Writes the selection manifest the distill reads. |

### Quickstart

Releases live on the **[releases page](https://git.lo.sh/kamal/source2rosetta/releases)**. For the offline path you need one file from there — the model (`model-<game>.json`) — plus the derive inputs, which ship in this repo under `mappings/`. Put the downloaded model wherever you like; the examples assume it's in the working directory.

```sh
# OFFLINE — derive gamedata + roll the model forward. No server, fully deterministic.
./target/release/source2rosetta --game cs2 produce \
    --seed         mappings/seed-cs2.json \
    --corpus-model model-cs2.json \
    --target       <build-dir> \
    --out-dir      out

# FULL — the same, plus it launches its own vanilla+bots server to validate on the live
# process and read field types for the typed netvars. Adding --game-dir is the only change.
./target/release/source2rosetta --game cs2 produce \
    --seed         mappings/seed-cs2.json \
    --corpus-model model-cs2.json \
    --target       <build-dir> \
    --game-dir     <cs2-install> \
    --out-dir      out
```

- `--target <dir>` (required) — the build **directory** to derive from; its libraries are searched by name, so pass the directory, not a bare `.so`.
- `--seed <bundle>` — one file bundling every derive input (catalogue + optional naming/offset/sig sections). The loose equivalent is `--catalogue <file>` plus the optional `--promotable` / `--candidates` / `--full-names` / `--extra-offsets` / `--extra-sigs`, all defaulting to empty — **so a brand-new game needs only a catalogue to start deriving.**
- Corpus signal — exactly one of `--corpus-model <model.json>` (the normal path: forward-derive from the model + the target binary, and roll the model N→N+1 as a sidecar) or `--corpus <dir>` (fingerprint the raw build binaries on the fly).

---

## Fork it & distill your own model

Nothing is hosted — fork it, `cargo build --release`, and point it at a build on disk. Two ways to run, depending on whether you already have a model:

- **Have a model** (downloaded from releases, or distilled): `--corpus-model model-<game>.json` forward-derives from the model plus *only the target binary* — no corpus on disk. This is the normal path.
- **No model yet:** distill one from a corpus of past builds. "Distilling" is what this project means by "training" — there's no ML (see [Derive](#2-derive) above); the model is the bundle of facts already described: vtable-alignment hops, reference-fingerprint windows, ABI-shape consensus, slot timelines.

```sh
# Distill a corpus into a model (streaming, bounded RAM even over Dota's ~1k builds).
./target/release/source2rosetta --game cs2 corpus-model \
    --seed   mappings/seed-cs2.json \
    --corpus corpus/binaries \
    --out    model-cs2.json
```

`--class-scope` (default `clean` — every real game class, enough for any modding offset to derive model-only) picks which classes get slot hops. **Whatever scope you distill with, `fold-model` and `produce`'s sidecar fold must use the same one.**

### Keeping a model fresh — the incremental fold

Once a model exists you never need the corpus again. `fold-model` rolls it forward one build, reading only the model plus the single new binary — identical to a full re-distill:

```sh
./target/release/source2rosetta --game cs2 fold-model \
    --model model-cs2.json \
    --seed  mappings/seed-cs2.json \
    --build <new-build-dir> \
    --out   model-cs2.next.json
```

`produce --corpus-model` runs exactly this fold as a sidecar, so a full build both derives *and* advances the model in one command. (`--class-scope` must match the model's.)

### Getting the corpus (only to bootstrap a model)

A corpus is a directory of past builds, one subdirectory of `.so` files per build (`corpus/binaries/<label>/*.so`). Fetch it yourself, one time:

1. Use **DepotDownloader** — the self-contained release binary from <https://github.com/SteamRE/DepotDownloader/releases>, **not** `dotnet tool install` (its NuGet package is pinned ancient).
2. Pull manifests from the **Linux binaries depot `2347773`** — *not* the content depot `2347770`. `2347773`'s manifest only advances when the binaries actually change, so its history already *is* the list of real recompiles; content micropatches only bump `2347770`. Read the manifest history off SteamDB, not the Steam client.
3. Download **oldest-first** (chronological = version order), then content-hash-dedup. `filter-corpus` further collapses code-identical builds and segments toolchain eras before you distill, so you never fingerprint the same code twice.

A **partial corpus is fine** — fewer labels is a shallower history, not a broken model; skip very old manifests if they're un-downloadable.

### Adding a game

Add a `profile::GameProfile` const (library set, schema-probe classes, launch spec, pawn anchor, dead-weight knobs) plus one `--game` clap-enum arm, then point `corpus-model` at that game's corpus. The rest is engine-generic.

---

## Artifacts, schemas & output formats

A full `produce` run writes a small, self-contained release set per game into `--out-dir`:

| File | What it is | When |
|------|-----------|------|
| `gamedata-<game>.json` | The **monolith** — the tiered function catalogue (signatures + vtable offsets) with provenance and live-validation folded inline | always |
| `netvars-<game>.json` | The **typed schema** — every SchemaSystem class → field → offset/type | full (`--game-dir`) runs only |
| `model-<game>.json` | The **per-game model** — the distilled facts derivation reads instead of the corpus (the shippable artifact) | when the run folds an existing model (`--corpus-model`) |
| `manifest.json` | Volatile release metadata: `{ version, artifacts: [...] }` | always |

Wall-clock and other volatile metadata live only in `manifest.json`; the monolith and schema carry no timestamp, so they're **byte-reproducible** — the same build in yields the same JSON out.

### `gamedata-<game>.json` — the monolith

```jsonc
{
  "meta": { "game_key", "game", "source_build", "version",
            "counts": { "core", "high_confidence", "experimental", "unresolved" } },
  "core":            { "<fn name>": <MonoEntry>, ... },
  "high_confidence": { "<fn name>": <MonoEntry>, ... },
  "experimental":    { "<fn name>": <MonoEntry>, ... },
  "unresolved":      { "<fn name>": { "reason", "detail" }, ... }
}
```

A **`MonoEntry`** is a locator (flattened to the top level) plus its grading. The locator obeys a strict **signature-XOR-offset** invariant — a virtual method ships as a bare integer `offset` (its RTTI vtable slot index), a non-virtual function as a `signature` object with the `library` it scans and a space-hex `linux` byte pattern with `?` wildcards:

```jsonc
{
  "signature": { "library": "server", "linux": "55 48 89 ? E5" },  // non-virtual: located scan pattern
  "offset": 158,                                                    // virtual: RTTI slot index (a bare int)
  "class": "CCSPlayerPawn",          // experimental offsets only: the vtable class, for an eyeball check
  "provenance": { "tier", "source", "confidence", "self_named", "by_value",
                  "rationale", "corroboration", "abi_drift", ... },  // grading; fields present by tier
  "validated": true                  // true = passed live validation · false = rejected · null = not validated (offline)
}
```

`validated: false` entries stay in the file for transparency but are **dropped by every renderer**.

> The `source2rosetta-gen` renderers write a slightly different *on-disk* shape (plural `{"signatures":{…}}` / `{"offsets":{…}}` for CounterStrikeSharp-family output). The keys above are the **canonical model JSON** as `gamedata-<game>.json` stores it.

### `netvars-<game>.json` — the typed schema

```jsonc
{
  "meta":    { "game_key", "source_build", "typed", "untyped" },
  "classes": { "<class>": { "<field>": { "offset", "type", "kind", "size", "name_hash" } } }
}
```

Offsets come from SchemaSystem tables (available offline); `type` / `kind` / `size` are read from the live process during a full run (`kind` ∈ `ref` | `ptr` | `fixed_array`). A full run refuses to ship a schema whose fields resolved wholesale-untyped rather than emit a typeless file — the same "stop loudly" contract.

### `model-<game>.json` — the per-game model

The distilled per-game **facts** — vtable-alignment hops, reference-fingerprint windows, ABI-shape consensus, slot timelines — the artifact derivation reads *instead of* the corpus. What "distill a model" produces (above).

### Rendering — the `gen` binary

The monolith and schema are format-neutral; **`source2rosetta-gen`** renders them, so the deriver never changes when a new consumer format is added.

**Gamedata** (`--from gamedata-<game>.json`, at a cumulative `--tier core | high_confidence | experimental`, default `high_confidence`): `cssharp` (CounterStrikeSharp), `metamod` (Metamod/SourceMod VDF), `modsharp`, `swiftly`, `plugify`, `model` (the canonical model re-serialized). **Schema** (`--netvars netvars-<game>.json`): `cs-sdk` (a typed C# SDK — one `static class` per schema class, `const int` field offsets tagged with their type), `netvars` (a flat `{ class: { field: offset } }` map). Full render walkthrough: **[gen binary README](crates/source2rosetta-core/README.md)**.

---

## Copyright

The build **corpus** — Valve's `.so` files (~86 GB) — is never shipped and never baked into a release. The published artifacts (model, gamedata, netvars) are *designed to* contain **derived facts** — vtable offsets, abstracted fingerprint statistics, and byte scan-patterns — rather than copies of the original code. That a statistic *about* code is a fact and not a copy is **the project's position, not settled law** — reverse-engineering Valve binaries under AGPL is exactly the territory a court hasn't ruled on, so use accordingly. Leaked or proprietary game source was deliberately kept out of the corpus and the naming pipeline; see [ATTRIBUTIONS.md](ATTRIBUTIONS.md).

## License

[AGPL-3.0](LICENSE). Built on a decade of community work — see **[ATTRIBUTIONS.md](ATTRIBUTIONS.md)** first.