kamal/source2rosetta
1
0
Fork 0
Code Issues Pull requests Projects Releases 55 Packages Wiki Activity Actions

initial commit
All checks were successful
CI / fuzz (push) Successful in 1m41s
Details
CI / lint (push) Successful in 16s
Details
CI / test (push) Successful in 22s
Details

Browse source
This commit is contained in:
Kamal Tufekcic 2026-07-27 10:12:04 +03:00
commit a2922b8bad
59 changed files with 2684583 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

245
README.md Normal file
View file

@ -0,0 +1,245 @@
# source2rosetta
**Re-derives Source-2 engine gamedata from stripped Valve binaries — and proves it on a live server.**
When Valve ships a CS2 or Dota 2 engine update, every Metamod / CounterStrikeSharp plugin breaks until someone hand-reverse-engineers fresh gamedata — function signatures, vtable offsets, netvar layouts. Historically that can be up to **weeks**. source2rosetta reads the stripped `.so` libraries a dedicated server maps and re-derives the whole surface in **minutes**, then launches its own vanilla server and *calls the functions* to prove they're right.
It's a **standalone Rust tool, not a plugin or a framework** — the gamedata it emits renders into whatever your stack already speaks (CounterStrikeSharp, Metamod/SourceMod, ModSharp, Swiftly, Plugify, or a typed C# SDK). CS2 and Dota 2 are both live-validated; a new game is a `--game` arm away.
## Docs
- **[ATTRIBUTIONS.md](ATTRIBUTIONS.md) — start here.** This tool stands on a decade of community reverse-engineering, catalogues, dumpers, and research. The credits come first because the work does.
- **🎯 Just want the gamedata?** → **[Grab a release and render it for your framework.](crates/source2rosetta-core/README.md)** Download the published CS2 / Dota 2 gamedata, point `source2rosetta-gen` at it, and get CounterStrikeSharp / Metamod / ModSharp / Swiftly / Plugify / a typed C# SDK in one command. No build, no corpus — the 30-second path, and what most people are here for.
- [CONTRIBUTING.md](CONTRIBUTING.md) — add or back-fill a gamedata entry.
- [LICENSE](LICENSE) — AGPL-3.0.
## Results
Ballpark from a recent build, on a 16-core desktop. These move build-to-build — treat them as orders of magnitude, not guarantees.
| | derived functions | typed schema | model | one-time distill |
|---|---|---|---|---|
| **CS2** | ~1,150 `core` + ~1,200 `high_confidence`, all live-validated, plus ~4,400 `experimental` name guesses | ~1,900 classes / ~12,300 fields | ~48 MB (a few MB gzipped) | ~15 min |
| **Dota 2** | ~1,900 `core` + ~1,000 `high_confidence`, plus ~6,100 `experimental` | ~2,960 classes / ~17,700 fields | ~570 MB | ~1 hr |
Both derive **0-dropped** — every offset and signature that ships passed live validation. Distilling the model is a one-time cost; after that, a per-build update (fold the new build in, re-derive) is **a couple of minutes** — the "minutes, not weeks" the headline is about.
---
## How it works — read → derive → validate → emit
### 1. Read — the whole server, not just `libserver`
A dedicated server maps roughly 22 shared libraries, and `libserver.so` is only a fraction of the reachable engine surface. source2rosetta reads **all of them** (multilib): `libserver`, `libengine2`, `libtier0`, `libnetworksystem`, `libschemasystem`, and the rest — each locator it emits carries the library it belongs to. ELF parsing is done by hand; `.rela.dyn` relocations are resolved so that `.data.rel.ro` pointer slots (zero on disk) come back as their true as-loaded values.
### 2. Derive
**Offsets & netvars — near-deterministic.** Vtable offsets come from **Itanium C++ RTTI**: the type hierarchy and vtable layout are read straight out of the binary, so a method's slot index is a fact, not a guess. Field layouts come from **Valve's own SchemaSystem** reflection tables — the engine emits class/field metadata (name, type, offset) as static data for its own use, and source2rosetta reads it directly. No fingerprints, no guessing.
**Non-virtual signatures — located, then verified.** A stripped, non-virtual function has no slot and no symbol, so it has to be *found*. source2rosetta computes a **recompilation-invariant fingerprint** of each catalogued function — CFG shape, call-graph degree, mnemonic histograms, imported-symbol references. These are deliberately **abstracted statistics, never raw bytes**. Locating a function in a new build is **nearest-history under a plain, unweighted L1 distance** over those features, accepted only within a small fixed recompile threshold. To be exact about what this is *not*: there is **no trained model, no machine learning, no learned or weighted metric, no embedding network** — it's a deterministic nearest-neighbour lookup against recent history, and the per-game "model" it reads is a bundle of derived *facts*, not a network. Every located address is then **re-verified independently of the match**: a fresh byte-signature is regenerated at the predicted address and confirmed to be **unique** in the target library and on a **function prologue** — the same check a loader does. A match that doesn't verify isn't shipped.
**ABI-shape guard.** A byte-signature sees a function's *body* drift and re-derives it, but it can't see the *argument list* change while the prologue stays recognisable — the sig still resolves and points at real code, yet a caller using the old prototype passes the wrong registers. source2rosetta recovers each function's observable **SysV-AMD64 ABI shape** (which argument registers are live-in, plus the return class) via a bounded backward-liveness pass and diffs it across builds, flagging exactly those prototype changes and marking struct-by-value (sret) returns that are unsafe to blind-call.
### 3. Validate — against a live server, not a spec
This is what separates source2rosetta from a static dumper. `produce` and `integration-test` **launch their own** vanilla dedicated server for the game (bots on an empty deathmatch for pawn games; a pawn-less game like Dota waits on a `ready_class` proxy) — no Steam, no separate instance, no human. Then, reading the running process through `/proc/<pid>/mem` (read-only ptrace — no injection, no debugger), it checks against ground truth:
- every **offset** lands on a real vtable slot, and every **signature** on live executable code;
- a gamedata function is **actually called** via ptrace to prove it's the semantically right function, not a plausible byte-match (pawn games);
- derived probes are **fuzzed across changing game state** for many iterations;
- field **types** are read from the live process for the typed netvars — and fields that are non-null live but zero on disk (e.g. `m_pSchemaBinding`) confirm the reader is seeing real live state, not stale disk bytes.
The contract is blunt: **"degrades or stops loudly, never lies."** An entry live validation confidently rejects is dropped, not shipped under a banner claiming it resolves; if the oracle can't run, it fails loudly rather than emit an unverified result.
### 4. Confidence tiers — nothing vanishes silently
The output is a per-game **monolith** in which every catalogue entry is accounted for, sorted into four tiers:
| tier | meaning |
|---|---|
| `core` | derived and, in a full run, **live-validated** — the load-bearing gamedata |
| `high_confidence` | corroborated names folded in as verified offsets/sigs (dictionary-exact or macOS ground-truth transfer) |
| `experimental` | the least-filtered band — every graded name guess, each with a **resolvable locator** but an **unverified name** |
| `unresolved` | catalogued but not confidently produced this build, with a reason (`sig-drifted`, `offset-low-conf`, …) and no locator |
A function that couldn't be derived this build shows up as `unresolved` with a reason — it never just disappears.
### 5. Emit
The canonical model and every format emitter live in `crates/source2rosetta-core`; the deriver writes format-neutral JSON and the tiny `source2rosetta-gen` binary renders it into any framework's shape (see [Artifacts & formats](#artifacts-schemas--output-formats)).
Two things to keep straight. **Multi-game:** the ELF / RTTI / SysV / SchemaSystem machinery is engine-generic; game-specific knobs (library set, launch spec, schema-probe classes, pawn anchor) live on a `profile::GameProfile`, and `--game <cs2|dota2>` selects one — adding a game is a `const` plus a clap arm. **Naming is not derivation:** AI-assisted name extrapolation exists but is producer-side dev tooling, not part of the shipped deriver, and every proposed name is gated on self-naming or ground-truth corroboration and then live-validated. A wrong *name* only mislabels a real slot; it never touches the **offset**, which comes from RTTI.
---
## Install & CLI usage
```sh
# The deriver (`source2rosetta`) — the root binary.
cargo build --release # → ./target/release/source2rosetta
# The renderer (`source2rosetta-gen`) lives in the core crate and is NOT built by the root
# build — build it explicitly (or use --workspace). See crates/source2rosetta-core/README.md.
cargo build --release -p source2rosetta-core # → ./target/release/source2rosetta-gen
```
`--game <cs2|dota2>` is a global flag (default `cs2`), given before the subcommand: `source2rosetta --game dota2 produce …`.
| command | one line |
|---|---|
| `produce` | The whole per-game build in one command: derive → fold → (with `--game-dir`) validate-live + typed netvars → roll the model forward, into `--out-dir`. **`--game-dir` present = full live-validated build; absent = fast offline build (gamedata + model only). That flag is the entire offline/full switch.** |
| `corpus-model` | Distill a corpus of past builds into one shippable model (vtable-alignment hops, reference fingerprints, slot timelines), so future derivation needs only the model + the target binary, not the corpus. |
| `fold-model` | Roll an existing model forward by ONE build (`model N + build → N+1`), reading only the model and that one binary — equal to a full re-distill. The production update path (also a sidecar inside `produce`). |
| `integration-test` | Stand-alone CI live oracle: launch a vanilla server, populate it, and verify derived gamedata against it — schema oracle, a semantic ptrace CALL on a live pawn, and (with `--gamedata`) a full validate-live plus optional live fuzzing. |
| `backfill` | Give an extrapolated name a real cross-build timeline — resolve its string anchor in every corpus build, or chain a vtable slot through the model — and report history depth + consistency (how a guess graduates to first-class). |
| `classify-change` | `--prev`/`--new` → `skip` / `normal` / `shift` + the exact % of function bodies that changed, comparing with position-dependent bytes masked so a pure layout shift reads as unchanged. Decides whether a build even warrants a re-derive. |
| `filter-corpus` | Collapse runs of code-identical builds to one representative, label each transition `normal`/`shift`, and segment the timeline into toolchain eras. Writes the selection manifest the distill reads. |
### Quickstart
Releases live on the **[releases page](https://git.lo.sh/kamal/source2rosetta/releases)**. For the offline path you need one file from there — the model (`model-<game>.json`) — plus the derive inputs, which ship in this repo under `mappings/`. Put the downloaded model wherever you like; the examples assume it's in the working directory.
```sh
# OFFLINE — derive gamedata + roll the model forward. No server, fully deterministic.
./target/release/source2rosetta --game cs2 produce \
--seed mappings/seed-cs2.json \
--corpus-model model-cs2.json \
--target <build-dir> \
--out-dir out
# FULL — the same, plus it launches its own vanilla+bots server to validate on the live
# process and read field types for the typed netvars. Adding --game-dir is the only change.
./target/release/source2rosetta --game cs2 produce \
--seed mappings/seed-cs2.json \
--corpus-model model-cs2.json \
--target <build-dir> \
--game-dir <cs2-install> \
--out-dir out
```
- `--target <dir>` (required) — the build **directory** to derive from; its libraries are searched by name, so pass the directory, not a bare `.so`.
- `--seed <bundle>` — one file bundling every derive input (catalogue + optional naming/offset/sig sections). The loose equivalent is `--catalogue <file>` plus the optional `--promotable` / `--candidates` / `--full-names` / `--extra-offsets` / `--extra-sigs`, all defaulting to empty — **so a brand-new game needs only a catalogue to start deriving.**
- Corpus signal — exactly one of `--corpus-model <model.json>` (the normal path: forward-derive from the model + the target binary, and roll the model N→N+1 as a sidecar) or `--corpus <dir>` (fingerprint the raw build binaries on the fly).
---
## Fork it & distill your own model
Nothing is hosted — fork it, `cargo build --release`, and point it at a build on disk. Two ways to run, depending on whether you already have a model:
- **Have a model** (downloaded from releases, or distilled): `--corpus-model model-<game>.json` forward-derives from the model plus *only the target binary* — no corpus on disk. This is the normal path.
- **No model yet:** distill one from a corpus of past builds. "Distilling" is what this project means by "training" — there's no ML (see [Derive](#2-derive) above); the model is the bundle of facts already described: vtable-alignment hops, reference-fingerprint windows, ABI-shape consensus, slot timelines.
```sh
# Distill a corpus into a model (streaming, bounded RAM even over Dota's ~1k builds).
./target/release/source2rosetta --game cs2 corpus-model \
--seed mappings/seed-cs2.json \
--corpus corpus/binaries \
--out model-cs2.json
```
`--class-scope` (default `clean` — every real game class, enough for any modding offset to derive model-only) picks which classes get slot hops. **Whatever scope you distill with, `fold-model` and `produce`'s sidecar fold must use the same one.**
### Keeping a model fresh — the incremental fold
Once a model exists you never need the corpus again. `fold-model` rolls it forward one build, reading only the model plus the single new binary — identical to a full re-distill:
```sh
./target/release/source2rosetta --game cs2 fold-model \
--model model-cs2.json \
--seed mappings/seed-cs2.json \
--build <new-build-dir> \
--out model-cs2.next.json
```
`produce --corpus-model` runs exactly this fold as a sidecar, so a full build both derives *and* advances the model in one command. (`--class-scope` must match the model's.)
### Getting the corpus (only to bootstrap a model)
A corpus is a directory of past builds, one subdirectory of `.so` files per build (`corpus/binaries/<label>/*.so`). Fetch it yourself, one time:
1. Use **DepotDownloader** — the self-contained release binary from <https://github.com/SteamRE/DepotDownloader/releases>, **not** `dotnet tool install` (its NuGet package is pinned ancient).
2. Pull manifests from the **Linux binaries depot `2347773`** — *not* the content depot `2347770`. `2347773`'s manifest only advances when the binaries actually change, so its history already *is* the list of real recompiles; content micropatches only bump `2347770`. Read the manifest history off SteamDB, not the Steam client.
3. Download **oldest-first** (chronological = version order), then content-hash-dedup. `filter-corpus` further collapses code-identical builds and segments toolchain eras before you distill, so you never fingerprint the same code twice.
A **partial corpus is fine** — fewer labels is a shallower history, not a broken model; skip very old manifests if they're un-downloadable.
### Adding a game
Add a `profile::GameProfile` const (library set, schema-probe classes, launch spec, pawn anchor, dead-weight knobs) plus one `--game` clap-enum arm, then point `corpus-model` at that game's corpus. The rest is engine-generic.
---
## Artifacts, schemas & output formats
A full `produce` run writes a small, self-contained release set per game into `--out-dir`:
| File | What it is | When |
|------|-----------|------|
| `gamedata-<game>.json` | The **monolith** — the tiered function catalogue (signatures + vtable offsets) with provenance and live-validation folded inline | always |
| `netvars-<game>.json` | The **typed schema** — every SchemaSystem class → field → offset/type | full (`--game-dir`) runs only |
| `model-<game>.json` | The **per-game model** — the distilled facts derivation reads instead of the corpus (the shippable artifact) | when the run folds an existing model (`--corpus-model`) |
| `manifest.json` | Volatile release metadata: `{ version, artifacts: [...] }` | always |
Wall-clock and other volatile metadata live only in `manifest.json`; the monolith and schema carry no timestamp, so they're **byte-reproducible** — the same build in yields the same JSON out.
### `gamedata-<game>.json` — the monolith
```jsonc
{
"meta": { "game_key", "game", "source_build", "version",
"counts": { "core", "high_confidence", "experimental", "unresolved" } },
"core": { "<fn name>": <MonoEntry>, ... },
"high_confidence": { "<fn name>": <MonoEntry>, ... },
"experimental": { "<fn name>": <MonoEntry>, ... },
"unresolved": { "<fn name>": { "reason", "detail" }, ... }
}
```
A **`MonoEntry`** is a locator (flattened to the top level) plus its grading. The locator obeys a strict **signature-XOR-offset** invariant — a virtual method ships as a bare integer `offset` (its RTTI vtable slot index), a non-virtual function as a `signature` object with the `library` it scans and a space-hex `linux` byte pattern with `?` wildcards:
```jsonc
{
"signature": { "library": "server", "linux": "55 48 89 ? E5" }, // non-virtual: located scan pattern
"offset": 158, // virtual: RTTI slot index (a bare int)
"class": "CCSPlayerPawn", // experimental offsets only: the vtable class, for an eyeball check
"provenance": { "tier", "source", "confidence", "self_named", "by_value",
"rationale", "corroboration", "abi_drift", ... }, // grading; fields present by tier
"validated": true // true = passed live validation · false = rejected · null = not validated (offline)
}
```
`validated: false` entries stay in the file for transparency but are **dropped by every renderer**.
> The `source2rosetta-gen` renderers write a slightly different *on-disk* shape (plural `{"signatures":{…}}` / `{"offsets":{…}}` for CounterStrikeSharp-family output). The keys above are the **canonical model JSON** as `gamedata-<game>.json` stores it.
### `netvars-<game>.json` — the typed schema
```jsonc
{
"meta": { "game_key", "source_build", "typed", "untyped" },
"classes": { "<class>": { "<field>": { "offset", "type", "kind", "size", "name_hash" } } }
}
```
Offsets come from SchemaSystem tables (available offline); `type` / `kind` / `size` are read from the live process during a full run (`kind` ∈ `ref` | `ptr` | `fixed_array`). A full run refuses to ship a schema whose fields resolved wholesale-untyped rather than emit a typeless file — the same "stop loudly" contract.
### `model-<game>.json` — the per-game model
The distilled per-game **facts** — vtable-alignment hops, reference-fingerprint windows, ABI-shape consensus, slot timelines — the artifact derivation reads *instead of* the corpus. What "distill a model" produces (above).
### Rendering — the `gen` binary
The monolith and schema are format-neutral; **`source2rosetta-gen`** renders them, so the deriver never changes when a new consumer format is added.
**Gamedata** (`--from gamedata-<game>.json`, at a cumulative `--tier core | high_confidence | experimental`, default `high_confidence`): `cssharp` (CounterStrikeSharp), `metamod` (Metamod/SourceMod VDF), `modsharp`, `swiftly`, `plugify`, `model` (the canonical model re-serialized). **Schema** (`--netvars netvars-<game>.json`): `cs-sdk` (a typed C# SDK — one `static class` per schema class, `const int` field offsets tagged with their type), `netvars` (a flat `{ class: { field: offset } }` map). Full render walkthrough: **[gen binary README](crates/source2rosetta-core/README.md)**.
---
## Copyright
The build **corpus** — Valve's `.so` files (~86 GB) — is never shipped and never baked into a release. The published artifacts (model, gamedata, netvars) are *designed to* contain **derived facts** — vtable offsets, abstracted fingerprint statistics, and byte scan-patterns — rather than copies of the original code. That a statistic *about* code is a fact and not a copy is **the project's position, not settled law** — reverse-engineering Valve binaries under AGPL is exactly the territory a court hasn't ruled on, so use accordingly. Leaked or proprietary game source was deliberately kept out of the corpus and the naming pipeline; see [ATTRIBUTIONS.md](ATTRIBUTIONS.md).
## License
[AGPL-3.0](LICENSE). Built on a decade of community work — see **[ATTRIBUTIONS.md](ATTRIBUTIONS.md)** first.