ship one record per function: merge the release set, gen reads it, descriptions as doc comments, gates for what was only claimed; v3.0
This commit is contained in:
Kamal Tufekcic
parent
71ce34edd2
commit
3410a79b6a
28 changed files with 30596 additions and 955 deletions
|
|
@ -1,13 +1,14 @@
|
|||
# source2rosetta-gen
|
||||
|
||||
Render a published [source2rosetta](../../README.md) gamedata release into whatever format your framework
|
||||
reads. `source2rosetta` does the hard part — deriving CS2 / Dota 2 gamedata from the stripped engine and
|
||||
validating it on a live server — and publishes a small set of JSON files per game. `source2rosetta-gen` turns those into
|
||||
CounterStrikeSharp, Metamod/SourceMod, ModSharp, Swiftly, Plugify, or a typed C# SDK, locally, in a second.
|
||||
Render a published [source2rosetta](../../README.md) release into whatever format your framework reads.
|
||||
`source2rosetta` does the hard part — deriving CS2 / Dota 2 gamedata from the stripped engine and
|
||||
validating it on a live server — and publishes **one file per game**, `rosetta-<game>.json`.
|
||||
`source2rosetta-gen` turns that into CounterStrikeSharp, Metamod/SourceMod, ModSharp, Swiftly, Plugify, a
|
||||
typed C# SDK, or the Dota script API, locally, in a second.
|
||||
|
||||
It's deliberately tiny: it links only `source2rosetta-core` (serde + the format emitters) — **no** ELF reader,
|
||||
no disassembler, no ptrace. So a consumer who "just wants the files" downloads one release + this small binary
|
||||
and generates exactly what they need, instead of every format being pre-baked into the release.
|
||||
no disassembler, no ptrace. So a consumer who "just wants the files" downloads one release + this small
|
||||
binary and generates exactly what they need, instead of every format being pre-baked into the release.
|
||||
|
||||
## Get it
|
||||
|
||||
|
|
@ -23,109 +24,140 @@ does **not** build it — use `-p source2rosetta-core` or `--workspace`.)
|
|||
|
||||
## Use it
|
||||
|
||||
Three of the published artifacts are `gen` inputs, one per `--` flag:
|
||||
|
||||
- `gamedata-<game>.json` (`--from`) — the derived gamedata (function signatures + vtable offsets), tiered by confidence.
|
||||
- `netvars-<game>.json` (`--netvars`) — the typed schema (field offsets + runtime types, plus the class base
|
||||
graph and per-type sizes).
|
||||
- `abi-<game>.json` (`--abi`) — declared parameter and return types, each re-judged against the footprint
|
||||
measured in that build. This is what a function TAKES, as opposed to where it is. See
|
||||
[Call shapes](#call-shapes----abi-abi-gamejson).
|
||||
|
||||
`bindings-<game>.json` ships beside them and `gen` does **not** render it — it is not locator data. It is
|
||||
what the binary declares about itself, in five sections: Pulse bindings (display name, description, call
|
||||
policy, and each binding's typed signature), entity-IO inputs and outputs, map-classname → C++ class, and
|
||||
console commands. Plain JSON, readable as-is.
|
||||
|
||||
Then point `gen` at whichever you need and pick a `--format`. Output goes to `--out`, or stdout if omitted.
|
||||
One input, one flag. `--format` says **who the output is for**; `--out` is a **directory**, because most
|
||||
formats write more than one file.
|
||||
|
||||
```sh
|
||||
# CounterStrikeSharp combined gamedata (the default)
|
||||
source2rosetta-gen --from gamedata-cs2.json --format cssharp --out gamedata.json
|
||||
R=https://git.lo.sh/kamal/source2rosetta/releases/download/cs2-latest
|
||||
curl -fsSLO $R/rosetta-cs2.json
|
||||
|
||||
# Metamod / SourceMod gamedata VDF (one .games.txt)
|
||||
source2rosetta-gen --from gamedata-cs2.json --format metamod --out csgo.games.txt
|
||||
# CounterStrikeSharp: the combined gamedata + typed call sites for the same functions
|
||||
source2rosetta-gen --from rosetta-cs2.json --format cssharp --out ./csharp
|
||||
|
||||
# Swiftly / ModSharp / Plugify gamedata
|
||||
source2rosetta-gen --from gamedata-cs2.json --format swiftly --out gamedata.json
|
||||
# Metamod:Source / SourceMod: the gamedata VDF + a C++ prototype header
|
||||
source2rosetta-gen --from rosetta-cs2.json --format metamod --out ./mm
|
||||
|
||||
# Typed C# SDK from the schema — one `static class` per engine class, `const` field offsets + types
|
||||
source2rosetta-gen --netvars netvars-cs2.json --format cs-sdk --out Schema.cs
|
||||
# A typed C# SDK from the schema — one `static class` per engine class, `const` offsets + types
|
||||
source2rosetta-gen --from rosetta-cs2.json --format cs-sdk --out ./sdk
|
||||
|
||||
# Flat netvar offset map (class -> field -> offset)
|
||||
source2rosetta-gen --netvars netvars-cs2.json --format netvars --out netvars.json
|
||||
# The Dota script API: ModDota's dota-data shape AND the TypeScript declarations
|
||||
source2rosetta-gen --from rosetta-dota2.json --format moddota --out ./dota
|
||||
```
|
||||
|
||||
Each run prints what it wrote.
|
||||
|
||||
## Formats
|
||||
|
||||
| `--format` | needs | output |
|
||||
**A framework gets two files, and it needs both.** The gamedata says *where* a function is; the call sites say
|
||||
*how to call it*. They were separate inputs when the release was four files; one artifact makes them one
|
||||
command.
|
||||
|
||||
| `--format` | writes | notes |
|
||||
|---|---|---|
|
||||
| `cssharp` *(default)* | `--from` | CounterStrikeSharp combined gamedata — **JSONC**: banner comments mean a strict JSON parser will reject it |
|
||||
| `metamod` | `--from` | Metamod:Source / SourceMod gamedata VDF (`.games.txt`) |
|
||||
| `modsharp` | `--from` | ModSharp gamedata JSON |
|
||||
| `swiftly` | `--from` | Swiftly gamedata JSON — **signature entries only**; vtable-offset entries are omitted, because that framework takes offsets through a separate file |
|
||||
| `plugify` | `--from` | Plugify gamedata JSON |
|
||||
| `model` | `--from` | the selected tiers flattened to one name → locator map (format-neutral; not a re-readable monolith) |
|
||||
| `cs-sdk` | `--netvars` | typed C# SDK — `static class` per schema class, `const int` field offsets tagged with their type |
|
||||
| `netvars` | `--netvars` | flat schema map, `{ class: { field: offset } }` |
|
||||
| `cssharp` *(default)* | `gamedata.json` + `RosettaFunctions.cs` | the gamedata is **JSONC** — banner comments, so a strict JSON parser will reject it. The `.cs` is `MemoryFunction*` fields / `VirtualFunction*` factories |
|
||||
| `metamod` | `<game_key>.games.txt` + `rosetta_prototypes.h` | the VDF also covers SourceMod. Metamod plugins are C++, so the prototypes are a header of `using X_t = RET (*)(…)` plus an `X_vtidx` constant per slot |
|
||||
| `modsharp` | `gamedata.json` + `RosettaCalls.cs` | `[AddressKey]` interface for its Roslyn generator, plus a vtable-dispatch class |
|
||||
| `swiftly` | `gamedata.json` + `prototypes.json` | **signature entries only** in the gamedata; that framework takes offsets through a separate file |
|
||||
| `plugify` | `gamedata.json` + `prototypes.json` | runtime type arrays (`{"paramTypes":["pointer","string"],"retType":"void"}`) |
|
||||
| `cs-sdk` | `Schema.cs` | typed C# SDK: `static class` per schema class, `const int` field offsets tagged with their type, plus the engine's own enums at their real width |
|
||||
| `netvars` | `netvars.json` | flat schema map, `{ class: { field: offset } }` |
|
||||
| `moddota` | `api.json` + `api.d.ts` | the VScript API in ModDota `dota-data`'s shape (their toolchain renders from it), plus TypeScript declarations for authors using the published packages as-is — one `interface` per class, Valve's own description as the doc comment |
|
||||
| `flat` | `gamedata-flat.json` | the selected tiers as one name → locator map, format-neutral |
|
||||
|
||||
### Call shapes — `--abi abi-<game>.json`
|
||||
**Every published artifact renders every format on this list.** The releases are always derived against a
|
||||
running server, so nothing here is conditional on how the artifact was made. (If you derive your own, that
|
||||
changes — see [below](#if-you-derived-the-artifact-yourself).)
|
||||
|
||||
The same framework ids, a different input: `--abi` renders **how to call** a function rather than where it
|
||||
is. The input picks the family, so `--abi … --format cssharp` emits typed call sites while
|
||||
`--from … --format cssharp` emits the gamedata those calls resolve through.
|
||||
### Which games a format covers
|
||||
|
||||
All five framework ids work here, exactly as they do for `--from`:
|
||||
**The artifact states its own game** (`meta.game_key`, `csgo` or `dota`) and the output follows it — there is
|
||||
no `--game` flag, because a second place to state one fact is a second place for it to be wrong.
|
||||
|
||||
```sh
|
||||
source2rosetta-gen --abi abi-cs2.json --format cssharp --out RosettaFunctions.cs
|
||||
source2rosetta-gen --abi abi-cs2.json --format metamod --out rosetta_prototypes.h
|
||||
source2rosetta-gen --abi abi-cs2.json --format modsharp --out RosettaCalls.cs
|
||||
source2rosetta-gen --abi abi-cs2.json --format swiftly --out prototypes.json
|
||||
source2rosetta-gen --abi abi-cs2.json --format plugify --out prototypes.json
|
||||
```
|
||||
Two formats are **game-keyed**, and for both the key is the game DIRECTORY the server runs out of, which is
|
||||
what `game_key` already holds:
|
||||
|
||||
| `--format` | output |
|
||||
|---|---|
|
||||
| `cssharp` | C# `MemoryFunction*` fields (signature) / `VirtualFunction*` factories (vtable slot) |
|
||||
| `metamod` | C++ header of `using X_t = RET (*)(…)`, plus an `X_vtidx` constant for a slot — Metamod plugins are C++ and take the **declared** types verbatim |
|
||||
| `modsharp` | C# `[AddressKey]` interface for its Roslyn generator (signature) + a vtable-dispatch class (slot) |
|
||||
| `swiftly` | JSON per-function type descriptors (`{"args":"ppf","ret":"v","call":"address"}`) |
|
||||
| `plugify` | JSON runtime type arrays (`{"paramTypes":["pointer","string","float"],"retType":"void"}`) |
|
||||
- `metamod` writes `Games { <game_key> { … } }`. The consuming plugin looks that section up by the engine's
|
||||
own `GetGameDir()` — see [cs2kz-metamod's reader][kz] — so `dota` is what a Dota 2 plugin will look for.
|
||||
Metamod takes Dota 2 as a first-class SDK target ([`dota.json`][mm], `define: DOTA`, `source2: true`).
|
||||
- `plugify` writes `{ "<game_key>": { … } }`, matched against the `S2SDK_GAME_NAME` its s2sdk plugin was
|
||||
BUILT with (default `csgo`).
|
||||
|
||||
Source for the two C# targets and for C++ because their type lists are **compile-time**; data for Swiftly
|
||||
and Plugify because theirs are resolved at runtime.
|
||||
The rest are game-neutral in shape: `modsharp` and `cssharp` carry no game key at all (flat, keyed only by
|
||||
platform), and `flat` / `cs-sdk` / `netvars` / `moddota` are plain data.
|
||||
|
||||
**The two locator forms are not interchangeable, and every output distinguishes them.** A signature
|
||||
resolves to one address; a vtable slot is entered through the object, so the framework reaches it by a
|
||||
different call entirely — `VirtualFunctionVoid(instance, slot)` rather than `GameData.GetSignature(key)`,
|
||||
`GetVFuncIndex` rather than `GetAddress`, `(*(void***)self)[idx]` rather than a scanned pointer. Roughly
|
||||
a quarter of a LIVE-derived manifest's call sites are vtable-located, so binding them all through the
|
||||
signature path would look up keys that live in the gamedata's `offsets` section and never in its
|
||||
`signatures` one.
|
||||
**Two consumers cannot run on Dota 2 at all**, and `gen` declines rather than write a file that can never
|
||||
load: CounterStrikeSharp resolves its binaries out of `<dir>/csgo/bin/`, and Swiftly initialises against the
|
||||
`csgo` game directory. `--force` renders anyway. It is a warning rather than a rule on purpose — that is a
|
||||
claim about somebody else's project, read out of their source at one point in time, and projects add games.
|
||||
|
||||
An **offline**-derived manifest emits none through the vtable path at all: a slot is recorded only once
|
||||
live validation has confirmed it is really a vtable slot and not a carried member offset, so an offline
|
||||
run states no slot rather than guess one. Same artifact shape, fewer vtable call sites — worth knowing
|
||||
before diffing two manifests produced different ways.
|
||||
**ModSharp is CS2-first but not excluded.** Its own paths are hardcoded (`../../csgo/steam.inf`), yet its
|
||||
gamedata carries no game key whatsoever — flat `Addresses` / `VFuncs`, platform-keyed — so the file rendered
|
||||
here is the same one whatever game the build targets.
|
||||
|
||||
**The receiver is always in the type list.** Where the declaration came from an Itanium-mangled symbol
|
||||
`this` is invisible, so it is prepended, spelled from the function's own class (`CBaseEntity*`, not
|
||||
`void*`) and marked `[this]` in the C++ header. It is a real register in the call frame — leaving it out
|
||||
shifts every argument by one.
|
||||
[kz]: https://github.com/KZGlobalTeam/cs2kz-metamod/blob/dev/src/utils/gameconfig.cpp
|
||||
[mm]: https://github.com/alliedmodders/hl2sdk-manifests/blob/master/manifests/dota.json
|
||||
|
||||
Only functions the deriver could stand behind are emitted: `status: verified` **or `lower-bound`** (the
|
||||
declaration passes registers the callee never reads and contradicts it in none — safe to call, and
|
||||
marked as such in every output), a receiver settled by evidence (the declaration names it, a
|
||||
live-validated vtable slot proves it, or the measurement independently agrees), and every parameter
|
||||
mappable onto an ABI class. A function whose return **nobody
|
||||
declared** is still emitted — otherwise Dota would lose 2,304 of its 3,732 call sites — but it is marked as such in every
|
||||
output (prose in the generated source, `ret_declared` / `retDeclared` in the data), and the value is
|
||||
documented as the raw return register rather than a typed result.
|
||||
### Descriptions
|
||||
|
||||
**Every call site is emitted with a sentence saying what the function is FOR**, wherever that target's
|
||||
readers hover: a C# XML `<summary>`, so IntelliSense shows it; a comment above the C++ typedef; a
|
||||
`description` field in the data formats. The prototype keeps a home of its own — a `<remarks>` in C#, the
|
||||
identity line in the header — so nothing is lost to make room. That is every emittable call site: 2,073 on
|
||||
CS2, 2,326 on Dota.
|
||||
|
||||
**Each one says whose sentence it is, and that is not decoration.** Some are Valve's own, read out of a
|
||||
registry in the binary (717 CS2 / 774 Dota); the rest are this project's reading of the build, and the two
|
||||
carry very different weight. So every generated source prints the origin beside the text, the data formats
|
||||
carry it as an id (`valve` / `derived` / `generated`), and `moddota` — which deliberately mirrors a shape the
|
||||
Dota ecosystem already publishes — keeps ours under keys of our own name rather than in the `description`
|
||||
field their toolchain renders as Valve's word.
|
||||
|
||||
**Valve's text always wins.** Where the binary documents a function, that is what ships; a generated
|
||||
description only ever fills a gap, so the two can never disagree in an output. Between Valve's own two
|
||||
registries the script one wins: a console registration's help text documents the COMMAND an operator types,
|
||||
while a script binding documents the function. In the `.d.ts`, a member Valve documents reads exactly as it
|
||||
did before — bare, the way the published types do — and only a gap Valve left is filled and marked.
|
||||
|
||||
`cs-sdk` and `netvars` render the schema, which is classes and field offsets. There are no functions in them
|
||||
to describe, so they carry none.
|
||||
|
||||
### What the call sites will and won't emit
|
||||
|
||||
Only functions the deriver could stand behind: `status: verified` **or `lower-bound`** (the declaration passes
|
||||
registers the callee never reads and contradicts it in none — safe to call, and marked as such in every
|
||||
output), a receiver settled by evidence (the declaration names it, a live-validated vtable slot proves it, or
|
||||
the measurement independently agrees), and every parameter mappable onto an ABI class. A function whose return
|
||||
**nobody declared** is still emitted — otherwise Dota would lose 2,304 of its 3,732 call sites — but it is
|
||||
marked as such in every output (`ret_declared` / `retDeclared` in the data formats, prose in the generated
|
||||
source), and the value is documented as the raw return register rather than a typed result.
|
||||
|
||||
**The two locator forms are not interchangeable, and every output distinguishes them.** A signature resolves to
|
||||
one address; a vtable slot is entered through the object, so the framework reaches it by a different call
|
||||
entirely — `VirtualFunctionVoid(instance, slot)` rather than `GameData.GetSignature(key)`, `GetVFuncIndex`
|
||||
rather than `GetAddress`, `(*(void***)self)[idx]` rather than a scanned pointer. Roughly a quarter of the
|
||||
call sites are vtable-located.
|
||||
|
||||
**A few entries carry BOTH**, which is worth knowing if you consume the gamedata rather than these call
|
||||
sites: `model::Entry` allows it and five CS2 `core` entries use it. Both locators are live-validated
|
||||
independently, and `validated: true` means both passed — an entry whose signature checked out but whose slot
|
||||
could not be reached ships `null`, never `true`. The call-site emitters here pick one form per function, so
|
||||
this only affects what you read out of the artifact directly.
|
||||
|
||||
**The receiver is always in the type list.** Where the declaration came from an Itanium-mangled symbol `this`
|
||||
is invisible, so it is prepended, spelled from the function's own class (`CBaseEntity*`, not `void*`) and
|
||||
marked `[this]` in the C++ header. It is a real register in the call frame — leaving it out shifts every
|
||||
argument by one.
|
||||
|
||||
**Two things `moddota` states honestly rather than guesses.** Parameters are declared `...args: any[]`,
|
||||
because the registry does not carry them — types appear only inside Valve's prose descriptions, inconsistently,
|
||||
in about a fifth of entries. It is visibly ugly on purpose: nobody should mistake these for complete
|
||||
declarations, and inventing plausible arity would emit declarations that lie rather than abstain. And
|
||||
`available` is always `server`, because a dedicated server never maps `libclient`, so this derivation cannot
|
||||
see the client side at all.
|
||||
|
||||
## Confidence tier
|
||||
|
||||
The gamedata formats (the `--from` ones) take a `--tier`, cumulative and defaulting to `high_confidence`:
|
||||
The locator half takes a `--tier`, cumulative and defaulting to `high_confidence`:
|
||||
|
||||
| `--tier` | includes |
|
||||
|---|---|
|
||||
|
|
@ -135,10 +167,40 @@ The gamedata formats (the `--from` ones) take a `--tier`, cumulative and default
|
|||
|
||||
```sh
|
||||
# only the rock-solid set:
|
||||
source2rosetta-gen --from gamedata-cs2.json --format cssharp --tier core --out gamedata.json
|
||||
source2rosetta-gen --from rosetta-cs2.json --format cssharp --tier core --out ./csharp
|
||||
```
|
||||
|
||||
The schema formats (`cs-sdk`, `netvars`) ignore `--tier`.
|
||||
The schema and script-API formats ignore `--tier`.
|
||||
|
||||
## What is in the artifact that `gen` does not render
|
||||
|
||||
`rosetta-<game>.json` is plain JSON and readable as-is, and it carries more than these formats consume: the
|
||||
Pulse binding registry (typed signatures, call policy, and a callable shim per binding), entity outputs,
|
||||
map classname → C++ class, ConVars with decoded flags, and the declared rows that belong to no function here
|
||||
(`surfaces.unjoined`). Those have no loader to render them into — read them directly. The per-function
|
||||
descriptions are in there too, and they DO reach the outputs, but only for the ~2,000 functions that become
|
||||
call sites; the artifact describes about twice as many. See the [artifacts section](../../README.md#artifacts-schemas--output-formats).
|
||||
|
||||
## If you derived the artifact yourself
|
||||
|
||||
Skip this if you downloaded the artifact — it is about `source2rosetta produce`, not about `gen`.
|
||||
|
||||
The releases are derived against a **running server**, which is where two whole surfaces come from. Run
|
||||
`produce` without `--game-dir` and it derives what it can offline, honestly, and states the rest as absent —
|
||||
so `gen` gets an artifact that is real but smaller, and three things follow:
|
||||
|
||||
- **`cs-sdk`, `netvars` and `moddota` have nothing to render**, and say so rather than writing an empty file.
|
||||
The schema's field TYPES are runtime-resolved, and a VScript binding's owning class is reached through a
|
||||
register loaded from memory, so neither is readable from the file alone. All three group by one or the
|
||||
other.
|
||||
- **No call site is reached through a vtable.** A slot is only recorded once live validation has confirmed it
|
||||
is really a vtable slot and not a carried member offset, so an offline run states no slot rather than guess
|
||||
one. Roughly a quarter of a live manifest's call sites are vtable-located; an offline one has none. Same
|
||||
shape, fewer entries — worth knowing before diffing two outputs made different ways.
|
||||
- **Nothing has been checked against a running server.** `validated` is `null` on every record rather than a
|
||||
verdict, and the locator half drops only what live validation confidently REJECTED — so an offline artifact
|
||||
keeps entries a live one would have thrown out. The tiers still mean what they mean; they just have not
|
||||
been tested.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue