# Luma — A Lua-Inspired Shader Language in Pure Odin > Cross-compiles to GLSL, HLSL, MSL, WGSL, and SPIR-V binary. > Ships as both a standalone CLI and an embeddable Odin library. Working name: **Luma** (Lua + lumen). Swap freely. --- ## 1. Language Design ### 1.1 Core Philosophy Lua's syntax is clean because it avoids ceremony. Shaders benefit from that — most shader code is short, math-heavy, and shouldn't fight the language. But shaders also need static typing, explicit binding layouts, and zero ambiguity about data flow. Luma keeps Lua's *feel* while being fully statically typed and shader-constrained. Key constraints inherited from the shader domain: - No recursion (call graphs must be statically resolvable) - No heap allocation (everything lives in registers or bound resources) - No closures, no first-class functions (functions are inlined or statically dispatched) - No dynamic dispatch, no interfaces, no polymorphism at runtime - All types known at compile time — type inference is a convenience, not a runtime feature ### 1.2 Syntax Overview ```lua -- Vertex shader entry point @entry(vertex) function transform(input: VertexInput) -> VertexOutput let pos = input.position let world_pos = uniforms.model * vec4(pos, 1.0) let clip_pos = uniforms.view_proj * world_pos return VertexOutput { position = clip_pos, uv = input.uv, normal = normalize((uniforms.normal_mat * vec4(input.normal, 0.0)).xyz), } end -- Fragment shader entry point @entry(fragment) function shade(input: FragmentInput) -> FragmentOutput let albedo = sample(material_tex, input.uv) let n = normalize(input.normal) let light = max(dot(n, light_dir), 0.0) return FragmentOutput { color = vec4(albedo.rgb * light, albedo.a), } end ``` ### 1.3 Syntax Rules **From Lua:** - `function ... end` blocks (no braces) - `let` for variable declarations - `if ... then ... elseif ... else ... end` - `for ... do ... end` (counted loops only — no iterators, supports `for i = start, stop do` and `for i = start, stop, step do`) - `while ... do ... end` - `--` line comments, `--[[ ]]` block comments - No semicolons - Struct construction via `TypeName { field = value }` **Departures from Lua:** - Explicit type annotations: `let x: float = 1.0` (inferred when unambiguous) - Return type arrows: `function foo(a: float) -> float` - No tables — replaced by structs, arrays, and buffer types - No metatables, no `self`, no `:` method syntax - No varargs, no multiple returns (single return value, use structs) - No `nil` — all values must be initialized - `!=` for not-equal (replacing Lua's `~=`) - `@attribute()` decorators for shader semantics (not Lua-native but visually clean) ### 1.3.1 Syntax Sugar Four sugars reduce boilerplate while desugaring to existing semantics. All are optional — the explicit form always works. **Auto-location numbering:** Struct fields used in entry point I/O get sequential `@location` values automatically. `@builtin` fields are skipped. Explicit `@location(N)` overrides and resets the counter to N+1. ```lua -- Sugar (locations inferred as 0, 1, 2): struct VertexInput position: vec3 normal: vec3 uv: vec2 end -- Equivalent explicit form: struct VertexInput @location(0) position: vec3 @location(1) normal: vec3 @location(2) uv: vec2 end ``` **Binding defaults:** `@group` defaults to 0. `@binding` auto-increments within its group. Only annotate to override. ```lua -- Sugar (group 0, bindings 0 and 1): uniform uniforms: Uniforms uniform material_tex: sampler2D -- Explicit override for a different group: @group(1) buffer lights: [16]Light ``` **`@varying` structs:** A single struct serves as both vertex output and fragment input. `@builtin` fields are automatically stripped when used as fragment input. ```lua @varying struct Interpolated @builtin(position) position: vec4 uv: vec2 normal: vec3 end @entry(vertex) function transform(input: VertexInput) -> Interpolated -- position, uv, normal all available as outputs end @entry(fragment) function shade(input: Interpolated) -> FragmentOutput -- input.uv and input.normal available; input.position is stripped end ``` **Inline entry points:** Simple shaders can declare I/O fields directly in the function signature instead of defining separate structs. ```lua -- Sugar: @entry(fragment) function shade(uv: vec2, normal: vec3) -> (color: vec4) let albedo = sample(material_tex, uv) return { color = vec4(albedo.rgb, 1.0) } end -- Desugars to anonymous input/output structs with auto-locations. -- Params become struct fields. Tuple return (name: Type, ...) becomes output struct. -- Anonymous struct literal { field = value } infers type from return context. ``` **Future idea — struct destructuring in entry points:** `function shade({uv, normal}: FragmentInput) -> FragmentOutput` would desugar to `let uv = input.uv; let normal = input.normal`. Natural for Lua users. Not planned for v0.1. **Future idea — implicit single-field return:** For single-field output structs, allow `return vec4(...)` instead of `return { color = vec4(...) }`. Type inferred from return context. Not planned for v0.1. ### 1.4 Type System **Scalar types:** `bool`, `int`, `uint`, `float`, `half` (maps to float16 where supported) **Vector types:** `vec2`, `vec3`, `vec4`, `ivec2`, `ivec3`, `ivec4`, `uvec2`, `uvec3`, `uvec4`, `bvec2`, `bvec3`, `bvec4` **Matrix types:** `mat2`, `mat3`, `mat4`, `mat2x3`, `mat3x4`, etc. **Sampler types:** `sampler2D`, `sampler3D`, `samplerCube`, `sampler2DArray`, `sampler2DShadow` **Separate texture/sampler types:** `texture2D`, `texture3D`, `textureCube`, `texture2DArray`, `sampler` Combined sampler types (e.g. `sampler2D`) are syntax sugar. During IR lowering, `sampler2D foo` expands to a `texture2D foo_tex` + `sampler foo_samp` pair. This matches WGSL, modern Vulkan, and SPIR-V best practice, which all expect separate texture and sampler objects. Users can also declare them separately for explicit control: ```lua @group(0) @binding(1) uniform tex: texture2D @group(0) @binding(2) uniform samp: sampler ``` **Composite types:** ```lua struct VertexInput @location(0) position: vec3 @location(1) normal: vec3 @location(2) uv: vec2 end struct Uniforms model: mat4 view_proj: mat4 normal_mat: mat4 end ``` **Buffer/resource bindings:** ```lua @group(0) @binding(0) uniform uniforms: Uniforms @group(0) @binding(1) uniform material_tex: sampler2D @group(1) @binding(0) buffer storage positions: []vec3 -- runtime-sized array (SSBO) ``` **Tuple return types (inline entry points only):** ```lua -- Parenthesized named fields as return type: function shade(uv: vec2) -> (color: vec4) -- Desugars to an anonymous output struct with field "color: vec4" ``` **Type inference:** Local variables infer from RHS. Function signatures always explicit. Struct fields always explicit. Numeric literals default to `float` unless context demands `int`/`uint`. Vector constructors infer component type from arguments. **Swizzling:** Standard — `v.xyz`, `v.rg`, `v.stpq`. Writable swizzles on the left-hand side. ### 1.5 Attribute System Attributes drive all shader-specific semantics. They're the bridge between clean syntax and messy GPU realities. | Attribute | Applies to | Purpose | |---|---|---| | `@entry(stage)` | function | Marks entry point. `stage`: vertex, fragment, compute, geometry, tessellation_control, tessellation_eval | | `@location(n)` | struct field | Vertex attribute or inter-stage location | | `@builtin(name)` | struct field | Built-in variable (position, vertex_id, frag_coord, etc.) | | `@group(n)` | binding decl | Descriptor set / bind group | | `@binding(n)` | binding decl | Binding index within group | | `@workgroup_size(x,y,z)` | function | Compute shader workgroup dimensions | | `@interpolation(mode)` | struct field | flat, smooth, noperspective | | `@push_constant` | binding decl | Push constant block (Vulkan) / root constant (DX12) | | `@early_depth` | function | Force early depth test | | `@spec(id)` | const decl | Specialization constant with pipeline-time ID | | `@varying` | struct | Stage-crossing I/O struct; builtins stripped for fragment input | ### 1.6 Built-in Functions Mirror the common subset across all targets. Organized by category: - **Math:** `abs`, `sign`, `floor`, `ceil`, `round`, `fract`, `mod`, `min`, `max`, `clamp`, `mix`, `step`, `smoothstep`, `sqrt`, `inversesqrt`, `pow`, `exp`, `exp2`, `log`, `log2`, `sin`, `cos`, `tan`, `asin`, `acos`, `atan`, `atan2` - **Vector:** `length`, `distance`, `dot`, `cross`, `normalize`, `reflect`, `refract`, `faceforward` - **Matrix:** `transpose`, `inverse`, `determinant` - **Texture:** `sample`, `sample_level`, `sample_grad`, `sample_compare`, `texel_fetch`, `texture_size` - **Atomic** (compute): `atomic_add`, `atomic_min`, `atomic_max`, `atomic_and`, `atomic_or`, `atomic_xor`, `atomic_exchange`, `atomic_compare_exchange` - **Compute:** `workgroup_barrier`, `storage_barrier` - **Derivative** (fragment): `dfdx`, `dfdy`, `fwidth` - **Pack/unpack:** `pack_unorm4x8`, `unpack_unorm4x8`, etc. (where supported) Builtins that don't exist on a target emit an error at compile time, not a silent fallback. ### 1.7 Control Flow Constraints - `for` loops must have statically determinable bounds OR use `@unroll` / `@unroll(N)` attribute. The parameterized form specifies max iterations and can be combined with dynamic bounds (e.g. `@unroll(16) for i = 0, numLights do`) — this tells the backend the max iteration count even when bounds aren't statically known, mapping to driver hints that dramatically improve codegen. Maps to: HLSL `[unroll(N)]`, GLSL `#pragma optionNV(unroll all)`, SPIR-V `LoopControl Unroll` with optional iteration count. Supports two-value `for i = 0, 10 do` (step defaults to 1) and three-value `for i = 0, 10, 2 do` (explicit step) - `while` loops allowed but flagged with a warning unless `@max_iterations(n)` is present - `if/elseif/else` fully supported (maps to branch/select depending on backend opt level) - No `goto`, no `break` to label, no `continue` (keep it simple — rethink with early-return patterns) - `return` anywhere in function body (backends handle control flow merge) ### 1.8 Include System Luma uses `#include` for multi-file shader composition. This is a pre-parse textual inclusion (like C), not a module/import system. Shaders are short — a full module system adds complexity that doesn't pay for itself. ```lua #include "common/types.luma" #include "common/lighting.luma" @entry(fragment) function shade(input: FragmentInput) -> FragmentOutput let result = compute_pbr_lighting(input) return FragmentOutput { color = result } end ``` **Semantics:** - `#include "path"` — relative to the including file's directory - `#include ` — relative to a configurable search path list (set via CLI `--include-dir` or library API) - Processed before lexing — the lexer sees a single flattened source string - Include guards via `#pragma once` (tracked by resolved absolute path) - Circular includes are a compile error - Max include depth of 32 (configurable) to catch runaway recursion **Implementation (`luma/preprocessor.odin`):** - Single pass before lexing: scan for lines starting with `#include` or `#pragma` - Resolve paths, read files, recursively expand - Maintain a source map that tracks which original file and line each position in the flattened source came from — this feeds into `Source_Span` so diagnostics report the correct file - `#line N "filename"` — overrides the source location for subsequent lines. Emitted into GLSL/HLSL output for driver error reporting. Since we already track a source map, emitting `#line` into text backend output is trivial and dramatically improves debugging for included files - No other preprocessor directives — no `#define`, no `#ifdef`. Compile-time constants (`const`) and target-conditional compilation (`@target(glsl)` blocks, if ever needed) are handled in the language proper, not a text preprocessor **CLI integration:** ``` luma compile shader.luma --target=spirv --include-dir=./common --include-dir=./shared ``` **Library API addition:** ```odin Compile_Options :: struct { // ... existing fields ... include_dirs: []string, // search paths for #include <...> file_reader: File_Reader, // callback for custom file resolution (optional) } // Default file reader uses core:os. Consumers can override for virtual filesystems, // in-memory shader libraries, or asset pipelines. File_Reader :: #type proc(path: string) -> (content: string, ok: bool) ``` --- ## 2. Project Structure ``` luma/ ├── build.sh # One-line: odin build . -out:luma ├── main.odin # CLI entry point — imports luma package ├── luma/ # Library package (the real meat) │ ├── shader.odin # Public API surface │ ├── preprocessor.odin # #include expansion, source map │ ├── lexer.odin # Tokenizer │ ├── token.odin # Token types and spans │ ├── parser.odin # Recursive descent → AST │ ├── ast.odin # AST node definitions │ ├── sema.odin # Type checking, name resolution, validation │ ├── symbols.odin # Symbol table, scopes │ ├── types.odin # Type representations and operations │ ├── ir.odin # IR node definitions │ ├── ir_builder.odin # AST → IR lowering │ ├── ir_opt.odin # Optimization passes on IR │ ├── emit_glsl.odin # IR → GLSL 450 source │ ├── emit_hlsl.odin # IR → HLSL SM6.0 source │ ├── emit_msl.odin # IR → MSL 2.4 source │ ├── emit_wgsl.odin # IR → WGSL source │ ├── emit_spirv.odin # IR → SPIR-V binary (pure Odin) │ ├── spirv_spec.odin # SPIR-V opcode/type constants (generated) │ ├── diagnostics.odin # Error/warning reporting with source spans │ └── common.odin # Shared utilities, string interning ├── vendor/ # Vendored C sources (SPIRV-Tools, optional) │ └── spirv-tools/ ├── tests/ │ ├── preprocessor_test.odin │ ├── lexer_test.odin │ ├── parser_test.odin │ ├── sema_test.odin │ ├── ir_test.odin │ ├── emit_test.odin │ └── shaders/ # Integration test shaders │ ├── basic_vertex.luma │ ├── basic_fragment.luma │ ├── compute_prefix_sum.luma │ └── pbr_fragment.luma └── tools/ └── gen_spirv_spec.odin # Generates spirv_spec.odin from SPIR-V JSON grammar ``` ### 2.1 Library API (`luma/shader.odin`) The library is the primary artifact. The CLI is a thin wrapper. ```odin package shader Target :: enum { GLSL_450, HLSL_SM6, MSL_2_4, WGSL, SPIR_V, } Opt_Level :: enum { None, // No optimization, faithful IR emission Basic, // Constant fold, dead code elim, copy propagation Aggressive, // + inlining, common subexpr elim, loop unroll } Compile_Options :: struct { target: Target, opt_level: Opt_Level, stage: Shader_Stage, // vertex, fragment, compute, etc. entry: string, // entry point name (default: auto-detect via @entry) debug: bool, // emit debug names/source mapping validate: bool, // run target-specific validation post-emit } Compile_Result :: struct { success: bool, output: []u8, // source bytes or SPIR-V binary diagnostics: []Diagnostic, } Diagnostic :: struct { level: Diagnostic_Level, message: string, span: Source_Span, } Diagnostic_Level :: enum { Error, Warning, Note } Source_Span :: struct { file: string, line_start: int, col_start: int, line_end: int, col_end: int, } // Primary entry point — source in, compiled output out. compile :: proc(source: string, options: Compile_Options) -> Compile_Result { ... } // Incremental / advanced usage — expose pipeline stages. parse :: proc(source: string) -> (AST, []Diagnostic) { ... } check :: proc(ast: AST) -> (Typed_AST, []Diagnostic) { ... } lower :: proc(tast: Typed_AST, opt: Opt_Level) -> IR_Module { ... } emit :: proc(ir: IR_Module, target: Target, opts: Compile_Options) -> ([]u8, []Diagnostic) { ... } ``` This pipeline-stage API matters. It lets consumers inspect or transform the IR before emission, which is critical for engine integration (custom intrinsics, patching bindings, reflection data extraction). ### 2.2 CLI (`main.odin`) ``` luma compile input.luma --target=spirv --stage=vertex --opt=basic -o output.spv luma compile input.luma --target=glsl -o output.glsl luma check input.luma # parse + typecheck only, report errors luma dump-ast input.luma # debug: print AST luma dump-ir input.luma --opt=basic # debug: print optimized IR luma reflect input.luma # emit binding reflection JSON luma compile input.luma --target=spirv --hash # output + SHA256 of canonical IR ``` Minimal `main.odin`: ```odin package main import shader "path/to/gpu/shader" import "core:os" import "core:fmt" import "core:flags" main :: proc() { // Parse CLI args, call shader.compile(), write output, report diagnostics. // ~80 lines of glue. } ``` --- ## 3. Frontend ### 3.1 Lexer (`lexer.odin`) Hand-written single-pass scanner. No regex, no tables — just a `switch` on the current byte with lookahead. Odin's `string` type (which is a `[]u8` view) makes this clean. Token categories: - **Keywords:** `function`, `end`, `let`, `return`, `if`, `then`, `elseif`, `else`, `for`, `while`, `do`, `in`, `and`, `or`, `not`, `true`, `false`, `struct`, `uniform`, `buffer`, `const` - **Symbols:** `(`, `)`, `[`, `]`, `{`, `}`, `.`, `,`, `:`, `->`, `=`, `==`, `!=`, `<`, `>`, `<=`, `>=`, `+`, `-`, `*`, `/`, `%`, `..`, `@`, `#` - **Literals:** integer, float, string (limited — shaders don't need strings, but useful for pragmas) - **Identifiers:** standard `[a-zA-Z_][a-zA-Z0-9_]*` - **Comments:** `--` to EOL, `--[[ ... ]]` Every token carries a `Source_Span`. The lexer is allocation-free — tokens reference slices of the original source string. Intern keywords into an enum; identifiers stay as string slices. Design decision: **the lexer skips nothing implicitly.** Newlines are tracked (for line counting) but not emitted as tokens. The parser handles statement boundaries via the grammar (Lua-style: statements end at `end`, `return`, or the next statement keyword — no semicolons, no newline-sensitivity). ### 3.2 AST (`ast.odin`) ```odin package shader AST_Node :: union { Ast_Module, Ast_Function, Ast_Struct, Ast_Binding, // uniform/buffer declarations Ast_Block, Ast_Let, // let x: T = expr Ast_Assign, Ast_Return, Ast_If, Ast_For, Ast_While, Ast_Call, Ast_Field_Access, Ast_Index, Ast_Swizzle, Ast_Unary, Ast_Binary, Ast_Literal, Ast_Ident, Ast_Struct_Literal, Ast_Attribute, } Ast_Function :: struct { name: string, params: []Ast_Param, return_type: ^Type_Expr, body: []^AST_Node, attributes: []Ast_Attribute, span: Source_Span, } Ast_Param :: struct { name: string, type: Type_Expr, attributes: []Ast_Attribute, // for inline entry points: @location override on params span: Source_Span, } Type_Expr :: union { Type_Named, // vec3, mat4, MyStruct Type_Array, // [N]T or []T (runtime-sized) Type_Tuple, // (name: Type, ...) — inline entry point return types } ``` The AST is a tree of heap-allocated nodes. Use Odin's `new()` with an arena allocator — the entire AST lives in one arena that's freed after compilation. No individual frees, no refcounting. ### 3.3 Parser (`parser.odin`) Recursive descent. Lua's grammar is LL(1) with minor exceptions (function call vs. assignment — resolved with one token of lookahead). Luma's grammar is even simpler since we drop tables, varargs, and multiple returns. Key parsing functions: ``` parse_module → (parse_struct | parse_function | parse_binding)* parse_function → attributes 'function' NAME '(' params ')' '->' return_type block 'end' parse_struct → attributes? 'struct' NAME fields 'end' parse_return_type → type | '(' NAME ':' type (',' NAME ':' type)* ')' parse_binding → attributes ('uniform' | 'buffer') NAME ':' type parse_block → statement* parse_statement → parse_let | parse_return | parse_if | parse_for | parse_while | parse_assign_or_call parse_let → 'let' NAME (':' type)? '=' expr parse_expr → parse_or parse_or → parse_and ('or' parse_and)* parse_and → parse_comparison ('and' parse_comparison)* parse_comparison → parse_addition (('==' | '!=' | '<' | '>' | '<=' | '>=') parse_addition)* parse_addition → parse_multiplication (('+' | '-') parse_multiplication)* parse_multiplication → parse_unary (('*' | '/' | '%') parse_unary)* parse_unary → ('-' | 'not') parse_unary | parse_postfix parse_postfix → parse_primary ('.' NAME | '[' expr ']' | '(' args ')')* parse_primary → NUMBER | IDENT | 'true' | 'false' | '(' expr ')' | struct_literal | anon_struct_literal ``` Additional parsing for syntax sugar: - `parse_return_type`: after `->`, if next token is `(`, parse tuple return `(name: Type, ...)` as `Type_Tuple` - `parse_params`: parse optional `@` attributes before each parameter for `@location` overrides on inline entry points - `parse_struct`: accepts preceding attributes (for `@varying`); currently attributes are parsed but discarded for structs - Anonymous struct literal: `{ field = value }` without type name prefix — parsed when `{` appears in expression position; type inferred from context during sema ```text (grammar summary continued) ``` Precedence is encoded in the call structure — no Pratt parser needed for this complexity level. Swizzle detection happens in semantic analysis, not parsing (`.xyz` parses as a field access; sema resolves whether it's a swizzle based on the LHS type). **Desugar pass (`desugar.odin`):** Runs on `Ast_Module` after parsing, before sema. For `@entry` functions with inline I/O (multiple primitive-typed params or tuple return type): synthesizes anonymous input/output structs, rewrites function signature, and rewrites body references from bare param names to `input.field` access. Error recovery: on parse error, skip tokens until we find a synchronization point (`function`, `end`, `struct`, `let`, `return`). Collect all errors — don't bail on the first one. --- ## 4. Semantic Analysis ### 4.1 Symbol Table (`symbols.odin`) Lexical scoping with a scope stack: ```odin Scope :: struct { parent: ^Scope, symbols: map[string]^Symbol, } Symbol :: struct { name: string, type: ^Resolved_Type, kind: Symbol_Kind, span: Source_Span, mutable: bool, } Symbol_Kind :: enum { Variable, Parameter, Function, Struct_Type, Builtin_Function, Binding, // uniform, buffer, sampler } ``` Module scope contains: struct types, functions, bindings, and all builtin functions (pre-populated). Function scope contains parameters. Block scope contains locals. ### 4.2 Type Checking (`sema.odin`) Two passes: **Pass 1 — Declaration collection:** Walk top-level declarations, register all struct types and function signatures. This allows forward references (function A calls function B defined later). **Pass 2 — Full check:** Walk function bodies with type inference: - **Arithmetic:** Vector/scalar promotion rules matching GLSL: `float * vec3 → vec3`, `mat4 * vec4 → vec4`. Disallow implicit int↔float conversion (explicit `float(x)` required). - **Swizzle resolution:** When sema encounters `expr.xyz` and `expr` is a vector type, resolve as swizzle. Validate component count and write-mask rules (no duplicate components in write swizzles). - **Function calls:** Resolve against declared functions and builtins. Builtins use overload sets (e.g., `dot(vec2,vec2)->float`, `dot(vec3,vec3)->float`, `dot(vec4,vec4)->float`). - **Struct literals:** Match field names and types. All fields must be initialized (no defaults). - **Shader validation:** - Entry points must have exactly one parameter (the input struct) and one return type (the output struct). - No recursion — build a call graph during checking and verify it's a DAG. - `@builtin` fields must use valid names for the declared stage. - Compute entry points must have `@workgroup_size`. - `sample()` without explicit LOD is an error in non-fragment stages (vertex, compute). Use `sample_level()` for explicit LOD in vertex shaders. This also applies to derivative functions (`dfdx`, `dfdy`, `fwidth`) which are fragment-only. Output: a `Typed_AST` — same shape as the AST but every expression node has a resolved `^Resolved_Type` attached. ### 4.3 Resolved Type Representation (`types.odin`) ```odin Resolved_Type :: union { Type_Scalar, Type_Vector, Type_Matrix, Type_Struct, Type_Array, Type_Sampler, Type_Void, } Type_Scalar :: struct { kind: Scalar_Kind, // Bool, Int, Uint, Float, Half } Type_Vector :: struct { elem: Scalar_Kind, size: int, // 2, 3, or 4 } Type_Matrix :: struct { elem: Scalar_Kind, cols: int, rows: int, } Type_Struct :: struct { name: string, fields: []Struct_Field, } Struct_Field :: struct { name: string, type: ^Resolved_Type, attributes: []Resolved_Attribute, } ``` --- ## 5. Intermediate Representation ### 5.1 Design The IR is a flat, SSA-form representation. Every value has a unique ID. No mutable variables — `let x = 1; x = 2` becomes two distinct SSA values with a phi at merge points (or, since shader control flow is simple, just let later assignments shadow earlier ones in the value numbering). ```odin IR_Module :: struct { functions: []IR_Function, structs: []IR_Struct, bindings: []IR_Binding, entry_points: []IR_Entry_Point, } IR_Function :: struct { name: string, params: []IR_Param, return_type: ^Resolved_Type, blocks: []IR_Block, is_entry: bool, stage: Shader_Stage, } IR_Block :: struct { label: int, instrs: []IR_Instr, } IR_Value :: distinct int // SSA value ID IR_Instr :: struct { result: IR_Value, // SSA value produced (0 if none) op: IR_Op, type: ^Resolved_Type, args: []IR_Arg, } IR_Op :: enum { // Constants Const_Int, Const_Float, Const_Bool, // Arithmetic Add, Sub, Mul, Div, Mod, Neg, // Comparison Eq, Neq, Lt, Gt, Lte, Gte, // Logical And, Or, Not, // Vector/matrix construction Vec_Construct, Mat_Construct, // Composite (value semantics — no memory access) Composite_Extract, // extract field/component from SSA value (maps to OpCompositeExtract) Composite_Insert, // produce new composite with one field/component replaced (maps to OpCompositeInsert) Vector_Shuffle, // swizzle / reorder vector components (maps to OpVectorShuffle) // Memory / bindings Load, // load from binding Store, // store to output Access_Chain, // struct field or array index (memory/binding access only) // Control flow Branch, Branch_Cond, Phi, Return, // Function Call, Call_Builtin, // Texture Sample, Sample_Level, Sample_Grad, Sample_Compare, Texel_Fetch, Texture_Size, // Compute Atomic_Op, Barrier, // Derivative Dfdx, Dfdy, Fwidth, } IR_Arg :: union { IR_Value, int, // immediate integer (swizzle mask, array index) f64, // immediate float string, // for builtin names, field names } // Address spaces — tracked on bindings and variables. // Required for WGSL (var), MSL (threadgroup), SPIR-V (StorageClass). // Planned addition — will be implemented as a separate phase after Phase 3. IR_Address_Space :: enum { Private, // function locals (default) Uniform, // constant buffers (read-only) Storage, // SSBOs (read-write) Workgroup, // shared compute memory Input, // stage inputs Output, // stage outputs } // Internal pointer type — not exposed in the language. // Required for SPIR-V emission (OpTypePointer, OpAccessChain returns pointer, OpLoad takes pointer). Type_Pointer :: struct { address_space: IR_Address_Space, pointee: ^Resolved_Type, } // Specialization constants — pipeline-time configuration values. // Syntax: @spec(0) const NUM_LIGHTS: int = 4 IR_Spec_Constant :: struct { name: string, spec_id: int, type: ^Resolved_Type, default_value: IR_Arg, // default if not specialized } ``` **Sampler splitting in IR:** Combined `sampler2D` bindings expand during lowering into two `IR_Binding` entries: one with kind `Texture` and one with kind `Sampler`, using consecutive binding slots. This makes the split canonical — backends emit directly from the IR without ad-hoc splitting logic. The GLSL backend recombines them when targeting GLSL 450 (which uses combined samplers). **Value vs memory semantics:** `Composite_Extract` and `Composite_Insert` handle field/component access on SSA values (no memory involved). `Access_Chain` + `Load`/`Store` are reserved for binding and memory access. `Access_Chain` produces a `Type_Pointer` internally; `Load` consumes a pointer and produces a value; `Store` consumes a pointer and a value. This distinction is critical for SPIR-V emission (`OpCompositeExtract` vs `OpAccessChain` + `OpLoad`) and produces cleaner code in text backends. Text backends can ignore pointer types — they only matter for SPIR-V. ### 5.2 Lowering (AST → IR) (`ir_builder.odin`) Walk the typed AST, emit IR instructions into the current block. Key translations: - `let x = expr` → evaluate expr, bind SSA value to name `x` in the value table - `x = expr` → new SSA value, update value table (or emit store if `x` is a binding output) - `if ... then ... else ... end` → conditional branch to then-block / else-block, merge block with phis - `for i = 0, 10 do ... end` → loop header (phi for i), body block, back-edge with increment (step defaults to 1; `for i = 0, 10, 2` uses step 2) - `return expr` → evaluate, emit `Return` - `a.field` → `Composite_Extract` when `a` is an SSA value (value semantics); `Access_Chain` + `Load` only for binding/memory access - `v.xyz` → `Vector_Shuffle(v, mask=[0,1,2])` - `sample(tex, uv)` → `Sample(texture, sampler, uv)` with separate texture + sampler operands - `uniform foo: sampler2D` → expands to `IR_Binding{kind=.Texture, name="foo_tex"}` + `IR_Binding{kind=.Sampler, name="foo_samp"}` with consecutive binding slots - `func(args)` → inline if small / `Call` with value args Struct flattening: entry point input/output structs are flattened to individual location-annotated values at the IR level. This is critical — backends need individual `@location` values, not struct pointers. **Auto-location assignment** (`build_entry_io`): When flattening I/O structs, maintain a `loc_counter` starting at 0. For each field: `@builtin` fields skip the counter; explicit `@location(N)` uses N and sets counter to N+1; unannotated fields get `loc_counter` and increment. This makes `@location` optional on I/O struct fields. **`@varying` struct handling** (`build_entry_io`): When building fragment inputs from a struct with `@varying`, skip fields that have `@builtin` annotations (e.g. `@builtin(position)` is vertex output only, not a fragment input). **Binding defaults** (`build_bindings`): Maintain a `map[int]int` of next-binding-per-group. Missing `@group` defaults to 0. Missing `@binding` uses the group's next auto-binding value. Explicit `@binding(N)` sets the counter to N+1. Uses `get_attribute_value(attrs, name) -> (int, bool)` to distinguish "absent" from "explicitly 0". **Edge case — mixed manual/auto bindings:** Explicit `@binding(N)` must update the per-group counter to `N+1` even when N jumps ahead. Example: `auto(0), auto(1), @binding(10), auto(11)`. The counter always tracks the highest-seen-plus-one within each group. This exact rule has caused bugs in multiple shader compilers — test it explicitly. **Group scoping:** `@group` always defaults to 0 when omitted, regardless of previous bindings. There is no implicit carry-over from the last explicit `@group`. Example: `@group(0) uniform a`, `@group(1) uniform b`, `uniform c` → c is in group 0, not group 1. This prevents surprising implicit state. ### 5.3 Optimization (`ir_opt.odin`) Each pass is a `proc(module: ^IR_Module)` — pure transformation, no side state. | Pass | Description | Priority | |---|---|---| | Constant folding | Evaluate `3.0 * 2.0` → `6.0` at compile time | Ship with v0.1 | | Dead code elimination | Remove unused SSA values (zero use count) | Ship with v0.1 | | Copy propagation | Replace `%2 = %1` usages of `%2` with `%1` | Ship with v0.1 | | Constant propagation | Propagate known values through operations | v0.2 | | Common subexpression elimination | Deduplicate identical operations in same block | v0.2 | | Function inlining | Replace `Call` with inlined body (all shader functions are candidates) | v0.2 | | Loop unrolling | Unroll `for` with known bounds under threshold | v0.3 | | Algebraic simplification | `x * 1.0 → x`, `x + 0.0 → x`, `x * 0.0 → 0.0` | v0.2 | | SROA | Scalar replacement of aggregates: `vec3(a,b,c).x` → `a`. High impact on generated code readability. | v0.2 | Don't over-invest here early. GPU drivers have excellent optimizers. Your job is to emit clean, correct IR — not to outdo the driver. The opts above are primarily for cleaner emitted source (humans read GLSL/HLSL output for debugging). --- ## 6. Backends ### 6.1 General Emission Strategy Each backend (`emit_*.odin`) is a single file containing: ```odin emit_TARGET :: proc(module: IR_Module, opts: Compile_Options) -> ([]u8, []Diagnostic) { ... } ``` Backends walk the IR_Module and emit text (for GLSL/HLSL/MSL/WGSL) or binary (for SPIR-V). They all share common challenges: - **Name mangling:** Odin-style names → target-legal names (avoid reserved words) - **Type mapping:** Luma types → target type syntax - **Binding layout:** `@group(n) @binding(m)` → target-specific qualifiers - **Entry point conventions:** Different targets handle inputs/outputs differently (varyings in GLSL, semantics in HLSL, `[[stage_in]]` in MSL, etc.) - **Builtin mapping:** `sample()` → `texture(tex, uv)` (GLSL), `tex.Sample(sampler, uv)` (HLSL), `tex.sample(sampler, uv)` (MSL), `textureSample(tex, sampler, uv)` (WGSL) ### 6.2 GLSL 450 Backend (`emit_glsl.odin`) Simplest backend — good first target. Mapping highlights: - `@location(n)` → `layout(location = n) in/out` - `@group(g) @binding(b)` → `layout(set = g, binding = b) uniform` - `@builtin(position)` → `gl_Position` (vertex out), `gl_FragCoord` (fragment in) - `vec3`, `mat4` → same names (GLSL native) - `sample(tex, uv)` → `texture(tex, uv)` - Entry point → `void main()` with flattened ins/outs as globals GLSL uses `#version 450` and assumes Vulkan semantics (separate sampler/texture where needed, or combined if the compiler detects paired usage). ### 6.3 HLSL SM6.0 Backend (`emit_hlsl.odin`) Mapping highlights: - `@location(n)` → `: TEXCOORD{n}` semantic (or custom semantics) - `@builtin(position)` → `: SV_Position` - `@group(g) @binding(b)` → `register(t/b/s{b}, space{g})` depending on resource type - `vec3` → `float3`, `mat4` → `float4x4` - `sample(tex, uv)` → `tex.Sample(sampler_state, uv)` - Entry point keeps its name, with input/output structs carrying semantics - `mix()` → `lerp()`, `fract()` → `frac()`, `mod()` → `fmod()` HLSL is the most verbose target due to semantic annotations. The backend maintains a mapping table from Luma builtins → HLSL equivalents. Note on DXIL: DXIL is LLVM bitcode and effectively requires DXC to produce. Don't burn time trying to emit it directly. Output HLSL and document that DXC is needed for the final step. This is what every non-Microsoft shader toolchain does (including Naga, shaderc, etc.). ### 6.4 MSL 2.4 Backend (`emit_msl.odin`) Mapping highlights: - `@location(n)` → `[[attribute(n)]]` (vertex in) or `[[user(locn{n})]]` (inter-stage) - `@builtin(position)` → `[[position]]` - `@group(g) @binding(b)` → `[[buffer(b)]]`, `[[texture(b)]]`, `[[sampler(b)]]` (MSL uses flat binding, group is metadata-only) - `vec3` → `float3` (metal namespace), `mat4` → `float4x4` - `sample(tex, uv)` → `tex.sample(sampler, uv)` - Entry point: `vertex`/`fragment`/`kernel` function with `[[stage_in]]` struct - Uniforms passed as function parameters with `constant` address space MSL's "everything is a function parameter" model means the backend must collect all bindings used by an entry point and emit them as parameters. This is a tree-walk over the call graph from each entry point. **Binding remap:** MSL requires resource binding indices to be contiguous per resource class (`[[texture(0)]]`, `[[texture(1)]]`, not `[[texture(0)]]`, `[[texture(5)]]`). The MSL backend must maintain a binding remap table during emission that reassigns indices to be contiguous within each class (texture, sampler, buffer). The IR's canonical bindings are preserved — remapping is local to MSL emission only. Note on AIR/metallib: Same situation as DXIL — Apple's proprietary format requires the Metal compiler. Emit MSL source. ### 6.5 WGSL Backend (`emit_wgsl.odin`) Mapping highlights: - `@location(n)` → `@location(n)` (direct match, lucky us) - `@group(g) @binding(b)` → `@group(g) @binding(b)` (also direct match) - `@builtin(position)` → `@builtin(position)` (wow) - `vec3` → `vec3`, `mat4` → `mat4x4` (explicit generic syntax) - `sample(tex, uv)` → `textureSample(tex, sampler, uv)` - Entry point: `@vertex fn`, `@fragment fn`, `@compute @workgroup_size(x,y,z) fn` - `let` → `var` or `let` depending on mutability (WGSL distinguishes mutable `var` from immutable `let`, matching Luma's semantics when combined with reassignment analysis) WGSL is the closest target to Luma's semantics. The attribute syntax was deliberately designed to be similar. This should be the easiest backend after GLSL. ### 6.6 SPIR-V Binary Backend (`emit_spirv.odin`) This is the only binary backend and the most complex. Pure Odin, no external dependencies. **Approach:** Emit SPIR-V binary directly using the specification's binary format. The format is well-documented (a stream of 32-bit words with a simple structure). SPIR-V module structure (emitted in order): 1. Magic number, version, generator ID, bound (max ID + 1) 2. `OpCapability` (Shader) 3. `OpExtInstImport` ("GLSL.std.450") 4. `OpMemoryModel` (Logical, GLSL450) 5. `OpEntryPoint` with interface variable IDs 6. `OpExecutionMode` (OriginUpperLeft for fragment, etc.) 7. Debug instructions (`OpName`, `OpMemberName`) if debug mode 8. Annotations (`OpDecorate` — locations, bindings, built-ins) 9. Type declarations (`OpTypeFloat`, `OpTypeVector`, `OpTypeStruct`, etc.) 10. Global variables (`OpVariable` for inputs, outputs, uniforms) 11. Function definitions (`OpFunction`, blocks of instructions, `OpFunctionEnd`) Implementation structure: ```odin SPIRV_Builder :: struct { words: [dynamic]u32, // final binary next_id: u32, // SSA ID allocator type_cache: map[Type_Key]u32, // dedup type declarations (structural hash key) const_cache: map[Const_Key]u32, // dedup constants ptr_cache: map[Ptr_Key]u32, // dedup OpTypePointer (keyed by address_space + pointee type ID) capabilities: [dynamic]u32, decorations: [dynamic][]u32, // deferred, emitted in annotation section type_section: [dynamic][]u32, global_section: [dynamic][]u32, func_section: [dynamic][]u32, } ``` **Type_Key:** Use a structural hash as the cache key — e.g. `vec3` → `hash("vec", 3, f32_id)`, `mat4x4` → `hash("mat", 4, 4, f32_id)`. This guarantees automatic dedup of `OpTypeFloat`, `OpTypeVector`, `OpTypeMatrix`, `OpTypeStruct` etc. without manual tracking. Generate `spirv_spec.odin` from the official SPIR-V JSON grammar (from the SPIRV-Headers repo). This gives you every opcode, capability, and enumerant as Odin constants. The generator (`tools/gen_spirv_spec.odin`) is ~150 lines — parse JSON with `core:encoding/json`, emit Odin source. **Validation:** Optionally vendor SPIRV-Tools for validation only (not generation). This is a post-emit step: write SPIR-V binary → call `spvValidateBinary()` via foreign import. But this is a nice-to-have — the first version can validate by round-tripping through `spirv-val` as an external tool in tests. --- ## 7. Vendoring Strategy ### 7.1 Pure Odin First The entire compiler can ship with **zero C dependencies.** Here's why: - **GLSL, HLSL, MSL, WGSL backends** — all text emission, pure Odin string building. - **SPIR-V backend** — binary format, emittable directly. The format is a flat word stream, not a compressed archive. No library needed. - **SPIR-V spec data** — generated from JSON, not a compiled dependency. ### 7.2 Optional Vendored C (SPIRV-Tools) SPIRV-Tools provides: - `spirv-val`: binary validation - `spirv-opt`: optimization passes (redundancy elim, inline, etc.) - `spirv-dis`: disassembly (useful for debugging) **If vendoring:** Clone the repo, compile the C++ sources to a static lib with Clang, write minimal Odin foreign bindings for `spvValidateBinary` and `spvOptimizeBinary`. Guard behind a build tag: ```odin when LUMA_USE_SPIRV_TOOLS { foreign import spirv_tools "vendor/spirv-tools/libSPIRV-Tools.a" @(default_calling_convention = "c") foreign spirv_tools { spvValidateBinary :: proc(env: rawptr, words: [^]u32, count: uint, diag: ^rawptr) -> i32 --- } } ``` **Recommendation:** Ship v0.1 without SPIRV-Tools. Use `spirv-val` as an external tool in CI/tests. Add vendored validation later if users want a single-binary experience with validation built in. ### 7.3 What NOT to Vendor - **DXC** — massive LLVM-based tool. Just emit HLSL. - **Metal compiler** — Apple-only, closed source. Just emit MSL. - **Tint** — Google's WGSL compiler. Large C++ codebase. Just emit WGSL. - **glslang** — if you have a SPIR-V emitter, you don't need GLSL→SPIR-V. And GLSL→SPIR-V is not your compiler's job — engines that want SPIR-V can use Luma's SPIR-V backend directly. --- ## 8. Reflection Shader compilers are only half-useful without reflection data. Engines need to know binding layouts, input/output locations, push constant sizes, etc. at build time. ### 8.1 Reflection Output The library exposes: ```odin Reflect_Info :: struct { entry_points: []Reflect_Entry, structs: []Reflect_Struct, bindings: []Reflect_Binding, } Reflect_Entry :: struct { name: string, stage: Shader_Stage, inputs: []Reflect_IO, outputs: []Reflect_IO, workgroup: [3]int, // [0,0,0] if not compute } Reflect_IO :: struct { name: string, type: string, // human-readable type name location: int, builtin: string, // empty if not a builtin } Reflect_Binding :: struct { name: string, group: int, binding: int, type: Binding_Type, // Uniform_Buffer, Storage_Buffer, Texture, Sampler struct_name: string, // for uniform buffers: which struct size: int, // byte size (for uniform buffers) array_size: int, // 0 = not an array, >0 = fixed array count (e.g. texture2D lights[16] → 16) descriptor_type: string, // Vulkan/DX12 descriptor type: "sampled_image", "storage_buffer", "uniform_buffer", "sampler" combined_id: int, // -1 if standalone, otherwise shared ID linking split texture+sampler pairs } Reflect_Spec_Constant :: struct { name: string, spec_id: int, type: string, // human-readable type name default: string, // default value as string } reflect :: proc(source: string) -> (Reflect_Info, []Diagnostic) { ... } ``` `Reflect_Binding_Type` uses separate `Texture` and `Sampler` variants (not `Combined_Image_Sampler`) to match the IR's canonical split representation. When `sampler2D foo` splits into `foo_tex` + `foo_samp`, both `Reflect_Binding` entries share the same `combined_id` so engines can reconstruct the logical pairing. `descriptor_type` maps to Vulkan/DX12 descriptor types: `texture2D` → `"sampled_image"`, `buffer` → `"storage_buffer"`, `uniform` → `"uniform_buffer"`, `sampler` → `"sampler"`. `Reflect_Info` includes `spec_constants: []Reflect_Spec_Constant` for pipeline-time configuration. The CLI's `luma reflect` command emits this as JSON. The `--spirv` flag outputs SPIR-V-specific reflection data (descriptor sets, push constants, vertex attributes) — saving engine developers from needing a separate SPIR-V reflection tool. Engines consume this at build time to auto-generate descriptor set layouts, pipeline layout structs, etc. --- ## 9. Testing Strategy ### 9.1 Unit Tests One test file per compiler stage: - `preprocessor_test.odin` — include resolution, pragma once, circular include detection, source map correctness. - `lexer_test.odin` — tokenize known strings, verify token types and spans. - `parser_test.odin` — parse snippets, verify AST structure. - `sema_test.odin` — type-check valid and invalid programs, verify diagnostics. - `ir_test.odin` — lower AST to IR, verify instruction sequences. - `emit_test.odin` — emit IR to each target, verify output contains expected patterns. Use `core:testing` and `expect()`. Keep tests fast — no file I/O, no external tools. ### 9.2 Integration Tests `tests/shaders/` contains `.luma` files that compile to all targets. A test harness: 1. Compiles each shader to all 5 targets 2. For GLSL: runs through `glslangValidator` (if available) 3. For SPIR-V: runs through `spirv-val` 4. For HLSL: optionally validates with DXC 5. For WGSL: optionally validates with Tint or `naga` 6. Compares output against golden files (checked in, updated deliberately) ### 9.3 Fuzz Testing The parser is the most exposed surface. Feed random byte sequences and verify no crashes (Odin's bounds checking helps here). Also fuzz the SPIR-V emitter — emit from random valid IR and validate with `spirv-val`. --- ## 10. Build-Order Roadmap ### Phase 1: Foundation (get something compiling) 1. `preprocessor.odin` — `#include` expansion with source map 2. `token.odin`, `lexer.odin` — tokenize a basic shader 3. `ast.odin`, `parser.odin` — parse into AST 4. `types.odin`, `symbols.odin`, `sema.odin` — typecheck 5. `emit_glsl.odin` — emit GLSL directly from typed AST (skip IR for now) 6. CLI with `compile` and `check` commands, `--include-dir` flag 7. First integration test: vertex + fragment shader → valid GLSL ### Phase 2: IR and Second Backend 8. `ir.odin`, `ir_builder.odin` — lower typed AST to IR 9. Rewrite GLSL backend to emit from IR 10. `ir_opt.odin` — constant folding + DCE 11. `emit_wgsl.odin` — WGSL backend (closest to Luma semantics) 12. `emit_hlsl.odin` — HLSL backend 13. Reflection API ### Phase 2.5: IR Improvements & Syntax Sugar 14. Refactor IR to split combined `sampler2D` into separate `texture2D` + `sampler` bindings during lowering 15. Add `Composite_Extract` / `Composite_Insert` IR expression kinds for value-semantic field access 16. Simplify WGSL and HLSL backends (remove ad-hoc sampler splitting) 17. Update reflection to use separate `Texture` / `Sampler` binding types 18. Update GLSL backend to recombine for GLSL 450 combined samplers 19. Auto-location numbering in `build_entry_io` (Sugar 1) 20. Binding defaults with auto-increment in `build_bindings` (Sugar 2) 21. `@varying` struct support — `attributes` on `Ast_Struct`, builtin stripping for fragment inputs (Sugar 3) 22. Inline entry points — `Type_Tuple`, `desugar.odin`, anonymous struct literals (Sugar 4) ### Phase 3: Full Coverage 23. `emit_msl.odin` — MSL backend 24. `spirv_spec.odin` generator + `emit_spirv.odin` — SPIR-V binary 25. Specialization constants (`@spec(id)` attribute, `IR_Spec_Constant`, backend mappings) 26. Additional opts (inlining, CSE, SROA, loop unroll) 27. Optional SPIRV-Tools vendoring for validation 28. `dump-ast`, `dump-ir` CLI commands 29. Fuzz testing, golden file tests ### Phase 3.5: Address Spaces & Compute 30. Add `IR_Address_Space` enum to IR (`Private`, `Uniform`, `Storage`, `Workgroup`, `Input`, `Output`) 31. `shared` keyword for workgroup memory in compute shaders 32. Update WGSL backend for `var`, MSL for `threadgroup`, SPIR-V for `StorageClass` ### Phase 4: Polish 33. Error recovery improvements 34. Source maps / debug info in SPIR-V 35. `luma reflect --spirv` for SPIR-V-specific reflection output 36. Documentation, examples, engine integration guide --- ## 11. Confirmed Design Decisions External review validated the following architectural choices: - **SSA IR before backends** — puts Luma in the same architectural class as SPIR-V, LLVM, and Naga. Each backend stays thin, optimization is centralized, language features don't multiply backend complexity. - **Entry point struct flattening** — necessary because GLSL, HLSL, and WGSL all handle inputs/outputs differently. Flattening to individual location-annotated values at the IR level avoids per-backend pain. - **No preprocessor macros** (`#define`, `#ifdef`) — the `const` approach is cleaner. Shader compilers that allow macros become maintenance nightmares. - **Library pipeline API** (`parse`/`check`/`lower`/`emit`) — exactly how real engines want to integrate shader compilers. Exposing pipeline stages lets consumers inspect or transform IR before emission. - **Deterministic compilation** — `luma compile --deterministic` guarantees stable SSA ordering, stable binding assignment, stable type IDs, and stable SPIR-V binary output. This matters for asset pipelines (build caching, content-addressed storage) and CI reproducibility. Implementation: avoid maps for ordering-sensitive output, use sorted iteration or ordered data structures where output order matters. With `--hash`, outputs a SHA256 of the canonical IR alongside the compiled output — useful for pipeline cache keys, asset deduplication, and build verification.