Literate Grammar Format Specification

This document specifies the literate .lime file format used by the lime-compose tool to assemble modular grammars for the Lime parser generator.

Overview

A literate grammar file is a Markdown document (.md) that interleaves human-readable documentation with machine-extractable grammar code and module metadata. The lime-compose tool reads one or more literate files, resolves inter-module dependencies, and produces a single .lime file suitable for the Lime parser generator.

This format replaces ad-hoc cat concatenation (as seen in the PostgreSQL example Makefile) with a structured, dependency-aware composition system.

File Structure

A literate grammar file has three kinds of content:

1. Markdown prose – documentation, headings, paragraphs, lists. Ignored by lime-compose; rendered by any Markdown viewer.
2. YAML metadata block – exactly one fenced block tagged yaml containing module identity, capabilities, and dependencies.
3. Lime code blocks – one or more fenced blocks tagged lime containing actual grammar directives and rules.

Everything outside of yaml and lime fenced code blocks is treated as documentation and discarded during composition.

Metadata Block

Each file must contain exactly one YAML metadata block:

```yaml
name: my-module
version: 1.0.0
description: Brief description of this module
provides: [tokens, precedence]
depends: [core-types]
```

Required Fields

Field	Type	Description
name	string	Unique module identifier (ASCII)

Optional Fields

Field	Type	Default	Description
version	string	0.0.0	Semantic version of this module
description	string	(empty)	One-line human-readable description
provides	list/string	[]	Capabilities this module exports
depends	list/string	[]	Capabilities this module requires

Naming Rules

Module names must match [A-Za-z_][A-Za-z0-9_.-]* (ASCII only).

Capabilities

Capabilities are arbitrary strings used to express inter-module contracts. A module lists what it provides and what it depends on. During composition, lime-compose verifies that every dependency is satisfied by some module's provides list.

Common capability conventions:

• tokens – the module defines token declarations
• precedence – the module defines operator precedence
• types – the module defines non-terminal type declarations
• rules-<section> – the module defines grammar rules for a section

Code Blocks

Grammar code lives in fenced blocks tagged lime:

```lime
%token IDENT.
%token ICONST.
```

Multiple code blocks per file are supported and are concatenated in order of appearance. An optional label may follow the lime tag:

```lime token-declarations
%token IDENT.
%token ICONST.
```

Labels are included as comments in the composed output for traceability.

Content Rules

• Code blocks contain standard Lime grammar directives: token, type, name, include, left, right, nonassoc, start_symbol, syntax_error, parse_failure, production rules, and C action code.
• Only ASCII characters are permitted in code blocks.
• Empty code blocks are allowed but serve no purpose.

Composition Process

When lime-compose processes input files, it performs these steps:

1. Parse each file, extracting the metadata block and all code blocks.
2. Validate that every module has a unique name and that no capability is provided by more than one module.
3. Resolve dependencies by checking that every depends entry is satisfied by some module's provides list.
4. Topological sort modules so that providers come before consumers. Modules with no dependency relationship are ordered alphabetically for deterministic output.
5. Compose by concatenating code blocks in resolved order, with provenance comments marking module boundaries.

Cyclic Dependencies

Cyclic dependencies cause lime-compose to exit with an error listing the modules involved in the cycle.

Duplicate Detection

• Duplicate module names across input files are rejected.
• Duplicate capability names provided by different modules are rejected.

Output Format

The composed output is a valid .lime file containing:

1. A header comment listing all composed modules and their versions.
2. For each module (in dependency order):
   • A banner comment identifying the module, version, and source file.
   • The concatenated code blocks from that module.

Example

Given two files:

tokens.md:

# SQL Tokens

Token definitions for the SQL parser.

```yaml
name: sql-tokens
version: 1.0.0
description: SQL token declarations
provides: [tokens, precedence]
```

## Non-keyword tokens

```lime token-declarations
%token IDENT.
%token ICONST.
%token SCONST.
```

## Precedence

```lime precedence
%left OR.
%left AND.
%right NOT.
```

grammar.md:

# SQL Grammar Rules

Production rules for basic SQL statements.

```yaml
name: sql-grammar
version: 1.0.0
description: SQL production rules
provides: [rules]
depends: [tokens]
```

## Statements

```lime
stmt ::= select_stmt.
stmt ::= insert_stmt.
```

Running:

lime-compose tokens.md grammar.md -o sql_parser.lime

Produces sql_parser.lime with tokens first (since grammar depends on tokens), followed by grammar rules, each section marked with provenance comments.

Tool Reference

lime-compose [options] <file.md>...

Option	Description
-o FILE	Write output to FILE (default: stdout)
-v, --verbose	Print resolution order to stderr
-n, --dry-run	Validate only, no output
--no-validate	Skip dependency validation
--list-modules	List modules and exit
-h, --help	Show help message