# EK9 compiler-validated corpus

The material the EK9 site (https://ek9.io) teaches from, as data files. **Every record was validated by the EK9 compiler, version 0.0.1-0**: each piece of EK9 in these files was compiled, and each wrong form was compiled and seen to fail with the error code it is paired with. The files are regenerated, with this datasheet and `manifest.json`, every time the site is built, from the same sources the compiler's own test gates check.

## Files

| file | records | what it is |
|---|---|---|
| `ek9-qa.jsonl` | 1225 | the Q&A corpus: question, alternate phrasings, answer, compiled EK9 example, keywords, topics and typical errors; the schema of /qa/qa.jsonl |
| `ek9-qa-chat.jsonl` | 5667 | the Q&A as chat conversations (system, user, assistant): one per question and one per alternate phrasing |
| `ek9-error-pairs.jsonl` | 2251 | wrong EK9, the error code the compiler reports for it, and right EK9: every Q&A typical-error quad and every fix in the error catalogue |
| `ek9-examples.jsonl` | 1248 | every whole EK9 program the site shows: the site examples (with their declared output), the error catalogue's failing sources and fixes, and the concept pages' own sources; `origin` says which |
| `ek9-examples.zip` | 2542 | every EK9 source the site compiles, as files in one zip: the site examples, the error catalogue, the concepts and the Q&A, each byte for byte the file its gate compiles, with a README; its record count is its number of files |

Each `.jsonl` file holds one JSON object per line, UTF-8, lines ending in `\n`. `manifest.json` gives each file's record count, size and SHA-256, the compiler version and the schema version (2); it carries no timestamp, so the same sources always give the same bytes.

## The fields of every file

Every key that appears in a file is listed here, with its type; a key marked *optional* is left out of a record that has no value for it. A key inside a list of objects is written `list[].key`. Strings are UTF-8 and may hold newlines (`\n`).

### `ek9-qa.jsonl`

| field | type | meaning |
|---|---|---|
| `id` | integer | the entry's number: 123 is the page QA0123 and the id Q123 that `ek9 -q` shows |
| `category` | string | the topic directory the entry lives in (one of the hub's categories) |
| `question` | string | the canonical question |
| `url` | string | the entry's page on the site |
| `alternatePhrasings` | string[] | other ways the same question is asked; the chat file has a conversation for each |
| `answer` | string | the answer in prose, with code blocks that are compiled too |
| `ek9Example` | string, optional | the entry's whole EK9 example, starting with its `#!ek9` line; it compiled, and if it declares an expected output it was run |
| `migrationContext` | string, optional | what the same task looks like in Java, Python or Kotlin, and what EK9 does instead |
| `keywords` | string[] | search keywords, sorted |
| `primaryTopics` | string[] | the language topics the entry teaches |
| `typicalErrors` | object[] | typical mistakes: each a quad of error code, wrong form, right form and explanation (the same quads are in the error pairs file); an empty list when none |
| `typicalErrors[].error` | string | the error code the compiler reports for `incorrect` |
| `typicalErrors[].correct` | string | a fragment of the entry's source: the right form |
| `typicalErrors[].incorrect` | string | the wrong form that replaces `correct` in that source; empty when the mistake is leaving `correct` out |
| `typicalErrors[].explanation` | string | why it is wrong |
| `companions` | object[] | the other files the example compiles with (a module spread over files, or a `dev/` test); an empty list when none |
| `companions[].filename` | string | the companion's file name |
| `companions[].source` | string | the companion's whole EK9 source |
| `companions[].dev` | boolean | true for a file kept under `dev/`, which only a test run compiles |
| `oracleToolHint` | object, optional | the EK9 MCP tool that answers the same question from a live workspace |
| `oracleToolHint.tool` | string, optional | the tool's name |
| `oracleToolHint.intent` | string, optional | when to use it |
| `oracleToolHint.description` | string, optional | what it returns |

### `ek9-qa-chat.jsonl`

| field | type | meaning |
|---|---|---|
| `messages` | object[] | the conversation, in order: the system prompt, the user turn and the assistant turn |
| `messages[].role` | string | `system`, `user` or `assistant` |
| `messages[].content` | string | the turn's text: the prompt; the question or one of its alternate phrasings; the answer with its example, typical errors and any tool hint |

### `ek9-error-pairs.jsonl`

| field | type | meaning |
|---|---|---|
| `id` | string | stable and unique: `qa:QA<nnnn>:<code>:<hash>` (`:2` ... for a quad an entry repeats) or `errorExamples:<code>/<fix>` |
| `code` | string | the error code the compiler reports for `wrong` |
| `classification` | string | the compiler's name for that error (`null` for a code of neither enum) |
| `wrong` | string | the wrong EK9: a fragment (`qaTypicalErrors`, empty when the mistake is a deletion) or a whole source (`errorExamples`) |
| `right` | string | the right EK9: the fragment `wrong` replaces, or a whole source that compiles |
| `explanation` | string | why `wrong` is wrong (a quad's explanation, or the fix's title) |
| `phase` | string | the compilation phase that reports the code, from the catalogue's failing example; `null` for a code the catalogue holds no example of |
| `source` | string | the page the record came from: the Q&A entry, or the code's entry in the error index |
| `origin` | string | `qaTypicalErrors` (a quad of a Q&A entry) or `errorExamples` (a fix in the error catalogue) |
| `validation` | string | how this record was checked, and by which test gate |

### `ek9-examples.jsonl`

| field | type | meaning |
|---|---|---|
| `id` | string | the source's path: in the site-examples corpus for `site` (unprefixed, as in schema 1), else its path in `ek9-examples.zip` (`errors/<CODE>/...`, `concepts/<slug>/...`) |
| `url` | string | the page that shows the source whole (for the error catalogue, the code's page) |
| `title` | string | what the example demonstrates (a fix's title; for a failing catalogue source, its code and what the code means) |
| `code` | string | the whole source as the site shows it: it starts with its `#!ek9` line, so it compiles as given, and has no compiler directives |
| `expectedOutput` | string, object or null | the declared output: a string, program name to output for a file with several programs, or `null` when none is declared |
| `kind` | string | `good` (compiles) or `bad` (fails with exactly `errors`) |
| `errors` | string[] | the error codes a `bad` source fails with; empty for `good` |
| `validation` | string | how this record was checked, and by which test gate |
| `origin` | string | `site` (a site example), `error-bad` (a failing source of the error catalogue), `error-good` (a fix in the catalogue) or `concept` (a concept page's own source) |
| `fixes` | string, optional | for `error-good` only: the error code the fix removes |

### `manifest.json`

| field | type | meaning |
|---|---|---|
| `name` | string | the corpus name |
| `schemaVersion` | integer | the schema version of every file here |
| `homepage` | string | where the corpus is published |
| `validatedBy` | object | what validated every record |
| `validatedBy.compiler` | string | `EK9` |
| `validatedBy.version` | string | the compiler version that compiled every record |
| `license` | object | the licence of the data |
| `license.spdx` | string | its SPDX identifier, `CC0-1.0` |
| `license.name` | string | its name |
| `license.url` | string | where the licence text is |
| `files` | object[] | the data files, in the order of the table above |
| `files[].name` | string | the file's name |
| `files[].records` | integer | its number of lines (for the zip, its number of files) |
| `files[].bytes` | integer | its size in bytes |
| `files[].sha256` | string | the SHA-256 of its bytes, lower-case hex |
| `files[].description` | string | what it holds |
| `datasheet` | object | this datasheet |
| `datasheet.name` | string | `README.md` |
| `datasheet.sha256` | string | the SHA-256 of the datasheet's bytes |

## How every record was validated

The checks below are the EK9 build's own test gates, run on the same sources these files are made from; a source that fails one fails the gate, so it never reaches these files.

### `ek9-qa.jsonl` and `ek9-qa-chat.jsonl`

- Every entry's EK9 example (and its companion files) compiles with no errors: each Q&A topic directory is compiled as one workspace.
- Every example that declares an expected output is compiled to JVM bytecode and run; its standard output, standard error and exit code are checked against the declaration.
- Every code block in an answer's prose is compiled too; one marked as a deliberate error must fail with the code it names.
- Every typical-error quad is checked as described for `ek9-error-pairs.jsonl` below.
- `ek9-qa.jsonl` is byte-for-byte the site's `/qa/qa.jsonl`. `ek9-qa-chat.jsonl` holds the same entries as conversations: a system prompt, the question (or one of its alternate phrasings) as the user turn, and the answer, example, typical errors and any tool hint as the assistant turn. Its record count is the number of entries plus the number of alternate phrasings.

### `ek9-error-pairs.jsonl`

Each record is `{id, code, classification, wrong, right, explanation, phase, source, origin, validation}`; `validation` states, for that record, how it was checked. `id` is stable: `qa:QA<nnnn>:<code>:<hash>` (the hash of `right` and `wrong` the quad gate keys the quad by; a quad an entry repeats with another explanation gets `:2`) or `errorExamples:<code>/<fix>`. There are two origins:

- `qaTypicalErrors`: a quad of a Q&A entry (`source` is its page). `wrong` and `right` are fragments: `right` occurs exactly once in the entry's compiled source, and replacing it with `wrong` makes `code` the **first** error the compiler reports (a cascade code must be reported on the edited lines). The unedited source compiles with no errors. `wrong` may be empty: the mistake is leaving `right` out.
- `errorExamples`: a fix in the compiler's error catalogue (`source` is the code's entry in the error index). `wrong` is a whole source that fails with exactly its declared error - that code, at that phase, nothing else - and `right` is a whole source that compiles with no errors through full code generation. A code with several fixes gives one record per fix. An example that only exists across several files is given as each file under a `// file: <name>` comment line.

`classification` is the compiler's name for the error; `phase` is the compilation phase that reports it, from the catalogue's failing example, and is `null` for a code the catalogue does not yet hold an example of. `explanation` is the quad's explanation, or the fix's title.

### `ek9-examples.jsonl`

Each record is `{id, url, title, code, expectedOutput, kind, errors, validation, origin}`, and a fix in the error catalogue also has `fixes`: `id` is the source's path, `url` the page that shows it, `code` the whole source as the site shows it (its `#!ek9` line first, no directives). `origin` says where it comes from:

- `site`: a site example (`id` is its path in the site-examples corpus, as in schema 1).
- `error-bad`: a failing source of the error catalogue (`id` is `errors/<CODE>/...`, its path in `ek9-examples.zip`; `url` the code's page).
- `error-good`: a fix in the error catalogue; `fixes` is the code it removes. Fixes are compiled, not run.
- `concept`: a concept page's own source (`id` is `concepts/<slug>/...`). A concept that shows a site or catalogue source refers to that record instead.

And `kind` says what the compiler does with it:

- `kind` `good`: compiled with no errors through full code generation; a declared `expectedOutput` (a string, or program name to output for a file with several programs) is what the program prints when run.
- `kind` `bad`: fails with exactly the error codes in `errors` - those errors, at their phase, nothing else (for a test-runner example of the catalogue, `ek9 -t` reports exactly that code).
- A source in a project directory (a `dev/` test beside the code it tests, or a module across several files) is compiled with the other files of its project. A project that holds an error source is an error example as a whole: each of its other files is `bad` too, with the project's `errors`, and reports nothing itself.

### `ek9-examples.zip`

One zip of every EK9 source the site compiles, each file byte for byte the source its gate compiles (front matter and `@Error` directives included): `site/<page>/...` (the site examples, projects in their layout with the `.txt` files their test runs read), `errors/<CODE>/...` (the error catalogue: `bad.ek9` or `bad/` fails on purpose with that code, each `good*` is a fix), `concepts/<slug>/...` (the concepts' own files) and `qa/<topic>/...` (the Q&A corpus), with a `README.md`. Its entries are sorted and carry one fixed timestamp, so its SHA-256 changes only when a source does. Its record count is its number of files.

## Licence

These files and this datasheet are dedicated to the public domain under CC0 1.0 (https://creativecommons.org/publicdomain/zero/1.0/), SPDX `CC0-1.0`: anyone, AI companies included, may copy, modify, train on and redistribute them, for any purpose, without asking. The EK9 compiler and runtime themselves are proprietary (https://ek9.io/LICENSE.txt).

## How to cite (optional)

No citation is required. If you want one:

> Limb, S. J. *EK9 compiler-validated corpus*, schema version 2, validated by the EK9 compiler version 0.0.1-0. https://ek9.io/data/