Packaging in EK9

This section covers the package construct.

The package construct is a flexible, gradual way to control what your programs consist of. It is a little like pom.xml in maven or package.json in npm, but builds the project model up more gradually. You don't need to know either of those: both package your software in some form and pull in the other packages you need.

The problem we are attempting to solve

If you only do small scale local development for yourself, you don't need packages at all. They solve problems that appear as software grows.

What makes up your whole program

The first issue a package solves is describing which source files to compile. You can put all your source in one file and compile that, and the examples show how far that goes. In time you will want to break the code up into several files, ideally with different module name spaces to give it more structure. How far to go is a matter of taste, practice, experience and tooling; it is easy to go from one extreme to the other.

Which files to compile

So, first issue: which files need compiling?

You may start with a 'grand plan' and fill in the blanks, or start coding and see where it takes you; EK9 supports either. The package has been developed in stages to support the second, fluid approach: prototypes, refactoring, and unit tests written as you go. Design note: start with code, then refactor explains why.

By default the EK9 compiler compiles only the single file you give it on the command line. Normally that file has at least one program you want to run, but not always.

When more files are needed, the package construct becomes useful: it lets you 'list' the files to compile, as shown below.

⚠ Important — what "single file" actually means

The CLI does not auto-discover sibling .ek9 files. If you run ek9 helloWorld.ek9 and the same directory also holds compA.ek9, traits.ek9 and so on, those are not compiled; only helloWorld.ek9 is. Its uses of symbols from the sibling files are reported as not resolved (E50001), as Step 2 below shows.

To compile multiple files you have two options:

1. Put the defines package block in the file containing the program(s) you intend to run. Add defines package / applyStandardIncludes <- true to helloWorld.ek9 itself, then run ek9 helloWorld.ek9. Use this when the project is built and run as a whole.
2. Use the -cd / -Cd dev-build flags (for example ek9 -cd helloWorld.ek9), which discover the full source tree whatever the package declaration says. They compile but do not run, and they also add debug instrumentation and the dev/ code, so they are not for release builds.

Important — a separate package.ek9 file does not work for running. It is tempting to put the defines package block in its own file and run ek9 -r MyProgram package.ek9. That fails with does not contain any programs: the CLI looks for the named program in the file you handed it. Keep the package block in the same file as the program(s) you want to run. A separate package file is fine for compile-only operations such as ek9 -P (packaging) or ek9 -Dp (dependency resolution).

EK9 IDE users: the IDE builds in dev-build mode, so its workspace shows every .ek9 source and compiles across files. To run a program it hands the CLI the program's own source file, so that file must contain a defines package block (or the program must be self-contained). Its Package, Deploy and Resolve-Dependencies operations use the project's package file directly, because they don't look up a program.

Compiling a single file by default is the first tier of EK9's graduated project model, in which complexity scales with the project:

1. One-off script — a single .ek9 file with one or two functions and a program. ek9 my-script.ek9 runs it. No directory layout, no package construct, no ceremony.
2. Small project — a dedicated directory holding several related .ek9 files, all in the same module. A defines package block (usually in the program's main file, or in a dedicated package.ek9 for compile-only use) brings in the siblings through includeFiles or applyStandardIncludes.
3. Multi-module system — a project tree with sub-directories, several defines module namespaces, perhaps external deps, and a package block that declares version / license / capabilities for publication.

As soon as your program needs siblings you move to tier two by adding the package construct, and from then on the package is the project model.

A worked example: from "Hello World" to multi-module

Here is a small project growing from one file to three files across two modules:

Step 1 — One-off script (tier 1). Single file, single module, one program:

#!ek9
defines module site.packaging.single

  defines program
    HelloWorld()
      stdout <- Stdout()
      stdout.println("Hello, World")

Download packaging_single.ek9

Hello, World

Save it as packaging_single.ek9 and run it with ek9 packaging_single.ek9. No package and no directory layout.

Step 2 — Add some shared code. The project has grown, and some records and traits are worth factoring out into packaging_nopackage_traits.ek9, in the same module:

#!ek9
defines module site.packaging.nopackage

  defines record
    BaseRecord as open
      anything as Integer: 90

      default operator ?

  defines trait
    Quack
      quack() as pure
        <- rtn as String?

  defines class
    Duck with trait of Quack
      override quack() as pure
        <- rtn as String: "Quack"

Download packaging_nopackage_traits.ek9

The program file, packaging_nopackage_hello.ek9, now uses BaseRecord and Duck from the other file:

#!ek9
defines module site.packaging.nopackage

  defines program
    HelloWorld()
      stdout <- Stdout()
      record <- BaseRecord()
      duck <- Duck()
      stdout.println(`Hello, World ${record.anything} ${duck.quack()}`)

Download packaging_nopackage_hello.ek9

The CLI cannot compile this with ek9 packaging_nopackage_hello.ek9: the CLI is still in single-file mode, so BaseRecord and Duck don't resolve:

$ ek9 -c packaging_nopackage_hello.ek9
ek9: compilation failed (exit 8)
EK9     : FULL_RESOLUTION: Errors in ./packaging_nopackage_hello.ek9
EK9     : Error   : E50001: 'BaseRecord' on line 7 position 16: not resolved
             See: ek9 -h E50001
EK9     : Error   : E50001: 'Duck' on line 8 position 14: not resolved
             See: ek9 -h E50001
EK9     : Compilation failed

Time to move to tier 2.

Step 3 — Tier 2: add a package. The package block goes inside the file containing the program, so the same command, ek9 packaging_withpackage_hello.ek9, still runs it:

#!ek9
defines module site.packaging.withpackage

  defines package
    version <- 1.0.0-0
    description <- "Simple samples"
    applyStandardIncludes <- true

  defines program
    HelloWorld()
      stdout <- Stdout()
      record <- BaseRecord()
      duck <- Duck()
      stdout.println(`Hello, World ${record.anything} ${duck.quack()}`)

Download packaging_withpackage_hello.ek9

The other file is unchanged (only its module is this project's):

#!ek9
defines module site.packaging.withpackage

  defines record
    BaseRecord as open
      anything as Integer: 90

      default operator ?

  defines trait
    Quack
      quack() as pure
        <- rtn as String?

  defines class
    Duck with trait of Quack
      override quack() as pure
        <- rtn as String: "Quack"

Download packaging_withpackage_traits.ek9

Hello, World 90 Quack

With the package block in packaging_withpackage_hello.ek9, ek9 packaging_withpackage_hello.ek9 compiles every .ek9 file in the directory tree (excluding dev/ for non-dev builds) and runs HelloWorld. The only addition was the four-line package block at the top of the program file.

You could put the package block in a separate package.ek9 file and use it as the target for compile-only operations such as ek9 -P package.ek9 (packaging) or ek9 -Dp package.ek9 (resolve dependencies). Running needs the package block beside the program, as the note above explains.

Convention — keep programs with the package. If your project has several defines program blocks (a small CLI tool with several subcommands, say), put them all in the file with the defines package block. Then ek9 helloWorld.ek9 compiles the whole tree and ek9 -r OtherProgram helloWorld.ek9 finds any program in the project, with no hunting for the file that hosts an entry point. The project then has:

• One entry-point file (e.g. helloWorld.ek9) — the package block + every defines program in the project.
• Supporting files (traits.ek9, compA.ek9, orders.ek9, …) — every other construct: records, classes, traits, functions, components, services, types, constants. These hold most of the code; the entry-point file orchestrates.

It is the same pattern as a single Main.java in a Java project or main.go in a Go module: the entry point is one well-known file and everything else is library code. EK9's per-file run rule reinforces the convention.

Step 4 — Tier 3: a second module. A richer example goes into a different module:

#!ek9
defines module site.packaging.twomodules.introduction

  defines class
    AnotherClass
      greeting() as pure
        <- rtn as String: "from another module"

Download packaging_twomodules_intro.ek9

The program's module references the class it uses:

references
  site.packaging.twomodules.introduction::AnotherClass

Full source: packaging_twomodules_hello.ek9 - Download packaging_twomodules_hello.ek9

Hello, World from another module

With applyStandardIncludes <- true already in place, the new file is picked up with no package edit, and references handles the cross-module visibility. The package still describes a single distributable unit (one version number, one license) even though the source spans several defines module blocks.

Define the list of files to be compiled

Say the content below is in a file called packaging_includes.ek9. It works rather like a 'Makefile' in 'C' and 'C++':

#!ek9
defines module site.packaging.includes

  references
    site.packaging.includes.maths::PI
    site.packaging.includes.physics::SPEED_OF_LIGHT

  defines package

    version <- 1.0.0-0
    description <- "Example of different constants in different modules."

    //We only want this file and these two to be compiled.
    includeFiles <- [
      "packaging_includes_maths.ek9",
      "packaging_includes_physics.ek9"
      ]

  defines program
    ShowConstants()
      stdout <- Stdout()
      stdout.println(`Pi is ${PI} and the speed of light is ${SPEED_OF_LIGHT}`)

Download packaging_includes.ek9

The two files it names each define a constant in a module of their own:

#!ek9
defines module site.packaging.includes.maths

  defines constant
    PI <- 3.14159

Download packaging_includes_maths.ek9

#!ek9
defines module site.packaging.includes.physics

  defines constant
    SPEED_OF_LIGHT <- 299792458

Download packaging_includes_physics.ek9

Pi is 3.14159 and the speed of light is 299792458

This suits a project with a few source files. It lets you take a one-file application and, over time, add features and split them across a small number of related files. Beyond a handful of files, remembering to add each new one becomes a burden.

It is aimed at small utility programs and tools that are a bit too big for a single source file, perhaps with some constructs shared by several small utilities.

Include all EK9 source files

Once your project reaches a certain size it has a directory of its own (small separate applications may have lived together, sharing little). You want all the related source code in that directory, with subdirectories for code that belongs together.

You can now stop listing the source files in includeFiles and write:

defines package
  version <- 1.0.0-0
  description <- "Simple samples"
  applyStandardIncludes <- true

Full source: packaging_withpackage_hello.ek9 - Download packaging_withpackage_hello.ek9

This one directive includes every file with the .ek9 suffix in the compilation: in the directory of the source file holding the package directive, and in all its subdirectories except /dev, which is included only in a development build. The build types are described later.

This 'directory' approach, with special named subdirectories, is closer to 'maven' or 'gradle'. So far your code uses only the standard libraries shipped with EK9 and has no external 'dependencies'. Those come next.

What about libraries

The second issue: you won't want or need to write all the code for your application yourself, or even with your team. Someone else may well have written code you would find useful, and made it available in the EK9 repository.

Equally, you may have developed constructs you want to package as a reusable library and make available within your own group or company, or publicly through the EK9 repository.

Either way, update your package directive to state the libraries (dependencies) your project needs.

Developers who want you to use their packages provide sample code in the /dev directory. Look in your library $HOME/.ek9/lib/{package-name}/dev for examples of how the developer intended the package to be used: expect tests and examples there, with documentation in the source code, to help you get started with the library.

Defining dependencies

defines module site.packaging.deps

  defines package

    version <- 1.0.0-0
    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }

Full source: packaging_deps.ek9 - Download packaging_deps.ek9

This project declares two dependencies, each at a specific version. EK9 also brings in any transitive dependencies, and the integrated build system detects circular dependencies between modules. EK9 is very strict on both dependency naming and version numbering, described below.

Dependencies and Libraries

To use or publish packages of constructs, you have to address version numbering and dependency naming.

Dependency Naming and Version Numbers

The EK9 repository has two main spaces for deploying modules.

Wild West

Naming public modules in the 'ekopen' space is a free for all: first come, first served. Corporates get no special treatment over freelance developers; there are no legal cases and no complaints, and whoever gets there first gets the name space. Names must have at least four parts, e.g. ekopen.network.support.utils. The developer who issues the first package to ekopen.network.support then controls all the final module names in that space: they can publish and deploy as many modules under it as they like, and only they can.

Civilisation

Here public module names are linked to a controlled DNS domain, so individuals or corporate entities can publish packages under their own name. Only the entity that controls the ek9lang.org domain can publish a package like org.ek9lang.core.functions. Domain validation uses the standard mechanism: the developer creates a suitable TXT DNS entry, the EK9 repository checks it, and the developer can then publish to that name space.

In Summary

If you are or represent a corporate entity, stay in Civilisation and keep control of your corporate brand, package and module names. If you like the free for all of the Wild West, don't complain about naming clashes; that's what it is like. If you don't like it, register a DNS domain and use Civilisation.

Version Numbering

Version numbers are the third issue. EK9 uses a strict form of Semantic Versioning:

• MAJOR.MINOR.PATCH-BUILD
• For features MAJOR.MINOR.PATCH-FEATURE-BUILD

No other variations are supported. Features explains why you might want a feature name in your version number.

• Each attempt to build a version increments the build number
• Each new patch released for a MAJOR.MINOR version increments the patch number
• Each new minor version released for a MAJOR version increments the minor number
• Each change to the public interface of the package increments the major number

The Public Interface

All constructs defined in the packaged module name are the public interface. In the example above, if class CRC and function reverse were defined in module ekopen.net.handy.tools, they would be its public interface.

To keep the same major version number, you cannot alter these constructs or their parameters in any way. No non public interface constructs can be used as parameters into or out of the public interface, and any transitive dependencies the module pulls in must also stay major version compatible.

In summary:

• The build number changes when builds fail and you fix a few minor things before the next patch release goes out
• A new patch number when a bug has been found and fixed
• A new minor release number when you add capabilities but the public interfaces (and transitive dependencies) stay the same: better performance, an alternative algorithm, or functionality that was stubbed in a previous release (with the public interface already there)
• A new major version number when the public interface changes, or a transitive dependency moves off its current major release number

Dependency Exclusions

The fourth issue: a transitive dependency is pulled in and has known defects, and you need to avoid that buggy version. This directive excludes a dependency when it has been pulled in:

defines module site.packaging.excludes

  defines package

    version <- 1.0.0-0
    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }

    excludeDeps <- {
      "ekopen.some.bad.dependency.pack": "ekopen.org.supertools.util"
      }

Full source: packaging_excludes.ek9 - Download packaging_excludes.ek9

The directive is a dictionary (map) of dependency names. It means: "exclude dependency 'ekopen.some.bad.dependency.pack' when it has been pulled in as a transitive dependency from 'ekopen.org.supertools.util'". It need not be a direct transitive dependency; it can be a dependency of a dependency, and so on.

Version Numbers for the module and program you are developing

Once your software is more than tooling for yourself and others use it, give it a version number. Your users will want to know when's the next release, and whether version x.y.z will have this feature or that, so versions are critical for managing releases, defects and improvements. Add the version directive to the package declaration, and align it with a source code repository such as git.

Here the version is 2.3.14-0. Each time you build, patch, or make minor or major improvements, the release number must change; the EK9 compiler can do that for you.

defines module site.packaging.version

  defines package

    version <- 2.3.14-0

    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }

    excludeDeps <- {
      "ekopen.some.bad.dependency.pack": "ekopen.org.supertools.util"
      }

Full source: packaging_version.ek9 - Download packaging_version.ek9

Once a version number is in place, the EK9 compiler can alter it as part of a build pipeline. When you supply software to others you need a reliable, repeatable and controlled build mechanism, not a build on your local PC.

With GitHub Actions, AWS, Azure or Hudson/Jenkins you can define the phases of a build, including changes to the version being built. The command line lists the options for updating the version number.

Publishing a module

Some of your code, a range of functions, classes or other constructs, may be useful to other people. It may already be in a git repository with contributions from other developers. Now you want to make specific versions available through the EK9 repository.

First decide Wild West or Civilisation and name your module accordingly, so that it is unique. Then add a few more directives.

The module below is named site.packaging.publish, as every compiled example on this site is; in a real project it would be named for its space, for example ekopen.math.simple.constants.

defines module site.packaging.publish

  defines package

    publicAccess <- true

    version <- 2.3.14-0

    description <- "Example of different constants in different modules."

    tags <- [
      "mathematics",
      "constants"
      ]

    license <- "MIT"

    //Pure computation - no system resources used. No 'capabilities' block is
    //declared, so capability enforcement is not activated for this package.
    //See the Capability Security section below for details.

    applyStandardIncludes <- true

    includeFiles <- [
      "License.txt",
      "Copyright.txt",
      "Authors.txt"
      ]

    //certain files like .gitignore we don't want to package
    applyStandardExcludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }

    excludeDeps <- {
      "ekopen.some.bad.dependency.pack": "ekopen.org.supertools.util"
      }

Full source: packaging_publish.ek9 - Download packaging_publish.ek9

• tags make the packaged module searchable (the fifth issue developers face: finding useful packages).
• publicAccess makes it publicly available.
• license states the software's license.
• applyStandardExcludes leaves out the standard set of files you don't want packaged, such as .gitignore.
• includeFiles is back: with applyStandardIncludes, all the ek9 source code and those three files are packaged.

Renamed to ekopen.math.simple.constants (if ekopen.math.simple is still available), the software can be packaged and uploaded to the EK9 repository. As the owners of ekopen.math.simple you can publish any number of packages under that structure, and no other developers can publish with that module naming. The same holds for a domain name you control: only your team can publish to that module name.

Importantly, once a version of a module is published it cannot be retracted, deleted or altered (other than by publishing a new version). Others may now reference it, and retracting it would break their builds.

How to publish

• First, follow the steps above to get your code to the point where it could be published
• Check with the EK9 repository that the module name is still available
• Sign up for an account with the EK9 repository, and save your credentials
• Use the ek9 command line compiler tool to publish.

Publishing to an EK9 public repository is not instant. The repository takes several steps to make sure the code is safe and meets basic standards: the content is scanned for viruses and checked against minimum standards (size, acceptable language), and the EK9 server compiles it to check that it compiles.

Capability Security: What Your Package Is Allowed to Do

Once you move from local development to publishing a module that other people depend on, you can add one more declaration inside defines package: the list of system capabilities your code uses. This is EK9's compile-time defence against supply chain attacks. The deploy command does not require it (it needs only publicAccess, version and license), but while the list is present the compiler holds the module's own code to it.

In npm, PyPI, Maven Central and crates.io, a dependency is free to open network connections, read your environment variables or write to your filesystem without telling anyone. A JSON parser and a credential-stealing trojan look identical in their published metadata, and supply chain attacks exploit exactly this gap. In EK9 a package can declare the system resources it uses, and while that declaration is present the compiler checks the module's own code against it.

When capability declarations are required

Capability enforcement applies to a package — which must then declare everything it uses — when all four of these are in its package block:

• publicAccess (true for the public EK9 repository, false for a private repository)
• version — consumers need a specific version to reference
• license — consumers need to know their legal obligations
• capabilities — the list of gated types your package uses (this declaration is what turns capability enforcement on for the module)

When any of the four is missing, enforcement is off and you can use any EK9 type. That is the case when you use the package block only for file organisation (applyStandardIncludes) or for dependency management (deps) and are not yet preparing to publish. Experimenting on your own machine carries no supply chain risk, so it carries no supply chain obligation.

The capabilities block itself says "I am entering the supply chain and accept the capability-review obligations." A published package that pre-dates the capability feature compiles unchanged until its author adds a capabilities block; there is no forced migration, and each module opts in when it is ready.

Test code under a dev/ directory is exempt. Tests need stdio, the filesystem and other system resources, and test code is never consumed by other packages, so it is not part of the supply chain.

Gated types: what you must declare

Gated types are built-in types that give access to system resources. Using one in a package under capability enforcement without declaring its capability is a compile error (E12010). The gated types are:

• Stdout, Stderr, Stdin — standard input and output streams
• TCP, UDP — raw network socket access (TCP covers TCPConnection, TCPHandler, TCPReadyHandler and TCPReadyObserver; UDP covers UDPPacket; either covers NetworkProperties)
• HTTPRequest, HTTPResponse — HTTP client and server types
• File, TextFile, FileSystem, FileSystemPath — filesystem access, each declared on its own
• EnvVars — environment variable access
• Networking — the gateway that reads network identities (addresses, hostnames, address ranges) from environment variables. Because it reads the environment, a package that uses it declares both org.ek9.lang::Networking and org.ek9.lang::EnvVars
• OS, Signals, Clock, SystemClock, Sensitive

Declare each gated type in the capabilities list by its fully-qualified name, with the module path org.ek9.lang::. Where a gated type has related helper types (declaring TCP permits TCPConnection and TCPHandler), declare only the primary type. The rich error message for E12010 lists the groupings.

Declaring capabilities in your package

The capabilities block sits inside defines package with version, license and deps. For a publishable package that reads environment variables and makes HTTP calls:

defines module site.packaging.httpclient

  defines package

    publicAccess <- true
    version <- 1.0.0-0
    description <- "HTTP client library"
    license <- "MIT"

    capabilities <- [
      "org.ek9.lang::TCP",
      "org.ek9.lang::HTTPRequest",
      "org.ek9.lang::HTTPResponse",
      "org.ek9.lang::EnvVars"
      ]

    deps <- {
      "org.ek9.collections": "1.0.0-0"
      }

Full source: packaging_httpclient.ek9 - Download packaging_httpclient.ek9

For a pure-computation package — a JSON parser, a string-manipulation library, a data-structure collection, a mathematical package — omit the capabilities block. Without it the fourth condition is not met, enforcement is off for the module, and the package compiles as it did before the capability feature. This is the recommended pattern for a library that touches no system resources:

defines module site.packaging.jsonparser

  defines package

    publicAccess <- true
    version <- 2.1.0-0
    description <- "Pure JSON parsing library"
    license <- "MIT"

    //No 'capabilities' block: pure computation, no gated types used.
    //Omitting the block is the current idiom for pure libraries.

    deps <- {
      "org.ek9.collections": "1.0.0-0"
      }

Full source: packaging_jsonparser.ek9 - Download packaging_jsonparser.ek9

Add a capabilities block, even with one entry, and the module comes under enforcement: every gated type it uses must be covered. You decide when a module moves from "unenforced" to "capability-governed", and no package that pre-dates the feature breaks.

Capability enforcement is per-module

Each package declares its own capabilities, and the compiler checks only that package's own code against them. Depending on another package does not inherit its capabilities, and a dependency that adds a capability does not cause compile errors up the dependency chain. Your module is responsible only for what your own code does.

Why this is worth the extra lines

The cost is a few extra lines in your package block, and an occasional compile error when you start using a new gated type. The benefit: while a capabilities block is present, every gated type the module's own code uses must be in it, so network access, your secrets or your filesystem show up in the package block.

A package with no capabilities block is not checked at all: it can use any gated type, and nothing about it appears in its package block. The compiler does not force a package to keep the block either. So when you review a dependency update, look first at whether it has a capabilities block; if it does, one that has been added, changed or removed is the change to check. For a dependency that carries the block, that is the difference between "the new version now declares TCP, why?" and noticing weeks later that data was being exfiltrated.

Many packages need no capabilities at all. When a dependency's capabilities block gains org.ek9.lang::TCP or org.ek9.lang::EnvVars where it had none before, that is the change to look at. You write the capability lines once per package, and the compiler checks them for as long as they are there.

The full design, the rationale, and detailed attack scenarios with their mitigations are in EK9_PACKAGE_CAPABILITY_SECURITY.md.

Summary

You can progress from small programs in single EK9 source files to medium, large and very large suites of software. They start small, gain dependencies, and can themselves be packaged and published. Each step adds a little more detail to the package construct as your applications and projects grow.

Final Steps!

The final section is on the command line compiler tool.