Packaging in EK9

This section will discuss the package construct in more detail.

The package construct is a flexible and gradual way to control what your programs will consist of. It is a little like the pom.xml in maven, but with a more gradual approach to creating a project model.
It does not matter if you don't know what a pom is or the npm - package.json. In short they are both mechanisms to help with packaging your software in some form or other, and also pulling in other packages you may need..

The problem we are attempting to solve

If you've only ever done small scale development or very limited applications/system utilities you're probably wondering what problem we are attempting to solve here with packages. In fact if you are still only doing small scale local development just for yourself - you don't even need to use packages.

What makes up your whole program

The first issue package aims to solve, is the way to describe what source files should be included for compilation. Clearly with EK9 it is possible to put all your source in one file and just compile that.

If you've looked at the examples, you can see you can actually get quite a long way with that approach. But in time you will probably want to break the code up in to several files. Hopefully you will use different module name spaces to give the code more structure.
This is a matter of taste, practice, experience, tooling - it's very easy to go from one extreme to another.

Which files to compile

So now we have our first issue, which files do we need to compile?

Depending on your approach to software development; you may start of with a 'grand plan' and fill in the blanks (i.e. you see the big picture and do lots of structure up front).
Alternatively you may just 'start coding' and see where it takes you! With the EK9 language you can take either approach.

Personally I prefer the latter, creating prototypes, pulling bits from here and there, refactoring, duplicating and then resolving duplications. So I tend to take a very dynamic and fluid approach to development and like to develop 'unit tests' to exercise the code written (sometimes before it is written). The EK9 package has been developed in a staged manner to support this approach.

By default the EK9 compiler will just compile the single file you gave it on the command line. Normally this will have at least one program you want to run - but not always.

But if there are multiple files that are needed. This is where the package construct starts to be useful. It enables the developer to 'list' the files they want to compile, as shown below.

⚠ Important — what "single file" actually means

The CLI does not auto-discover sibling .ek9 files. If you run ek9 helloWorld.ek9 and your project also contains compA.ek9, traits.ek9, etc. in the same directory, those are not compiled — only helloWorld.ek9 is. References to symbols in the sibling files will fail with E03030: ... reference does not resolve.

To compile multiple files you have two options:

  1. Put the defines package block in the file containing the program(s) you intend to run. Add defines package / applyStandardIncludes <- true to helloWorld.ek9 itself, then run ek9 helloWorld.ek9. This is the right answer for nearly every multi-file project.
  2. Use the -cd / -Cd dev-build flags (e.g. ek9 -cd helloWorld.ek9) which force full source-tree discovery regardless of any package declaration. Note these flags also enable debug instrumentation, so they are not appropriate for release builds.

Important — a separate package.ek9 file does not work for running. A common reflex is to put the defines package block in its own file (for cleanliness) and try ek9 -r MyProgram package.ek9. That fails: the CLI's run validation looks for the named program in the file you handed it. Since package.ek9 contains no programs you'll get does not contain any programs. The package block must live in the same file as the program(s) you want to run. (A separate package file is fine for compile-only operations like ek9 -P packaging or ek9 -Dp dependency resolution — but not for run.)

EK9 IDE users: the IDE always runs in dev-build mode internally, so its workspace shows every .ek9 source automatically and the in-IDE compile always succeeds across files. When the IDE then forks the CLI for a Run operation it hands the CLI the program's own source file — and that file must therefore contain a defines package block (or the program must be self-contained) for the multi-file compile to succeed. The IDE's Package / Deploy / Resolve-Dependencies operations use the project's package file directly because they don't need a program lookup.

The single-file-by-default rule is part of EK9's graduated project model — complexity scales with the project, not the other way round:

  1. One-off script — a single .ek9 file with one or two functions and a program. ek9 my-script.ek9 just runs it. No directory layout, no package construct, no ceremony.
  2. Small project — a dedicated directory holding several related .ek9 files, all in the same module. Add a defines package block (commonly inside the program's main file, or in a dedicated package.ek9) so the compiler picks up siblings via includeFiles or applyStandardIncludes.
  3. Multi-module system — a project tree with sub-directories, multiple defines module namespaces, possibly external deps, and the package block now declares version / license / capabilities for publication.

The CLI's "single file" default is the no-ceremony tier-one experience. As soon as your program needs siblings you graduate to tier two by introducing the package construct — and from that point on the package is the project model.

A worked example: from "Hello World" to multi-module

The progression isn't theoretical. Here's a small project's evolution from one file to three files across two modules, mirroring what most projects look like in practice:

Step 1 — One-off script (tier 1). Single file, single module, one program:

#!ek9
defines module simple

  defines program
    HelloWorld()
      stdout <- Stdout()
      stdout.println("Hello, World")
//EOF

Save as helloWorld.ek9, run with ek9 helloWorld.ek9. No package, no directory layout. Just a script.

Step 2 — Add some shared code. The project has grown — there are records and traits worth factoring out. We add traits.ek9 in the same module: 

#!ek9
defines module simple

  defines record
    BaseRecord as open
      anything <- 90
      ...

  defines trait
    Quack
      quack() as pure
        <- rtn as String?

helloWorld.ek9 now references BaseRecord and Duck from the trait file. The CLI cannot compile this with ek9 helloWorld.ek9 any more — the named file's references to simple::BaseRecord won't resolve because the CLI is still in single-file mode. Time to graduate to tier 2.

Step 3 — Tier 2 graduation: add a package. The package block goes inside the file containing the program (so the same command, ek9 helloWorld.ek9, continues to work for running): 

#!ek9
defines module simple

  defines package
    version <- 1.0.0-0
    description <- "Simple samples"
    applyStandardIncludes <- true

  defines program
    HelloWorld()
      stdout <- Stdout()
      stdout.println("Hello, World")
//EOF

With the package block now in helloWorld.ek9, ek9 helloWorld.ek9 compiles every .ek9 file in the directory tree (excluding dev/ for non-dev builds) and then runs HelloWorld. The other files are unchanged — the only addition was the four-line package block at the top of the program file.

You could alternatively put the package block in a separate package.ek9 file and use that as the target for compile-only operations like ek9 -P package.ek9 (packaging) or ek9 -Dp package.ek9 (resolve dependencies). But running requires the package block to live alongside the program — see the gotcha note above.

Convention — colocate programs with the package. If your project has multiple defines program blocks (e.g. a small CLI tool with several subcommands), put them all in the same file as the defines package block. That way ek9 helloWorld.ek9 compiles the whole tree and ek9 -r OtherProgram helloWorld.ek9 finds any program in the project — no hunting through files to remember which one happens to host which entry point. This naturally factors a project into:

It's the same pattern as a single Main.java in a Java project, or main.go in a Go module — the entry point is one well-known file and everything else is library code. EK9's per-file run rule reinforces this convention rather than fighting it.

Step 4 — Tier 3: a second module. The project introduces a richer example, factored into a different module: 

#!ek9
defines module introduction

  defines class
    AnotherClass
      ...

defines module simple

  references introduction::AnotherClass

With applyStandardIncludes <- true already in place, the new file is picked up automatically — no package edit is required. The references mechanism handles the cross-module visibility. The package construct in package.ek9 still describes a single distributable unit (one version number, one license) even though the source spans multiple defines module blocks.

The lesson: at no point did we have to rewrite earlier code or rename anything. Each tier added a new concept (multiple files; then a package; then a second module) without disturbing what came before. The package construct is purely additive — and it stays out of your way until your project actually needs it.

Define the list of files to be compiled

Assumes the content below is stored in a file called ConstantRef1.ek9. This is a bit like a 'Makefile' used by other languages like 'C' and 'C++' for example. 

#!ek9
defines module introduction

  defines package

    description <- "Example of different constants in different modules."

    //We only want this file and these three to be compiled up.
    includeFiles <- [
      "ConstantRef2.ek9",
      "ConstantRef3.ek9",
      "ConstantRef4.ek9"
      ]
      ...
//EOF

Now for projects that require just a few source files this approach is quite suitable, it has enabled the developer to take a simple application with one source file, refactor it (over a period of time). Adding features and capabilities and then break those features and capabilities up and use a small number of related source files. This approach might work for a handful of source files, but it soon becomes a burden to remember to add in the new source code files you have created.

This approach is really aimed at small utility programs and tools that are really just a bit too big for a single source file. Maybe there are some reusable elements needed by several little utilities and some constructs need to be shared.

Just include all EK9 source files

Once your project gets to a certain size, you will probably have it all in a directory on the filesystem that is dedicated just to that project. Smaller separate applications might have all been stored together but were actually separate and shared very little. But now you've reached the point where you want a dedicated directory for all the related source code. Moreover you want subdirectories for specific code that are best stored near each other.

You now need to make a slight modification to your package construct. You will be relieved to know that you can stop listing all the source files now in the includeFiles directive. Now you can just write the following: 

#!ek9
defines module introduction
  defines package

    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true
    ...
//EOF

This single directive above means that any and all files that have the .ek9 file suffix will now be included in the compilation build. This applies to the current directory where the source file with the package directive is and all sub directories (though not quite true) it will exclude all files in sub directories of /dev unless the type of compilation is a development build. The different types of build will be discussed later.

You may ask, why not just do this as the default. The main reason is 'different use cases'. Sometimes you just want to knock up a quick utility (maybe a single source file). Why mess about with package when a simple program and a couple of functions are all that is needed? See EK9 purpose for more details on this reasoning, EK9 is trying to make easy things 'easy' and hard things possible and 'fairly easy'.

This 'directory' based approach with special named subdirectories is more the like 'maven' or 'gradle' approach. At this point all your code only uses the standard libraries shipped with EK9 and as such has no external 'dependencies'. Those 'dependencies' and 'libraries' are the subject of the next paragraph.

What about libraries

This is the second issue developers face; in general you won't want or need to write all the code for your application yourself or even with your team. You will probably find someone else has written code that you would find useful. That code will most probably have been made available by that developer (or developers) in the EK9 repository.

Conversely, maybe you've developed a number of constructs and want to package those up as a reusable library and make that available either within your own development group/company or publicly via the EK9 repository.

You must now update your package directive again to state what libraries (dependencies) you need included in your project.

Now if the developers of those dependencies are the sort of developers that really want you to use their packages - they will have probably provided lots of sample code in the /dev directory. Take a look in your library $HOME/.ek9/lib/{package-name}/dev. There should be some good examples of how the developer intended you to use the package they have provided.
You'd expect quite a few tests and examples to be located in the dev/ directory with lots of documentation in the source code. These examples and snippets should really help you get started with how you use that library.

Defining dependencies
#!ek9
defines module introduction

  defines package

    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }
      ...
//EOF

In the above example we have declared that our project has two dependencies. Also note that we have declared the specific version of the dependency. EK9 will also bring in any transitive dependencies. As part of its integrated build system, the compiler will also detect any circular dependencies between modules, ensuring your project has a sound architectural foundation. EK9 is very strict on both dependency naming and version numbering.
There is more in the dependency naming and the version numbering in later paragraphs.

Dependencies and Libraries

If you want to use or publish packages of constructs then the issue of version numbering and dependency naming has to be addressed.

Dependency Naming and Version Numbers

There are two main spaces for deploying modules in the EK9 repository.

Wild West

The naming of public modules in the 'ekopen' space is a 'free for all', first come first served, no corporates that can't mix it with freelance devs. No legal cases, no complaints, whoever gets there first gets the name space. But they must have a depth of at least four parts i.e ekopen.network.support.utils. The developer that issues the first package to ekopen.network.support then controls all the final module names under that space. They can publish and deploy as many modules under that space as they like and only they can do this.

Civilisation

This is the naming of public modules where there is a link to a controlled DNS domain. This is controlled so that individuals or corporate entities can publish packages with their associated name. For example only the entity that controls the ek9lang.org domain name can publish a package like org.ek9lang.core.functions. The mechanism of domain validation is a standard one. The developer wishing to publish to a controlled space like this has to create a suitable TXT DNS entry. The EK9 repository will then check that and that developer will be able publish to that name space.

In Summary

If you are or represent a corporate entity stay in Civilisation, do not stray into the Wild West. In Civilisation you will be able to retain control of your corporate brand naming and main package names and module names. If you are a developer in the Wild West and like the free for all - then don't complain about naming clashes - that's what it is like. If you don't like it; register a DNS domain with some provider and use Civilisation.

Version Numbering

Version number are the third issue developers face; EK9 uses a form of strict Semantic Versioning as follows:

There are no other supported variations. See features for more information on why you might want a feature name in your version number.

The Public Interface

All constructs defined in the packaged module name are the public interface. So as an example: Using ekopen.net.handy.tools in the example above, if class CRC and function reverse were defined in module ekopen.net.handy.tools; then they would be the public interface.
If you wished to retain the same major version number - it is not possible to alter these constructs or their parameters in any way. Moreover no other non public interface constructs can be used as parameters into or out of that public interface. Any transitive dependencies that the module pulled in must also be major version compatible.

This may appear overly strict or draconian, but it is necessary to ensure dependency consistency. A developer needs to be assured that version 4.5.6-21 can be used in place of 4.4.1-9 and that not only will the public code interface be the same but also the transitive dependencies will also remain major version compatible. The EK9 compiler enforces this by failing the build if dependency resolution would require using a dependency with a higher major version than what was specified, as this indicates a breaking API change.

So in summary:

Dependency Exclusions

There will come a time when a transitive dependency (the fourth issue developers face) is pulled in and it has known defects; you need to avoid that specific buggy version. You can use the directive below to be explicit about excluding a dependency when it has been pulled in. 

#!ek9
defines module introduction

  defines package

    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }

    excludeDeps <- {
      "ekopen.some.bad.dependency.pack": "ekopen.org.supertools.util"
      }
      ...
//EOF

As you can see above the directive is very simple, it is just a dictionary (map) of dependency names. The above statement means: "exclude dependency 'ekopen.some.bad.dependency.pack' when it has been pulled in as a transitive dependency from 'ekopen.org.supertools.util'". It does not have to be a direct transitive dependency it could be a dependency of a dependency etc.

Version Numbers for the module and program you are developing

In general, if/when the software you are developing has become more than just a bit of tooling for yourself; and other users are now using it you will probably want to give your software a version number.

This becomes critical for managing releases/defect/improvements - your customers or people that just use your software will want to know when's the next release. Will version x.y.z have this feature or that feature.
The version directive is now added to the package declaration, but you should probably align this with some sort of source code repository like git or something like that.

Now add in your version number, as shown below we've decided this version is 2.3.14-0. Now each time you build, patch, make minor improvements or major improvements you must alter the release number.
But don't worry the EK9 compiler can be used to automate that - so there is no need to do it by hand. 

#!ek9
defines module introduction

  defines package

    version <- 2.3.14-0

    description <- "Example of different constants in different modules."

    //No need for includeFiles for source code anymore.
    applyStandardIncludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }

    excludeDeps <- {
      "ekopen.some.bad.dependency.pack": "ekopen.org.supertools.util"
      }
      ...
//EOF

Now maybe you're thinking it's going to be a pain altering that version number all the time. EK9 has a mechanism to deal with that for you. Once you've put a version number in the EK9 compiler can alter that for you as part of a build pipeline.
In general when you get to the point of supplying your software to others, you need a reliable/repeatable and controlled build mechanism (rather than just building it on your local PC!).

By using something like GitHub Actions, AWS, Azure or Hudson/Jenkins you can define phases of a build - and this can include alterations to the version being built. See the command line for details on the command line options for updating the version number.

Publishing a module

You may feel that the code you've developed, either just a range of functions, classes or other constructs could be useful to other people. You may already have a git repository with the code on and may other developers have contributed to that. Now you want to make specific versions available via the EK9 repository.

As mentioned earlier - decide Wild West or Civilisation and name your module accordingly. But now we need to add a couple more directives (as well as naming the module to be unique). 

#!ek9
defines module ekopen.math.simple.constants

  defines package

    publicAccess <- true

    version <- 2.3.14-0

    description <- "Example of different constants in different modules."

    tags <- [
      "mathematics",
      "constants",
      ]

    license <- "MIT"

    //Pure computation — no system resources used. No 'capabilities' block is
    //declared, so capability enforcement is not activated for this package.
    //See the Capability Security section below for details.

    applyStandardIncludes <- true

    includeFiles <- [
      "License.txt",
      "Copyright.txt",
      "Authors.txt"
      ]

    //certain files like .gitignore we don't want to package
    applyStandardExcludes <- true

    deps <- {
      "ekopen.network.support.utils": "1.6.1-9",
      "ekopen.net.handy.tools": "3.2.1-0"
      }

    excludeDeps<- {
      "ekopen.some.bad.dependency.pack": "ekopen.org.supertools.util"
      }
      ...
//EOF

By adding in the tags the packaged module will be searchable (the fifth issue developers face - how to easily find useful stuff) and by adding publicAccess it has been made publicly available. We've also stated what license the software is. By using the directive applyStandardExcludes we can exclude specific files. You will probably have noticed the the includeFiles has been reintroduced; this in conjunction with applyStandardIncludes will ensure that all ek9 source code in addition to those three files all get packaged.

By changing the name of the module to ekopen.math.simple.constants (assuming ekopen.math.simple is still available) the software can now be packaged and uploaded to the EK9 repository. As the developer(s) that now own ekopen.math.simple you can publish any number of packages to under that structure. No other developers can publish with that module naming.

It's the same process if you use and control a domain name, only your development team can publish to that module name.

Importantly once a version of a module is published - that's it; it cannot be retracted/deleted or altered (other than publishing a new version). Others may now have referenced it and if you could retract the software you would break their builds.

How to publish

Publishing to a EK9 public repository is not instant - the content being published must be scanned for viruses and also ensure it means the minimum standards (i.e. size, minimal/acceptable profanities). Note that when they are published the EK9 server will also compile them to check they will actually compile.

There are a number of stages and steps the 'repository' will take to ensure that the code provided is safe and meets and number of basic standards.

Capability Security: What Your Package Is Allowed to Do

Once you progress from local development to publishing a module that other people will depend on, EK9 requires one additional declaration inside defines package: the list of system capabilities your code uses. This is EK9's compile-time defence against supply chain attacks.

In every other ecosystem — npm, PyPI, Maven Central, crates.io — a dependency is free to open network connections, read your environment variables, or write to your filesystem without telling anyone. A JSON parser and a credential-stealing trojan look identical in their published metadata. Every major supply chain attack of the last few years has exploited exactly this gap. EK9 closes it at the compiler, before a single line of a compromised dependency runs on your machine.

When capability declarations are required

Your package opts into publishable capability enforcement — and therefore must declare its capabilities — when all four of these are present in the package block:

When any of those four is missing — for example, if you're just using the package block for file organisation with applyStandardIncludes or for dependency management with deps but not yet preparing for publication — capability enforcement is not activated. You can use any EK9 type freely. This matches how most developers actually work: experimenting locally on your own machine has no supply chain risk, so it incurs no supply chain obligation.

The presence of the capabilities block itself is the signal "I am entering the supply chain and accepting the capability-review obligations." An existing published package that pre-dates the capability feature continues to compile unchanged until its author adds a capabilities block. There is no forced retroactive migration: each module opts in on its own schedule when it is ready to.

Test code under a dev/ directory is also exempt. Tests naturally need access to stdio, filesystem, and other system resources, and test code is never consumed by other packages, so it is not part of the supply chain that capability security is defending.

Gated types: what you must declare

EK9 maintains a list of gated types — built-in types that represent access to system resources. Using one of these in a publishable package without declaring the corresponding capability is a compile error (E12010). The common gated types are:

Each gated type is declared as a fully-qualified name inside the capabilities list using the module path org.ek9.lang::. When a gated type has related helper types (for example, declaring TCP automatically permits TCPConnection and TCPHandler), you only need to declare the primary type. The rich error message for E12010 lists the groupings for each primary type.

Declaring capabilities in your package

The capabilities block sits inside defines package alongside version, license, and deps. For a publishable package that reads environment variables and makes HTTP calls: 

#!ek9
defines module com.example.httpclient

  defines package

    publicAccess <- true
    version <- 1.0.0-0
    description <- "HTTP client library"
    license <- "MIT"

    capabilities <- [
      "org.ek9.lang::TCP",
      "org.ek9.lang::HTTPRequest",
      "org.ek9.lang::HTTPResponse",
      "org.ek9.lang::EnvVars"
      ]

    deps <- {
      "org.ek9.collections": "1.0.0-0"
      }
//EOF

For a pure-computation package — a JSON parser, a string-manipulation library, a data-structure collection, a mathematical package — the simplest path is to omit the capabilities block entirely. Without that block the fourth publishable condition is not met, capability enforcement is not activated for the module, and the package compiles as it would have before the capability feature landed. This is the recommended pattern for any library that truly touches no system resources: 

#!ek9
defines module com.example.jsonparser

  defines package

    publicAccess <- true
    version <- 2.1.0-0
    description <- "Pure JSON parsing library"
    license <- "MIT"

    //No 'capabilities' block: pure computation, no gated types used.
    //Omitting the block is the current idiom for pure libraries.

    deps <- {
      "org.ek9.collections": "1.0.0-0"
      }
//EOF

As soon as you add a capabilities block — even with just one entry — the module enters the enforcement regime and every gated type it uses must be covered. This gives you explicit control over when a module transitions from "unenforced legacy" to "capability-governed" and avoids breaking any package that pre-dates the feature.

Capability enforcement is per-module

Each publishable package declares its own capabilities, and the compiler verifies that only the package's own code respects those declarations. Depending on another package does not inherit that package's capabilities, and it does not automatically propagate compile errors up the dependency chain when a dependency adds a new capability. Each module is a self-contained enforcement context: your module is responsible only for what your own code does.

This is a deliberate simplification. The compiler's job ends at the module boundary — "does this module's declared capabilities cover its own gated type usage?" is the only question it answers. Visibility into what your dependencies declare is a concern for the repository and your review workflow, not the compiler's static analysis. When you pull in a new version of a dependency you can inspect its package block (it's plain EK9 source, always legible) to see what capabilities it declares, and the EK9 repository tooling is designed to surface capability changes between versions so a human can review them at upgrade time.

The practical consequence: each of your published packages independently declares what its own code does. If a library you depend on adds a new capability in v2, your build does not break automatically. Instead, the repository's version-diff view highlights the change, and you decide whether to upgrade. This trades the strict compile-time tripwire for a simpler compiler and a review workflow that fits how developers actually evaluate dependency updates in practice.

Why this is worth the extra lines

The cost of capability security is a few extra lines in your package block and an occasional compile error when you start using a new gated type. The benefit is structural: it is no longer possible for a compromised dependency to silently add network access, read your secrets, or exfiltrate data without every consumer seeing a compile error first. A malicious package update that would silently own your build on every other platform cannot load on EK9 until a human reviews and acknowledges the new capability. That is the difference between "my build broke, what happened?" and "three weeks later we noticed data was being exfiltrated."

Most packages need zero capabilities. When a previously-pure package suddenly declares org.ek9.lang::TCP or org.ek9.lang::EnvVars, every consumer is alerted through compiler errors before the new version runs. That is the single largest supply chain defence improvement you can get without changing your development habits at all — you write the capability line once per publishable package, and the compiler handles the rest from that point forward.

The full design, rationale, and detailed examples of attack scenarios and their mitigations are documented in EK9_PACKAGE_CAPABILITY_SECURITY.md.

Summary

Hopefully you will have seen how you can easily progress from small programs in single EK9 source files, through to medium, large and very large suites of software. These start of simply; but can have dependencies added and can themselves be wrapped up and published. Each aspect of this just requires the progression of augmenting the package construct with a little more detail as your applications/projects grow for small simple programs through to large modules/applications.

Final Steps!

The final section is on the command line compiler tool.