Layered System - dynamic Java code processing

The dynamic layered system powers both the 'scc' command and the IntelliJ plugin. It provides a complete language processing environment, including parsing and model generation framework that supports robust Java emulation, incremental Java code-transformations with or without type resolution and code validation. Grammars support a variety of formats, including Java, StrataCode, HTML, XML, and CSS. At the API level, the LayeredSystem has a srcPath, an external classPath, a system classPath, and a classLoader.

The srcPath specifies directories which are scanned for source files using languages, and file-processors registered for specific extensions, paths, and patterns. Languages parse the files, build the "abstract syntax tree", then use a multi-step process: init, start, validate, process, transform to produce the output file. File processors, just copy the most specific version of the file from the source path to the build directory.

The externalClassPath and classPath both contain compiled Java jar and work like Java's normal classpath. The classPath classes are actually loaded using a ClassLoader but the externalClassPath types are loaded by parsing the .class files. If you mix them, files in the Java classPath cannot depend on files in the external classPath.

A single consistent set of APIs exist for accessing and manipulating types whether they are in source form, or compiled form see javadoc for the classes: ModelUtil, LayeredSystem, ParseUtil, Language, Parser.

Layers

You can use StrataCode as a development tool from code by constructing a LayeredSystem and using the APIs (See the Simple Parser example) but it's much more powerful when you use it to manage source and class files using layers. The LayeredSystem has a layerPath which specifies the list of directories to search for a given layer name. Layers are hierarchical path names, and layer definition files are named with the file name = the directory name. For example, "a.b" refers to the layer definition file a/b/b.sc in your layerPath.

The layered system supports a set of inactive layers, and an optional 'type index' for all layers it finds in the layer path. It also supports a set of active layers, those that you are building and running (e.g. as specified on the scc command line or through an IDE run configuration).

Layer definition files

Layers are defined with layer definition files which are coded in interpreted StrataCode (a superset of Java), not XML or Groovy like most build systems. There is one syntax and toolset you can use to manage all aspects of your development. You have the benefits of static typing and IDE support to build and validate your entire project at once.

Languages and file processors

Layer definition files can add new file types to be processed of two different types: languages, and file processors. Languages add new types of files that are parsed and transformed during the build. File processors are applied in layer order to merge the directory trees found in layers into the one build directory, letting you build up any resulting project by picking the "latest" file. By defining a new file in a new layer, you can override a file in the previous directory tree when generating the build directory.

When you need to evaluate variables or insert or modify a section of a file in a layer snippets, you can define a new language for your format based on the TemplateLanguage. StrataCode defines scxml, schtml, scsh, etc. New formats you can use to generate xml, html, and sh files. You can build one of these formats from a couple of lines of code in your layer definition file, or by customizing the TemplateLanguage grammar and model objects like HTMLLanguage in a few hundred to a couple of thousand lines of code.

Layer srcPath

Layer's can add srcPath entries, directory trees that are used to find src files. The default srcPath includes the layer directory itself. RepositoryPackage's which define new src modules can add additional directories to a layer's srcPath.

There are two types of source files in a Layer - those which prefix the package prefix of the layer and those that do not. If the generated file needs to be put into a package specific directory, like a .class file, that language or file processor will prepend the package. If it's a file in the document root, it will not prepend the package.

Mixing two styles of source organization

Layer can be used with a typical java project directory organization (i.e. src/main/java/...), but also offer new ways of organizing files to eliminate unnecessary directories and organize modules based on features.

Most Java projects are known for having lots of nearly empty directories. When you set a packagePrefix prefix on a layer, the layer directory contains files already in that package. This makes source trees easier to navigate.

Most Java projects use a heavily type-specific organization of assets - e.g. separating java from component and build configuration, and document root files. Even files that are part of the same feature are no where near each other. This also increases the overhead of creating new modules so Java modules tend to be more coarse grained.

Locality of reference - keeping all assets of a feature together - however is a great way to make code easier to navigate and read, especially when you are extending or patching an existing system but often long-term as well. StrataCode's ability to distribute files that are in the same directory to different remote directories based on their type, let's you take maximum advantage of locality of reference without compromising security or control in the production system. It's ability to merge directories and later merge layers, let's you evolve code in a way that keeps it separate while it's in development, and easily merged later when it's "part of the core" and no longer changing.

Active layers

To run a StrataCode application, you specify a set of active layers to either the scc command or an IntelliJ run configuration. All of the dependent layers (those specified in the 'extends' clause of the layer) are pulled in to create a single stack of layers that's ordered based on dependencies. Layers can define their own classes and object instances using Java's 'class' and StrataCode's 'object' operators, called layer components. You can layer components for "build time" logic including definitions of file processors, languages, or repositories containing assets of information stored in git or maven. These are called layer components. They can also modify the classes or objects of any layers they extend. For example, you might change the version of a dependency in some base layer, disable a component, or replace it entirely by redefining a new class or instance to replace it.

Changes to existing practices

With most build systems, you start a new project by copying the project boilerplate or running a new project wizard. With StrataCode, you simply extend the base layer, set any required properties and override any features you want to change.

With most build systems, you are forced to specify version numbers of packages you don't care about, just so you have control to customize them should you need to. With StrataCode, the default is to inherit the base package's version so you remove redundancy right there. If you have dependent layers which both set a version, the layer order is automatically defined so you get the right version. If you have unrelated layers which both set a version, the order you choose for them defines the dependency order. That typically just requires adding a new dependency to establish precedence of packages.

Build/run steps

Here are the high-level steps involved in building/running a StrataCode application:

  • All layers in the layer stack are created
  • All init methods are run in stack order. Any process or runtime constraints are defined here. Layers can include or exclude themselves from specific processes and runtimes.
  • If there's more than one runtime/process, the stack is split into separate stacks and those new stacks are created and initialized.
  • All layer start methods are run. At this point, the srcPath, and classPath for the layer should be defined.
  • All layer validate methods are run. One more step for additional error checking that's dependent on having all types resolveable. Layers should ensure their basic requirements are satisfied and might look for any incompatible layersor configuration to be sure this is a valid stack.

You can mix and match two code styles in your layer definition files: component based, or using init, start, validate methods and the lower level APIs.

Layer components

Layer components are objects defined inside of the layer definition file. They are created and merged together when the layer stack is created. By the time the start method runs, the layer components will have been initialized and merged so the last layer's change to each property will be reflected in the component. It's a very simple and powerful way to define build properties that can be easily overridden downstream. Using the @BuildInit annotation you can even inject them into the generated code.

Layer components can be basic objects to hold information, or they can extend components specifically designed to augment the LayeredSystem.

Main layer components include: LayerFileProcessor, Language, RepositoryPackage, MvnRepositoryPackage.

With RepositoryPackage, you define the url of a package directory. You can set srcPaths, classPaths, webPaths, testPaths, variables to extend the system.

With MvnRepositoryPackage, you can use type = 'git-mvn'; and set the url to point to a git repository. That will define a source package using the maven pom.xml in the git directory root to build the package's dependencies.

If your package directory is more complex, you can nest RepositoryPackage components. For example you can use a RepositoryPackage to retrieve a source dir, and a MvnReposioryPackage inside to add a maven artifact from a sub-directory of the git package.

Defining init, start methods

Your Layer objects are being created and executed using the same dynamic code engine as with dynamic objects and layers. That means that you have the full power of defining Java classes, extending libraries etc. that you have with regular Java. In general, we like to keep these files as declarative as possible so the layer component method is preferred. But layer definition files are essentially interpreted Java that's run to generate your project's build and run configuration.

To use this power, just add code to your Layer's class constructor, or it's init, start, or validate methods as needed. When you use layer components, under the hood it's init, start, and validate methods are being called during those lifecycle phases of the layer so it's easy to mix, match, and debug, both styles.

Layer fields

Layers can add fields to store properties which can be used during the build/run cycle. Instead of duplicating a version number in a number of mvn or git URLs, define a new field and build those URLs on the fly.

The "scc" Command

The scc command takes a list of layers, finds any changed source files, generates new code or copies files into the layer's build directory. It then compiles changed java files, and runs main methods and starts processes, or opens urls configure in the layers you specify. Frameworks also generate standalone shell scripts that can be run without the StrataCode runtime environment. The goal of the scc command is to be an easy way to validate, build and run any configuration of layers through one interface.

StrataCode maintains an index of all generated code so it can validate itself and clean up after itself from run to run and detect when generated files have been modified to avoid losing changes.

When you provide the scc command the layer or set of layers you want to run, it includes all dependent layers automatically and validates the configuration. If you run "scc a b c" it applies the layers in that order - so c is the last layer. This is called "the build layer". The results are by default stored in the last layer's build or dynbuild directories depending on whether the last layer is compiled or dynamic.

If your layers are stored under different root directories, set the layerPath to include each root using the -lp option. When searching for a layer directory, the directories in the layer path are searched in order like Java's classpath. You can then use layer names relative to the directory in the layer path.

The layer path cannot contain zip or jar files currently. They must be expanded directories.

External repository integrations

Repositories in StrataCode store code in source or compiled form that is maintained in external systems. StrataCode provides several repository implementations: git, maven, scp, and a generic URL download over http or ftp. See: IRepositoryManager,

To learn more about the StrataCode apis, you can read the javadoc for the IRepositoryManager interface and the classes which implement that interface: AbstractRepositoryManager, GitRepositoryManager, MvnRepositoryManager, etc. RepositoryManagers are registered in the LayeredSystem with a type, e.g. 'git', 'mvn'. This type corresponds to the type property in the RepositoryPackage.

Your layer definition files can either use the repository apis directly to add new packages, or you can define instance of these repository components in your layer using the object tag. When you use layer components, you can customize those package definitions easily in subsequent layers by simplying overriding properties of that component.

Here are some example layer definition files that add repositories using the component style:

A repository which adds a compiled maven package:

file: log4j/core/core.sc 
log4j.core extends sys.std {
   compiledOnly = true;
   hidden = true;

   codeType = sc.layer.CodeType.Framework;

   // Log4j complains if we include two incompatible implementation classes and so higher level 
   // packages need to define the impl library.  Here we only pull in the api.

   object slf4jApiPkg extends MvnRepositoryPackage {
      url = "mvn://org.slf4j/slf4j-api/1.7.25";
   }

   // Supplying a default log4j - this can be overridden if frameworks need a specific version (e.g. jetty/lib)
   object log4jPkg extends MvnRepositoryPackage {
      url = "mvn://org.apache.logging.log4j/log4j-core/2.15.0-rc1";
   }

   // The resourceFileProcessor layer component is defined in sys.std 
   // and defines standard Java rules for copying src files into the 
   // classpath so they can be loaded  as Java resources.  
   // Some layers treat properties as config files
   // which are put into the build-dir (with some config prefix).  Here
   // we ensure the log4j.properties file is treated as a resource, 
   // by modifying that instance and adding a new file pattern.
   resourceFileProcessor {
      {
         // Treat this file as a resource so it goes in the classpath
         addPatterns("log4j\\.properties", "log4j2\\.properties");
         // TODO: add log4j2.xml and .json here?
      }
   }

   // This method is run when the layer is initialized.  
   public void init() {
      // log4j is automatically excluded from these runtimes
      excludeRuntimes("js", "android", "gwt");
   }
}
Referencing the module as source stored in 'git':
file: jool/lib/lib.sc 
jool.lib {
   // With Java8 this was in the standard rt but it moved in Java 11
   object annotationPkg extends MvnRepositoryPackage {
     url = "mvn://javax.annotation/javax.annotation-api/1.3.2";
   }
   object joolPkg extends MvnRepositoryPackage {
      packageName = "org.jooq/jool";
      type = "git-mvn";
      // problems with auth using this URL
      //url = "git@github.com:jOOQ/jOOL.git";
      url = "https://github.com/jOOQ/jOOL.git#4fa207f6a4cae65b2b0c79c70b0662a068c69ce3";

      srcPaths = {"src/main/java"};
   }
}
Define a git package which downloads a maven project where the maven module we pull in is a sub-directory of the git directory:
file: ticketmonster/core/core.sc 
ticketmonster.core {
   object ticketMonsterPkg extends RepositoryPackage {
      packageName = "ticketmonster";
      type = "git";
      url = "https://github.com/jboss-developer/ticket-monster.git";

      // No src for this package - only subpackages
      definesSrc = false;
      definesClasses = false;

      // Defining the sub-package as a relative path inside of the above
      // ticket-monster main.  This is where the maven definition lives.
      object demoPkg extends MvnRepositoryPackage {
         url = "demo"; 

         srcPaths = {"src/main/java", "src/main/resources"};
         webPaths = {"src/main/webapp"};

         // The POM file assumes we have installed JBoss EAP so marks 
         // dependencies as 'provided'.  We'll just suck them in here but
         // perhaps they should be included in some base layer we extend?
         includeProvided = true;

         // Need to use the jboss repositories defined in this POM
         useRepositories = true;
      }
   }
}

Dynamic and compiled layers

A layer becomes dynamic when you prefix the layer's name with the dynamic attribute in its definition file or use the -dyn option before the layer name on the command line. A layer which extends another dynamic layer either directly or indirectly is also by definition dynamic. As your system system evolves, you typically make layers dynamic during development, then remove the dynamic keyword and compile them into the core for deployment.

Using the -dyn option helps you temporarily run the system with one or more layers as dynamic.

Running StrataCode applications with the interpreter

To run any StrataCode application, just run scc with the relative path name of the layers involved. You set the -lp (layer path) option to include the layer root directories. StrataCode searches for the first layer definition file which matches the layer name you provide. It then figures out what needs to be compiled, compiles it if necessary and then runs the main. This usually takes place all in one process - the same Java virtual machine. Any layer names you specify after -dyn option are treated as 'dynamic layers', even if they do not specify the 'dynamic' keyword. Layers they extend are also treated as dynamic, unless they specify compiledOnly=true in their layer definition file. This is a quick way to run a feature in dynamic mode when you intend to make live customizations to the code. You can change the source files, and do 'cmd.refresh();' or run with the program editor and press refresh there.

The command-line interpreter runs in two different modes: script, and edit. Use edit mode when you are building a program and script mode when you do not intend to change the existing program, but instead need to run a test script or look at property values without staging a change to the underlying file. In script mode, if you add a field or a method it's still staging a change to the underlying file because ultimately that representation is required to apply that kind of change. But for both edit and script modes, use "cmd.save()" to actually save changes (and be careful because it's possible to override parts of your actual source code that way by mistake).

If you have a client-server application, the command interpreter offers features to run code on the client, server or both by targeting a specific runtime with a command. Use the cmd.scopeContextName in conjunction with the scopeContextName query parameter (available in test-mode only) to target a specific browser window with commands.

The StrataCode Classpath

When you start the scc command, you specify a classpath used to find precompiled classes. These core classes can be used by any of the layers using Java's normal import mechanism. You can't use the modify operator with these classes except to set annotations in explicitly designated annotation layers.

Layers can add to the system classpath to include any classes they require for their runtime.

If you use the @MainSettings annotation to specify a run script for your applications main method(s), or extend a framework that does, StrataCode will put the classpath into the generated run scripts for your application. When your application consists of only compiled layers, you can run with only the core StrataCode runtime, only a small amount of code needed when you use data binding etc. In this environment StrataCode generates the run script to run your standalone application.

Multiple processes and runtimes

Complex system development requires building systems which span more than one process, for example the client and server parts of a tightly coupled application or different services in a micro-services framework. They also span more than one runtime environment - e.g. Javascript in the browser, Java on the server, Android, or Java Desktop.

StrataCode layers let you make the process of developing in such environments relatively seamless. You can use the scc command to compile and run all processes in your application in one step. Each layer of code in your application specifies the framework layers it depends upon. That organization naturally helps StrataCode partition the layers into separate stacks, one for each process. Sometimes an application layer needs to declare whether it is included or excluded from a process or runtime to create the proper stacking of the layers.

StrataCode takes care of building and running all processes to ease development workflows, all from a single command: scc list-of-layers .

Command line and multiple processes/runtimes

The command interpreter by default will run all statements on all applicable runtimes or processes. When a method or property only exists in one runtime or process, it's only sent to that runtime/process. You can set the cmd.targetRuntime property to specify a specific runtime or process name for subsequent commands.

StrataCode initialization

If you are customizing a StrataCode framework, the low-level details will be helpful:

The scc process goes through the following initialization phases:

  • Finds all layer definition files, and the extended layers computes the list of initial system layers
  • Sorts the list in dependency order - detects and flags cycles as errors. When there are no dependencies between two layers, compiled layers are put in front of dynamic layers, and framework layers are put ahead of application layers.
  • Constructs each layer object and call the initialize method in each Layer.
  • Computes the runtimes required for this list of layers (e.g. java only, js only, or java and js)
  • Finds the first build layer in the stack for the first runtime and starts all layers before the build layer.
  • When starting a layer the layer first adds any repository packages it depends on. Those packages are installed if necessary via git, http, or maven, gradle, etc.
  • The layer then updates its library classpath to include pre-compiled jars it depends on. The library classpath is added to the system immediately after the layer is started.
  • Once all layers prior to the build layer are started, the build layer computes the set of top-level directories to prepare (if necessary as in android) and then process. The pre-process phase is used to generate any .java files that need to exist for the main Java files to be parsed and compiled (e.g. android's resource files).
  • For each required phase the .dep file for that phase is used to direct the incremental compile. This file contains a cache of all source files, the generated files made from those source files, and any dependencies detected in the code on other types. If this file is not found, the global BuildInfo is not found, or the set of layers used to do the last build has changed, a full build is performed.
  • Any files that have changed since the last build are re-parsed and the .dep files are updated with new dependencies.
  • For StrataCode files or Java files that are pre-processed, the new Java file is generated into the buildSrc directory - (e.g. buildLayer/build/java or buildLayer/build/js)
  • All changed files are compiled by javac or the internal java compiler.
  • Any processing specific to the current runtime is performed - e.g. generating Javascript files from the Java
  • This process is repeated for each build layer in each runtime until the final build layer in each runtime reached. Once each buildLayer is compiled, it's buildDir is added to the system classpath in front of any previous build layers. SC only loads the compiled version for a class once it's determined that class will no longer be modified in a subsequent build layer. That way, it is ensured it always has the right compiled version of each class.
  • If any errors have occurred scc exits (or optionally prompts to retry the build to speed up the fix-recompile process)
  • Upon a successful build, scc starts the interpreter and any Main methods and postBuild commands that were registered in the framework layers. If you use the -c option, it compiles but does not run your application. It's easy to have StrataCode generate a shell script and the requisite jar file to run and deploy your app as a standard Java application.

Options

You can force all files to be compiled with the -a option or compile a specific file with the -f option.

The -cp option sets the classpath StrataCode uses to find binary descriptions of classes.

The -lp option specifies the layer path.

The -v family of options: -vs, -vsa, -vb, -vba, -vh, -vha turn on verbose for sync, data binding, and html respectively.

By default, scc runs all main classes defined in framework layers, after compiling by loading the generated classes into the same VM. The -rs option execs the scripts generated by the main commands. The -r pattern option lets you only run main methods matching a specific pattern. Each of the -r options consumes the remaining options and provides them to the executed command. If you use -nr it suppresses running of main.

If you use the -nw option, it suppresses the opening of the default browser window for a Javascript application.

The -t option is like -r but runs tests. You can specify an individual class name, or a pattern matching all test classes (e.g. -t .* to run all tests or just use -ta)

More information can be found in the Javadoc for the LayeredSystem class or with scc -help.

Layer Properties

When building framework layers, the starting point for customization is the layer definition file. The layer definition file has logic that would ordinarily be in an ant file or buried in some file your IDE maintains per project. Because framework layers are shared by application layers, you do not have to specify this information for each project. And yet all options can be adjusted by downstream layers so they are always configurable as necessary.

The most common thing framework layers do is to add compiled class files to the classpath. Classes added this way cannot yet be modified with the modify operation but are available otherwise as normal StrataCode types.

Layer definition files also can import classes which are then visible without explicit imports in downstream code (unless a layer along the chain sets "inheritImports=false" to turn that off).

More options:

buildDir

Specifies the buildDirectory where .class files go. By default, it is the build directory in the layer's folder itself. This way, the layer is completely self contained. Clearly the build directory itself cannot contain source.

buildSrcDir

Specifies the directory where generated .java files go.

buildLayer

Any layer which has a build directory is considered to have buildLayer=true by default. Any layer without a build directory has buildLayer=false unless set explicitly in the layer definition file, or built explicitly (scc [-c] layerName). Any buildLayer=true layer compiles all files that have changed since any previous buildLayer. In other words, buildLayers are themselves chained, the latter inheriting the contents of the former. This lets you share compiled versions of some layers with subsequent layers so you can optimize what gets built when you change a given layer.

buildSeparate

By default, .sc files in a layer cannot be used as compiled classes in the layer definition file. Occasionally you need to do this in a layer. If you mark that layer with buildSeparate, it is compiled and its classes are put into the classpath of the system before subsequent layer def files are processed allowing them to find these classes. This means that you cannot replace those classes though with the modify operation. Structurally StrataCode assumes you cannot reload a .java .class file since, though technically some JVMs do allow limited versions of class reloading

classPath

Java jar files or directories containing .class files in package organization. The list of jars/directories you add is searched in order to find classes available to code in this layer or any downstream layers. Once a layer's classpath has been searched, it looks for any classes defined in layers that layer extends. Thus a downstream layer can override a compiled class in an upstream layer, but only if that layer and class file are in place the first time that class is accessed. StrataCode does not unload any class files, except as normally garbage collected by the JVM, so a restart is required when you add a new layer with compiled classes in it that override classes that have already been loaded.

exportImports

Default is true.

Should the imports in the layer file be visible to layers which extend this layer

inheritImports

Default is true.

Should this layer use the imports from any extended layers.

inheritPackage

Default is true.

Should this layer use the package prefix of its extend layers? If there are more than one conflicting package prefix in the list of extend layers, the last one wins.

copyPlainJavaFiles

Default is true.

Any Java files do not use any extensions are not modified. They can either be compiled from the source directory or copied into the build directory. Doing the later ensures you have a single folder containing the entire source for your project, nice if you are editing the Java files with an IDE.

excludedFiles

A list of Java regular expressions for files to ignore during the build process. By default:

   public List<string> excludedFiles = Arrays.asList("build", "out", ".*.sctd", ".git");

StrataCode will ordinarily ignore files which are not in a language it understands. The "sctd" files are in this list because it currently does not compile those files, they are interpreted instead. That will be removed once they are compiled like normal StrataCode/Java files.

compiledOnly

Indicates this layer does not support dynamic mode. Any attempt to make it dynamic is ignored and all of the files are compiled anyway. You can turn off dynamic mode on a per class basis using the CompilerSettings annotation.

transparent

When you mark a layer as transparent, it only affects how the editor displays the types for that layer. It does not affect the runtime behavior of the application. A transparent layer shows all objects and properties defined in the layers which the transparent layer extends. It includes properties in other transparent layers found along the way and so works recursively.

useGlobalImports

By default a layer only sees the imports which are defined by layers which it extends, either directly or indirectly. An import defined in a totally independent layer would not be accessible. There are some cases however where you might want to define a global layer that can see all base layers without having to extend a bunch of layers explicitly. For example, the temporary layer created with the -i option can see and modify any type. For these layers, set useGlobalImports=true so that they can see all of the imports as well.

codeType

A metadata property stored in the layer for use by the editor. Lets you tag layers as having certain types of code assets (e.g. domain model, style, or UI only). These properties also help sort layers so framework layers do not get intermixed with application layers for example.