GWT

StrataCode Build and Packaging System

Layers make a great way to manage build and run configurations. The stack of layers defines a way to run code in the right order to assemble your source path, and classpath. You can use components in your layers to work more declaratively and override settings in a base-layer just as you can properties in your application classes.

With layers as your build and packaging system, you can:

Any new build system needs rich features to be sufficient for this task. StrataCode can:

Layered System - Dynamic Java Code Processing

The dynamic layered system powers both the 'scc' command and the IntelliJ plugin. It provides a complete language processing environment, including parsing and model generation framework that supports robust Java emulation, incremental Java code-transformations with or without type resolution and code validation. Grammars support a variety of formats, including Java, StrataCode, HTML, XML, and CSS. At the API level, the LayeredSystem has a srcPath, an external classPath, a system classPath, and a classLoader.

The srcPath specifies directories which are scanned for source files using languages, and file-processors registered for specific extensions, paths, and patterns. Languages parse the files, build the "abstract syntax tree", then use a multi-step process: init, start, validate, process, transform to produce the output file. File processors, just copy the most specific version of the file from the source path to the build directory.

The externalClassPath and classPath both contain compiled Java jar and work like Java's normal classpath. The classPath classes are actually loaded using a ClassLoader but the externalClassPath types are loaded by parsing the .class files. If you mix them, files in the Java classPath cannot depend on files in the external classPath.

A single consistent set of APIs exist for accessing and manipulating types whether they are in source form, or compiled form see javadoc for the classes: ModelUtil, LayeredSystem, ParseUtil, Language, Parser.

Layers

You can use StrataCode as a development tool from code by constructing a LayeredSystem and using the APIs but it's much more powerful when manage source and class files using layers. The LayeredSystem has a layerPath which specifies the list of directories to search for a given layer name. Layers are hierarchical path names, and layer definition files are named with the file name = the directory name. For example, "a.b" refers to the layer definition file a/b/b.sc in your layerPath.

The layered system supports a set of inactive layers by indexing all layers, and all files in your layer path. It also supports a set of active layers, those that you are building and running (e.g. as specified on the scc command line or through an IDE run configuration).

Layer Definition Files

Layers are defined with layer definition files which are coded in interpreted StrataCode (a superset of Java), not XML or Groovy like most build systems. There is one syntax and toolset you can use to manage all aspects of your development. You have the benefits of static typing and IDE support to build and validate your entire project at once.

Languages and File Processors

Layer definition files can add new file types to be processed of two different types: languages, and file processors. Languages add new types of files that are parsed and transformed during the build. File processors are applied in layer order to merge the directory trees found in layers into the one build directory, letting you build up any resulting project by picking the "latest" file. By defining a new file in a new layer, you can override a file in the previous directory tree when generating the build directory.

When you need to evaluate variables or insert or modify a section of a file in a layer snippets, you can define a new language for your format based on the TemplateLanguage]. StrataCode defines scxml, schtml, scsh, etc. New formats you can use to generate xml, html, and sh files. You can build one of these formats from a couple of lines of code in your layer definition file, or by customizing the TemplateLanguage grammar and model objects like HTMLLanguage in a few hundred to a couple of thousand lines of code.

Layer srcPath

Layer's can add srcPath entries, directory trees that are used to find src files. The default srcPath includes the layer directory itself. RepositoryPackage's which define new src modules can add additional directories to a layer's srcPath.

There are two types of source files in a Layer - those which prefix the package prefix of the layer and those that do not. If the generated file needs to be put into a package specific directory, like a .class file, that language or file processor will prepend the package. If it's a file in the document root, it will not prepend the package.

Mixing Two Styles of Source Organization

Layer can be used with standard projects directory trees, but also offer new ways of organizing files to elimiate extra directories and organize modules based on features.

Most Java projects are known for having lots of nearly empty directories. The packagePrefix prefix can be used to eliminate many of them, making directories easier to navigate.

Most Java projects suffer from overly using type-specific organization of assets. This is where you break apart the storage of a file so you keep "likes with likes". The classic overuse of type-specific organization is the windows registry which makes it very hard to find and manage assets for a particular application.

In Java projects, build assets, configuration, document root files, and java files are no where near each other, even if they implement the same feature. You could create a new module for a new feature but features commonly cut across types so this requires more code. The more types you have, the more complex the contracts you make visible to clients. So we largely accept coarse grained modules and give up locality of reference for features.

Locality of reference however is a great way to make code easier to navigate and read, especially when you are extending or patching an existing system but often long-term as well. StrataCode's ability to distribute files that are in the same directory to different remote directories based on their type, let's you take maximum advantage of locality of reference without compromising security or control in the production system. It's ability to merge directories and later merge layers, let's you evolve code in a way that keeps it separate while it's in development, and easily merged later when it's "part of the core" and no longer changing.

Active Layers

To run a StrataCode application, you specify a set of active layers to either the scc command or an IntelliJ run configuration. All of the dependent layers (those specified in the 'extends' clause of the layer) are pulled in to create a single stack of layers that's ordered based on dependencies. Layers can define their own classes and object instances using Java's 'class' and StrataCode's 'object' operators, called layer components. You can layer components for "build time" logic including definitions of file processors, languages, or repositories containing assets of information stored in git or maven. These are called layer components. They can also modify the classes or objects of any layers they extend. For example, you might change the version of a dependency in some base layer, disable a component, or replace it entirely by redefining a new class or instance to replace it.

Changes to Existing Practices

With most build systems, you start a new project by copying the project boilerplate or running a new project wizard. With StrataCode, you simply extend the base layer, set any required properties and override any features you want to change.

With most build systems, you are forced to specify version numbers of packages you don't care about, just so you have control to customize them should you need to. With StrataCode, the default is to inherit the base package's version so you remove redundancy right there. If you have dependent layers which both set a version, the layer order is automatically defined so you get the right version. If you have unrelated layers which both set a version, the order you choose for them defines the dependency order. That typically just requires adding a new dependency to establish precedence of packages.

Build/Run Steps

Here are the high-level steps to building/running a StrataCode application:

You can mix and match two code styles in your layer definition files: component based, or using init, start, validate methods and the lower level APIs.

Layer Components

You define layer components using the object keyword using any class. Main layer components include: LayerFileProcessor, Language, RepositoryPackage, MvnRepositoryPackage.

With RepositoryPackage, you define the url of a package directory. You can set srcPaths, classPaths, webPaths, testPaths, variables to extend the system.

With MvnRepositoryPackage, you can use type = 'git-mvn'; and set the url to point to a git repository. That will define a source package using the maven pom.xml in the git directory root to build the package's dependencies.

If your package directory is more complex, you can nest RepositoryPackage components. For example you can use a RepositoryPackage to retrieve a source dir, and a MvnReposioryPackage inside to add a maven artifact from a sub-directory of the git package.

Init, Start Methods

Your Layer objects are being created and executed using the same dynamic code engine as with dynamic objects and layers. That means that you have the full power of defining Java classes, extending libraries etc. that you have with regular Java. In general, we like to keep these files as declarative as possible so the layer component method is preferred. But layer definition files are essentially interpreted Java that's run to generate your project's build and run configuration.

To use this power, just add code to your Layer's class constructor, or it's init, start, or validate methods as needed. When you use layer components, under the hood it's init, start, and validate methods are being called during those lifecycle phases of the layer so it's easy to mix, match, and debug, both styles.

Layer Fields

Layers can add fields to store properties which can be used during the build/run cycle. Instead of duplicating a verison number in a number of mvn or git URLs, define a new field and build those URLs on the fly.

The "scc" Command

The scc command takes a list of layers, finds any changed source files, generates new code or copies files into the layer's build directory. It then compiles changed java files, and runs main methods and starts processes, or opens urls configure in the layers you specify. Frameworks also generate standalone shell scripts that can be run without the StrataCode runtime environment. The goal of the scc command is to be an easy way to validate, build and run any configuration of layers through one interface.

StrataCode maintains an index of all generated code so it can validate itself and clean up after itself from run to run and detect when generated files have been modified to avoid losing changes.

When you provide the scc command the layer or set of layers you want to run, it includes all dependent layers automatically and validates the configuration. If you run "scc a b c" it applies the layers in that order - so c is the last layer. This is called "the build layer". The results are by default stored in the last layer's build or dynbuild directories depending on whether the last layer is compiled or dynamic.

If your layers are stored under different root directories, set the layerPath to include each root using the -lp option. When searching for a layer directory, the directories in the layer path are searched in order like Java's classpath. You can then use layer names relative to the directory in the layer path.

The layer path cannot contain zip or jar files currently. They must be expanded directories.

External Repository Integrations

Repositories in StrataCode store code in source or compiled form that is maintained in external systems. StrataCode provides several repository implementations: git, maven, scp, and a generic URL download over http or ftp. See: IRepositoryManager,

To learn more about the StrataCode apis, you can read the javadoc for the IRepositoryManager interface and the classes which implement that interface: AbstractRepositoryManager, GitRepositoryManager, MvnRepositoryManager, etc. RepositoryManagers are registered in the LayeredSystem with a type, e.g. 'git', 'mvn'. This type corresponds to the type property in the RepositoryPackage.

Your layer definition files can either use the repository apis directly to add new packages, or you can define instance of these repository components in your layer using the object tag. When you use layer components, you can customize those package definitions easily in subsequent layers by simplying overriding properties of that component.

Here are some example layer definition files that add repositories using the component style:

A repository which adds a compiled maven package:

file: log4j/log4j.sc
// Pull in the log4 layer
log4j extends sys.std {
   compiledOnly = true;

   // Log4j complains if we include two incompatible implementation classes and so higher level 
   // packages need to define the impl library.  Here we only pull in the api.

   object slf4jApiPkg extends MvnRepositoryPackage {
      url = "mvn://org.slf4j/slf4j-api/1.7.0";
   }

   // The resourceFileProcessor layer component is defined in sys.std 
   // and defines standard Java rules for copying src files into the 
   // classpath so they can be loaded  as Java resources.  
   // Some layers treat properties as config files
   // which are put into the build-dir (with some config prefix).  Here
   // we ensure the log4j.properties file is treated as a resource, 
   // by modifying that instance and adding a new file pattern.
   resourceFileProcessor {
      {
         // Treat this file as a resource so it goes in the classpath
         addPatterns("log4j\\.properties");
      }
   }

   // This method is run when the layer is initialized.  
   public void init() {
      // log4j is automatically excluded from these runtimes
      excludeRuntimes("js", "android", "gwt");
   }
}
Referencing the module as source stored in 'git':
file: jool/lib/lib.sc
jool.lib {
   object joolPkg extends MvnRepositoryPackage {
      packageName = "org.jooq/jool";
      type = "git-mvn";
      url = "git@github.com:jOOQ/jOOL.git";

      srcPaths = {"src/main/java"};
   }
}
Define a git package which downloads a maven project where the maven module we pull in is a sub-directory of the git directory:
file: ticketmonster/core/core.sc
ticketmonster.core {
   object ticketMonsterPkg extends RepositoryPackage {
      packageName = "ticketmonster";
      type = "git";
      url = "git@github.com:jboss-developer/ticket-monster.git";

      // No src for this package - only subpackages
      definesSrc = false;
      definesClasses = false;

      object demoPkg extends MvnRepositoryPackage {
         url = "demo"; 

         srcPaths = {"src/main/java", "src/main/resources"};
         webPaths = {"src/main/webapp"};

         // The POM file assumes we have installed JBoss EAP so marks 
         // dependencies as 'provided'.  We'll just suck them in here but
         // perhaps they should be included in some base layer we extend?
         includeProvided = true;

         // Need to use the jboss repositories defined in this POM
         useRepositories = true;
      }
   }
}

Dynamic and Compiled Layers

A layer becomes dynamic when you prefix the layer's name with the dynamic attribute in its definition file or use the -dyn option before the layer name on the command line. A layer which extends another dynamic layer either directly or indirectly is also by definition dynamic. As your system system evolves, you typically make layers dynamic during development, then remove the dynamic keyword and compile them into the core for deployment.

Using the -dyn option helps you temporarily run the system with one or more layers as dynamic.

Running StrataCode Applications With the Interpreter

To run any StrataCode application, just run scc with the relative path name of the layers involved. You set the -lp (layer path) option to include the layer root directories. StrataCode searches for the first layer definition file which matches the layer name you provide. It then figures out what needs to be compiled, compiles it if necessary and then runs the main all in the same JVM. If you specify the -dyn option, any layer names you provide are treated as dynamic layers even if they do not have the dynamic keyword.

Be careful when you save changes using either the editor or the command line interpreter. It's possible that you've just changed source files in your layers folder. If you run scc with the -i option, by default all changes go into a new temporary layer, a safe sandbox for making any change.

The StrataCode Classpath

When you start the scc command, you specify a classpath used to find precompiled classes. These core classes can be used by any of the layers using Java's normal import mechanism. You can't use the modify operator with these classes except to set annotations in explicitly designated annotation layers.

Layers can add to the system classpath to include any classes they require for their runtime.

If you use the @MainSettings annotation to specify a run script for your applications main method(s), or extend a framework that does, StrataCode will put the classpath into the generated run scripts for your application. When your application consists of only compiled layers, you can run with only the core StrataCode runtime, only a small amount of code needed when you use data binding etc. In this environment StrataCode generates the run script to run your standalone application.

Multiple Processes and Runtimes

Complex system development requires building systems which span more than one process, for example the client and server parts of a tightly coupled application or different services in a micro-services framework. They also span more than one runtime environment - e.g. Javascript in the browser, Java on the server, Android, or Java Desktop.

StrataCode layers let you make the process of developing in such environments relatively seamless. You can use the scc command to compile and run all processes in your application in one step. Each layer of code in your application specifies the framework layers it depends upon. That organization naturally helps StrataCode partition the layers into separate stacks, one for each process. Sometimes an application layer needs to declare whether it is included or excluded from a process or runtime to create the proper stacking of the layers.

StrataCode takes care of building and running all processes to ease development workflows, all from a single command: scc list-of-layers .

StrataCode Initialization

If you are customizing a StrataCode framework, the low-level details will be helpful:

The scc process goes through the following initialization phases:

Options

You can force all files to be compiled with the -a option or compile a specific file with the -f option.

The -cp option sets the classpath StrataCode uses to find binary descriptions of classes.

The -lp option specifies the layer path.

The -v family of options: -vs, -vsa, -vb, -vba, -vh, -vha turn on verbose for sync, data binding, and html respectively.

By default, scc runs all main classes defined in framework layers, after compiling by loading the generated classes into the same VM. The -rs option execs the scripts generated by the main commands. The -r pattern option lets you only run main methods matching a specific pattern. Each of the -r options consumes the remaining options and provides them to the executed command. If you use -nr it suppresses running of main.

If you use the -nw option, it suppresses the opening of the default browser window for a Javascript application.

The -t option is like -r but runs tests. You can specify an individual class name, or a pattern matching all test classes (e.g. -t .* to run all tests or just use -ta)

More information can be found in the Javadoc for the LayeredSystem class or with scc -help.

Layer Properties

When building framework layers, the starting point for customization is the layer definition file. The layer definition file has logic that would ordinarily be in an ant file or buried in some file your IDE maintains per project. Because framework layers are shared by application layers, you do not have to specify this information for each project. And yet all options can be adjusted by downstream layers so they are always configurable as necessary.

The most common thing framework layers do is to add compiled class files to the classpath. Classes added this way cannot yet be modified with the modify operation but are available otherwise as normal StrataCode types.

Layer definition files also can import classes which are then visible without explicit imports in downstream code (unless a layer along the chain sets "inheritImports=false" to turn that off).

More options:

buildDir

Specifies the buildDirectory where .class files go. By default, it is the build directory in the layer's folder itself. This way, the layer is completely self contained. Clearly the build directory itself cannot contain source.

buildSrcDir

Specifies the directory where generated .java files go.

buildLayer

Any layer which has a build directory is considered to have buildLayer=true by default. Any layer without a build directory has buildLayer=false unless set explicitly in the layer definition file, or built explicitly (scc [-c] layerName). Any buildLayer=true layer compiles all files that have changed since any previous buildLayer. In other words, buildLayers are themselves chained, the latter inheriting the contents of the former. This lets you share compiled versions of some layers with subsequent layers so you can optimize what gets built when you change a given layer.

buildSeparate

By default, .sc files in a layer cannot be used as compiled classes in the layer definition file. Occasionally you need to do this in a layer. If you mark that layer with buildSeparate, it is compiled and its classes are put into the classpath of the system before subsequent layer def files are processed allowing them to find these classes. This means that you cannot replace those classes though with the modify operation. Structurally StrataCode assumes you cannot reload a .java .class file since, though technically some JVMs do allow limited versions of class reloading

classPath

Java jar files or directories containing .class files in package organization. The list of jars/directories you add is searched in order to find classes available to code in this layer or any downstream layers. Once a layer's classpath has been searched, it looks for any classes defined in layers that layer extends. Thus a downstream layer can override a compiled class in an upstream layer, but only if that layer and class file are in place the first time that class is accessed. StrataCode does not unload any class files, except as normally garbage collected by the JVM, so a restart is required when you add a new layer with compiled classes in it that override classes that have already been loaded.

exportImports

Default is true.

Should the imports in the layer file be visible to layers which extend this layer

inheritImports

Default is true.

Should this layer use the imports from any extended layers.

inheritPackage

Default is true.

Should this layer use the package prefix of its extend layers? If there are more than one conflicting package prefix in the list of extend layers, the last one wins.

copyPlainJavaFiles

Default is true.

Any Java files do not use any extensions are not modified. They can either be compiled from the source directory or copied into the build directory. Doing the later ensures you have a single folder containing the entire source for your project, nice if you are editing the Java files with an IDE.

excludedFiles

A list of Java regular expressions for files to ignore during the build process. By default:

   public List<string> excludedFiles = Arrays.asList("build", "out", ".*.sctd", ".git");

StrataCode will ordinarily ignore files which are not in a language it understands. The "sctd" files are in this list because it currently does not compile those files, they are interpreted instead. That will be removed once they are compiled like normal StrataCode/Java files.

compiledOnly

Indicates this layer does not support dynamic mode. Any attempt to make it dynamic is ignored and all of the files are compiled anyway. You can turn off dynamic mode on a per class basis using the CompilerSettings annotation.

transparent

When you mark a layer as transparent, it only affects how the editor displays the types for that layer. It does not affect the runtime behavior of the application. A transparent layer shows all objects and properties defined in the layers which the transparent layer extends. It includes properties in other transparent layers found along the way and so works recursively.

useGlobalImports

By default a layer only sees the imports which are defined by layers which it extends, either directly or indirectly. An import defined in a totally independent layer would not be accessible. There are some cases however where you might want to define a global layer that can see all base layers without having to extend a bunch of layers explicitly. For example, the temporary layer created with the -i option can see and modify any type. For these layers, set useGlobalImports=true so that they can see all of the imports as well.

codeType and codeFunction

Two metadata properties stored in the layer for use by the editor. Lets you tag layers as having certain types of code assets (e.g. declarative only) or certain types of code functions (i.e. UI, style, domain model, etc.). These properties also help sort layers so framework layers do not get intermixed with application layers for example.

TODO: should we use these properties to restrict the feature set available to a layer? It would help enforce best practices and predictability of the code.