Version 3 Internal Reference Spec

=Summary of Version 2 Implementation= The object model for Context Free 2 is that there is a CFDG object which contains all of the information in a cfdg file and a Renderer object which generates a stream of primitive shapes using the information in the CFDG object. Both the CFDG and the Renderer classes hide their implementation details behind a pointer to a derived class that has all the details. The CFDG class has a ParseFile factory method that uses flex and yacc to parse the cfdg files (in C mode). ParseFile uses a Builder object to store state used in parsing the cfdg file and provide helper methods for the parser. Since yacc generates C-code there is a suite of glue functions in yglue.cpp that call Builder methods on the behalf of the parser's C code.

Ultimately the CFDG object contains an STL container full of rules. Each rule contains an STL container full of replacements. The Renderer object keeps an STL container of non-primitive shapes that it needs to expand and primitive shapes and paths that have been issued from the rule expansion algorithm. The non-primitive shapes are stored in an ordered STL container ordered by the shapes "area", the determinant of the affine transform in its world state. The Renderer run algorithm gets the shape with the largest area, queries the CFDG object for a rule that expands that shape and then expands the shape using the rule. Primitive shapes and paths found in the rule are placed in the primitive shape STL container. All the rest are put in the non-primitive container for subsequent expansion.

Primitive shapes and path elements are converted into smaller form, called a finished shape, before being put in the finished shapes STL container. Finished shapes contain just enough information to sort properly against other finished shapes and make a call into the Canvas class drawing methods. All doubles are converted to floats, the color is converted to an rgb8, and the random seed is thrown away.

Loops in rules and path command loops in paths are stored as loopBegin/loopEnd pseudo-replacements. The Renderer run algorithm executes these loop instructions. The path processing method processShape handles path command loops in a similar manner. Path operation loops are unrolled fully during parsing and the complete vertex output of the loop is stored in the agg::path_storage object.

The primitive shape/path container is stored in the order that the shapes/paths were generated. But the Canvas output algorithm requires that the shapes/paths be ordered by Z and then by generation order, so the container is sorted on Z before output to a Canvas.

Expressions in cfdg files are reduced to a number by the parser using standard yacc methods for parsing expressions

Random number tricks
Context Free does not use a common random number generator. Instead each shape has a 48-bit PRNG seed as part of its state. The value of the seed is derived from the parent shape's seed. This allows the output to be resolution independent. In order to keep thing interesting and prevent gaming of the PRNG sequence each shape replacement has a 48-bit piece of entropy that gets XOR-ed into the parent shape's seed to produce the child shape's seed. Early versions of Context Free generated this entropy directly from the bits of the IEEE-754 doubles in the replacement modification. This proved to be a bad choice as different CPU architectures would produce slightly different bits for the same floating point math. Later versions used the ASCII (and later utf-8) text of the cfdg file to generate entropy. Care was taken so that a cfdg file written with an older version of Context Free had the same entropy in newer versions.

=Overview of Version 3 Changes= Version 3 keeps the same CFDG & Renderer object model as version 2, with implementation details hidden by pointers to derived classes. But the parser is pure C++ using flex & bison. The flex/bison C++ models requires a driver class to glue together the two halves of the parser and the Builder class steps into this function. Since the parser is C++ it can access Builder class resources directly, so all interesting code in yglue.cpp has migrated into the Builder class. The purpose for switching the parser from C to C++ is because the parser now compiles cfdg files into an abstract syntax tree (AST). In principle the AST can be compiled to some virtual machine or x86 instructions, but right now the Renderer class interprets it directly.

There are two AST class hierarchies: ASTexpression and ASTreplacement. ASTexpression represents all expressions: numeric expressions, shape adjustments, shape specifications (shape names and parameters), and path operation parameters. ASTreplacement represents all syntax structures from individual shape replacements, to rules, to the entire cfdg file. It may be desirable to split the ASTreplacement hierarchy into two halves, and ASTreplacement hierarchy for syntax elements that are within rules or paths, and an ASTrule hierarchy that is for syntax elements that are global scope.

The Renderer object has two pieces of new state that are new: a stack of runtime information and a global random number generator seed. The global PRNG seed is directly analogous to the PRNG seed at the base of the shape stack in the version 2 execution algorithm, there is nothing new here. But the new execution model for version 3 requires that this PRNG seed be explicitly called out in order to make the Context Free PRNG tree feature work. The runtime stack is required for the two main new features of version 3: parameters and random numbers. A shape's parameters are placed at the base of the runtime stack. Any variables that are defined that are random and/or dependent on information already on the stack are themselves stored on the stack. Loop indices are also stored on the stack. Variables bound to constant expressions are not stored on the stack, the evaluated expression is part of the AST.

Entropy and Variations
Version 3 radically changes how cfdg files are parsed and executed. The old parser had accumulated cruft to maintain variation continuity across versions. This cruftiness threatened to grow exponentially in version 3. Instead we are dropping variation continuity with version 2 and sweeping away all the cruft. Version 3 seeks to simplify entropy generation.

No More Finished Shapes
Version 2 is able to pickle paths into small finished shape objects because the bulk of the information about paths, the agg::path_storage and the path command info is stored elsewhere. This paradigm breaks down with version 3 because paths can take parameters or use the random functions. Each instance of a given path could have different path storage info or different commands. To keep the finished shape paradigm we would have to generate path storage and path commands for each instance of a path, even if there are a million of them. Instead we have gotten rid of finished shapes and used the same shape class for shapes that are terminals and shapes that are non-terminals. When it is time to draw a path shape to a Canvas we traverse the path AST, generating an agg::path_storage and issuing Canvas::path method calls for each path command. If the path has no parameters and doesn't use random functions then the agg::path_storage is kept after the first traversal and ASTpathOps are no-ops on subsequent path traversals.

=The Stack= The stack is declared as a std::vector. Numeric, transform, and rule expressions are laid out on the stack in a compact manner on eight byte boundaries. Numeric n-tuples consume n stack entries. Transform expressions consume 22 stack entries. Rule expressions consume one stack entry, a pointer to a parameter block plus a header. The same layout is used for parameter blocks, so they can simply be copied onto the stack at rule invocation. Parameters, loop variables and non-constant defined variables are stored on the stack and referenced via ASTvariable objects. Constant defined variables are copied directly into expressions that reference them.

=ASTexpression Class Hierarchy= The ASTexpression hierachy all share a few virtual functions that perform operations on expressions:
 * int evaluate(double* dest array, int size, renderer pointer) const
 * This function evaluates numeric n-tuples, storing the result in the destination array. The return value is the number of doubles stored in the array or -1 if there is not enough room or some other problem arises. If a null pointer is given for the destination array then nothing is stored (or really evaluated), but the required size is returned. If a null pointer is given for the renderer and the expression requires this information then a DeferUntilRuntime exception is thrown.


 * void evaluate(Modification& mod, string* param, double* stroke width, bool just check, int& seed index, renderer pointer) const
 * Evaluates transform expressions, storing the result in the Modification reference. If a parameter or stroke width adjustment is encountered and null pointers were given for that data then a CfdgError exception is thrown. If just_check is true then the adjustment is checked for errors but the Modification is not changed. The seed index is used for inserting entropy string into the Modification PRNG seed. If a null pointer is given for the renderer and the expression requires this information then a DeferUntilRuntime exception is thrown.


 * const StackType* evalArgs(renderer pointer)
 * Evaluates shape expressions, producing a rule parameter block.


 * flatten(std vector& dest)
 * This method traverses a tree of expressions and dumps pointers to the leaf nodes in a std vector.


 * void entropy(string& entString) const
 * This functions takes whatever entropy information the expression has and appends it to the provided entropy string. The string accumulates all of the entropy text for an expression and this information is eventually munged into a PRNG seed by the evaluate method.


 * void simplify
 * This method reduces constant subexpressions in numeric expressions. For transform expressions and shape specifications this method simplifies the arguments or parameters. If this can be reduced then this is deleted and the reduced form is returned. Otherwise this is returned.

Safety issues with flatten and simplify
Since flatten and simplify potentially delete this, methods that call them must take ownership of the expression and make sure that the results are deleted or placed where they can be deleted when the Builder and CFDGImple objects are deleted. Constructors that call flatten or simplify must delete the results if they throw an exception. Even if the constructor throws before calling flatten or simplify they must delete the expression argument, as it is not safe for the exception handler to delete it.

Derived classes
The classes in the ASTexpression class hierarchy are:
 * ASTexpression(location& where, bool isConstant, typeEnum type)
 * Base class for all the rest. ASTexpression is not an abstract class, instances of it are created as place holders for values the exist purely on the stack: shape parameters and loop variables. It provides a default implementation for the transform-type evaluate method that throws a CfdgError exception.


 * ASTfunction(string& name, ASTexpression* arguments, PRNG seed& builder seed, location& nameLoc, location& argsLoc)
 * Evaluates built-in functions. May evaluate user-defined functions if that feature is added. The PRNG seed referenced is owned by Builder and is initialized directly by the current variation. The seed is only used by the rand_static function at ASTfunction construction time.


 * ASTruleSpecifier(int name, string& name, ASTexpression* arguments, location& where)
 * Stores information about a shape reference. The constructor extracts and stores the entropy text for the shape specification and then simplifies the arguments.


 * ASTcons(ASTexpression* left, ASTexpression* right)
 * Groups together two expressions preserving order. The evaluate method tries to store the left-side expression into the destination array, followed by the right-side expression. The transform-type evaluation method does this as well, but it should probably just throw an exception if called. The flatten method implements the heart of the flatten algorithm. The flatten method also checks if either left or right is a canonical form transform expression and flattens it in canonical order if it is. This is the only check for canonical transform expressions anywhere, which means that transform expressions must have an ASTcons at the top in order to guarantee canonical form reordering.


 * ASTreal(string& number, location where, bool negative) and ASTreal(double number, location where)
 * Stores a numeric one-tuple value along with textual form of the number, which is used for generating entropy text. The negative argument indicates that the negative of the number is to be stored. This is hold-over code that tries to preserve entropy generation to match what version 2.1 generated. Since version 3 is going to break variation compatibility with version 2 this can be dropped. The second constructor is used to generate numeric expressions from evaluated functions.


 * ASTvariable(int name, string& name, location where)
 * Stores a reference to a value that is on the stack. The Builder MakeVariable method generates these when the parser finds an identifier that is bound to an expression that is stored on the stack. If the bound expression is constant then MakeVariable returns a copy of the constant data.


 * ASToperator(char operator, ASTexpression* left, ASTexpression* right'')
 * Stores a numeric expression or a transform expression of the form left op right or op left. The only valid form for transform expressions is left + right.


 * ASTparen(ASTexpression* inner expression)
 * The only purpose of this class is to add the signature of a parenthesized expression to the entropy text of an expression. It always deletes itself in the simplify method.


 * ASTmodTerm(int mod type, ASTexpression* arguments, location& where) or
 * ASTmodTerm(int mod type, string& text, location& where)
 * Stores the information for a single shape adjustment term. The constructor does some work to determine if the term affects both the XY affine transform and the Z affine transform.


 * ASTmodification(ASTexpression* mod expression, string& name, location where)
 * This class is where transform expression get parked after they are parsed. The constructor simplifies the expression, puts an ASTcons at the top of the expression containing its entropy text, flattens it (which canonical form parts have canonical order), and then tries to evaluate constant portions of the expression. Ultimately the object contains the constant part of the transform expression and an std vector of the remainng ASTmodTerms that need to be evaluated at runtime.


 * ASTmodRef(Modification& mod, int modification class, location where)
 * This class stores references to constant transform expressions. If a variable is bound to a constant transform expression then references to that variable are converted to ASTmodRef objects. This class is only used to point to a constant transform-type variable definition. For other types the constant data is copied. We could drop this class and use ASTmodification to hold a copy of the constant data. Alternately, we could change this class to a generic reference class that works for all types of constant data.


 * ASTselect(auto_ptr args, location& where)
 * This class implements the select function.

=ASTreplacement Class Hierarchy= The ASTreplacement hierachy all share a few virtual functions that perform operations on syntactic elements above the level of expressions:
 * traverse(Shape& parent, agg::affine_transform transform, Renderer* renderer) const
 * Performs whatever action is required to traverse the node in the AST. The renderer reference is used to access the runtime stack and global PRNG seed. The affine transform is used in traversing AST trees of path operations to facilitate unrolling of path operation loops.

ASTreplacement classes get ownership of objects passed to their constructors and destroy them either at construction time (by sucking out their information), or at destruction time. The constructors guarantee that they will destroy their arguments if they throw an exception within the constructor.

Derived classes
The classes in the ASTreplacement class hierarchy are:
 * ASTreplacement(int name, string& name, auto_ptr mods, location& where)
 * Base class, used within rules to represent individual shape replacements and in paths to represent subpaths. The mods argument is stored in an ASTmodification.


 * ASTloop(string& count, auto_ptr mods, location& where) or
 * ASTloop(int name, string& name, location& nameLoc, auto_ptr loopArgs, location& argsLoc, auto_ptr mods)
 * Stores loops. The loopArgs argument are one to three numbers for the start, end, and step values of the loop. The global parameter list and stack index are state in the Builder that are used to setup the loop variable, give its position in the runtime stack, and insert it into the stack of identifiers currently in-scope. ASTloop objects contain the ASTreplacements for the loop body and the finally body.


 * ASTtransform(auto_ptr mods, location& where)
 * Stores transform blocks.


 * ASTdefine(const ASTparameter& def, location where)
 * Evaluates parameter expression and pushes it onto the runtime stack. This object is inserted into a body when a definition is not constant. Constant definitions do not generate ASTdefine objects.


 * ASTrule(int name, double weight)
 * Stores the contents of a rule or path. The traverse method sets the global PRNG seed in the renderer to the seed contained in the parent shape. This class contains a mutable pointer to an ASTcompiledPath object. This pointer is non-null only if the shape is a path and the path has no parameters and never uses the random number function. The ASTcompiledPath object is a cached version of the path's agg::path_storage. This class used to be split into two sub-classes, one for rules and one for paths. It has a traverPath method that came from the path sub-class. This traversePath method can probably be merged with the virtual traverse method.


 * ASTshape(int name, bool isPath, location& where)
 * Stores rules contained within a shape declaration along with the shape parameter information.


 * ASTpathOp(string& pathOp, auto_ptr args, bool positional, location& where)
 * Stores a path operation. The traverse method compiles the operation into the path_storage if the containing path is being compiled, otherwise it does nothing. The positional argument tells the constructor whether the path op arguments are a version 2 list of ASTmodTerms or a version 3 list of numeric expressions.


 * ASTpathCommand(string& command, auto_ptr mods, location& where)
 * Stores a path command. ASTpathCommand is a leaf node and implements ASTreplacement Actions. ASTpathCommand has several static agg path convertors that should probably by moved to aggCanvas to facilitate multi-threaded drawing. ASTpathCommand objects have a mutable member of type ASTpathCommand::CommandInfo. The data member is mutable because a path command within a loop may cover multiple groups of path operations. The ASTpathCommand object is not copyable, so instead the CommandInfo member of the single ASTpathCommand is mutated during the execution of the loop.


 * ASTif(auto_ptr condition, location& where)
 * Stores if statements. Contains the contents of the then body and the else body.


 * ASTswitch(auto_ptr select, location& where)
 * Stores switch statements. Contains a std::map member to store each case statement. Also stores the else block that is executed if the indexed case is not found in the map.

=Other Classes=

ASTparameter
ASTparameter is a record of a single parameter, loop variable, or defined variable. It contains an ASTexpression pointer for numeric and rule parameters/variables and an ASTmodification for transform parameters/variables. If the ASTparameter is for a parameter or a non-constant variable then the mStackIndex member indicates where on the stack the value should live. The Builder sets this up and uses this information when making ASTvariable objects.

ASTrepContainer
This is a helper class that is used as a member variable in ASTreplacement classes that contain other replacements (ASTrule, ASTloop, etc.). It has an ASTparameterList member that contains all parameters and variables that are in the scope of the contained replacements. It also has a traverse method that has the same signature as the ASTreplacement traverse method. The traverse method handles popping run-time-evaluated definitions off of the run-time stack.

CommandInfo
This struct contains information about the path operations covered by a path command: a weak pointer to the agg::path_storage object, an index into the path storage, and all of the state from the ASTpathCommand object that issued the CommandInfo.

ASTcompiledPath
Object of this class contain the agg::path_storage for a path, a std::deque, and an iterator for the deque. The object also has an ASTpathCommand that is the implicit terminal command in paths with no terminal command.

Builder
Contains state and helper functions for generating the AST. The completed AST is moved from the Builder object to the CFDGImpl object and the Builder object is destroyed.

pathIterator
Contains the AGG vertex pipelines and helper methods for traversing and rasterizing paths. Used by Bounds for bounding paths and aggCanvas for drawing paths.