class Scanner: public ScannerBase
{
    public:
        enum Tokens
        {
            IDENTIFIER = 0x100,
            NUMBER
        };
    // ... (etc, as generated by flexc++)
        

#include <iostream>
#include "Scanner.h"

using namespace std;

int main()
{
    Scanner scanner;        // define a Scanner object

    while (int token = scanner.lex())   // get all tokens
    {
        string const &text = scanner.matched();
        switch (token)
        {
            case Scanner::IDENTIFIER:
                cout << "identifier: " << text << '\n';
            break;

            case Scanner::NUMBER:
                cout << "number: " << text << '\n';
            break;

            default:
                cout << "char. token: `" << text << "'\n";
            break;
        }
    }
}
        

    g++ --std=c++11 *.cc
        

    a.out < main.cc
        

2. QUICK START: FLEXC++ and BISONC++

To interface flexc++ to the bisonc++(1) parser generator proceed as follows:

• Specify a grammar that can be processed by bisonc++(1). Assuming that the scanner and parser are developed in, respectively, the sub-directories scanner and parser, then a simple grammar specification that can be used with the scanner developed in the previous section is, e.g., write the file parser/grammar:

  
  %scanner                ../scanner/Scanner.h
  %scanner-token-function d_scanner.lex()
  
  %token IDENTIFIER NUMBER CHAR
  
  %%
  
  startrule:
      startrule tokenshow
  |
      tokenshow
  ;
  
  tokenshow:
      token
      {
          std::cout << "matched: " << d_scanner.matched() << '\n';
      }
  ;
  
  token:
      IDENTIFIER
  |
      NUMBER
  |
      CHAR
  ;
      
  

• Write a scanner specification file. E.g.,

  
  %%
  
  [ \t\n]+                            // skip white space chars.
  [0-9]+                              return Parser::NUMBER;
  [[:alpha:]_][[:alpha:][:digit:]_]*  return Parser::IDENTIFIER;
  .                                   return Parser::CHAR;
      
  

  This causes the scanner to return Parser tokens to the generated parser.

• Add the line

  
  #include "../parser/Parserbase.h"
      
  

  to the file scanner/Scanner.ih

• Write a simple main function in the file main.cc. E.g.,

  
  #include "parser/Parser.h"
  
  int main(int argc, char **argv)
  {
      Parser parser;
  
      parser.parse();
  }
      
  

• Generate a scanner in the scanner subdirectory:

  
  flexc++ lexer
      
  

• Generate a parser in the parser subdirectory:

  
  bisonc++ grammar
      
  

• Compile all sources:

  
  g++ --std=c++0x *.cc */*.cc
      
  

• Execute the program, providing it some source file to be processed:

  
  a.out < main.cc
      
  

3. GENERATED FILES

Flexc++ generates four files from a well-formed input file:

• A file containing the implementation of the lex member function and its support functions. By default this file is named lex.cc.

• A file containing the scanner's class interface. By default this file is named Scanner.h. The scanner class itself is generated once and is thereafter `owned' by the programmer, who may change it ad-lib. Newly added members (data members, function members) will survive future flexc++ runs as flexc++ will never rewrite an existing scanner class interface file, unless explicitly ordered to do so.

• A file containing the interface of the scanner class's base class. The scanner class is publicly derived from this base class. It is used to minimize the size of the scanner interface itself. The scanner base class is `owned' by flexc++ and should never be hand-modified. By default the scanner's base class is provided in the file Scannerbase.h. At each new flexc++ run this file is rewritten unless flexc++ is explicitly ordered not to do so.

• A file containing the implementation header. This file should contain includes and declarations that are only required when compiling the members of the scanner class. By default this file is named Scanner.ih. This file, like the file containing the scanner class's interface is never rewritten by flexc++ unless flexc++ is explicitly ordered to do so.

4. OPTIONS

Where available, single letter options are listed between parentheses following their associated long-option variants. Single letter options require arguments if their associated long options require arguments as well. Options affecting the class header or implementation header file are ignored if these files already exist. Options accepting a `filename' do not accept path names, i.e., they cannot contain directory separators (/); options accepting a 'pathname' may contain directory separators.

Some options may generate errors. This happens when an option conflicts with the contents of an existing file which flexc++ cannot modify (e.g., a scanner class header file exists, but doesn't define a name space, but a --namespace option was provided). To solve the error the offending option could be omitted, the existing file could be removed, or the existing file could be hand-edited according to the option's specification. Note that flexc++ currently does not handle the opposite error condition: if a previously used option is omitted, then flexc++ does not detect the inconsistency. In those cases you may encounter compilation errors.

• --baseclass-header=filename (-b)
  Use filename as the name of the file to contain the scanner class's base class. Defaults to the name of the scanner class plus base.h

  It is an error if this option is used and an already existing scanner-class header file does not include `filename'.

• --baseclass-skeleton=pathname (-C)
  Use pathname as the path to the file containing the skeleton of the scanner class's base class. Its filename defaults to flexc++base.h.

• --case-insensitive
  Use this option to generate a scanner case insensitively matching regular expressions. All regular expressions specified in flexc++'s input file are interpreted case insensitively and the resulting scanner object will case insensitively interpret its input.

  When this option is specified the resulting scanner does not distinguish between the following rules:

  
          First       // initial F is transformed to f
          first
          FIRST       // all capitals are transformed to lower case chars
                  
  

  With a case-insensitive scanner only the first rule can be matched, and flexc++ will issue warnings for the second and third rule about rules that cannot be matched.

  Input processed by a case-insensitive scanner is also handled case insensitively. The above mentioned First rule is matched for all of the following input words: first First FIRST firST.

  Although the matching process proceeds case insensitively, the matched text (as returned by the scanner's matched() member) always contains the original, unmodified text. So, with the above input matched() returns, respectively first, First, FIRST and firST, while matching the rule First.

• --class-header=filename (-c)
  Use filename as the name of the file to contain the scanner class. Defaults to the name of the scanner class plus the suffix .h

• --class-name=className
  Use className (rather than Scanner) as the name of the scanner class. Unless overridden by other options generated files will be given the (transformed to lower case) className* name instead of scanner*.

  It is an error if this option is used and an already existing scanner-class header file does not define class `className'

• --class-skeleton=pathname (-C)
  Use pathname as the path to the file containing the skeleton of the scanner class. Its filename defaults to flexc++.h.

• --construction (-K)
  Write details about the lexical scanner to the file `rules-file'.output. Details cover the used character ranges, information about the regexes, the raw NFA states, and the final DFAs.

• --debug (-d)
  Provide lex and its support functions with debugging code, showing the actual parsing process on the standard output stream. When included, the debugging output is active by default, but its activity may be controlled using the setDebug(bool on-off) member. Note that #ifdef DEBUG macros are not used anymore. By rerunning flexc++ without the --debug option an equivalent scanner is generated not containing the debugging code.

• --filenames=genericName (-f)
  Generic name of generated files (header files, not the lex-function source file, see the --lex-source option for that). By default the header file names will be equal to the name of the generated class.

• --help (-h)
  Write basic usage information to the standard output stream and terminate.

• --implementation-header=filename (-i)
  Use filename as the name of the file to contain the implementation header. Defaults to the name of the generated scanner class plus the suffix .ih. The implementation header should contain all directives and declarations only used by the implementations of the scanner's member functions. It is the only header file that is included by the source file containing lex()'s implementation. User defined implementation of other class members may use the same convention, thus concentrating all directives and declarations that are required for the compilation of other source files belonging to the scanner class in one header file.

  It is an error if this option is used and an already 'filename' file does not include the scanner class header file.

• --implementation-skeleton=pathname (-I)
  Use pathname as the path to the file containing the skeleton of the implementation header. Its filename defaults to flexc++.ih.

• --lex-skeleton=pathname (-L)
  Use pathname as the path to the file containing the lex() member function's skeleton. Its filename defaults to flexc++.cc.

• --lex-function-name=funname
  Use funname rather than lex as the name of the member function performing the lexical scanning.

• --lex-source=filename (-l)
  Define filename as the name of the source file to contain the scanner member function lex. Defaults to lex.cc.

• --matched-rules (-'R')
  The generated scanner will write the numbers of matched rules to the standard output. It is implied by the --debug option. Displaying the matched rules can be suppressed by calling the generated scanner's member setDebug(false) (or, of course, by re-generating the scanner without using specifying --matched-rules).

• --max-depth=depth (-m)
  Set the maximum inclusion depth of the lexical scanner's specification files to depth. By default the maximum depth is set to 10. When more than depth specification files are used the scanner throws a Max stream stack size exceeded std::length_error exception.

• --namespace=identifier
  Define the scanner class in the namespace identifier. By default no namespace is used. If this options is used the implementation header is provided with a commented out using namespace declaration for the requested namespace. In addition, the scanner and scanner base class header files also use the specified namespace to define their include guard directives.

  It is an error if this option is used and an already scanner-class header file does not define namespace identifier.

• --no-baseclass-header
  Do not write the file containing the scanner's base class interface even if it doesn't yet exist. By default the file containing the scanner's base class interface is (re)written each time flexc++ is called.

• --no-lines
  Do not put #line preprocessor directives in the file containing the scanner's lex function. By default #line directives are entered at the beginning of the action statements in the generated lex.cc file, allowing the compiler and debuggers to associate errors with lines in your grammar specification file, rather than with the source file containing the lex function itself.

• --no-lex-source
  Do not write the file containing the scanner's predefined scanner member functions, even if that file doesn't yet exist. By default the file containing the scanner's lex member function is (re)written each time flexc++ is called. This option should normally be avoided, as this file contains parsing tables which are altered whenever the grammar definition is modified.

• --own-tokens (-T)
  The tokens returned as well as the text matched when flexc++ reads its input files(s) are shown when this option is used.

  This option does not result in the generated program displaying returned tokens and matched text. If that is what you want, use the --print-tokens option.

• --print-tokens (-t)
  The tokens returned as well as the text matched by the generated lex function are displayed on the standard output stream, just before returning the token to lex's caller. Displaying tokens and matched text is suppressed again when the lex.cc file is generated without using this option. The function showing the tokens (ScannerBase::print__) is called from Scanner::printTokens, which is defined in-line in Scanner.h. Calling ScannerBase::print__, therefore, can also easily be controlled by an option controlled by the program using the scanner object.

  This option does not show the tokens returned and text matched by flexc++ itself when reading its input s. If that is what you want, use the --own-tokens option.

• --regex-calls
  Show the function call order when parsing regular expressions (this option is normally not required. Its main purpose is to help developers understand what happens when regular expressions are parsed).

• --show-filenames (-F)
  Write the names of the files that are generated to the standard error stream.

• --skeleton-directory=pathname (-S)
  Defines the directory containing the skeleton files. This option can be overridden by the specific skeleton-specifying options (-B -C, -H, and -I).

• --target-directory=pathname
  Specifies the directory where generated files should be written. By default this is the directory where flexc++ is called.

• --usage (-h)
  Write basic usage information to the standard output stream and terminate.

• --verbose(-V)
  The verbose option generates on the standard output stream various pieces of additional information, not covered by the --construction and --show-filenames options.

• --version (-v)
  Display flexc++'s version number and terminate.

FILES

Flexc++'s default skeleton files are in /usr/share/flexc++.
By default, flexc++ generates the following files:

• Scanner.h: the header file containing the scanner class's interface.
• Scannerbase.h: the header file containing the interface of the scanner class's base class.
• Scanner.ih: the internal header file that is meant to be included by the scanner class's source files (e.g., it is included by lex.cc, see the next item's file), and that should contain all declarations required for compiling the scanner class's sources.
• lex.cc: the source file implementing the scanner class member function lex (and support functions), performing the lexical scan.

SEE ALSO

bisonc++(1), flexc++api(3), flexc++input(7)

BUGS

None reported

ABOUT flexc++

Flexc++ was originally started as a programming project by Jean-Paul van Oosten and Richard Berendsen in the 2007-2008 academic year. After graduating, Richard left the project and moved to Amsterdam. Jean-Paul remained in Groningen, and after on-and-off activities on the project, in close cooperation with Frank B. Brokken, Frank undertook a rewrite of the project's code around 2010. During the development of flexc++, the lookahead-operator handling continuously threatened the completion of the project. By now, the project has evolved to a level that we feel it's defensible to publish the program, although we still tend to consider the program in its experimental stage; it will remain that way until we decide to move its version from the 0.9x.xx series to the 1.xx.xx series.

COPYRIGHT

AUTHOR