TcLex is a lexer (lexical analyzer) generator extension to Tcl. It is inspired by Unix and GNU lex and flex, which are "tools for generating programs that perform pattern-matching on text". TcLex is very similar to these programs, except it uses Tcl philosophy and syntax, whereas the others use their own syntax and are used in conjunction with the C language. People used to lex or flex should then feel familiar with tcLex.
TcLex is a small extension (the Windows compiled version is about 20kb, and the source is about 150kb), because it extensively uses the Tcl library. However, the current doesn't use Tcl's regexp code anymore but a patched version is now included in tcLex, which makes it slightly bigger (by a few KB).
TcLex should work with Tcl 8.0 and later. TcLex will NEVER work with earlier versions, because it uses Tcl 8.0's "object" system for performance.
The most interesting features are:
Flex is a tool for generating lexical analyzers using the C language. It uses its own syntax to specify rules, start conditions, regular expressions... Rules codes are written in C. Flex is a command-line tool that takes lexer definition files as input and outputs C source files.
TcLex resembles flex on many points, but its philosophy differs slightly. tcLex is intended to be used from within Tcl rather than as an external command-line tool. Rather than a specific syntax, it uses Tcl syntax to specify lexers constructs using regular Tcl strings, lists, and substitution rules. As a regular Tcl command, it can be used to generate lexers on the fly rather than statically generate Tcl sources. Lexers then become new Tcl control structures like procs and switches that can be used from any Tcl script.
While flex is most suited for performance-tuned lexers writing, tcLex is perfect for interactively exploring the power of lexers using higher level concepts. TcLex compares to flex the same way Tcl compares to C, and goes beyond by turning lexers into a new control structure, rather than keeping them a tool for string processing. Thanks to the many concepts shared with flex, tcLex can also be regarded as a tool for rapid prototyping of lexers intended to be converted to flex, the same way Tcl can be used for rapid prototyping of applications which in turn can be partly rewritten in C for performance issues.
Contrary to flex, tcLex allows programmers to write lexers with the ease of use allowed by the Tcl scripting language: dynamically, interactively, programmatically, rather than statically. Such written lexers are immediately available as regular Tcl commands, and thus can be tested interactively from within a Tcl shell. The rather high-level nature of lexers perfectly suits Tcl philosophy.
TcLex was designed as a powerful mixture of Tcl commands regexp, switch and proc. It borrows many of its syntactical constructs to the switch command, extending it to introduce many concepts taken from flex. Lexers are made available to programmers as new Tcl commands that can be transparently used in a wide variety of situations, the same way proc creates new command from Tcl scripts.