On most systems, tcLex takes the form of a loadable library that contains a Tcl package. To load tcLex, just include this piece of code in your Tcl scripts:
package require tcLex
Tcl's package managing facility also provides many useful features such as versionning. The current tcLex version is 1.1. The above line will load the tcLex package with the highest version number. If you want to ensure that the correct version is loaded (to avoid incompatibilities between versions), append the desired version number to the package command:
will load any package whose version number is equal to or greater than 1.1. Adding the -exact switch tells that you want the package with the same version as specified:package require tcLex 1.1
package require -exact tcLex 1.1
This option should only be used in specific cases. Packages are supposed to be compatible with other packages with the same major version number but with lower minor.
Precise version control avoids loading a package that is incompatible with the desired one, and also check for the presence of this package. A good package command gives precious information to the user of a software about the packages it depends on.
If loading is successful, a new Tcl command becomes available in the interpreter: the lexer command. The reason why tcLex doesn't define a new namespace is that it creates only one new command whose name is unlikely to conflict with any existing one.
lexer option ?args ... args?
The lexer command creates lexical analyzers (lexers) commands. It creates a new command named name, which can contain namespaces delimiters (::). The newly created command replaces any existing one.
Lexer's syntax is a mixture of the Tcl commands proc and switch. Like proc, it creates a new Tcl command. Like switch, such defined commands are built around a specific control structure similar to switch blocks.
Lexer takes a subcommand name as its first argument. This can be any of the following options:
lexer ?create? name ?option ...? rules lexer ?create? name ?option ...? rule ?rule ...?
The create keyword is optional for compatibility with tcLex 1.0, but its use is strongly encouraged as it may become mandatory in future major versions and it also avoids any name conflict with lexers name.
The create subcommand, as its name says, is used to create new lexers. It creates a new command named name, which can contain namespaces delimiters (::). The newly created command replaces any existing one. The return value is empty. On these points, lexer create resembles proc.
Lexer takes a variable number of options, each starting by '-'. Some options take an extra argument as value. Any of these options can be abbreviated if the abbreviation is non ambiguous. The following options are currently supported:
Following the options list are the rules. They can be specified using two syntaxes
(like switch). The first uses a separate argument for each of the rules specifiers;
this form is convenient if substitutions are desired on some of the patterns or
commands. The second form places all of the rules specifiers into a single argument;
the argument must have proper list structure, with the elements of the list being the
rules specifiers. The second form makes it easy to construct multiple rules, since
the braces around the whole list make it unnecessary to include a backslash at the end
of each line. Since the rules specifiers are in braces in the second form, no command
or variable substitutions are performed on them; this makes the behavior of the second
form different than the first form in some cases.
lexer current
This returns the name of the currently active lexer. This can be used as a convenience from within lexers script, and is most useful when lexers are renamed or aliased.
Lexers are made up of rules which define the behavior of the lexer depending on its input. Rules are specified using the following syntax (rules specifiers):
conditionlist regexp matchvars actionwhere:
Conditionlist is a list of conditions during which the rule is active. Every condition must have been specified by either -inclusiveconditions or -exclusiveconditions switches. If not, it raises an error.
Regexp is a regular expression that uses the same syntax as Tcl's regexp command. It can contain parenthesized subexpressions that correspond to different parts of the matched string. If the rule matches (ie if the rexexp matches) the input string, the variables specified by matchvars are filled with corresponding substrings (the same way regexp does, see its man page).
Action is a Tcl script that is evaluated when the rule matches the input string. This script is evaluated in the lexer's context, the same used by -(pre|mid|post)script and where the matchvars live.
lexerName option ?arg arg ...?
The lexer creates new lexer commands that are used like regular Tcl commands. These command takes in turn subcommands that perform several operations.
Option and the args determine the exact behavior of the command. The following commands are possible for lexers:
These commands can be invoked from any script. They start a new processing sequence.
These commands can only be used during processing (when lexerName is active), else they raise an error.
Lexers' main goal is to read and consume an input string, and to perform a number of tasks according to this input. To do so, lexers are built around a loop that globally performs the following jobs:
Designing a lexer involves writing rules. Rules are made of several parts:
Lexers regular expressions are like standard regular expressions, and are covered by the Tcl regexp man page.
Actions are standard Tcl scripts that can do anything, including calling the lexer's subcommands to query or modify its state.
Conditions are one of the key features of lexers, because they tell when a rule is active and can be tried. Basically, conditions are just states that lexers enter and leave. There can be only one active condition for a given active lexer. Thus, to be active, a rule must have the current condition in its active conditions list.
In tcLex, the list of valid conditions must be given at lexer creation time. The options -inclusiveconditions (or -ic) and -exclusiveconditions (or -ec) are used to specify the list of inclusive or exclusive conditions that can be used by the lexer. In turn, every rule must specify a list of conditions during which they are active. Conditions are just arbitrary strings.
During processing, the current condition can change. The lexer subcommand begin tells the lexer to enter the specified condition, whereas end tells it to leave the current condition and return to the previous. Conditions are managed in a stack structure, which can be queried using the conditions subcommand.
As written above, there are two kinds of conditions: inclusive and exclusive. At the first sight, the difference between inclusive and exclusive conditions isn't very obvious, and especially how and why they are used. Let's see the following example:
lexer lexclusive -exclusiveconditions {foo} {
foo {b} {char} {puts "$char matched within foo"}
{} a {char} {puts "$char matched, entering foo"; [lexer current] begin foo}
{} . {char} {puts "$char matched"}
}
lexer linclusive -inclusiveconditions {foo} {
foo {b} {char} {puts "$char matched within foo"}
{} a {char} {puts "$char matched, entering foo"; [lexer current] begin foo}
{} . {char} {puts "$char matched"}
}
We see that the only difference between both lexers is the condition "foo" being inclusive or exclusive. This slight difference in the definition can make a huge difference in the behavior. We can observe that if we eval both lexers with the same simple string:
% lexclusive eval abcd a matched, entering foo b matched within foo % % linclusive eval abcd a matched, entering foo b matched within foo c matched d matched
Here we see that the last 2 characters of the string, 'c' and 'd', get matched by linclusive but not by lexclusive. The reason is that after character 'b' has been matched, both lexer enter the condition "foo", which is exclusive in lexclusive and inclusive in linclusive. When both lexers try to match the next characters 'c' and 'd', lexclusive can't find an appliable rule so evals the default rule (which does nothing, see below Default rule handling), whereas linclusive can apply the rules with empty conditions list even within condition "foo". So the main difference between inclusive and exclusive conditions is that inclusive conditions activates rule with empty conditions list.
Since inclusive and exclusive conditions yelds slightly different results, they are often used in specific, distinct situations. Exclusive conditions are used to write "sub-lexers" to apply a different set of rules to some part of the string. Inclusive conditions are used to override some rules in some situations while maintaining the overall behavior. A good example is the Tcl language itself. Tcl parsing rules are slightly different whether the strings are enclosed between brackets, braces or quotes, but many rules still apply (backslash substitution for example). These rules could be expressed as rules with empty conditions list that are still active within inclusive conditions.
As stated before, lexers are built around a main processing loop, which goal is to consume the whole input string. To do so, it tries to find a matching rule, and if it succeeds, it evals its action and consume the matched chars.
Since actions can contain arbitrary command, they can modify the lexer's state. Thus, this state may differ before and after the action has been evaluate. The lexer's state can be modified on some points:
Changing the active is the most simple case. Rejection and input modification are more complex.
Any matching rule can be rejected from within its action. Rejection means that the lexer must behave as if the rule didn't match at all. So rejecting a rule tells the lexer to go on looking for a matching rule. The complexity comes from the fact that the rejected rule can modify the lexer's state before.
Rejection is very useful in a number of cases where normal rule handling would overcomplicate the lexer. Let's consider the following example:
lexer count \
-pre {
set nbLines 1
set nbChars 0
} \
-post {
puts "number of lines: $nbLines"
puts "number of chars: $nbChars"
} {
{} \n {} {incr nbLines; [lexer current] reject}
{} . {} {incr nbChars}
}
This simple lexer is a character and line counter for strings. At the end of the processing, it prints the total number of lines and characters in the input string. The first line is used to match and count the newline characters. The second counts characters. But since a newline is also a character, it must be counted as such. A first possibility is to add the character counting code to the line counting rule. But although it seems useless in this simple example, it can be very tedious and error-prone in more complex lexers (for instance if the character-counting rule is rewritten to add new features), as duplicating code should be avoided. The second possibility is using rejection. In this example we tell the lexer to reject the line-counting rule, so that it proceeds to the next matching rule which is the character-counting rule. That way, newline chars can be used for line counting while still being seen as regular characters. The real power with reject is that it can be conditionally triggered and cascaded. This provides some sort of inheritance between rules.
The last way to act on the lexer state is acting on the input. The input string can't be modified (unlike with flex), but its consumption can be modified. The lexer subcommands input and unput are respectively used to get extra characters from, and to put back consumed characters to the input. This can allow for dynamic rule writing where characters are not consumed by rule matching but programmatically. A practical use would be to write a rule that only matches the beginning of a sequence, and then manually inputs extra characters in order to perform conditional action. This can help a lot grouping many similar rules into one generic rule. Since such consumed characters can be put back by and rules can be rejected, this provide a very powerful way to write dynamic rules.
Flex defines a default rule that is evaluated when none matched. Its default behavior is to consume the first unmatched character and copy it to the output. TcLex lexers' default behavior is slightly different: it consumes one char and does nothing else. Literally, the default rule is:
* . {} {}
This is because lexers have no idea what the "output" can be. The result can take any form and its handling is not tcLex's responsibility but the lexer writer's. You can override the default behavior by defining your own default rule.
One of the most powerful features of tcLex is incremental processing. In short, it allows a lexer to be interrupted and resumed depending on the availability of input data. Given the non-threaded, event-based nature of the Tcl language, it thus provides a very powerful data-driven processing scheme. Applications range from interactive user input processing to socket-delivered document parsing (eg. HTML).
The main strength of tcLex's incremental processing facility is its transparency. Lexers need not be specially designed (except from some performance-related precautions and some specific limtations, see Bugs and Limitations below) to be used incrementally, processing scheme being chosen at run-time. Lexers keep state information between sessions, thus allowing concurrent use of several lexing processes. This state processing is completely transparent to users, and such lexers behave like interruptible procs that could keep their state alive -- local variables for example.
The key to incremental processing lies in the three dedicated lexers subcommands: start, continue and finish. A typical incremental processing session is initiated by start, then additional data is processed by further continue commands, finally processing terminates by finish when no more data is available. Each subcommand returns a result that can be partial (start, continue) or final (finish). TcLex's incremental processing is designed so that the final result of an incremental processing session (returned by finish) is the same as the result of a one-pass session (using eval command) on the same complete string.
As an example is worth a thousand words, here is a typical incremental processing session:
# create a lexer named lx
lexer create lx ...
# channel contains the name of an open Tcl channel (eg. file or socket)
# first initiate the process with an empty string. We could also give
# a non-empty string taken from the input channel.
set input ""
lx start token $input
# then get strings from the channel
while {![eof $channel]} {
# update the input string
append input "[gets $channel]\n" ;# don't forget the trailing newline
lx continue $token $input
}
# all data has been read, finish
# result should be equal to [lx eval $input]
set result [lx finish $token $input]
Every start/continue command should return a valid partial result. This could be for example a partial HTML tree ready to be rendered in a web browser. These results are returned by Tcl scripts specified by -midscript, or through a -resultvariable-specified Tcl variable. These can be used to generate a well-formatted result from a partial internal result (eg. an uncomplete tree) so that it can be properly used by outer code.
Another example, this time chunk-oriented rather than line-oriented (using read rather than gets, ie reading a fixed number of chars rather than whole lines):
# create a lexer named lx
lexer create lx ...
# channel contains the name of an open Tcl channel (eg. file or socket)
set chunkSize 1024 ;# number of chars to read at a time
# first initiate the process with an empty string. We could also give
# a non-empty string taken from the input channel.
set input ""
lx start token $input
# then get strings from the channel
while {![eof $channel]} {
# update the input string
append input [read $channel $chunkSize]
lx continue $token $input
}
# all data has been read, finish
# result should be equal to [lx eval $input]
set result [lx finish $token $input]
Using the input subcommand during incremental processing is potentially dangerous: input cannot return more characters than available at the time the command is issued. Thus, if input wants more characters than available, it will get less characters than during one-pass processing. This contradicts the principle that one-pass and incremental processing yields identical results. Correcting this problem would require a wholely redesigned incremental processing scheme (event-driven for instance, where lexers would get the input themselves rather than be manually called via start/continue/finish).