switch to 'new' expression parser, remove support for old parser

provide parser files in source tree, so flex/bison are not need to build
update Makefile to use simpler techniques to build parser
update README to remove references to old vs. new parsers
remove version comparison tool used for flex/bison programs


git-svn-id: http://svn.digium.com/svn/asterisk/trunk@6420 f38db490-d61c-443f-a65b-d21fe96a405b

1 files changed, 147 insertions, 221 deletions

diff --git a/doc/README.variables b/doc/README.variables
index e072dc8be..fc96bd446 100755
--- a/doc/README.variables
+++ b/doc/README.variables
@@ -13,15 +13,6 @@ Asterisk has user-defined variables and standard variables set
 by various modules in Asterisk. These standard variables are
 listed at the end of this document.
 
-NOTE: During the Asterisk build process, the versions of bison and
-flex available on your system are probed. If you have versions of
-flex greater than or equal to 2.5.31, it will use flex to build a 
-"pure" (re-entrant) tokenizer for expressions. If you use bison version 
-greater than 1.85, it will use a bison grammar to generate a pure (re-entrant) 
-parser for $[] expressions. 
-Notes specific to the flex parser are marked with "**" at the beginning
-of the line.
-
 ___________________________
 PARAMETER QUOTING: 
 ---------------------------
@@ -131,13 +122,6 @@ considered as an expression and it is evaluated. Evaluation works similar to
 (but is done on a later stage than) variable substitution: the expression 
 (including the square brackets) is replaced by the result of the expression 
 evaluation. 
-Note: The arguments and operands of the expression MUST BE separated 
-by at least one space. 
-** Using the Flex generated tokenizer, this is no longer the case. Spaces
-** are only required where they would seperate tokens that would normally
-** be merged into a single token. Using the new tokenizer, spaces can be
-** used freely.
-
 
 For example, after the sequence: 
 
@@ -146,26 +130,9 @@ exten => 1,2,Set(koko=$[2 * ${lala}])
 
 the value of variable koko is "6".
 
-** Using the new Flex generated tokenizer, the expressions above are still
-** legal, but so are the following:
-** exten => 1,1,Set(lala=$[1+2])
-** exten => 1,2,Set(koko=$[2*    ${lala}])
-
-And, further:
-
-exten => 1,1,Set(lala=$[1+2]);
-
-will not work as you might have expected. Since all the chars in the single 
-token "1+2" are not numbers, it will be evaluated as the string "1+2". Again,
-please do not forget, that this is a very simple parsing engine, and it
-uses a space (at least one), to separate "tokens".
-
-** Please note that spaces are not required to separate tokens if you have
-** Flex version 2.5.31 or higher on your system.
- 
 and, further:
 
-exten => 1,1,Set,"lala=$[  1 +    2   ]";
+exten => 1,1,Set,(lala=$[  1 +    2   ]);
 
 will parse as intended. Extra spaces are ignored.
 
@@ -227,17 +194,17 @@ with equal precedence are grouped within { } symbols.
              Return the results of multiplication, integer division, or
              remainder of integer-valued arguments.
 
-**   - expr1
-**          Return the result of subtracting expr1 from 0.
-**          This, the unary minus operator, is right associative, and
-**          has the same precedence as the ! operator.
-**
-**   ! expr1
-**          Return the result of a logical complement of expr1.
-**          In other words, if expr1 is null, 0, an empty string,
-**          or the string "0", return a 1. Otherwise, return a "0". (only with flex >= 2.5.31)
-**          It has the same precedence as the unary minus operator, and
-**          is also right associative.
+     - expr1
+            Return the result of subtracting expr1 from 0.
+            This, the unary minus operator, is right associative, and
+            has the same precedence as the ! operator.
+
+     ! expr1
+            Return the result of a logical complement of expr1.
+            In other words, if expr1 is null, 0, an empty string,
+            or the string "0", return a 1. Otherwise, return a 0.
+            It has the same precedence as the unary minus operator, and
+            is also right associative.
 
      expr1 : expr2
              The `:' operator matches expr1 against expr2, which must be a
@@ -256,43 +223,36 @@ with equal precedence are grouped within { } symbols.
              before the regex match is made, beginning and ending double quote
              characters are stripped from both the pattern and the string.
 
-**    expr1 =~ expr2
-**           Exactly the same as the ':' operator, except that the match is
-**           not anchored to the beginning of the string. Pardon any similarity
-**           to seemingly similar operators in other programming languages!
-**           (only if flex >= 2.5.31). The ":" and "=~" operators share the 
-**           same precedence.
-
-**    expr1 ? expr2 :: expr3
-**           Traditional Conditional operator. If expr1 is a number that evaluates
-**           to 0 (false), expr3 is result of the this expression evaluation.
-**           Otherwise, expr2 is the result.
-**           If expr1 is a string, and evaluates to an empty string, or the two
-**           characters (""), then expr3 is the result. Otherwise, expr2 is the result.
-**           In Asterisk, all 3 exprs will be "evaluated"; if expr1 is "true",
-**           expr2 will be the result of the "evaluation" of this expression.
-**           expr3 will be the result otherwise. This operator has the lowest
-**           precedence.
+      expr1 =~ expr2
+             Exactly the same as the ':' operator, except that the match is
+             not anchored to the beginning of the string. Pardon any similarity
+             to seemingly similar operators in other programming languages!
+             The ":" and "=~" operators share the same precedence.
+
+      expr1 ? expr2 :: expr3
+             Traditional Conditional operator. If expr1 is a number
+	     that evaluates to 0 (false), expr3 is result of the this
+	     expression evaluation.  Otherwise, expr2 is the result.
+	     If expr1 is a string, and evaluates to an empty string,
+	     or the two characters (""), then expr3 is the
+	     result. Otherwise, expr2 is the result.  In Asterisk, all
+	     3 exprs will be "evaluated"; if expr1 is "true", expr2
+	     will be the result of the "evaluation" of this
+	     expression.  expr3 will be the result otherwise. This
+	     operator has the lowest precedence.
 
 Parentheses are used for grouping in the usual manner.
 
 Operator precedence is applied as one would expect in any of the C
 or C derived languages.
 
-The parser must be generated with bison (bison is REQUIRED - yacc cannot 
-produce pure parsers, which are reentrant)  The same with flex, if flex
-is at 2.5.31 or greater; Re-entrant scanners were not available before that
-version.
-
-
-
 Examples
 
-** "One Thousand Five Hundred" =~ "(T[^ ]+)"
-**	returns: Thousand
+ "One Thousand Five Hundred" =~ "(T[^ ]+)"
+	returns: Thousand
 
-** "One Thousand Five Hundred" =~ "T[^ ]+"
-**	returns: 8
+ "One Thousand Five Hundred" =~ "T[^ ]+"
+	returns: 8
 
  "One Thousand Five Hundred" : "T[^ ]+"
 	returns: 0
@@ -303,34 +263,37 @@ Examples
  "3075551212":"...(...)"
 	returns: 555
 
-** ! "One Thousand Five Hundred" =~ "T[^ ]+"
-**	returns: 0 (because it applies to the string, which is non-null, which it turns to "0",
-                    and then looks for the pattern in the "0", and doesn't find it)
+ ! "One Thousand Five Hundred" =~ "T[^ ]+"
+	returns: 0 (because it applies to the string, which is non-null,
+                    which it turns to "0", and then looks for the pattern
+                    in the "0", and doesn't find it)
 
-** !( "One Thousand Five Hundred" : "T[^ ]+" )
-**	returns: 1  (because the string doesn't start with a word starting with T, so the
-                     match evals to 0, and the ! operator inverts it to 1 ).
+ !( "One Thousand Five Hundred" : "T[^ ]+" )
+	returns: 1  (because the string doesn't start with a word starting
+                     with T, so the match evals to 0, and the ! operator
+                     inverts it to 1 ).
 
  2 + 8 / 2
 	returns 6. (because of operator precedence; the division is done first, then the addition).
 
-** 2+8/2
-**	returns 6. Spaces aren't necessary.
+ 2+8/2
+	returns 6. Spaces aren't necessary.
 
-**(2+8)/2
-**	returns 5, of course.
+(2+8)/2
+	returns 5, of course.
 
-Of course, all of the above examples use constants, but would work the same if any of the
-numeric or string constants were replaced with a variable reference ${CALLERIDNUM}, for
-instance.
+Of course, all of the above examples use constants, but would work the
+same if any of the numeric or string constants were replaced with a
+variable reference ${CALLERIDNUM}, for instance.
 
 __________________________
 NUMBERS VS STRINGS
 --------------------------
 
-Tokens consisting only of numbers are converted to 64-bit numbers for most of the
-operators. This means that overflows can occur when the numbers get above 18 digits.
-Warnings will appear in the logs in this case.
+Tokens consisting only of numbers are converted to 64-bit numbers for
+most of the operators. This means that overflows can occur when the
+numbers get above 18 digits.  Warnings will appear in the logs in this
+case.
 ___________________________
 CONDITIONALS
 ---------------------------
@@ -349,7 +312,6 @@ above, eg :
 
 	exten => 1,2,gotoif($[${CALLERID} = 123456]?2|1:3|1)
 
-
 Example of use : 
 
 exten => s,2,Set(vara=1)
@@ -369,44 +331,18 @@ exten => s,6,GotoIf($[ "${CALLERIDNUM}"  = "3071234567" & &  "${CALLERIDNAME}" :
 
 You may see an error in /var/log/asterisk/messages like this:
 
-May  3 15:58:53 WARNING[1234455344]: ast_yyerror(): syntax error: parse error; Input:
- "3072312154"  : "3071234567" & & "Steves Extension" : "Privacy Manager"
- ^^^^^^^^^^^^^^^^^^^^^^^^^^^
-                                  ^
-
-The first line shows the string passed to the expression parser. This
-string is the result of the variable replacements, etc. This way, you
-can see the actual string that went into the parser.
-
-The second line usually shows a string of '^' chars, that show what's
-been legally parsed so far.
-
-And the third line shows where the parser was (lookahead token lexing,
-etc), when the parse hit the rocks. A single '^' here. The error is
-going to be somewhere between the last '^' on the second line, and the
-'^' on the third line. That's right, in the example above, there are two
-'&' chars, separated by a space, and this is a definite no-no!
-
-** WITH FLEX >= 2.5.31, this has changed slightly. The line showing the 
-** part of the expression that was successfully parsed has been dropped,
-** and the parse error is explained in a somewhat cryptic format in the log.
-** 
-** The same line in extensions.conf as above, will now generate an error 
-** message in /var/log/asterisk/messages that looks like this:
-**
-** Jul 15 21:27:49 WARNING[1251240752]: ast_yyerror(): syntax error: parse error, unexpected TOK_AND, expecting TOK_MINUS or TOK_LP or TOKEN; Input:
-** "3072312154"  = "3071234567" & & "Steves Extension" : "Privacy Manager" 
-**                                ^
-**
-** The log line tells you that a syntax error was encountered. It now
-** also tells you (in grand standard bison format) that it hit an "AND" (&)
-** token unexpectedly, and that was hoping for for a MINUS (-), LP (left parenthesis),
-** or a plain token (a string or number).
-** 
-** As before, the next line shows the evaluated expression, and the line after
-** that, the position of the parser in the expression when it became confused,
-** marked with the "^" character.
+Jul 15 21:27:49 WARNING[1251240752]: ast_yyerror(): syntax error: parse error, unexpected TOK_AND, expecting TOK_MINUS or TOK_LP or TOKEN; Input:
+"3072312154"  = "3071234567" & & "Steves Extension" : "Privacy Manager" 
+                               ^
+
+The log line tells you that a syntax error was encountered. It now
+also tells you (in grand standard bison format) that it hit an "AND"
+(&) token unexpectedly, and that was hoping for for a MINUS (-), LP
+(left parenthesis), or a plain token (a string or number).
 
+The next line shows the evaluated expression, and the line after
+that, the position of the parser in the expression when it became confused,
+marked with the "^" character.
 
 ___________________________
 NULL STRINGS
@@ -443,11 +379,12 @@ INCOMPATIBILITIES
 The asterisk expression parser has undergone some evolution. It is hoped
 that the changes will be viewed as positive. 
 
-The "original" expression parser had a simple, hand-written scanner, and 
-a simple bison grammar. This was upgraded to a more involved bison grammar,
-and a hand-written scanner upgraded to allow extra spaces, and to generate
-better error diagnostics. This upgrade required bison 1.85, and a [art of the user
-community felt the pain of having to upgrade their bison version.
+The "original" expression parser had a simple, hand-written scanner,
+and a simple bison grammar. This was upgraded to a more involved bison
+grammar, and a hand-written scanner upgraded to allow extra spaces,
+and to generate better error diagnostics. This upgrade required bison
+1.85, and part of the user community felt the pain of having to
+upgrade their bison version.
 
 The next upgrade included new bison and flex input files, and the makefile
 was upgraded to detect current version of both flex and bison, conditionally
@@ -465,18 +402,18 @@ of possible concern with "legacy" extension.conf files:
 
 1. Tokens separated by space(s).
    Previously, tokens were separated by spaces. Thus, ' 1 + 1 ' would evaluate
-  to the value '2', but '1+1' would evaluate to the string '1+1'. If this
-  behavior was depended on, then the expression evaluation will break. '1+1'
-  will now evaluate to '2', and something is not going to work right.
-  To keep such strings from being evaluated, simply wrap them in double 
-  quotes: '  "1+1" '
+   to the value '2', but '1+1' would evaluate to the string '1+1'. If this
+   behavior was depended on, then the expression evaluation will break. '1+1'
+   will now evaluate to '2', and something is not going to work right.
+   To keep such strings from being evaluated, simply wrap them in double 
+   quotes: '  "1+1" '
 
 2. The colon operator. In versions previous to double quoting, the
    colon operator takes the right hand string, and using it as a 
    regex pattern, looks for it in the left hand string. It is given
    an implicit ^ operator at the beginning, meaning the pattern 
    will match only at the beginning of the left hand string. 
-     If the pattern or the matching string had double quotes around
+   If the pattern or the matching string had double quotes around
    them, these could get in the way of the pattern match. Now,
    the wrapping double quotes are stripped from both the pattern 
    and the left hand string before applying the pattern. This
@@ -499,98 +436,87 @@ of possible concern with "legacy" extension.conf files:
    but they were not documented. The symbolic operators, <=, >=, and !=
    should be used instead.
 
-**5. flex 2.5.31 or greater should be used. Bison-1.875 or greater. In
-**   the case of flex, earlier versions do not generate 'pure', or 
-**   reentrant C scanners. In the case of bison-1.875, earlier versions
-**   didn't support the location tracking mechanism.
+5.  Added the unary '-' operator. So you can 3+ -4 and get -1.
 
-**    http://ftp.gnu.org/gnu/bison/bison-1.875.tar.bz2
-**    http://prdownloads.sourceforge.net/lex/flex-2.5.31.tar.bz2?download
-**	or http://lex.sourceforge.net/
+6.  Added the unary '!' operator, which is a logical complement.
+    Basically, if the string or number is null, empty, or '0',
+    a '1' is returned. Otherwise a '0' is returned.
 
-**6.  Added the unary '-' operator. So you can 3+ -4 and get -1.
+7.  Added the '=~' operator, just in case someone is just looking for
+    match anywhere in the string. The only diff with the ':' is that
+    match doesn't have to be anchored to the beginning of the string.
 
-**7.  Added the unary '!' operator, which is a logical complement.
-**    Basically, if the string or number is null, empty, or '0',
-**    a '1' is returned. Otherwise a '0' is returned.
+8.  Added the conditional operator  'expr1 ? true_expr :: false_expr'
+    First, all 3 exprs are evaluated, and if expr1 is false, the 'false_expr'
+    is returned as the result. See above for details. 
 
-**8.  Added the '=~' operator, just in case someone is just looking for
-**    match anywhere in the string. The only diff with the ':' is that
-**    match doesn't have to be anchored to the beginning of the string.
-
-**9.  Added the conditional operator  'expr1 ? true_expr :: false_expr'
-**    First, all 3 exprs are evaluated, and if expr1 is false, the 'false_expr'
-**    is returned as the result. See above for details. 
-
-**10. Unary operators '-' and '!' were made right associative.
+9.  Unary operators '-' and '!' were made right associative.
 
 --------------------------------------------------------
 DEBUGGING HINTS FOR $[  ]  EXPRESSIONS
 --------------------------------------------------------
 
-** THE FOLLOWING UTILITIES ARE PROVIDED ONLY FOR SYSTEMS THAT
-** HAVE FLEX-2.5.31 OR GREATER, AND USE THE UPGRADED LEXER!!!
-**
-** There are two utilities you can build to help debug the $[ ] in
-** your extensions.conf file.
-**
-** The first, and most simplistic, is to issue the command:
-**
-** make testexpr2
-**
-** in the top level asterisk source directory. This will build
-** a small executable, that is able to take the first command line
-** argument, and run it thru the expression parser. No variable
-** substitutions will be performed. It might be safest to wrap
-** the expression in single quotes...
-**
-** testexpr2 '2*2+2/2'
-**
-** is an example.
-**
-** And, in the utils directory, you can say:
-**
-** make check_expr
-**
-** and a small program will be built, that will check the file
-** mentioned in the first command line argument, for any expressions
-** that might be have problems when you move to flex-2.5.31.
-** It was originally designed to help spot possible incompatibilities
-** when moving from the pre-2.5.31 world to the upgraded version of
-** the lexer.
-**
-** But one more capability has been added to check_expr, that might
-** make it more generally useful. It now does a simple minded evaluation of
-** all variables, and then passes the $[] exprs to the parser. If there are
-** any parse errors, they will be reported in the log file. You can use
-** check_expr to do a quick sanity check of the expressions in your 
-** extensions.conf file, to see if they pass a crude syntax check.
-**
-** The "simple-minded" variable substitution replaces ${varname} variable
-** references with '555'. You can override the 555 for variable values,
-** by entering in var=val arguments after the filename on the command line.
-** So...
-**
-** check_expr /etc/asterisk/extensions.conf CALLERIDNUM=3075551212 DIALSTATUS=TORTURE EXTEN=121
-**
-** will substitute any ${CALLERIDNUM} variable references with 3075551212, any ${DIALSTATUS} 
-** variable references with 'TORTURE', and any ${EXTEN} references with '121'.
-** If there is any fancy stuff going on in the reference, like ${EXTEN:2}, then the
-** override will not work. Everything in the ${...} has to match. So, to substitute
-** #{EXTEN:2} references, you'd best say:
-**
-** check_expr /etc/asterisk/extensions.conf CALLERIDNUM=3075551212 DIALSTATUS=TORTURE EXTEN:2=121
-**
-** on stdout, you will see something like:
-**
-** OK -- $[ "${DIALSTATUS}"  = "TORTURE" | "${DIALSTATUS}" = "DONTCALL" ] at line 416
-**
-** In the expr2_log file that is generated, you will see:
-**
-** line 416, evaluation of $[  "TORTURE"  = "TORTURE" | "TORTURE" = "DONTCALL"  ] result: 1
-**
-** check_expr is a very simplistic algorithm, and it is far from being guaranteed 
-** to work in all cases, but it is hoped that it will be useful.
+There are two utilities you can build to help debug the $[ ] in
+your extensions.conf file.
+
+The first, and most simplistic, is to issue the command:
+
+make testexpr2
+
+in the top level asterisk source directory. This will build a small
+executable, that is able to take the first command line argument, and
+run it thru the expression parser. No variable substitutions will be
+performed. It might be safest to wrap the expression in single
+quotes...
+
+testexpr2 '2*2+2/2'
+
+is an example.
+
+And, in the utils directory, you can say:
+
+make check_expr
+
+and a small program will be built, that will check the file mentioned
+in the first command line argument, for any expressions that might be
+have problems when you move to flex-2.5.31.  It was originally
+designed to help spot possible incompatibilities when moving from the
+pre-2.5.31 world to the upgraded version of the lexer.
+
+But one more capability has been added to check_expr, that might make
+it more generally useful. It now does a simple minded evaluation of
+all variables, and then passes the $[] exprs to the parser. If there
+are any parse errors, they will be reported in the log file. You can
+use check_expr to do a quick sanity check of the expressions in your
+extensions.conf file, to see if they pass a crude syntax check.
+
+The "simple-minded" variable substitution replaces ${varname} variable
+references with '555'. You can override the 555 for variable values,
+by entering in var=val arguments after the filename on the command
+line.  So...
+
+ check_expr /etc/asterisk/extensions.conf CALLERIDNUM=3075551212 DIALSTATUS=TORTURE EXTEN=121
+
+will substitute any ${CALLERIDNUM} variable references with
+3075551212, any ${DIALSTATUS} variable references with 'TORTURE', and
+any ${EXTEN} references with '121'.  If there is any fancy stuff
+going on in the reference, like ${EXTEN:2}, then the override will
+not work. Everything in the ${...} has to match. So, to substitute
+#{EXTEN:2} references, you'd best say:
+
+ check_expr /etc/asterisk/extensions.conf CALLERIDNUM=3075551212 DIALSTATUS=TORTURE EXTEN:2=121
+
+on stdout, you will see something like:
+
+ OK -- $[ "${DIALSTATUS}"  = "TORTURE" | "${DIALSTATUS}" = "DONTCALL" ] at line 416
+
+In the expr2_log file that is generated, you will see:
+
+ line 416, evaluation of $[  "TORTURE"  = "TORTURE" | "TORTURE" = "DONTCALL"  ] result: 1
+
+check_expr is a very simplistic algorithm, and it is far from being
+guaranteed to work in all cases, but it is hoped that it will be
+useful.
 
 ---------------------------------------------------------
 Asterisk standard channel variables