Improve parsing of % followed immediately by identifier

If it's not a recognised directive the scanner now emits MODULO and then
rescans what follows, and if the parser then gives a syntax error we
report it as an unknown directive. This means that `a%b` is now allowed
in an expression, and that things like `%std::vector<std::string>` now
give an error rather than being quietly ignored.

Fixes #300
Fixes #368

CHANGES.current

@@ -7,6 +7,15 @@ the issue number to the end of the URL: https://github.com/swig/swig/issues/
 Version 4.1.0 (in progress)
 ===========================
 
+2022-02-15: olly
+	    #300 #368 Improve parser handling of % followed immediately by
+	    an identifier.  If it's not a recognised directive the scanner
+	    now emits MODULO and then rescans what follows, and if the parser
+	    then gives a syntax error we report it as an unknown directive.
+	    This means that `a%b` is now allowed in an expression, and that
+	    things like `%std::vector<std::string>` now give an error rather
+	    than being quietly ignored.
+
 2022-02-10: olly
 	    [Tcl] https://sourceforge.net/p/swig/bugs/1207/
 	    https://sourceforge.net/p/swig/bugs/1213/

Examples/test-suite/constant_expr.i

@@ -14,4 +14,13 @@ isPointer = false
 };
 };
 
+/* Regression test for #300, fixed in 4.1.0.
+ *
+ * Now `a%b` without a space after the `%` is handled as a modulus operator,
+ * but it gave a cryptic `Syntax error in input(1)` before SWIG 3.0.4, and from
+ * SWIG 3.0.4 until 4.1.0, `Unknown directive '%a'`.
+ */
+int a;
+int test2(int b = 9%a);
+
 %}

Examples/test-suite/errors/pp_unknowndirective.i

@@ -1,10 +1,5 @@
 %module xxx
 
-/* Regression test for bug introduced in 3.0.4 and fixed in 3.0.6 - the '%std'
- * here led to SWIG calling abort().
- */
-%typemap(jstype) std::vector<std::string>, const %std::vector<std::string>&, std::vector<std::string>&  "List<String>"
-
 /* This used to give the rather cryptic "Syntax error in input(1)." prior to
  * SWIG 3.0.4.  This testcase checks that the improved message is actually
  * issued.

Examples/test-suite/errors/pp_unknowndirective.stderr

@@ -1 +1 @@
-pp_unknowndirective.i:12: Error: Unknown directive '%remane'.
+pp_unknowndirective.i:7: Error: Unknown directive '%remane'.

Examples/test-suite/errors/pp_unknowndirective4.i

@@ -0,0 +1,7 @@
+%module xxx
+
+/* Regression test for bug #368 introduced in 3.0.4 and fully fixed in 4.1.0.
+ * The `%std` here led to SWIG calling abort() before 3.0.6, and was quietly
+ * ignored in from 3.0.6 until 4.1.0.
+ */
+%typemap(jstype) std::vector<std::string>, const %std::vector<std::string>&, std::vector<std::string>&  "List<String>"

Examples/test-suite/errors/pp_unknowndirective4.stderr

@@ -0,0 +1 @@
+pp_unknowndirective4.i:7: Error: Unknown directive '%std'.

Source/CParse/cscanner.c

@@ -1031,8 +1031,23 @@ int yylex(void) {
       if (strcmp(yytext, "%warn") == 0)
 	return (WARN);
 
-      /* Note down the apparently unknown directive for error reporting. */
+      /* Note down the apparently unknown directive for error reporting - if
+       * we end up reporting a generic syntax error we'll instead report an
+       * error for his as an unknown directive.  Then we treat it as MODULO
+       * (`%`) followed by an identifier and if that parses OK then
+       * `cparse_unknown_directive` doesn't get used.
+       *
+       * This allows `a%b` to be handled in expressions without a space after
+       * the operator.
+       */
       cparse_unknown_directive = NewString(yytext);
+      String *stext = NewString(yytext + 1);
+      Seek(stext,0,SEEK_SET);
+      Setfile(stext,cparse_file);
+      Setline(stext,cparse_line);
+      Scanner_push(scan,stext);
+      Delete(stext);
+      return (MODULO);
     }
     /* Have an unknown identifier, as a last step, we'll do a typedef lookup on it. */