[minleejae]

Purpose

Several MySQL 8.4 DML statements fail to parse, and several INSERT and REPLACE forms render differently in toString() and StatementDeParser:

• Index hints: SELECT a FROM t IGNORE INDEX FOR ORDER BY (i1) ORDER BY a, SELECT a FROM t FORCE INDEX FOR JOIN (i1) JOIN s ON t.a = s.a, USE INDEX (), a second hint on the same table (USE INDEX (i1) IGNORE INDEX (i2)) and IGNORE KEY (i1) fail.
• WEIGHT_STRING: SELECT WEIGHT_STRING('x' AS CHAR(3)) fails, also with AS BINARY(n).
• INSERT and REPLACE: INSERT INTO t VALUES ROW(1, 2), ROW(3, 4), INSERT INTO t TABLE s, INSERT INTO t () VALUES () and REPLACE INTO t () VALUES () fail, although the queries VALUES ROW(1, 2) and TABLE s parse.
• DELETE: DELETE FROM t1 USING t1 JOIN t2 ON t1.id = t2.id fails. PostgreSQL's DELETE FROM t USING s JOIN u ON … fails the same way.
• INSERT INTO t PARTITION (p0) (a) VALUES (1) renders as INSERT INTO t (a) PARTITION (p0) VALUES (1). MySQL rejects that, and JSqlParser cannot parse it again.
• StatementDeParser renders INSERT INTO t (a, b) VALUES (1, 2) AS new ON DUPLICATE KEY UPDATE … and REPLACE INTO t(a, b) VALUES (1, 2), while toString() uses single spaces.

Current design (As-is)

• Index hints: Table holds one immutable MySQLIndexHint (action, INDEX/KEY and the names). The grammar reads at most one hint per table and has no FOR scope. IGNORE before KEY is read as an alias of the table. Table.appendTo and SelectDeParser.visit(Table) each render the hint.
• WEIGHT_STRING: CHAR(77 USING utf8mb4) is a Function with a keyword argument. AS is not accepted inside a function call.
• INSERT sources: INSERT … VALUES ROW(…) and INSERT … TABLE s already go through Select(), which yields Values and TableStatement. The alias check before the source reads VALUES (before ROW) and TABLE as an alias of the target, so the source is never reached. The column list requires at least one column.
• INSERT rendering: Insert.toString() and InsertDeParser are two hand-written copies, and they disagree:
  • Both render PARTITION after the column list and OVERRIDING; toString() adds a second space.
  • The deparser renders the priority before the optimizer hint; the grammar and toString() put the hint first.
  • For ON DUPLICATE KEY UPDATE … WHERE …, the deparser drops the WHERE.
  • The ON CONFLICT action is appended through its toString(), so the deparser's expression visitor never sees it.
  • toString() leaves double spaces before OUTPUT and the SET … AS new row alias, and ON DUPLICATE KEY UPDATE NOTHING renders with extra spaces.
  • Insert.duplicateUpdateSets is never written.
• REPLACE rendering: UpsertDeParser renders the column list without a space, and it renders ON DUPLICATE KEY UPDATE only when there is no SET.
• DELETE … USING: USING is FromItem (, FromItem)* and cannot take a JOIN. Delete.toString() and DeleteDeParser render the target's joins after the USING list, so DELETE FROM t1, t2 USING t1, t2 WHERE … renders as DELETE FROM t1 USING t1, t2, t2 WHERE …. DeleteDeParser drops the aliases of the target tables that toString() keeps. StatementVisitorAdapter casts every USING item to Table, which fails for a subquery.

Target design (To-be)

• Index hints:
  • New MySQLIndexHint.Scope enum (JOIN, ORDER_BY, GROUP_BY), with a 4-argument constructor; the 3-argument constructor stays, with no scope.
  • Table.getIndexHints() holds every hint in source order. getIndexHint() returns the first one, and setHint() replaces the list with one hint (or clears it). New setIndexHints, withIndexHints, addIndexHint.
  • One list production, MySQLIndexHints(), is used for FROM items, joined tables and the UPDATE target. One render helper, Table.appendIndexHintsTo, is used by toString() and SelectDeParser. SelectValidator checks the index names of every hint.
  • USE INDEX () has an empty name list. IGNORE/FORCE with an empty list are rejected, as the server rejects them.
  • IGNORE followed by KEY is no longer taken as an alias, as for IGNORE INDEX, so IGNORE KEY (i1) is a hint.
• WEIGHT_STRING: Function.getCastType() holds the AS CHAR(n) / AS BINARY(n) type as a ColDataType, built as CAST('x' AS CHAR(3)) builds it, with an AST node spanning CHAR(3). Both renderers write it after the argument. The grammar accepts exactly CHAR(n) or BINARY(n) with an integer length, after the single argument of WEIGHT_STRING; other types, CHARACTER SET, other functions and several arguments are rejected.
• INSERT sources:
  • The alias check no longer takes VALUES ROW( or TABLE <name> as an alias. The sources reuse the query models: Values with ROW(…) items, as for VALUES ROW(1, 2), and TableStatement. VALUE ROW(…) stays rejected, since only VALUES starts a row constructor list.
  • A new production InsertColumnList() accepts (), and INSERT and REPLACE share it. INSERT INTO t () VALUES () has an empty, non-null getColumns(); without a column list it stays null. REPLACE keeps its ParenthesedExpressionList, with an AST node spanning the parentheses.
• INSERT rendering:
  • New Insert.appendTo(builder, withItemRenderer, selectRenderer, expressionRenderer, selectItemRenderer) writes the clauses in grammar order. toString() passes builder::append; InsertDeParser passes its visitors.
  • PARTITION follows the table, before the column list. The optimizer hint comes before the priority.
  • InsertDuplicateAction, InsertConflictAction and OracleMultiInsertClause get the same callback appendTo, and UpdateSet.appendUpdateSetsTo(builder, sets, expressionRenderer) renders the assignments. The deparser keeps ON DUPLICATE KEY UPDATE … WHERE … and visits the ON CONFLICT action.
  • The unused private duplicateUpdateSets field is removed. getDuplicateUpdateSets() returns the action's update sets, as before.
  • ValuesStatementDeParser renders the AS new row alias with one space. UpsertDeParser renders REPLACE INTO t (a, b) and the ON DUPLICATE KEY UPDATE action through InsertDuplicateAction.appendTo.
• DELETE … USING:
  • New Delete.getUsingJoins() holds the joins after the USING items, for MySQL's DELETE FROM t1 USING t1 JOIN t2 … and PostgreSQL's DELETE FROM t USING s JOIN u …. A comma list without joins keeps today's getUsingFromItemList().
  • getJoins() keeps the target's joins, as in DELETE FROM t1, t2 USING …. Both renderers write them before USING.
  • DeleteDeParser renders target tables with their aliases.
  • TablesNamesFinder and StatementVisitorAdapter visit the USING items and joins, and DeleteValidator validates the USING joins like the target's joins.
• AST nodes: the new parsed objects that implement ASTNodeAccess (the WEIGHT_STRING cast type, REPLACE's column list, the USING joins) carry nodes spanning their source. ASTNodeCoverageTest walks every form above.
• Grammar: no new tokens or reserved words. Three productions (MySQLIndexHints, InsertColumnList/ParenthesedInsertColumnList, WeightStringCastType), two semantic lookahead helpers (isInsertTableSourceAhead, isFunctionCastTypeAhead), and two follower conditions in the alias check (VALUES ROW(, IGNORE KEY). The grammar choice conflict check passes. The new forms parse in every dialect, like the index hints and VALUES ROW(…) queries today; no existing reading changes.
• Compatibility:
  • New API: MySQLIndexHint.Scope, Table.getIndexHints(), Function.getCastType(), Delete.getUsingJoins(), Insert.appendTo(…) and the callback appendTo methods. Table.getIndexHint()/setHint()/withHint() keep working.
  • Rendering changes, all toward the parsed text: INSERT … PARTITION order and spacing; ON DUPLICATE KEY UPDATE NOTHING (InsertDuplicateAction.toString() is now NOTHING); the deparser's WHERE, hint order and spacing; target joins of DELETE before USING; target aliases in DeleteDeParser.
  • A Delete built with both joins and USING items now renders the joins before USING. Joins of the USING clause belong in usingJoins.
• usage.rst documents the hint list and scope, getCastType(), the insert sources and getUsingJoins().

Validation

• 109 new tests:

  • MySqlIndexHintTest (22): the reported statements with Dialect.MYSQL and without a dialect, exact toString() and deparser output, every scope, USE KEY, IGNORE KEY, USE INDEX (), several hints, hints on joined tables and the UPDATE target, FOR UPDATE after a hint, rejected forms, statement boundary, TablesNamesFinder, validation and model edits.
  • WeightStringFunctionTest (20): AS CHAR(n) and AS BINARY(n) with and without a dialect, the same ColDataType as CAST, the cast type's node, the CHAR(… USING …) keyword argument, nesting, rejected forms (other types, CHARACTER SET, a non-integer length, several arguments, DISTINCT, other functions), statement boundary and the model.
  • MySqlInsertSourceTest (18): VALUES ROW(…), TABLE s and () VALUES () for INSERT and REPLACE, AST fields, REPLACE's column list node, TABLE still as an alias before other sources, other dialects, rejected forms (including VALUE ROW(…)), statement boundaries, TablesNamesFinder and validation.
  • InsertRenderingTest (32): identical toString(), deparser and reparse output for PARTITION, row aliases, ON DUPLICATE KEY UPDATE … WHERE, NOTHING, OUTPUT, ON CONFLICT, BY NAME, WITH, Oracle multi-table INSERT and REPLACE/UPSERT; visited expressions in both actions and the SET list; appendTo callbacks; getDuplicateUpdateSets().
  • DeleteUsingJoinTest (17): MySQL and PostgreSQL USING joins, several targets, comma lists followed by joins, target alias rendering, unchanged forms, rejected forms, statement boundary, TablesNamesFinder, StatementVisitorAdapter, validation and the model.
  • ASTNodeCoverageTest walks the new forms.

• Java 17 offline Gradle spotlessApply check: 12,738 tests, 12,713 passed, 25 skipped, zero failures/errors. The grammar choice conflict check passes.

• Compared with master (4a970a34e) on a corpus of 188 MySQL 8.4 and 166 PostgreSQL 18 statements other than DDL/DCL, each accepted by the server, with the MySQL or PostgreSQL dialect and without a dialect:

  • Fixed with Dialect.MYSQL and without a dialect (11 each):
    • now parsed: the two index-hint queries, WEIGHT_STRING('x' AS CHAR(3)), INSERT … VALUES ROW(…), INSERT INTO t TABLE s, INSERT INTO t () VALUES () and DELETE FROM t1 USING t1 JOIN t2 …;
    • now rendered the same by toString() and the deparser: INSERT INTO t PARTITION (p0) (a) VALUES (1) (which also parses again), both INSERT … AS new … statements and REPLACE INTO t (a, b) VALUES (1, 2).
  • 0 regressed and 0 changed in status, statement class or rendering. The PostgreSQL results are unchanged.

• Compared with master (4a970a34e) on a corpus of 635 MySQL 8.4 and PostgreSQL 18 DDL/DCL statements, each accepted by the server, plus variants, scripts and the keyword sweeps: 0 regressed, 0 changed.

• On 395 INSERT, REPLACE, UPSERT and DELETE statements taken from the test suite and the corpora, every statement that master parses (312) still parses, and 4 more parse. toString() and StatementDeParser differ for 34 of them on master and for none on this branch.

• The statements of the new tests were run against MySQL 8.4.11 (63 accepted and 31 rejected forms) and PostgreSQL 18.3 (11 forms). Every form the parser accepts with the dialect is accepted by the server's grammar, and every form it rejects is a syntax error on the server. One leniency stays as on master: MySQL rejects a row alias after a VALUES ROW(…) list (INSERT INTO t (a, b) VALUES ROW(1, 2), ROW(3, 4) AS new …), which master already parses.

AI assistance: prepared with Claude Code.