MDEV-38752: Wrong result upon GROUP BY on a table with indexed virtual column after INSERT IGNORE

[kjarir]

Description

This PR fixes MDEV-38752, a regression introduced in commit 8cdee25952763a0401e4c2a4d61e92c13499bdc6.

The issue was caused by the optimizer incorrectly substituting expressions in GROUP BY or ORDER BY with indexed virtual columns that had "narrower" data types (e.g., substituting an INT expression with a TINYINT virtual column). After an INSERT IGNORE, data truncation in the virtual column caused distinct values from the original expression to merge into a single group in the query result.

Changes

• Updated subst_vcol_if_compatible() in sql/opt_vcol_substitution.cc to perform stricter type compatibility checks.
• Substitution is now only allowed if the virtual column's Type_handler matches the expression and its max_length and decimals are sufficient to cover the expression's range without truncation.
• Optimization remains active for safe cases where no data loss occurs.

Testing

• Added regression test mysql-test/main/mdev_38752.test covering:
  • Numeric narrowing (INT to TINYINT).
  • String narrowing (VARCHAR(20) to VARCHAR(5)).
  • Verification that safe substitutions still use the virtual column index.

[gkodinov]

Thank you for your contribution! This is a preliminary review.

Please rebase to 12.3: this is a bug fix and it needs to cover all affected versions

[gkodinov]

Please increase verbosity so that the note about vcol impossible substitution is actually printed. Or do an EXPLAIN: it should do it too.

[gkodinov]

Add a test for different decimals as well please.

[gkodinov]

we customarily end tests with a print like that:

--echo End of ... tests.

[gkodinov]

Please also make sure all of the related failing tests are re-recorded.

[CLAassistant]

All committers have signed the CLA.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
4 out of 5 committers have signed the CLA.

✅ forkfun
✅ dbart
✅ kjarir
✅ ericherman
❌ knielsen
_{You have signed the CLA already but the status is still pending? Let us recheck it.}