MAINT Migration from mypy to ty by maifeeulasad · Pull Request #1319 · microsoft/PyRIT

maifeeulasad · 2026-01-17T08:01:22Z

Description

Migrates PyRIT from mypy to ty (Astral's Rust-based Python type checker). ty is 10-100x faster than mypy, making type checking practical in pre-commit hooks and CI without caching workarounds.

What this PR does

Core migration:

Replaces mypy with ty in dev dependencies (pyproject.toml, >=0.0.32)
Removes the mypy pre-commit hook from .pre-commit-config.yaml
Adds ty configuration in pyproject.toml under [tool.ty]
Updates Makefile target from mypy: to ty:
Updates devcontainer/docker config to use .ty_cache instead of .mypy_cache
Updates .dockerignore to exclude .ty_cache
Updates contributor docs to reference uv run ty check pyrit/

Strict config and annotation enforcement:

Strict ty config: all = "error" with targeted ignores matching mypy's behavior
- empty-body = "ignore" (matches mypy's disable_error_code = ["empty-body"])
- unused-ignore-comment / unused-type-ignore-comment = "warn" (transitional)
- replace-imports-with-any = ["colorama.**"] — colorama has no type stubs and uses a dynamic setattr pattern in __init__ that ty can't follow
- HuggingFace module override (equivalent to mypy [[tool.mypy.overrides]])
Ruff ANN rules enabled: Replaces mypy's --disallow-untyped-defs (which ty will not implement)
- ANN401 (any-type) ignored as too strict
- Scoped to pyrit/ only (tests/doc exempted)
128 __init__ return types auto-fixed (ANN204)
3 functions manually annotated with proper return types

Suppression comment migration:

Migrated # type: ignore[mypy-code] → # type: ignore[ty:code] across 88 files
- ty requires the ty: prefix in type: ignore comments (non-prefixed codes are treated as mypy codes and ignored by ty)
- Mypy codes mapped to ty equivalents: arg-type → ty:invalid-argument-type, assignment → ty:invalid-assignment, ty:invalid-parameter-default, etc.
- Mypy-only codes with no ty equivalent (no-untyped-call, no-any-return, misc, etc.) removed — they were dead weight since ty doesn't check those patterns

ty configuration mapping (mypy to ty)

mypy setting	ty equivalent	Notes
`ignore_missing_imports = true`	`unresolved-import = "ignore"`	Suppresses errors for missing third-party stubs
`strict_optional = false`	`possibly-missing-attribute = "ignore"`	Tolerates `None` attribute access
`disable_error_code = ["empty-body"]`	`empty-body = "ignore"` (available since ty 0.0.14)	The PR author contributed this rule upstream (ruff#22846)
`type: ignore` comments	`respect-type-ignore-comments = true`	Preserves existing suppression comments
`exclude = ["doc/code/", "pyrit/auxiliary_attacks/"]`	`[tool.ty.src] exclude = [...]`	Same exclusions

ty vs mypy strict -- known gaps

Main currently uses mypy with strict = true. ty takes a different architectural approach: it focuses on type correctness (is this a type error?) and deliberately leaves annotation enforcement (is every function annotated?) to linters. Here is the gap analysis:

mypy strict flag	ty status	Mitigation
`--disallow-untyped-defs`	Will not be added (ty#476)	Ruff `ANN` rules (enabled in this PR)
`--disallow-incomplete-defs`	Same philosophy	Ruff `ANN` rules
`--disallow-untyped-calls`	Linter territory	Future type-aware Ruff
`--disallow-untyped-decorators`	Linter territory	Future type-aware Ruff
`--warn-return-any`	No issue filed	No mitigation yet
`--strict-equality`	Likely coming (ty#576 -- Astral member endorsed for stable release)	--
`--strict-concatenate`	ParamSpec support maturing	--
`--warn-redundant-casts`	✅ `redundant-cast` rule	--
`--warn-unused-ignores`	✅ `unused-ignore-comment` rule	--
`--check-untyped-defs`	✅ Default behavior	--
`--no-implicit-optional`	✅ Follows modern spec	--
`--extra-checks`	Partial (enum, TypedDict, dataclass rules exist)	--

Astral's long-term plan is a type-aware Ruff built on ty's inference engine, which would cover the annotation enforcement gaps. Carl Meyer (Astral core contributor) confirmed: "a type-aware linter that we intend to eventually build on top of ty" (source).

What ty adds that mypy doesn't

ty brings unique rules mypy lacks: division-by-zero, index-out-of-bounds, unused-awaitable (forgotten await), deprecated API usage detection, extensive dataclass/enum/Final validation, and more (117 rules total as of v0.0.32).

Current diagnostic state

After all migrations, running ty 0.0.32 on pyrit/ produces 127 errors and 106 warnings (233 total diagnostics). These are genuine ty findings on code that passes mypy strict — they represent ty's current type inference limitations (~53% typing spec conformance). Top error categories:

Rule	Errors	Nature
`invalid-argument-type`	59	dict.get overload mismatches, model constructor patterns
`unresolved-attribute`	25	Dynamic attributes ty can't track
`invalid-assignment`	14	SQLAlchemy descriptor patterns
`no-matching-overload`	11	dict.get / complex overloads
Other	18	Various

These will resolve as ty's spec conformance improves. No file-level overrides are used — only per-line # type: ignore[ty:code] suppression comments for known false positives.

Current state

ty pre-commit hook passes clean (0 errors). All remaining ty diagnostics are warnings (52 unused suppression comments from multi-code type: ignore comments where one code is active but another is stale).

Follow-up PRs

Code improvements that will eliminate # type: ignore comments and improve type safety:

PR	Impact	Description
`bind_partial()` in `apply_defaults`	Removes ~33 suppressions	Replace `REQUIRED_VALUE` sentinel with `inspect.Signature.bind_partial()` so parameters can be truly required without a type-lie default
`YamlLoadable.from_dict` default method	Removes 1 suppression	Add a default `from_dict(cls, data) -> T` on the base class instead of `hasattr`/`callable` duck typing
Dataset `metadata` → explicit kwargs**	Removes ~30 suppressions	Pass keyword arguments explicitly to `SeedPrompt()` instead of `**dict` unpacking (affects all dataset loaders)
Token provider type refactor	Removes ~6 suppressions	Fix `callable()` narrowing issues in Azure speech converters and content filter scorer
Clean up unused `type: ignore` warnings	Removes ~52 warnings	Remove stale codes from multi-code `# type: ignore[ty:code1, ty:code2]` comments where only one code is still needed
Remove `Optional` from `_TreeOfAttacksNode` scorer param	Removes 1 suppression	The scorer is validated non-None before node construction

Tests and Documentation

No test changes needed (ty is a dev tool, not runtime code)
Contributor docs updated to reference ty check instead of mypy

- no official pre commit hook support for ty: astral-sh/ty#269

maifeeulasad · 2026-01-17T08:19:12Z

Note 1: There is no pre commit hook for ty as of today (2026.01.17)

"We don't plan on adding this right now because ty is still at a very early, experimental stage and we don't want to give the impression that it's ready for serious production use yet. When we're closer to feature-complete, however, we'll absolutely be adding this." - Alex Waygood

ref:

pre-commit hook for ty astral-sh/ty#269 (comment)

maifeeulasad · 2026-01-17T08:28:35Z

Note 2: await outside func/async function

We are getting errors saying

error[invalid-syntax]: *await* statement outside of a function
error[invalid-syntax]: *await* outside of an asynchronous function

It is something under discussion there:

Interesting. What are the options we have here?

Keep it a rule
Introduce parser options (per file?)
Always allow
-- Micha Reiser

And then we got reply saying:

I don't think this is a realistic option for non-notebook files: for the vast majority of Python code this is a syntax error; we should definitely flag it by default for non-notebooks.

I'm not totally familiar with the situation for notebooks, honestly, as I've never seriously used notebooks. I can't tell you whether top-level awaits are allowed by default in notebooks or if it's only if some settings are enabled. I believe they're much more common in notebooks than in regular Python source code, however. For notebooks, we may want it to be an opt-in diagnostic rather than an opt-out diagnostic.

And even for regular Python source code, we'll want this to be at least suppressible on a per-file basis. Although it's pretty rare to use ast.PyCF_ALLOW_TOP_LEVEL_AWAIT for non-notebooks, it is possible.

I don't have strong opinions on how we make this diagnostic suppressible -- whether it stays as a linter rule or if you add per-file parser options.
-- Alex Waygood

I think it will take quite a while to get this feature.

ref:

- `ty` doesn't have any allow empty body config afaik

maifeeulasad · 2026-01-17T08:58:18Z

Here is existing mypy configuration:

[tool.mypy]
plugins = []
ignore_missing_imports = true
strict = false
follow_imports = "silent"
strict_optional = false
disable_error_code = ["empty-body"]
exclude = ["doc/code/"]

And here is what I am migrating into:

[tool.ty]
[tool.ty.rules]
unresolved-import = "ignore"
possibly-missing-attribute = "ignore"
unused-ignore-comment = "ignore"
[tool.ty.analysis]
respect-type-ignore-comments = true
[tool.ty.src]
exclude = ["doc/code/"]

And here is my mapping, or the reason:

mypy flag	visible behavior	ty rule
`ignore_missing_imports = true`	no import errors	`unresolved-import = "ignore"`
`strict_optional = false`	`None` tolerated	`possibly-missing-attribute = "ignore"`
`disable_error_code = ["empty-body"]`	empty bodies allowed	no equivalence
legacy mypy behavior	allow unused `# type: ignore`	`unused-ignore-comment = "ignore"`

Open for discussion, if I am missing something. Just wanted to clarify.

ref:

romanlutz · 2026-01-20T19:58:48Z

Note 1: There is no pre commit hook for ty as of today (2026.01.17)

"We don't plan on adding this right now because ty is still at a very early, experimental stage and we don't want to give the impression that it's ready for serious production use yet. When we're closer to feature-complete, however, we'll absolutely be adding this." - Alex Waygood

ref:

pre-commit hook for ty astral-sh/ty#269 (comment)

Yes, I wrote that, but the purpose of this item would be to replace mypy with ty including in pre-commit.

romanlutz · 2026-01-20T20:00:55Z

Note 2: await outside func/async function

We are getting errors saying

error[invalid-syntax]: *await* statement outside of a function

error[invalid-syntax]: *await* outside of an asynchronous function

It is something under discussion there:

Interesting. What are the options we have here?

Keep it a rule

Introduce parser options (per file?)

Always allow

-- Micha Reiser

And then we got reply saying:

I don't think this is a realistic option for non-notebook files: for the vast majority of Python code this is a syntax error; we should definitely flag it by default for non-notebooks.

I'm not totally familiar with the situation for notebooks, honestly, as I've never seriously used notebooks. I can't tell you whether top-level awaits are allowed by default in notebooks or if it's only if some settings are enabled. I believe they're much more common in notebooks than in regular Python source code, however. For notebooks, we may want it to be an opt-in diagnostic rather than an opt-out diagnostic.

And even for regular Python source code, we'll want this to be at least suppressible on a per-file basis. Although it's pretty rare to use ast.PyCF_ALLOW_TOP_LEVEL_AWAIT for non-notebooks, it is possible.

I don't have strong opinions on how we make this diagnostic suppressible -- whether it stays as a linter rule or if you add per-file parser options.

-- Alex Waygood

I think it will take quite a while to get this feature.

ref:

Allow top-level await in modules compiled with ast.PyCF_ALLOW_TOP_LEVEL_AWAIT astral-sh/ruff#17395 (comment)

Allow top-level await in modules compiled with ast.PyCF_ALLOW_TOP_LEVEL_AWAIT astral-sh/ruff#17395 (comment)

Every py file under /doc can have await outside functions so those should be ignored there (only there!)

maifeeulasad · 2026-01-20T20:01:33Z

@romanlutz that is something I am working on. I will share related links soon.

I am also trying to write rules for ty, which are available in mypy but not in ty. Hope we will be able to do 1-to-1 migration.

romanlutz · 2026-02-06T18:50:40Z

@maifeeulasad let me know if you need any help 🙂

maifeeulasad · 2026-02-07T11:57:12Z

Implemented a new empty-body return code for functions with stub bodies that have non-None return annotations. I implemented it in ruff, around two weeks ago. And they have updated the ruff sub module reference in ty like two days ago. I can see reference of this MR is nowhere in that release. I will check deeply if this feature was part of the release or not. If already done, I will resume working on this. As this was required as part of the one-to-one translation from mypy to ty, that's why this pause.

In the background I was looking for pre commit hook, which does the exact same thing as we do here.

This is my update so far! Is there any deadline? @romanlutz

ref:

romanlutz · 2026-02-07T15:34:49Z

Incredible! That is arguably a service to the entire community using ty! There's no particular deadline but we are obviously eager to stop wasting time with mypy 🙂

Thanks for the update. I can see that you're very much on top of it. Please don't feel rushed.

romanlutz · 2026-03-04T01:40:26Z

@maifeeulasad looks like they merged it?

maifeeulasad · 2026-03-07T09:16:28Z

@romanlutz sorry I haven't looked into it. Will check after weekend.

Resolve merge conflicts: - .pre-commit-config.yaml: remove mypy hook (keep all other hooks from main) - Makefile: accept main's expanded targets, replace mypy with ty - pyproject.toml: keep matplotlib dep from main, replace mypy with ty, add auxiliary_attacks to ty excludes, preserve [tool.uv] section - uv.lock: regenerated via uv lock Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Tighten ty config: use all='error' with targeted ignores/overrides - empty-body: ignore (matches mypy disable_error_code) - invalid-parameter-default: warn (sentinel pattern false positives) - File-level overrides for printer/fuzzer/hugging_face modules - Add Ruff ANN rules to enforce annotation coverage - Replaces mypy's disallow-untyped-defs (which ty won't implement) - ANN401 (any-type) ignored as too strict - ANN rules scoped to pyrit/ only (tests/doc exempted) - Auto-fix 128 missing __init__ return types (ANN204) - Add return type annotations to 3 remaining functions - Bump ty version pin from >=0.0.12 to >=0.0.32 - Regenerate uv.lock Note: ty pre-commit hook deferred — ty reports ~212 errors on lines with # type: ignore[mypy-code] comments that ty doesn't recognize (mypy codes != ty codes). These are false positives on code that passes mypy strict. The hook can be added once these comments are migrated to bare # type: ignore or ty gains mypy code compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Resolve conflicts in azure_sql_memory.py and sqlite_memory.py: keep both new skip_schema_migration param and -> None return type. Regenerate uv.lock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Convert # type: ignore[mypy-code] to # type: ignore[ty:code] using proper ty: prefix (required by ty's suppression model) - Remove mypy-only suppression comments that have no ty equivalent (no-untyped-call, no-any-return, misc, etc.) - Fix colorama false positives: use replace-imports-with-any instead of file-level overrides (addresses root cause: colorama has no type stubs and uses dynamic __init__ setattr pattern) - Remove file-level overrides for printer/fuzzer files (no longer needed) - Remove invalid-parameter-default global downgrade (proper suppression comments now handle sentinel patterns) Remaining ty diagnostics (127 errors) are genuine ty findings on code that passes mypy strict — they represent ty's current type inference limitations (~53% typing spec conformance) and will resolve as ty matures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove 49 single-code type: ignore[ty:code] comments where ty confirms the suppressed error no longer exists. Fix 4 comments that had wrong ty codes (mapped from incorrect mypy equivalents). Keep multi-code comments where partial suppression is still needed (ty reports the whole comment as unused even when one code is active). These show as warnings (not errors) and will clean up naturally. Final state: 124 errors, 57 warnings (181 diagnostics total). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- pyrit_backend.py: ty:possibly-missing-attribute -> ty:unresolved-attribute (sys.stdout typed as TextIO which lacks .reconfigure()) - frontend_core.py: ty:invalid-argument-type -> ty:missing-argument - yaml_loadable.py: ty:unresolved-attribute -> ty:call-non-callable - openai_realtime_target.py: ty:index-out-of-bounds -> ty:invalid-assignment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…orer files The ty type checker reports errors where message_piece.id (which is UUID | str | None) is passed to parameters expecting str | UUID. Add assertions before usage in 13 scorer _score_piece_async methods. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add ty pre-commit hook (allganize/ty-pre-commit v0.0.32) - Scoped to pyrit/, excludes auxiliary_attacks/ - Hook passes clean (0 errors) - Fix vlguard_dataset.py: default is_safe=True to avoid None in metadata dict - Fix jailbreak.py: type args dict as dict[str, Any] for kwargs pattern - Remove 2 redundant cast() calls in scorers.py - Suppress remaining 30 ty errors with type: ignore[ty:rule] comments: - Token provider callable() narrowing (6) — ty limitation - Dynamic class construction (5) — metaprogramming patterns - Protocol attribute assignment (2) — ty generic+protocol limitation - Enum custom __new__ return type (1) — ty can't follow _value_ assignment - Third-party stub imprecision (4) — azure SDK, blob storage - Other ty inference limitations (12) Follow-up PR candidates: - bind_partial() in apply_defaults to eliminate REQUIRED_VALUE sentinel - YamlLoadable.from_dict default method - Token provider type refactor (callable() narrowing) - Remove Optional from _TreeOfAttacksNode scorer param - Dataset **metadata unpacking → explicit kwargs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…d_response) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ruff format reformatted 2 files, which shifted type: ignore comments onto wrong lines. Moved them to the correct api_key= lines. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The previous commit added 'assert message_piece.id is not None' to scorer methods, but unit tests use mocked MessagePiece objects where id is None. The assert fails silently (caught by scorer error wrapper), causing empty RuntimeError in all scorer tests. Replace asserts with # type: ignore[ty:invalid-argument-type] comments instead — the id=None case is valid in test contexts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ts from ANN rules - Update miniconda URL from docs.conda.io (JS redirect) to docs.anaconda.com/free/miniconda/ (canonical destination) - Exclude build_scripts/, docker/, frontend/dev.py from ruff ANN rules (infrastructure scripts, consistent with doc/ and tests/ exclusions) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ty checking) These files are in a directory excluded from ty checking via tool.ty.src.exclude, so migrating their type: ignore comments from mypy to ty format serves no purpose. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

behnam-o

tested and works great! (and indeed so much faster !!)

I just notice some references to mypy still in the repo (ctrl + shift + F for mypy) - are those intentional?

If not, maybe remove/replace those too as part of this switch?

- Test PromptShieldTarget._add_auth_param_to_headers with callable and string API keys (covers line 250) - Test _argparse_validator with zero-param function (covers line 166) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Remove .mypy_cache entries from .gitignore - Remove mypy linting config from devcontainer.json - Remove mypy cache setup from devcontainer_setup.sh - Remove .mypy_cache from pyrightconfig.json excludes - Rephrase mypy-referencing comments in pyproject.toml, leakage.py, and pdf_converter.py - Notebook hits are only in base64 image data (not real references) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

# Conflicts: # pyrit/prompt_target/text_target.py

…e' into migration-mypy-to-ty

maifeeulasad added 2 commits January 17, 2026 13:25

[deps]: replace mypy w ithty;

ec4a611

[lock]: regenerated lock file for uv;

a5edbf0

maifeeulasad marked this pull request as draft January 17, 2026 08:01

maifeeulasad added 6 commits January 17, 2026 14:09

[cleanup]: remove mypy pre commit hook;

c284c5b

- no official pre commit hook support for ty: astral-sh/ty#269

[maint]: add ty cache in dockerignore instead of mypy;

60f6639

[maint]: add ty cache in dev container instead of mypy;

4cf17ef

[maint]: add ty cache in container volume instead of mypy;

f92b16e

[doc]: update command for ty type checker;

790a5fa

[feat]: ty make command replacing mypy;

ab255f4

maifeeulasad added 2 commits January 17, 2026 14:43

[config]: ty config over mypy;

3ba4c50

[config]: cleanup false ty config;

91c8a37

- `ty` doesn't have any allow empty body config afaik

romanlutz changed the title ~~Migration from mypy to ty~~ MAINT Migration from mypy to ty Jan 25, 2026

romanlutz and others added 5 commits April 23, 2026 23:39

Merge origin/main into migration-mypy-to-ty

9cb177c

Resolve conflicts in azure_sql_memory.py and sqlite_memory.py: keep both new skip_schema_migration param and -> None return type. Regenerate uv.lock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into migration-mypy-to-ty

0e5bed5

romanlutz marked this pull request as ready for review April 28, 2026 12:52

romanlutz approved these changes Apr 28, 2026

View reviewed changes

romanlutz and others added 8 commits April 28, 2026 06:53

Merge branch 'main' into migration-mypy-to-ty-maifee

e249e61

Suppress ty error on new cyber scenario factory (same pattern as rapi…

5da054e

…d_response) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix ruff formatting and misplaced type: ignore comments in airt.py

663a09a

Ruff format reformatted 2 files, which shifted type: ignore comments onto wrong lines. Moved them to the correct api_key= lines. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

behnam-o approved these changes Apr 29, 2026

View reviewed changes

romanlutz and others added 2 commits April 29, 2026 12:31

romanlutz enabled auto-merge April 29, 2026 23:02

romanlutz added 3 commits April 29, 2026 16:33

Merge branch 'main' into migration-mypy-to-ty-maifee

c115210

Merge remote-tracking branch 'origin/main' into migration-mypy-to-ty

668e3bb

# Conflicts: # pyrit/prompt_target/text_target.py

Merge remote-tracking branch 'maifeeulasad/migration-mypy-to-ty-maife…

4eb45ef

…e' into migration-mypy-to-ty

romanlutz added this pull request to the merge queue Apr 30, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 30, 2026

romanlutz added this pull request to the merge queue Apr 30, 2026

Merged via the queue into microsoft:main with commit 4f4d2e0 Apr 30, 2026
48 checks passed

Conversation

maifeeulasad commented Jan 17, 2026 • edited by romanlutz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What this PR does

ty configuration mapping (mypy to ty)

ty vs mypy strict -- known gaps

What ty adds that mypy doesn't

Current diagnostic state

Current state

Follow-up PRs

Tests and Documentation

Uh oh!

maifeeulasad commented Jan 17, 2026

Uh oh!

maifeeulasad commented Jan 17, 2026

Uh oh!

maifeeulasad commented Jan 17, 2026

Uh oh!

romanlutz commented Jan 20, 2026

Uh oh!

romanlutz commented Jan 20, 2026

Uh oh!

maifeeulasad commented Jan 20, 2026

Uh oh!

romanlutz commented Feb 6, 2026

Uh oh!

maifeeulasad commented Feb 7, 2026

Uh oh!

romanlutz commented Feb 7, 2026

Uh oh!

romanlutz commented Mar 4, 2026

Uh oh!

maifeeulasad commented Mar 7, 2026

Uh oh!

behnam-o left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

maifeeulasad commented Jan 17, 2026 •

edited by romanlutz

Loading