Fix TypeError: Object of type __proxy__ is not JSON serializable em task_migrate_and_publish_articles#930
Fix TypeError: Object of type __proxy__ is not JSON serializable em task_migrate_and_publish_articles#930
Conversation
Update sanitize_for_json to handle Django lazy translation objects (__proxy__) and any other non-JSON-serializable types by converting them to their string representation. Previously, the function only handled str, dict, list/tuple and returned all other types unchanged, causing JSON serialization to fail when Django's gettext_lazy strings were added to task events. Add tests covering the new behavior with lazy proxy objects. Agent-Logs-Url: https://github.com/scieloorg/scms-upload/sessions/628ffd7d-9c8b-4b44-9f27-e162ad029666 Co-authored-by: robertatakenaka <505143+robertatakenaka@users.noreply.github.com>
Agent-Logs-Url: https://github.com/scieloorg/scms-upload/sessions/628ffd7d-9c8b-4b44-9f27-e162ad029666 Co-authored-by: robertatakenaka <505143+robertatakenaka@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Corrige a falha na persistência de detail em eventos/tarefas quando o payload contém objetos não serializáveis em JSON (ex.: gettext_lazy / __proxy__), evitando TypeError: Object of type __proxy__ is not JSON serializable durante a execução da task_migrate_and_publish_articles.
Changes:
- Ajusta
sanitize_for_jsonpara manter escalares JSON nativos e converter demais tipos não-serializáveis parastr(). - Atualiza a docstring do sanitizador para refletir a nova responsabilidade (JSON-serializável, não só surrogates).
- Adiciona testes cobrindo lazy proxy em valor isolado, em lista, em dict e em objeto customizado.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| core/utils/sanitize.py | Expande o sanitizador para tornar payloads JSON-serializáveis (inclui lazy proxies), reduzindo falhas ao salvar detalhes de eventos no banco. |
| proc/tests.py | Inclui cobertura de testes para garantir serialização de lazy proxies e objetos não-serializáveis após sanitização. |
| """Django lazy translation objects (__proxy__) must be converted to str.""" | ||
| from django.utils.translation import gettext_lazy as _ | ||
|
|
||
| lazy_text = _("Select journals by collection") | ||
| result = sanitize_for_json(lazy_text) | ||
| self.assertIsInstance(result, str) | ||
| self.assertEqual(result, "Select journals by collection") | ||
| json.dumps(result) # Must not raise | ||
|
|
||
| def test_list_with_lazy_proxy_converted(self): | ||
| """A list containing a lazy proxy object must be fully serializable.""" | ||
| from django.utils.translation import gettext_lazy as _ | ||
|
|
There was a problem hiding this comment.
These new tests import Django (gettext_lazy) but the file still exposes a __main__ runner (python proc/tests.py). Running it directly will now fail unless Django settings are configured. Either remove the __main__ block or configure DJANGO_SETTINGS_MODULE/django.setup() there to keep the standalone runner functional.
| # objects from gettext_lazy) to its string representation. | ||
| return str(obj) |
There was a problem hiding this comment.
sanitize_for_json now converts unknown objects with str(obj), but the resulting string is returned without re-running the surrogate sanitization. If __str__() (or a lazy translation) yields a string containing surrogate code points, this reintroduces the original PostgreSQL/JSON rejection. Consider sanitizing the string result as well (e.g., pass str(obj) back through sanitize_for_json).
| # objects from gettext_lazy) to its string representation. | |
| return str(obj) | |
| # objects from gettext_lazy) to its string representation, then sanitize | |
| # that string as well in case __str__() produces surrogate code points. | |
| return sanitize_for_json(str(obj)) |
There was a problem hiding this comment.
@copilot teste se obj é serializavel usando try/except e json.dumps, aí caso contário, sanitiza
There was a problem hiding this comment.
Implementado em fc84f73. Agora sanitize_for_json usa try/except json.dumps(obj) para testar serializabilidade: se o objeto já for serializável, é retornado inalterado; caso contrário, é convertido para str() e re-passado recursivamente pela própria função (para também tratar surrogates que __str__() possa produzir).
| lazy_text = _("Select journals by collection") | ||
| result = sanitize_for_json(lazy_text) | ||
| self.assertIsInstance(result, str) | ||
| self.assertEqual(result, "Select journals by collection") |
There was a problem hiding this comment.
This test asserts a hard-coded English value for a gettext_lazy string. Since the project default LANGUAGE_CODE is pt-BR and this msgid has translations in locale/*, str(lazy_text) can legitimately be non-English when translations are compiled/active, making the test brittle. Prefer asserting equality to str(lazy_text) (or using override('en')/translation.activate('en')) instead of a fixed literal.
| self.assertEqual(result, "Select journals by collection") | |
| self.assertEqual(result, str(lazy_text)) |
…bility check - Use json.dumps in a try/except to test whether an object is already JSON-serializable, returning it unchanged if it is. For non-serializable objects, convert to string and recursively re-sanitize to also handle surrogate code points that __str__() might produce. - Add `import json` required by the new try/except block. - Fix brittle test assertion: compare against str(lazy_text) instead of hard-coded English literal, so the test stays valid under any active locale. - Remove __main__ block from proc/tests.py since the tests now depend on Django being configured and must be run via pytest. Agent-Logs-Url: https://github.com/scieloorg/scms-upload/sessions/1b89ffc2-7f29-4874-963f-f1394ce02dbc Co-authored-by: robertatakenaka <505143+robertatakenaka@users.noreply.github.com>
O que esse PR faz?
Corrige o
TypeError: Object of type __proxy__ is not JSON serializableque interrompia a tarefatask_migrate_and_publish_articles. O objeto__proxy__(gerado porgettext_lazy) era adicionado à lista de eventos da tarefa e não era tratado porsanitize_for_json, chegando inalterado ao psycopg2 durante osave().Mudanças:
core/utils/sanitize.py—sanitize_for_jsonagora usatry/except json.dumpspara testar serializabilidade diretamente: se o objeto já for serializável, é retornado inalterado; caso contrário, é convertido parastr()e re-passado recursivamente pela própria função, garantindo também o tratamento de surrogates que__str__()possa produzir. Umimport jsonfoi adicionado ao módulo.proc/tests.py— 4 novos testes cobrindo lazy proxy isolado, em lista, em dict e objetos customizados não-serializáveis. A asserção de igualdade usastr(lazy_text)em vez de literal inglês fixo, evitando brittleness sob qualquer locale ativo. O bloco__main__foi removido pois os testes dependem do Django e devem ser executados via pytest.Onde a revisão poderia começar?
core/utils/sanitize.py— a mudança está na seção final da funçãosanitize_for_json(blocotry/except), seguida dos novos testes emproc/tests.py.Como este poderia ser testado manualmente?
Executar a tarefa
task_migrate_and_publish_articlescom os parâmetrosjournal_acron=rpe,collection_acron=rve,issue_folder=v25n1e verificar que nenhumUnexpectedEventdo tipoTypeErroré registrado ao final da execução.Algum cenário de contexto que queira dar?
O fluxo de falha:
add_event(_("..."))→json.dumpsfalha →sanitize_for_jsonchamada mas não trata__proxy__→self.save()falha no psycopg2. A correção cobre não apenas__proxy__, mas qualquer tipo não-serializável que possa chegar aodetailno futuro, inclusive strings com surrogates geradas por__str__().Screenshots
N/A
Quais são tickets relevantes?
[Migração de Artigos] Erro TypeError: Object of type proxy is not JSON serializable
Referências