Soft deprecation of the project?

[iliabylich]

So, Prism is out and it's:

1. correct
2. comes out of the box with MRI (and IIRC with other Ruby implementations)
3. faster (I think on TruffleRuby/JRuby it's absurdly faster because it doesn't run in interpreted mode like it does with parser)
4. always up-to-date
5. written in a language with perfect interop capabilities

On the other side backporting changes to parser becomes more and more difficult. MRI doesn't use yacc/bison anymore, instead there's Lrama that is also an LALR(1) parser generator but it has more "syntax sugar" that Racc doesn't have, and so the grammar will diverge more and more over time. And if (or once) parse.y is deprecated it will be even more complex to "translate" grammar changes.

I personally believe that parser should be soft-deprecated. What do you guys think? @kddnewton @bbatsov @koic @palkan @marcandre @mbj

And if you agree do you see anything we could do to parser to make transition easier?

[mbj]

The real use case of this gem left for me is multi version capability. Prism is not shipped with all the ruby versions my use cases still have to support, basically all non EOL (MRI) rubies.

If this gem where to stop supporting future ruby versions I'd be fine. Cannot speak for the rest of the ecosystem OFC.

[kddnewton]

In general I think this is the right long-term direction, but I'm uncomfortable soft deprecating until we mirror the tree rewriting functionality, since so many consumers use that. I think it could be added to the Ruby API of Prism relatively easily, but I haven't had time to look into this yet.

I'm not sure if there are any other APIs that I'm not thinking about, but that's the main one I know we are missing.

[bbatsov]

Indeed. I don't see how we can deprecated parser without the rewriting functionality. RuboCop also relies on the multi-version capability quite a lot, although I guess we can live without it if push comes to shove.

One other question - if parser gets soft deprecated will it still be updated to track newer Ruby releases? I think that a realistic time-frame for full migration to Prism is probably 2-3 years (for the projects that are still active, that is), so parser will definitely require some maintenance is the mean time.

[iliabylich]

One other question - if parser gets soft deprecated will it still be updated to track newer Ruby releases?

My initial thought was to stop doing it. At this point parser looks to me a lot like a (bad) polyfill and extending it more to be compliant with new grammar features feels like a disservice to the community.

[mbj]

I think we need to put evolutionary pressure on and NOT release compatibility with new versions.

[bbatsov]

I'm not super fond of measures that will be quite disruptive to the end users - to me it seems just wild to stop supporting something without providing a complete alternative for it. E.g. what happens if the rewriting is still not supported by Prism when Ruby 3.5 comes out?

At this point parser looks to me a lot like a (bad) polyfill and extending it more to be compliant with new grammar features feels like a disservice to the community.

Breaking existing apps is not exactly in the service of the community either. ;-)

I'm not arguing that parser should go away - I'm just concerned about abandoning the project prematurely and causing pain for the users of tool depending on it.

[mbj]

To play a bit of devils advocate here: How is not supporting a new future ruby release "breaking existing apps"?

[bbatsov]

Okay, "causing problems for the existing apps". I'm fairly certain RuboCop's users won't be exactly happy if we tell them we don't really support newer Ruby releases for whatever reasons. :-) And I'm not sure that simply stopping work on parser will motivate everyone to implement whatever is missing, etc. Also - how is the translation layer (Prism -> Parser) going to work when Parser stops tracking upstream changes and the available nodes in both libraries are out of sync?

Moving fully to Prism's native AST is going to be quite the time investment for many projects. I'd like for us to get there, but ideally I don't want us to be pressured into a timeline for this given that everyone working on RuboCop does this in their spare time.

[mbj]

Is there a translation layer from prosm to this parsers AST somewhere?

[palkan]

Is there a translation layer from prism to this parsers AST somewhere?

Prism supports AST translation for popular parsers: https://github.com/ruby/prism/tree/main/lib/prism/translation

And if (or once) parse.y is deprecated it will be even more complex to "translate" grammar changes.

I think, deprecating Parser as soon as parse.y is deprecated makes sense. Otherwise the effort required to port Ruby changes to Parser would be enormous.

how is the translation layer (Prism -> Parser) going to work when Parser stops tracking upstream changes and the available nodes in both libraries are out of sync?

From my experience, adding new nodes could be done relatively easy (compared to upgrading the grammar and lexer). I see, how, in theory, we can use Prism for newer Ruby versions as a parser and a Parser-compatible AST builder.

So, if we can keep Parser Ruby internals and APIs work in new Ruby versions (without providing the corresponding syntax support), that would smooth the transition.

To sum up, I would treat soft-deprecation as follows: stop adding new rubyXY.y files, but maintain compatibility with potential Ruby changes (I mean something breaking, like kwargs separation, etc.).

[iliabylich]

To sum up, I would treat soft-deprecation as follows: stop adding new rubyXY.y files, but maintain compatibility with potential Ruby changes (I mean something breaking, like kwargs separation, etc.).

Yes, that's more or less what I proposed (probably the word "deprecate" was a bit misleading). I'm not proposing to yank parser from rubygems.org, only to stop backporting new grammar changes.

I personally never touched rewriting API, shouldn't it "just work" with an AST constructed by Prism's compatibility layer?

Breaking existing apps is not exactly in the service of the community either. ;-)

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

[palkan]

shouldn't it "just work" with an AST constructed by Prism's compatibility layer?

Yeah, it works right now. Well, at least, for Ruby Next.

I think, what could be done in terms of ensuring this compatibility is to make it possible to run rewriter tests via Prism and the translation layer. I can take a look at this (since I've already done the same for Ruby Next).

[mbj]

Is there an unparser equivalent yet?

[kddnewton]

Truthfully I don't know if parse.y is going to be deprecated, or if that will happen any time soon at all. I imagine it could stick around for quite a long time. That being said, it is getting further away from strict bison/racc syntax, so I know it's quite a maintenance burden to hold on to.

My understanding is that it shouldn't break anything. If you are on Ruby 3.3+ use Prism (probably with a compatibility layer but it will be a part of the tooling, fully transparent for end users), otherwise use legacy parser gem. Or is it going to be a nightmare in terms of configuration and testing?

I think this would work, but we will want to test the translation layer even more. I think there are issues with string escapes at the moment, but I'm not sure.

I think, what could be done in terms of ensuring this compatibility is to make it possible to run rewriter tests via Prism and the translation layer. I can take a look at this (since I've already done the same for Ruby Next).

That would be really great.

Is there an unparser equivalent yet?

I haven't managed to get it over the finish line yet, but yes https://github.com/ruby-syntax-tree/syntax_tree-prism is that work. It's almost done.

[iliabylich]

Is there an unparser equivalent yet?

Probably no, but I've found "untokenizer" made by @koic 😄