gh-127833: Use productionlist nodes to implement the grammar-snippet directive

[AA-Turner]

Split into individual commits for readability. I contend that productionlist nodes are semantically the right thing to use here, the only reason we didn't before was due to a limitation in Sphinx.

A

• Issue: Align the grammar documentation with Python's actual grammar #127833

---

📚 Documentation preview 📚: https://cpython-previews--130376.org.readthedocs.build/

[encukou]

Thank you! This looks like the way to go. I left a few notes inline.

conf.py still has needs_sphinx = '8.1.3', and I got a very worrying result with that version: the entire reference directory was missing, without any error message I could see. Is that something you want to look into?

[encukou]

The extension defines two directives, one with the new syntax but also a CompatProductionList that overrides Sphinx's .. productionlist::. We have syntax highlighting in existing productionlists too.

I added code to merge the lines of each multi-line production into a single production node.
Upstream Sphinx might need a similar change.

[encukou]

OK, now each production should should have complete rawsource.

[AA-Turner]

Sorry for the delay on my part. I've refactored to split the parsing of content lines into individual productions into its own method, as I think it is sufficiently complex to warrant a dedicated function (and this massively simplifies the creation of the actual production node).

One open question is if group_dict is the right representation, as my understanding is that each dict must have exactly one key -- perhaps an enum would be a better fit? I'm not sure though, and we mustn't let perfect be the enemy of the good.

A

[encukou]

Don't worry about delays; I've had more important things to deal with too :)

Do you think a (type, content) tuple makes it clearer? Like this: AA-Turner/cpython@docs/productionlist-grammar...encukou:cpython:docs/productionlist-grammar

I think Enum would be overengineering; we'd need to duplicate the names from the regex (unless we build the regex from the enum -- more overengineering).

[encukou]

I've pushed this change to the PR, and updated the branch to fix a conflict.

[AA-Turner]

@encukou I don't see any new commits...

[encukou]

I think that was a GitHub glitch.