Add a module for monitoring GC statistics

[sergey-miryanov]

Feature or enhancement

Proposal:

I propose adding a new internal module to read the gathered statistics from GC.
It is supposed to work out-of-process, so we can read and show statistics in non-intrusive way and without any pauses of the monitored process.

To achieve this goal, we need to follow these steps:

1. Extend _PyDebugOffset with a pointer and size to the GC stats from _gc_runtime_state.
2. Add more data to the GC stats(see below).
3. Add a small module that connects to the process you're monitoring and displays statistics.

I have a working prototype.

I have added the following extra data to GC stats:

• ts_start - the timestamp of the GC collect start
• ts_stop - the timestamp of the GC collect stop
• heap_size - the number of live objects.
• work_to_do - an internal value from incremental GC. It controls the number of objects that are processed by one increment.
• object_visits - the number of objects that were visited during GC work.
• objects_transitively_reachable - the number of objects that were reachable from the global and local roots when incremental GC is called.
• objects_not_transitively_reachable - the number of objects that were reachable while increment was constructed.

With this module we can build tools, that gathers stats in the "real-time" and output it to various formats. For example, we can format data to use in Perfetto UI (image from dpo post):

I want to create two PRs, one for _PyDebugOffset and GC stats changes, and one for new module.

I want to start from GIL-enabled build.

cc @pablogsal @markshannon @nascheme @colesbury

Also link to the issue #131253

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

https://discuss.python.org/t/add-a-module-for-monitoring-gc-statistics/106695

Linked PRs

• GH-146527: Add more data to GC statistics and add it to PyDebugOffsets #146532