gh-146385: Switch back to `re` to detect shlex.quote slow path by bonzini · Pull Request #146408 · python/cpython

bonzini · 2026-03-25T08:59:31Z

Commit 06a26fd ("gh-118761: Optimise import time for shlex (#132036)") when the input has to be quoted. This is because the regular expression search was able to short-circuit at the first unsafe character.

Go back to the same algorithm as 3.13, but make the "import re" and compilation of the regular expression lazy.

Testing s.isascii() makes shlex.quote() twice as fast in the non-ASCII case, but costs up to 25% of the full run time (because it necessitates an earlier isinstance check) if the string is ASCII. The latter is probably the common case, so drop the check.

Issue: Performance regression in shlex.quote from 3.13 to 3.14 #146385

bedevere-app · 2026-03-25T08:59:35Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

python-cla-bot · 2026-03-25T08:59:36Z

All commit authors signed the Contributor License Agreement.

picnixz · 2026-03-25T09:03:30Z

@AA-Turner Can you check if the results are consistent on Windows as well please?

picnixz · 2026-03-25T09:05:03Z

Here is what I suggest:

For 3.14, we can fix the regression (I will consider it a bug) if this reproduces on multiple systems. We might have other things at play here depending on where Python is built.
For 3.15, we can use lazy imports to make it cleaner if necessary.

I will also try to check what happens on older systems (openSUSE + older GCC/clang).

Commit 06a26fd ("pythongh-118761: Optimise import time for ``shlex`` (python#132036)") when the input has to be quoted. This is because the regular expression search was able to short-circuit at the first unsafe character. Go back to the same algorithm as 3.13, but make the "import re" and compilation of the regular expression lazy. Testing s.isascii() makes shlex.quote() twice as fast in the non-ASCII case, but costs up to 25% of the full run time (because it necessitates an earlier isinstance check) if the string *is* ASCII. The latter is probably the common case, so drop the check.

bedevere-app · 2026-03-25T09:11:11Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

bonzini · 2026-03-25T09:14:30Z

Yeah I didn't use lazy import to cater for backports. Would you prefer the lazy import version to be in this fix PR, or in a follow-up?

picnixz · 2026-03-25T09:16:23Z

For now, let's check whether the regression is consistent across platforms.

bonzini · 2026-03-25T09:30:36Z

For now, let's check whether the regression is consistent across platforms.

It is, and also it's really a complexity change. shlex.quote('a ' + 'b' * N) will always be slower in 3.14+ if you make N high enough. (And in practice, 10 is enough to have a substantial effect).

picnixz · 2026-03-25T09:39:56Z

What I want to know is why Adam reported numbers that went in the other direction. Ideally I also want to know the stdev of the numbers using pyperf (which I will do this w-e)

picnixz · 2026-03-25T09:41:23Z

I suggested in #132036 (comment) to replace the re call if we had something more efficient, and Adam found out that it was faster using bytes.translate so I'd like to know which data it was tested on:

With this, we go from ~26 microseconds for quote to ~2 (with only safe characters) and ~6 (with characters needing quotation).

vstinner

I would prefer a more regular global variable for _find_unsafe regex:

_find_unsafe = None

def quote(s):
    """Return a shell-escaped version of the string *s*."""
    global _find_unsafe

    if not s:
        return "''"

    if _find_unsafe is None:
        # deferred import and compilation to optimize import time
        import re
        _find_unsafe = re.compile(r'[^\w@%+=:,./-]', re.ASCII).search

    if _find_unsafe(s) is None:
        return s

    # use single quotes, and put single quotes into double quotes
    # the string $'b is then quoted as '$'"'"'b'
    return "'" + s.replace("'", "'\"'\"'") + "'"

vstinner · 2026-03-25T21:10:28Z

Can you provide a benchmark testing the no-quote and quote cases?

I suggested in #132036 (comment) to replace the re call if we had something more efficient, and Adam found out that it was faster using bytes.translate so I'd like to know which data it was tested on

Sadly, Adam didn't provide the benchmark code.

bonzini · 2026-03-25T21:18:51Z

Can you provide a benchmark testing the no-quote and quote cases?

It's in the linked issue #146385, though it's quite basic (using timeit).

I used this idiom because it decays to exactly the code in 3.13, but I can certainly change it if there's consensus.

picnixz · 2026-03-25T21:42:36Z

It's in the linked issue #146385, though it's quite basic (using timeit).

Yes, but I wanted the pyperf benchmark to see the numbers in terms of stdev and mean.

vstinner · 2026-03-25T22:19:52Z

I converted the timeit benchmark to a pyperf benchmark:

import pyperf
import shlex

runner = pyperf.Runner()
runner.bench_func('quote: with spaces', shlex.quote, 'the quick brown fox jumps over the lazy dog')
runner.bench_func('no quote: without spaces', shlex.quote, 'thequickbrownfoxjumpsoverthelazydog')
runner.bench_func('quote: non-ASCII', shlex.quote, 'mötley')
runner.bench_func('no quote: short', shlex.quote, 'a')

I ran the benchmark on Python 3.13 and 3.15 provided by Fedora 43 (optimized by LTO+PGO). I was lazy and didn't use CPU isolation for this benchmark.

py313:

quote: with spaces: Mean +- std dev: 413 ns +- 28 ns
no quote: without spaces: Mean +- std dev: 378 ns +- 29 ns
quote: non-ASCII: Mean +- std dev: 423 ns +- 32 ns
no quote: short: Mean +- std dev: 136 ns +- 4 ns

py315:

quote: with spaces: Mean +- std dev: 481 ns +- 24 ns
no quote: without spaces: Mean +- std dev: 344 ns +- 14 ns
quote: non-ASCII: Mean +- std dev: 200 ns +- 11 ns
no quote: short: Mean +- std dev: 305 ns +- 3 ns

Comparison table using Python 3.13 as reference:

Benchmark	py313	py315
quote: with spaces	413 ns	481 ns: 1.17x slower
no quote: without spaces	378 ns	344 ns: 1.10x faster
quote: non-ASCII	423 ns	200 ns: 2.11x faster
no quote: short	136 ns	305 ns: 2.24x slower
Geometric mean	(ref)	1.03x slower

Results in a REPL:

>>> print(shlex.quote('the quick brown fox jumps over the lazy dog'))  # quote
'the quick brown fox jumps over the lazy dog'
>>> print(shlex.quote('thequickbrownfoxjumpsoverthelazydog'))  # no quote
thequickbrownfoxjumpsoverthelazydog
>>> print(shlex.quote('mötley'))  # quote
'mötley'
>>> print(shlex.quote('a'))  # no quote
a

vstinner · 2026-03-25T22:26:46Z

I ran a benchmark on this PR using Python built with gcc -O3 (without LTO, without PGO).

ref:

quote: with spaces: Mean +- std dev: 463 ns +- 43 ns
no quote: without spaces: Mean +- std dev: 285 ns +- 25 ns
quote: non-ASCII: Mean +- std dev: 209 ns +- 14 ns
no quote: short: Mean +- std dev: 258 ns +- 18 ns

change:

quote: with spaces: Mean +- std dev: 371 ns +- 17 ns
no quote: without spaces: Mean +- std dev: 371 ns +- 4 ns
quote: non-ASCII: Mean +- std dev: 363 ns +- 31 ns
no quote: short: Mean +- std dev: 139 ns +- 14 ns

Comparison table:

Benchmark	ref	change
quote: with spaces	463 ns	371 ns: 1.25x faster
no quote: without spaces	285 ns	371 ns: 1.30x slower
quote: non-ASCII	209 ns	363 ns: 1.73x slower
no quote: short	258 ns	139 ns: 1.86x faster
Geometric mean	(ref)	1.01x faster

bonzini · 2026-03-25T23:19:36Z

Thanks, the results make sense - as mentioned in the description of the PR, I am ignoring the non-ASCII case on purpose and optimizing for the other three, given the typical uses of shlex; the geometric mean is overall 1.2x faster for those.

bedevere-app bot mentioned this pull request Mar 25, 2026

Performance regression in shlex.quote from 3.13 to 3.14 #146385

Open

bedevere-app bot added the awaiting review label Mar 25, 2026

picnixz requested review from AA-Turner and picnixz March 25, 2026 09:05

bonzini force-pushed the fix146385 branch from 769d1e0 to 3de138e Compare March 25, 2026 09:11

vstinner reviewed Mar 25, 2026

View reviewed changes

Uh oh!

Conversation

bonzini commented Mar 25, 2026 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app bot commented Mar 25, 2026

Uh oh!

python-cla-bot bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz commented Mar 25, 2026

Uh oh!

picnixz commented Mar 25, 2026

Uh oh!

bedevere-app bot commented Mar 25, 2026

Uh oh!

bonzini commented Mar 25, 2026

Uh oh!

picnixz commented Mar 25, 2026

Uh oh!

bonzini commented Mar 25, 2026

Uh oh!

picnixz commented Mar 25, 2026

Uh oh!

picnixz commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

vstinner commented Mar 25, 2026

Uh oh!

bonzini commented Mar 25, 2026

Uh oh!

picnixz commented Mar 25, 2026

Uh oh!

vstinner commented Mar 25, 2026

Uh oh!

vstinner commented Mar 25, 2026

Uh oh!

bonzini commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bonzini commented Mar 25, 2026 •

edited by bedevere-app bot

Loading

python-cla-bot bot commented Mar 25, 2026 •

edited

Loading

picnixz commented Mar 25, 2026 •

edited

Loading