argparse optionals with nargs='?', '*' or '+' can't be followed by positionals

[bethard]

BPO	9338
Nosy	@rhettinger, @jwilk, @merwok, @berkerpeksag, @kernc, @jacksonriley, @iritkatriel
Files	test_pos_after_var_args.patch issue9338_argparse.patch issue9338_3.patch test_9338.py issue9338_7.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-07-23.10:46:27.420>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = "argparse optionals with nargs='?', '*' or '+' can't be followed by positionals"
updated_at = <Date 2022-02-17.10:51:52.599>
user = 'https://bugs.python.org/bethard'

bugs.python.org fields:

activity = <Date 2022-02-17.10:51:52.599>
actor = 'wdoekes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2010-07-23.10:46:27.420>
creator = 'bethard'
dependencies = []
files = ['18328', '19691', '30296', '30297', '30350']
hgrepos = []
issue_num = 9338
keywords = ['patch']
message_count = 25.0
messages = ['111270', '111280', '111282', '112511', '121013', '121720', '122016', '141639', '141660', '166237', '166239', '189198', '189200', '189470', '189472', '189867', '189868', '200858', '200862', '216990', '285866', '314213', '355748', '370265', '408332']
nosy_count = 19.0
nosy_names = ['rhettinger', 'bethard', 'wrobell', 'jwilk', 'eric.araujo', 'TD22057', 'wdoekes', 'catherine', 'elsdoerfer', 'Kotan', 'berker.peksag', 'paul.j3', 'phawkins', 'kernc', 'sebix', 'extmind', 'jacksonriley', 'iritkatriel', 'kottalovag']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9338'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-53584: Prevent variable-nargs options from stealing required positional args #146513

[bethard]

[From the old argparse tracker: http://code.google.com/p/argparse/issues/detail?id=20]

You can't follow a nargs='+' optional argument with a positional argument:

>>> import argparse
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--badger', nargs='+')
>>> parser.add_argument('spam')
>>> parser.parse_args('--badger A B C D'.split())
usage: PROG [-h] [--badger BADGER [BADGER ...]] spam
PROG: error: too few arguments

Ideally, this should produce:

>>> parser.parse_args('--badger A B C D'.split())
Namespace(badger=['A', 'B', 'C'], spam='D')

The problem is that the nargs='+' causes the optional to consume all the arguments following it, even though we should know that we need to save one for the final positional argument.

A workaround is to specify '--', e.g.:

>>> parser.parse_args('--badger A B C -- D'.split())
Namespace(badger=['A', 'B', 'C'], spam='D')

The problem arises from the fact that argparse uses regular-expression style matching for positional arguments, but it does that separately from what it does for optional arguments.

One solution might be to build a regular expression of the possible things a parser could match. So given a parser like::

  parser = argparse.ArgumentParser()
  parser.add_argument('-w')
  parser.add_argument('-x', nargs='+')
  parser.add_argument('y')
  parser.add_argument('z', nargs='*')

the regular expression might look something like (where positionals have been replaced by the character A)::

(-w A)? (-x A+)? A (-w A)? (-x A+)? A* (-w A)? (-x A+)?

Note that the optionals can appear between any positionals, so I have to repeat their regular expressions multiple times. Because of this, I worry about how big the regular expression might grow to be for large parsers. But maybe this is the right way to solve the problem.

[bethard]

This is definitely a different bug from the one you just marked it as a duplicate of.

[merwok]

I read too fast, I’m sorry for that.

[catherine]

Here's a unit test for the simplest cases.

[merwok]

Looks good to me. Do you want to propose a code patch too, and/or more tests for non-simple cases?

[Kotan]

My attached patch adds the "--" between the optionals and the arguments, if there are optionals which have variable length and at least some positional argument can be provided.

Patch is for python 3.2 svn revision 86553.

[merwok]

Patch looks good, thanks! Does it integrate Catherine’s tests?

[wrobell]

is there a chance to fix this issue?

[merwok]

There is. Someone wanting to help could reply to the question I asked :)