New grammar rule often imples expanded reduce rule

As seen in _cmp() of python3.8/distutils/version.py with optimization -O2

.circleci/config.yml

@@ -36,10 +36,11 @@ jobs:
           # fallback to using the latest cache if no exact match is found
           - v2-dependencies-
 
-    # This is based on your 1.0 configuration file or project settings
+    - run:
-    - run: pip install --user --upgrade setuptools
+        command: | # Use pip to install dependengcies
-    - run: pip install --user -e .
+          pip install --user --upgrade setuptools
-    - run: pip install --user -r requirements-dev.txt
+          pip install --user -e .
+          pip install --user -r requirements-dev.txt
 
     # Save dependency cache
     - save_cache:

.github/FUNDING.yml

@@ -0,0 +1,12 @@
+# These are supported funding model platforms
+
+github: [rocky]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

.gitignore

@@ -10,6 +10,7 @@
 /.pytest_cache
 /.python-version
 /.tox
+.mypy_cache
 /.venv*
 /README
 /__pkginfo__.pyc

.travis.yml

@@ -1,9 +1,9 @@
 language: python
 
 python:
-  - '3.5'
+#  - '3.5'
-  - '2.7'
+#  - '2.7'
-  - '3.4'
+#  - '3.4'
   - '3.6'
   - '3.8'
 

NEWS.md

@@ -1,3 +1,50 @@
+3.7.1: 2020-6-12 Fleetwood66
+====================================================
+
+Released to pick up new xdis version which has fixes to read bytestings better on 3.x
+
+* Handle 3.7+ "else" branch removal adAs seen in `_cmp()` of `python3.8/distutils/version.py` with optimization `-O2`
+* 3.6+ "with" and "with .. as" grammar improvements
+* ast-check for "for" loop was missing some grammar rules
+
+3.7.0: 2020-5-19 Primidi 1st Prairial - Alfalfa - HF
+====================================================
+
+The main impetus for this release is to pull in the recent changes from xdis.
+We simplify imports using xdis 4.6.0.
+
+There were some bugfixes to Python 3.4-3.8. See the ChangeLog for details
+
+
+3.6.7: 2020-4-27 xdis again
+===========================
+
+More upheaval in xdis which we need to track here.
+
+3.6.6: 2020-4-20 Love in the time of Cholera
+============================================
+
+The main reason for this release is an incompatablity bump in xdis which handles
+3.7 SipHash better.
+
+* Go over "yield" as an expression precidence
+* Some small alignment with code in decompyle3 for "or" and "and" was done
+
+
+3.6.5: 2020-4-1 April Fool
+==========================
+
+Back port some of the changes in decompile3 here which mostly helps 3.7 and 3.8 decompilation, although this may also help 3.6ish versions too.
+
+- Handle nested `async for in for...`  and better async comprehension detection via `xdis`.  Still more work is needed.
+- include token number in listings when `-g` and there is a parser error
+- remove unneeded `Makefile`s now that remake 4.3+1.5dbg is a thing that has `-c`
+- Bug in finding annotations in functions with docstrings
+- Fix bug found by 2.4 sre_parse.py testing
+- Fix `transform` module's  `ifelseif` bugs
+- Fix bug in 3.0 name module detection
+- Fix docstring detection
+
 3.6.4: 2020-2-9 Plateau
 =======================
 

README.rst

@@ -211,7 +211,7 @@ however that the magic of a released version is usually the same as
 the *last* candidate version prior to release.
 
 There are also customized Python interpreters, notably Dropbox,
-which use their own magic and encrypt bytcode. With the exception of
+which use their own magic and encrypt bytecode. With the exception of
 the Dropbox's old Python 2.5 interpreter this kind of thing is not
 handled.
 
@@ -230,7 +230,7 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back ehre.
+* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back here.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
 * https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details

__pkginfo__.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2018 Rocky Bernstein <rocky@gnu.org>
+# Copyright (C) 2018, 2020 Rocky Bernstein <rocky@gnu.org>
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -21,9 +21,20 @@
 # less elegant than having it here with reduced code, albeit there
 # still is some room for improvement.
 
+# Python-version | package | last-version |
+# -----------------------------------------
+# 2.5            | pip     |  1.1         |
+# 2.6            | pip     |  1.5.6       |
+# 2.7            | pip     | 19.2.3       |
+# 2.7            | pip     |  1.2.1       |
+# 3.1            | pip     |  1.5.6       |
+# 3.2            | pip     |  7.1.2       |
+# 3.3            | pip     | 10.0.1       |
+# 3.4            | pip     | 19.1.1       |
+
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015-2019 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015-2020 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
 classifiers =  ["Development Status :: 5 - Production/Stable",
@@ -58,7 +69,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ["spark-parser >= 1.8.9, < 1.9.0",
-                      "xdis >= 4.2.2, < 4.3.0"]
+                      "xdis >= 4.6.1, < 4.8.0"]
 
 license            = "GPL3"
 mailing_list       = "python-debugger@googlegroups.com"

admin-tools/how-to-make-a-release.md

@@ -55,7 +55,7 @@
 # Make packages and tag
 
     $ . ./admin-tools/make-dist-older.sh
-    $ git tag release-python-2.4-$VERSION
+	$ pyenv local 3.8.3
 	$ twine check dist/uncompyle6-$VERSION*
     $ . ./admin-tools/make-dist-newer.sh
 	$ twine check dist/uncompyle6-$VERSION*

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.17 3.2.6 3.1.5 3.4.10 3.7.6 3.8.1'
+export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.18 3.2.6 3.1.5 3.4.10 3.7.7 3.8.3'

admin-tools/setup-master.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-PYTHON_VERSION=3.7.6
+PYTHON_VERSION=3.7.7
 
 # FIXME put some of the below in a common routine
 function finish {

appveyor.yml

@@ -53,6 +53,7 @@ install:
   # compiled extensions and are not provided as pre-built wheel packages,
   # pip will build them from source using the MSVC compiler matching the
   # target Python version and architecture
+  - "%CMD_IN_ENV% pip install git+git://github.com/rocky/python-uncompyle6.git#egg=uncompyle6-3.6.6"
   - "%CMD_IN_ENV% pip install -r requirements.txt"
 
 build_script:

pytest/test_token.py

@@ -1,14 +1,13 @@
-from uncompyle6 import PYTHON_VERSION
 from uncompyle6.scanners.tok import Token
 
 
 def test_token():
     # Test token formatting of: LOAD_CONST None
     t = Token("LOAD_CONST", offset=0, attr=None, pattr=None, has_arg=True)
-    expect = "             0  LOAD_CONST               None"
+    expect = "0  LOAD_CONST               None"
     # print(t.format())
     assert t
-    assert t.format() == expect
+    assert t.format().strip() == expect.strip()
 
     # Make sure equality testing of tokens ignores offset
     t2 = Token("LOAD_CONST", offset=2, attr=None, pattr=None, has_arg=True)
@@ -17,8 +16,8 @@ def test_token():
     # Make sure formatting of: LOAD_CONST False. We assume False is the 0th index
     # of co_consts.
     t = Token("LOAD_CONST", offset=1, attr=False, pattr=False, has_arg=True)
-    expect = "             1  LOAD_CONST               False"
+    expect = "1  LOAD_CONST               False"
-    assert t.format() == expect
+    assert t.format().strip() == expect.strip()
 
 
 if __name__ == "__main__":

pytest/testdata/if-2.7.right

@@ -1,12 +1,12 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_if.py
 
-   6         0  LOAD_NAME             0  'True'
+ L.   6         0  LOAD_NAME             0  'True'
-             3  POP_JUMP_IF_FALSE    15  'to 15'
+                3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   7         6  LOAD_NAME             1  'False'
+ L.   7         6  LOAD_NAME             1  'False'
-             9  STORE_NAME            2  'b'
+                9  STORE_NAME            2  'b'
-            12  JUMP_FORWARD          0  'to 15'
+               12  JUMP_FORWARD          0  'to 15'
-          15_0  COME_FROM            12  '12'
+             15_0  COME_FROM            12  '12'
-            15  LOAD_CONST               None
+               15  LOAD_CONST               None
-            18  RETURN_VALUE     
+               18  RETURN_VALUE     

pytest/testdata/ifelse-2.7.right

@@ -1,15 +1,15 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_ifelse.py
 
-   3         0  LOAD_NAME             0  'True'
+ L.   3         0  LOAD_NAME             0  'True'
-             3  POP_JUMP_IF_FALSE    15  'to 15'
+                3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   4         6  LOAD_CONST               1
+ L.   4         6  LOAD_CONST               1
-             9  STORE_NAME            1  'b'
+                9  STORE_NAME            1  'b'
-            12  JUMP_FORWARD          6  'to 21'
+               12  JUMP_FORWARD          6  'to 21'
 
-   6        15  LOAD_CONST               2
+ L.   6        15  LOAD_CONST               2
-            18  STORE_NAME            2  'd'
+               18  STORE_NAME            2  'd'
-          21_0  COME_FROM            12  '12'
+             21_0  COME_FROM            12  '12'
-            21  LOAD_CONST               None
+               21  LOAD_CONST               None
-            24  RETURN_VALUE     
+               24  RETURN_VALUE     

pytest/validate.py

@@ -12,8 +12,7 @@ import functools
 from uncompyle6 import PYTHON_VERSION, PYTHON3, IS_PYPY, code_deparse
 
 # TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
-from xdis.bytecode import Bytecode
+from xdis import Bytecode, get_opcode
-from xdis.main import get_opcode
 
 opc = get_opcode(PYTHON_VERSION, IS_PYPY)
 Bytecode = functools.partial(Bytecode, opc=opc)

setup.py

@@ -4,40 +4,54 @@ import sys
 """Setup script for the 'uncompyle6' distribution."""
 
 SYS_VERSION = sys.version_info[0:2]
-if not ((2, 6) <= SYS_VERSION  <= (3, 9)):
+if not ((2, 6) <= SYS_VERSION <= (3, 9)):
     mess = "Python Release 2.6 .. 3.9 are supported in this code branch."
-    if ((2, 4) <= SYS_VERSION <= (2, 7)):
+    if (2, 4) <= SYS_VERSION <= (2, 7):
-        mess += ("\nFor your Python, version %s, use the python-2.4 code/branch." %
+        mess += (
-                 sys.version[0:3])
+            "\nFor your Python, version %s, use the python-2.4 code/branch."
+            % sys.version[0:3]
+        )
     elif SYS_VERSION < (2, 4):
-        mess += ("\nThis package is not supported for Python version %s."
+        mess += (
-                 % sys.version[0:3])
+            "\nThis package is not supported for Python version %s." % sys.version[0:3]
+        )
     print(mess)
     raise Exception(mess)
 
-from __pkginfo__ import \
+from __pkginfo__ import (
-    author,           author_email,       install_requires,          \
+    author,
-    license,          long_description,   classifiers,               \
+    author_email,
-    entry_points,     modname,            py_modules,                \
+    install_requires,
-    short_desc,       VERSION,            web,                       \
+    license,
-    zip_safe
+    long_description,
+    classifiers,
+    entry_points,
+    modname,
+    py_modules,
+    short_desc,
+    VERSION,
+    web,
+    zip_safe,
+)
 
 from setuptools import setup, find_packages
+
 setup(
-       author             = author,
+    author=author,
-       author_email       = author_email,
+    author_email=author_email,
-       classifiers        = classifiers,
+    classifiers=classifiers,
-       description        = short_desc,
+    description=short_desc,
-       entry_points       = entry_points,
+    entry_points=entry_points,
-       install_requires   = install_requires,
+    install_requires=install_requires,
-       license            = license,
+    license=license,
-       long_description   = long_description,
+    long_description=long_description,
-       long_description_content_type = "text/x-rst",
+    long_description_content_type="text/x-rst",
-       name               = modname,
+    name=modname,
-       packages           = find_packages(),
+    packages=find_packages(),
-       py_modules         = py_modules,
+    py_modules=py_modules,
-       test_suite         = 'nose.collector',
+    test_suite="nose.collector",
-       url                = web,
+    url=web,
-       tests_require      = ['nose>=1.0'],
+    tests_require=["nose>=1.0"],
-       version            = VERSION,
+    version=VERSION,
-       zip_safe           = zip_safe)
+    zip_safe=zip_safe,
+)

test/.gitignore

@@ -1,3 +1,8 @@
 /.coverage
 /.python-version
 /nohup.out
+/pycdc
+/test_pycdc_tests.sh
+/test_uncompyle2.py
+/test_unpy33.py
+/test_unpy37.py

test/Makefile

@@ -78,7 +78,7 @@ check-3.8: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.8 --syntax-verify $(COMPILE)
 
 # FIXME
-#: this is called when running under pypy3.5-5.8.0 or pypy2-5.6.0
+#: this is called when running under pypy3.5-5.8.0, pypy2-5.6.0, or pypy3.6-7.3.0
 5.8 5.6:
 
 #: Check deparsing only, but from a different Python version
@@ -95,12 +95,19 @@ check-bytecode-2:
         --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
 
 #: Check deparsing bytecode 3.x only
+# check-bytecode-3:
+# 	$(PYTHON) test_pythonlib.py \
+#         --bytecode-3.0 \
+# 	--bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
+# 	--bytecode-3.4 --bytecode-3.5 --bytecode-3.6 \
+#         --bytecode-3.7  \
+# 	--bytecode-pypy3.2 --bytecode-pypy3.6 --bytecode-3.8
+
+# FIXME: Until we shaked out problems with xdis...
 check-bytecode-3:
-	$(PYTHON) test_pythonlib.py --bytecode-3.0 \
+	$(PYTHON) test_pythonlib.py \
-	--bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
 	--bytecode-3.4 --bytecode-3.5 --bytecode-3.6 \
-        --bytecode-3.7  \
+        --bytecode-3.7  --bytecode-3.8
-	--bytecode-pypy3.2 --bytecode-pypy3.6 --bytecode-3.8
 
 #: Check deparsing on selected bytecode 3.x
 check-bytecode-3-short:
@@ -339,8 +346,8 @@ check-bytecode-pypy3.6: 7.1
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
 
 #: PyPy 5.0.x with Python 3.6.9
-check-bytecode-pypy3.6: 7.2
+check-bytecode-pypy3.6: 7.2 7.3
-7.2:
+7.3 7.2:
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6-run --verify-run
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
 

test/add-test.py

@@ -19,6 +19,7 @@ for path in py_source:
     else:
         cfile = "bytecode_%s%s/%s" % (version, suffix, short) + "c"
     print("byte-compiling %s to %s" % (path, cfile))
-    py_compile.compile(path, cfile)
+    optimize = 2
+    py_compile.compile(path, cfile, optimize=optimize)
     if isinstance(version, str) or version >= (2, 6, 0):
         os.system("../bin/uncompyle6 -a -T %s" % cfile)

test/run-and-email.sh

@@ -3,32 +3,55 @@
 function displaytime {
     printf "ran in "
     local T=$1
-    local D=$((T/60/60/24))
+    ((D=T/60/60/24))
-    local H=$((T/60/60%24))
+    ((H=T/60/60%24))
-    local M=$((T/60%60))
+    ((M=T/60%60))
-    local S=$((T%60))
+    ((S=T%60))
-    (( $D > 0 )) && printf '%d days ' $D
+    (( D > 0 )) && printf '%d days ' $D
-    (( $H > 0 )) && printf '%d hours ' $H
+    (( H > 0 )) && printf '%d hours ' $H
-    (( $M > 0 )) && printf '%d minutes ' $M
+    (( M > 0 )) && printf '%d minutes ' $M
-    (( $D > 0 || $H > 0 || $M > 0 )) && printf 'and '
+    (( D > 0 || H > 0 || M > 0 )) && printf 'and '
     printf '%d seconds\n' $S
 }
 
-. ../admin-tools/pyenv-newer-versions
+bs=${BASH_SOURCE[0]}
+if [[ $0 != $bs ]] ; then
+    echo "This script should not be *sourced* but run through bash"
+    exit 1
+fi
+
+mydir=$(dirname $bs)
+cd $mydir
+
+branch=$(cat ../.git/HEAD | cut -d'/' -f 3)
+if [[ $branch == 'python-2.4' ]]; then
+    . ../admin-tools/pyenv-older-versions
+elif [[ $branch == 'master' ]]; then
+    . ../admin-tools/pyenv-newer-versions
+else
+    echo &1>2 "Error git branch should either be 'master' or 'python-2.4'; got: '$branch'"
+    exit 1
+fi
+
+MAIN="test_pyenvlib.py"
 
 USER=${USER:-rocky}
 EMAIL=${EMAIL:-rb@dustyfeet.com}
+WHAT="uncompyle6 ${MAIN}"
 MAX_TESTS=${MAX_TESTS:-800}
+export BATCH=1
+
 typeset -i RUN_STARTTIME=$(date +%s)
 
 # PYVERSIONS="3.5.6"
-actual_versions=""
+MAILBODY=/tmp/${MAIN}-mailbody-$$.txt
+# for VERSION in 3.3.7 ; do
 for VERSION in $PYVERSIONS ; do
     typeset -i rc=0
-    LOGFILE=/tmp/pyenvlib-$VERSION-$$.log
+    LOGFILE=/tmp/${MAIN}-$VERSION-$$.log
 
     case "$VERSION" in
-	3.7.6 | 3.8.1 | 3.1.5 | 3.0.1 )
+	3.7.7 | 3.8.2 | 3.1.5 | 3.0.1 )
 	    continue
 	    ;;
 	3.5.9 )
@@ -44,7 +67,15 @@ for VERSION in $PYVERSIONS ; do
 	    MAX_TESTS=800
 	    ;;
 	3.6.10 )
-	    MAX_TESTS=1300  # about 2139 exist
+	    # MAX_TESTS=1300  # about 2139 exist
+	    # fails on _pyio.cpython-36.opt-1.pyc
+	    MAX_TESTS=34
+	    ;;
+	2.4.6 )
+	    MAX_TESTS=600
+	    ;;
+	2.5.6 )
+	    MAX_TESTS=600
 	    ;;
 	2.6.9 )
 	    MAX_TESTS=1300
@@ -58,11 +89,13 @@ for VERSION in $PYVERSIONS ; do
 
     if ! pyenv local $VERSION ; then
 	rc=1
+	mailbody_line="pyenv local $VERSION not installed"
+	echo $mailbody_line >> $MAILBODY
     else
       echo Python Version $(pyenv local) > $LOGFILE
       echo "" >> $LOGFILE
       typeset -i ALL_FILES_STARTTIME=$(date +%s)
-      cmd="python ./test_pyenvlib.py --max ${MAX_TESTS} --syntax-verify --$VERSION"
+      cmd="python ./${MAIN} --max ${MAX_TESTS} --syntax-verify --$VERSION"
       echo "$cmd" >>$LOGFILE 2>&1
       $cmd >>$LOGFILE 2>&1
       rc=$?
@@ -72,20 +105,25 @@ for VERSION in $PYVERSIONS ; do
 
       typeset -i ALL_FILES_ENDTIME=$(date +%s)
       (( time_diff =  ALL_FILES_ENDTIME - ALL_FILES_STARTTIME))
-      displaytime $time_diff >> $LOGFILE
+      time_str=$(displaytime $time_diff)
+      echo ${time_str}. >> $LOGFILE
     fi
 
-    SUBJECT_PREFIX="pyenv weak verify (max $MAX_TESTS) for"
+    SUBJECT_PREFIX="$WHAT (max $MAX_TESTS) for"
     if ((rc == 0)); then
+	mailbody_line="Python $VERSION ok; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION ok" ${USER}@localhost
     else
+	mailbody_line="Python $VERSION failed; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${USER}@localhost
-	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${EMAIL}
+	tail -v $LOGFILE | mail -s "$HOST $SUBJECT_PREFIX $VERSION not ok" ${EMAIL}
     fi
+    echo $mailbody_line >> $MAILBODY
     rm .python-version
 done
 
 typeset -i RUN_ENDTIME=$(date +%s)
 (( time_diff =  RUN_ENDTIME - RUN_STARTTIME))
 elapsed_time=$(displaytime $time_diff)
-echo "Run complete $elapsed_time for versions $actual_versions" | mail -s "pyenv weak verify in $elapsed_time" ${EMAIL}
+echo "${WHAT} complete; ${elapsed_time}." >> $MAILBODY
+cat $MAILBODY | mail -s "$HOST $WHAT ${elapsed_time}." ${EMAIL}

test/simple-decompile-code-test.py

@@ -3,7 +3,7 @@
 from __future__ import print_function
 
 from uncompyle6.main import decompile
-from xdis.magics import sysinfo2float
+from xdis import sysinfo2float
 import sys, inspect
 
 def uncompyle_test():

test/simple_source/bug31/04_def_annotate.py

@@ -66,6 +66,14 @@ def div(a: dict(type=float, help='the dividend'),
     """Divide a by b"""
     return a / b
 
+# From 3.7.6 functools.py
+# Bug is in picking up the annotation.
+def f(a:"This is a new annotation"):
+    """This is a test"""
+    assert f.__annotations__['a'] == "This is a new annotation"
+
+f(5)
+
 class TestSignatureObject1():
     def test_signature_on_wkwonly(self):
         def test(*, a:float, b:str, c:str = 'test', **kwargs: int) -> int:

test/simple_source/bug33/02_pos_args.py

@@ -1,6 +1,8 @@
+# Python 3.3+
+#
 # From Python 3.3.6 hmac.py
 # Problem was getting wrong placement of positional args.
-# In 3.6+ paramter handling changes
+# In 3.6+ parameter handling changes
 
 # RUNNABLE!
 

test/simple_source/bug35/04_call_function.py

@@ -2,6 +2,7 @@
 # Bug was code not knowing which Python versions
 # have kwargs coming before positional args in code.
 
+"""This program is self-checking!"""
 # RUNNABLE!
 
 def tometadata(self, metadata, schema, Table, args, name=None):

test/simple_source/bug36/02_var_annotate.py

@@ -1,6 +1,9 @@
 # 3.6+ type annotations on variables
 from typing import List
 
+# This test program is part of the uncompyle6 test suite
+# tests STORE_ANNOTATION and SETUP_ANOTATIONS
+
 # RUNNABLE!
 y = 2
 x: bool

test/simple_source/bug37/02_async_for_generator.py

@@ -6,3 +6,11 @@ def make_arange(n):
 
 async def run(m):
     return [i async for i in m]
+
+# From 3.7.6 test_coroutines.py
+async def run_list(pair, f):
+    return [i for pair in p async for i in f]
+
+# FIXME: add this. It works in decompyle3
+# async def run_gen():
+#     return (i async for i in f if 0 < i < 4)

test/simple_source/bug37/03_else_removal.py

@@ -0,0 +1,12 @@
+# From python3.8/distutils/version.py with optimization -O2
+# The bug was that the other "else" constant propagated removed.
+
+# NOTE: this program needs to be compile with optimization
+def _cmp (b, c):
+    if b:
+        if c:
+            return 0
+        else:
+            return 1
+    else:
+        assert False, "never get here"

test/simple_source/bug37/04_async.py

@@ -1,23 +0,0 @@
-# from 3.7 test_contextlib_async.py
-# Bugs were not adding "async" when a function is a decorator,
-# and a misaligment when using "async with as".
-@_async_test
-async def test_enter(self):
-    self.assertIs(await manager.__aenter__(), manager)
-
-    async with manager as context:
-        async with woohoo() as x:
-            x = 1
-            y = 2
-        assert manager is context
-
-# From 3.7.6 test_coroutines.py
-# Bug was different form of code for "async with" below
-class CoroutineTest():
-    def test_with_8(self):
-        CNT = 0
-        async def foo():
-            nonlocal CNT
-            async with CM():
-                CNT += 1
-                return

test/simple_source/bug37/04_grammar.py

@@ -0,0 +1,87 @@
+# From test_grammar.py
+# RUNNABLE!
+def check_syntax_error(statement):
+    try:
+        compile(statement, '<test string>', 'exec')
+    except SyntaxError:
+        return
+    assert False
+
+def test_yield():
+    # Requires parentheses as call argument
+    def g():
+        f((yield 1), 1)
+    def g():
+        f((yield from ()))
+    def g():
+        f((yield from ()), 1)
+    def g():
+        f((yield 1))
+
+    # Allowed as standalone statement
+    def g():
+        yield 1
+
+    def g():
+        yield from ()
+
+    # Allowed as RHS of assignment
+    def g():
+        x = yield 1
+
+    def g():
+        x = yield from ()
+
+    # Ordinary yield accepts implicit tuples
+    def g():
+        yield 1, 1
+
+    def g():
+        x = yield 1, 1
+
+    # 'yield from' does not
+    check_syntax_error("def g(): yield from (), 1")
+    check_syntax_error("def g(): x = yield from (), 1")
+    # Requires parentheses as subexpression
+    def g():
+        1, (yield 1)
+
+    def g():
+        1, (yield from ())
+
+    check_syntax_error("def g(): 1, yield 1")
+    check_syntax_error("def g(): 1, yield from ()")
+    # Requires parentheses as call argument
+    def g():
+        f((yield 1))
+
+    def g():
+        f((yield 1), 1)
+
+    def g():
+        f((yield from ()))
+
+    def g():
+        f((yield from ()), 1)
+
+    check_syntax_error("def g(): f(yield 1)")
+    check_syntax_error("def g(): f(yield 1, 1)")
+    check_syntax_error("def g(): f(yield from ())")
+    check_syntax_error("def g(): f(yield from (), 1)")
+    # Not allowed at top level
+    check_syntax_error("yield")
+    check_syntax_error("yield from")
+    # Not allowed at class scope
+    check_syntax_error("class foo:yield 1")
+    check_syntax_error("class foo:yield from ()")
+    # Check annotation refleak on SyntaxError
+    check_syntax_error("def g(a:(yield)): pass")
+
+test_yield()
+
+# From test_types.py
+# Bug was needing parens around (yield 2)
+def gen_func():
+    yield 1
+    return (yield 2)
+gen = gen_func()

test/simple_source/bug37/10_async.py

@@ -0,0 +1,70 @@
+# from 3.7 test_contextlib_async.py
+# Bugs were not adding "async" when a function is a decorator,
+# and a misaligment when using "async with ... as".
+
+"""This program is self-checking!"""
+import asyncio
+from contextlib import asynccontextmanager, AbstractAsyncContextManager
+import functools
+
+
+def _async_test(func):
+    """Decorator to turn an async function into a test case."""
+
+    @functools.wraps(func)
+    def wrapper(*args, **kwargs):
+        coro = func(*args, **kwargs)
+        loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(loop)
+        try:
+            return loop.run_until_complete(coro)
+        finally:
+            loop.close()
+            asyncio.set_event_loop(None)
+
+    return wrapper
+
+
+state = []
+
+
+@asynccontextmanager
+async def woohoo():
+    state.append(1)
+    yield 42
+    state.append(999)
+
+
+@_async_test
+async def test_enter():
+    class DefaultEnter(AbstractAsyncContextManager):
+        async def __aexit__(*args):
+            return
+            # await super().__aexit__(*args)
+
+    manager = DefaultEnter()
+    got_manager = await manager.__aenter__()
+    # print(got_manager, manager)
+    assert got_manager is manager
+
+    async with manager as context:
+        async with woohoo() as x:
+            x = 1
+            y = 2
+        assert manager is context
+
+
+# From 3.7.6 test_coroutines.py
+# Bug was different form of code for "async with" below
+class CoroutineTest:
+    def test_with_8(self):
+        CNT = 0
+
+        async def foo():
+            nonlocal CNT
+            async with CM():
+                CNT += 1
+                return
+
+
+test_enter()

test/simple_source/expression/09_yield_from.py

@@ -17,9 +17,9 @@ def ybug(g):
 # From 3.5.1 _wakrefset.py
 #
 #  3.5:
-#  withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+#  with ::= expr SETUP_WITH POP_TOP suite_stmts_opt
-#              POP_BLOCK LOAD_CONST COME_FROM
+#           POP_BLOCK LOAD_CONST COME_FROM
-#              WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+#           WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
 
 def __iter__(self, IterationGuard):

test/simple_source/looping/08_while1_if_continue.py

@@ -7,3 +7,40 @@ def readline (self):
                 continue
 
         return
+
+# From 2.4.6 sre.py
+# Bug has to do with "break" not being recognized
+# and is a JUMP_FORWARD.
+def _parse(a, b, source, state):
+    while 1:
+        if b:
+            while 1:
+                break
+        else:
+            raise
+
+def _parse2(source, state):
+    while 1:
+        if a:
+            if b:
+                while 1:
+                    this = 1
+                    break
+                continue
+
+        while 1:
+            if b:
+                break
+
+        x = 3
+
+# Bug was in 2.3 decompilation
+def _parse3(source, state):
+    while 1:
+        if a:
+            if b:
+                x = 1
+                while 1:
+                    if a:
+                        break
+                    raise

test/simple_source/stmts/03_if_elif.py

@@ -1,11 +1,13 @@
 # 2.6.9 symbols.py
 # Bug in 2.6 is having multple COME_FROMs due to the
 # "and" in the "if" clause
+
+# RUNNABLE
 if __name__:
     if __file__ and __name__:
         pass
-    elif __name__:
+    elif not __name__:
-        pass
+        assert False
 
 # 2.6.9 transformer.py
 # Bug in 2.6 is multple COME_FROMs as a result
@@ -21,3 +23,20 @@ elif __file__:
     assert __name__ or __file__
 else:
     pass
+
+# From 3.3.7 test_binop.py
+# Bug was in ifelsestmt(c) ensuring b+=5 is not in "else"
+# Also note: ifelsetmtc should not have been used since this
+# this is not in a loop!
+def __floordiv__(a, b):
+    if a:
+        b += 1
+    elif not b:
+        return a
+    b += 5
+    return b
+
+assert __floordiv__(1, 1) == 7
+assert __floordiv__(1, 0) == 6
+assert __floordiv__(0, 3) == 8
+assert __floordiv__(0, 0) == 0

test/simple_source/stmts/04_withas.py

@@ -16,3 +16,25 @@ def withas_bug(self, nested, a, b):
     with self.assertRaises(ZeroDivisionError):
         with nested(a(), b()) as (x, y):
                 1 // 0
+
+# From 3.7.7 test_functools.py
+# Bug is a unreachable code after "return"
+def test_invalid_registrations(x):
+    return
+    with x:
+        x = 1
+
+# From 3.7.7 test_re.py
+# Bug was hooking in c_with.
+def test_re_tests(tests):
+    for t in tests:
+        with a:
+            continue
+
+# Adapted from 3.8 distutils/command/config.py
+# In 3.8 the problem was in handling "with .. as" code
+def _gen_temp_sourcefile(x, a, headers, lang):
+    with x as y:
+        if a:
+            y = 2
+    return 5

test/stdlib/.gitignore

@@ -1 +1,3 @@
 /.python-version
+/runun33.sh
+/runun7.sh

test/stdlib/2.5-exclude.sh

@@ -81,6 +81,6 @@ SKIP_TESTS=(
 )
 # About 265 tests in 14 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_doctest.py]=1 # Fails on ppc64le
 fi

test/stdlib/2.6-exclude.sh

@@ -1,4 +1,12 @@
 SKIP_TESTS=(
+    # ifelsestmt is borked in:
+    #   if filename == 'srcfile':
+    #      return srcfile
+    #   if filename == 'destfile':
+    #      return destfile
+    #   assert 0  # shouldn't reach here.
+    [test_shutil.py]=1
+
 
     [test___all__.py]=1  # it fails on its own
     [test___all__.py]=1 # it fails on its own
@@ -59,6 +67,7 @@ SKIP_TESTS=(
 
     [test_scriptpackages.py]=1 # it fails on its own
     [test_select.py]=1 # test takes too long to run: 11 seconds
+
     [test_socket.py]=1 # test takes too long to run: 12 seconds
     [test_startfile.py]=1 # it fails on its own
     [test_structmembers.py]=1 # it fails on its own
@@ -85,7 +94,7 @@ SKIP_TESTS=(
 )
 # About 305 unit-test files in about 12 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     # Fails in crontab environment?
     # Figure out what's up here
     SKIP_TESTS[test_aifc.py]=1

test/stdlib/2.7-exclude.sh

@@ -1,4 +1,6 @@
 SKIP_TESTS=(
+    [test_cgi.py]=1 # FIXME: Works on c90ff51
+
     [test_bsddb3.py]=1 # test takes too long to run: 110 seconds
     [test_compile.py]=1  # Code introspects on co_consts in a non-decompilable way
     [test_curses.py]=1  # Possibly fails on its own but not detected
@@ -25,6 +27,7 @@ SKIP_TESTS=(
     [test_ssl.py]=1  #
     [test_subprocess.py]=1 # Runs ok but takes 22 seconds
     [test_sys_settrace.py]=1 # Line numbers are expected to be different
+
     [test_tokenize.py]=1 # test takes too long to run: 19 seconds
     [test_traceback.py]=1 # Line numbers change - duh.
     [test_unicode.py]=1  # Too long to run 11 seconds
@@ -34,7 +37,7 @@ SKIP_TESTS=(
 )
 # 334 unit-test files in about 15 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     # Fails in crontab environment?
     # Figure out what's up here
     SKIP_TESTS[test_array.py]=1
@@ -43,6 +46,7 @@ if (( batch )) ; then
     SKIP_TESTS[test_doctest2.py]=1 # a POWER thing?
     SKIP_TESTS[test_httplib.py]=1  # Ok, but POWER has problems with this
     SKIP_TESTS[test_pdb.py]=1 # Ok, but POWER has problems with this
+    SKIP_TESTS[test_tarfile.py]=1 # test can take over 15 seconds to run on an overloaded POWER7 system
 
     # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
     SKIP_TESTS[test_base64.py]=1

test/stdlib/3.2-exclude.sh

@@ -17,8 +17,6 @@ SKIP_TESTS=(
     [test_peepholer.py]=1
     [test_pep352.py]=1
 
-    [test_quopri.py]=1 # TypeError: Can't convert 'bytes' object to str implicitly
-
     [test_runpy.py]=1
 
     [test_ssl.py]=1 # too installation specific
@@ -31,9 +29,8 @@ SKIP_TESTS=(
 
 )
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     # Fails in crontab environment?
     # Figure out what's up here
     SKIP_TESTS[test_exception_variations.py]=1
-    SKIP_TESTS[test_quopri.py]=1
 fi

test/stdlib/3.3-exclude.sh

@@ -1,7 +1,19 @@
 SKIP_TESTS=(
-    [test_atexit.py]=1  # The atexit test starting at 3.3 looks for specific comments in error lines
+    # FIXME: Did this work sometime in the past ?
+    #   for elem in g(s):
+    #     if not tgt and isOdd(elem): continue
+    # is erroneously:
+    #   for elem in g(s):
+    #    if tgt or isOdd(elem):
+    #       pass
+    #    else:
+    #       tgt.append(elem)
+    [test_itertools.py]=1
 
-    [test_buffer.py]=1  # parse error
+    [test_buffer.py]=1  # FIXME: Works on c90ff51
+    [test_cmath.py]=1  # FIXME: Works on c90ff51
+
+    [test_atexit.py]=1  # The atexit test starting at 3.3 looks for specific comments in error lines
 
     [test_cmd_line.py]=1 # too long?
     [test_concurrent_futures.py]=1  # too long?
@@ -30,6 +42,7 @@ SKIP_TESTS=(
 
     [test_nntplib.py]=1
 
+    [test_pep352.py]=1  # test failures
     [test_peepholer.py]=1
     [test_poll.py]=1  # test takes too long to run: 11 seconds
     [test_pty.py]=1  # FIXME: Needs grammar loop isolation separation
@@ -57,7 +70,7 @@ SKIP_TESTS=(
 )
 # About 300 unit-test files in about 20 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_ftplib.py]=1  # Runs too long on POWER; over 15 seconds
     SKIP_TESTS[test_idle.py]=1  # No tk
     SKIP_TESTS[test_pep352.py]=1  # UnicodeDecodeError may be funny on weird environments

test/stdlib/3.4-exclude.sh

@@ -1,11 +1,29 @@
 SKIP_TESTS=(
+    # FIXME: Did this work sometime in the past ?
+    #   for elem in g(s):
+    #     if not tgt and isOdd(elem): continue
+    # is erroneously:
+    #   for elem in g(s):
+    #    if tgt or isOdd(elem):
+    #       pass
+    #    else:
+    #       tgt.append(elem)
+    [test_itertools.py]=1
+
+    [test_buffer.py]=1  # FIXME: Works on c90ff51
+    [test_cmath.py]=1  # FIXME: Works on c90ff51
+    [test_strftime.py]=1  # FIXME: Works on c90ff51
+
     [test___all__.py]=1  # it fails on its own
     [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
+
     [test_cmd_line.py]=1 # takes too long to run
     [test_concurrent_futures.py]=1  # too long?
+
     [test_configparser.py]=1  # Doesn't terminate
     [test_ctypes.py]=1 # it fails on its own
     [test_curses.py]=1 # Investigate
+
     [test_dbm_gnu.py]=1   # fails on its own
     [test_devpoll.py]=1 # it fails on its own
     [test_descr.py]=1   # test assertion errors
@@ -14,33 +32,44 @@ SKIP_TESTS=(
     [test_doctest2.py]=1
     [test_doctest.py]=1  # test assert failures
     [test_docxmlrpc.py]=1
+
     [test_enum.py]=1 # compile syntax?
     [test_exceptions.py]=1
+
     [test_faulthandler.py]=1
     [test_file_eintr.py]=1   # parse error
     [test_fork1.py]=1 # too long
+
     [test_gdb.py]=1 # it fails on its own
     [test_grammar.py]=1 # parse error
+
     [test_httplib.py]=1 # it fails on its own
+
     [test_import.py]=1 # it fails on its own
     [test_io.py]=1
     [test_ioctl.py]=1 # it fails on its own
     [test_inspect.py]=1 # Syntax error Investigate
+
     [test_logging.py]=1 # Too long to run
     [test_long.py]=1 # FIXME: Works on c90ff51
+
     [test_modulefinder.py]=1  # test assertion error
     [test_multiprocessing_fork.py]=1 # doesn't terminate
     [test_multiprocessing_forkserver.py]=1 # doesn't terminate
     [test_multiprocessing_main_handling.py]=1  # doesn't terminate
     [test_multiprocessing_spawn.py]=1 # doesn't terminate
+
     [test_nntplib.py]=1 # too long to run
+
     [test_peepholer.py]=1 # control flow?
     [test_pep352.py]=1 # test assert failures
     [test_pickle.py]=1 # test assert failures
     [test_pkgimport.py]=1 # long
     [test_poll.py]=1 # Too long to run: 11 seconds
     [test_pydoc.py]=1 # test assertion failures
+
     [test_runpy.py]=1 # Too long:
+
     [test_select.py]=1 # Too long: 11 seconds
     [test_selectors.py]=1  # Too long: 11 seconds
     [test_signal.py]=1 # Too long: 22 seconds
@@ -50,12 +79,15 @@ SKIP_TESTS=(
     [test_subprocess.py]=1 # Too long
     [test_symtable.py]=1 # Investigate bad output
     [test_sys_settrace.py]=1 # test assert failures
+
     [test_tcl.py]=1 # May be implementation specific. On POWER though it fails
     [test_threading.py]=1 # Too long
     [test_threadsignals.py]=1 # Too long to run: 12 seconds
     [test_timeout.py]=1 # Too long to run: 19 seconds
     [test_traceback.py]=1 # introspects on code
+
     [test_urllib2net.py]=1 # Doesn't terminate
+
     [test_zipfile64.py]=1
     [test_zlib.py]=1
 )
@@ -66,5 +98,4 @@ if (( batch )) ; then
     # Figure out what's up here
     SKIP_TESTS[test_exception_variations.py]=1
     SKIP_TESTS[test_mailbox.py]=1 # Takes to long on POWER; over 15 secs
-    SKIP_TESTS[test_quopri.py]=1
 fi

test/stdlib/3.5-exclude.sh

@@ -1,11 +1,7 @@
 SKIP_TESTS=(
     [test_buffer.py]=1  # FIXME: Works on c90ff51
-    [test_decorators.py]=1  # FIXME: Works on c90ff51
+    [test_platform.py]=1  # FIXME: Works on c90ff51
-    [test_platform.py]=1 # FIXME: Works on c90ff51
+    [test_pyclbr.py]=1  # FIXME: Works on c90ff51
-    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
-    [test_tempfile.py]=1 # FIXME: Works on c90ff51
-    [test_uu.py]=1 # FIXME: Works on c90ff51
-    [test_ftplib.py]=1 # FIXME: Works on c90ff51
 
     [test___all__.py]=1 # it fails on its own
     [test_aifc.py]=1  #
@@ -100,6 +96,7 @@ SKIP_TESTS=(
     [test_sys_settrace.py]=1 # test assert fail
 
     [test_tcl.py]=1  # it fails on its own
+    [test_tempfile.py]=1  # test assertion failures
     [test_thread.py]=1
     [test_threading.py]=1
     [test_timeout.py]=1
@@ -115,6 +112,7 @@ SKIP_TESTS=(
     [test_urllib2net.py]=1 # it fails on its own
     [test_urllibnet.py]=1 # it fails on its own
     [test_urlparse.py]=1 # test assert error
+    [test_uu.py]=1 # test assert error
 
     [test_winreg.py]=1 # it fails on its own
     [test_winsound.py]=1 # it fails on its own
@@ -129,7 +127,7 @@ SKIP_TESTS=(
 )
 # About 260 unit-test in about 16 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_asyncore.py]=1 # Ok, but takes more than 15 seconds to run
     SKIP_TESTS[test_bisect.py]=1
     SKIP_TESTS[test_compileall.py]=1  # Something weird on POWER
@@ -137,10 +135,10 @@ if (( batch )) ; then
     SKIP_TESTS[test_distutils.py]=1
 
     SKIP_TESTS[test_exception_variations.py]=1
+    SKIP_TESTS[test_ioctl.py]=1 # it fails on its own
     SKIP_TESTS[test_poplib.py]=1 # May be a result of POWER installation
 
-    SKIP_TESTS[test_quopri.py]=1
+    SKIP_TESTS[test_sysconfig.py]=1 # POWER extension fails
-    SKIP_TESTS[test_ioctl.py]=1 # it fails on its own
     SKIP_TESTS[test_tarfile.py]=1 # too long to run on POWER 15 secs
     SKIP_TESTS[test_venv.py]=1 # takes too long 11 seconds
 fi

test/stdlib/3.6-exclude.sh

@@ -1,13 +1,12 @@
 SKIP_TESTS=(
     [test_ast.py]=1 # FIXME: Works on c90ff51
-    [test_binop.py]=1 # FIXME: Works on c90ff51
+    [test_cmath.py]=1  # FIXME: Works on c90ff51
-    [test_complex.py]=1 # FIXME: Works on c90ff51
     [test_format.py]=1 # FIXME: Works on c90ff51
     [test_ftplib.py]=1 # FIXME: Works on c90ff51
     [test_slice.py]=1 # FIXME: Works on c90ff51
     [test_sort.py]=1 # FIXME: Works on c90ff51
     [test_timeit.py]=1 # FIXME: Works on c90ff51
-    [test_os.py]=1 # FIXME: Works on c90ff51
+    [test_os.py]=1 # parse error FIXME: Works on c90ff51
 
     [test___all__.py]=1  # it fails on its own
     [test_aifc.py]=1  #
@@ -49,6 +48,7 @@ SKIP_TESTS=(
     [test_datetime.py]=1 # it fails on its own
     [test_dbm_ndbm.py]=1 # it fails on its own
     [test_decimal.py]=1
+    [test_decorators.py]=1 # control-flow failures
     [test_descr.py]=1   # syntax error: Investigate
     [test_devpoll.py]=1 # it fails on its own
     [test_dict.py]=1 # it fails on its own
@@ -106,7 +106,8 @@ SKIP_TESTS=(
     [test_nntplib.py]=1 # test takes too long to run: 31 seconds
     [test_normalization.py]=1 # it fails on its own
 
-    [test_ordered_dict.py]= # it fails on its own
+    [test_optparse.py]=1 # test fails
+    [test_ordered_dict.py]=1 # it fails on its own
     [test_ossaudiodev.py]=1 # it fails on its own
 
     [test_pdb.py]=1 # Probably introspection
@@ -122,8 +123,6 @@ SKIP_TESTS=(
     [test_pyclbr.py]=1 # it fails on its own
     [test_pydoc.py]=1 # it fails on its own
 
-    [test_quopri.py]=1      # AssertionError: b'123=four' != '123=four'
-
     [test_random.py]=1 # it fails on its own
     [test_range.py]=1
     [test_regrtest.py]=1 # test takes too long to run: 12 seconds
@@ -147,6 +146,7 @@ SKIP_TESTS=(
     [test_startfile.py]=1 # it fails on its own
     [test_statistics.py]=1 # it fails on its own
     [test_string_literals.py]=1
+    [test_strftime.py]=1 # test assertion failures
     [test_strtod.py]=1 # it fails on its own
     [test_struct.py]=1  # test assertion errors
     [test_subprocess.py]=1
@@ -200,8 +200,10 @@ SKIP_TESTS=(
 )
 # 236 unit-test files in about 13 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_codeccallbacks.py]=1
+    SKIP_TESTS[test_complex.py]=1 # Something funky with POWER8
+
     # locale on test machine is probably customized
     SKIP_TESTS[test__locale.py]=1
 fi

test/stdlib/3.7-exclude.sh

@@ -1,6 +1,22 @@
 SKIP_TESTS=(
+    # FIXME: Did this work sometime in the past ?
+    #   for elem in g(s):
+    #     if not tgt and isOdd(elem): continue
+    # is erroneously:
+    #   for elem in g(s):
+    #    if tgt or isOdd(elem):
+    #       pass
+    #    else:
+    #       tgt.append(elem)
+    [test_itertools.py]=1
+
+    # Fails on decompyle3 as well.
+    # complicated control flow and "and/or" expressions
+    [test_pickle.py]=1
+
     [test_builtin.py]=1 # FIXME works on decompyle6
     [test_context.py]=1 # FIXME works on decompyle6
+    [test_doctest2.py]=1 # FIXME works on decompyle6
     [test_format.py]=1 # FIXME works on decompyle6
     [test_marshal.py]=1 # FIXME works on decompyle6
     [test_normalization.py]=1 # FIXME works on decompyle6
@@ -9,12 +25,11 @@ SKIP_TESTS=(
     [test_slice.py]=1 # FIXME works on decompyle6
     [test_sort.py]=1 # FIXME works on decompyle6
     [test_statistics.py]=1 # FIXME works on decompyle6
-    [test_string_literals.py]=1 # FIXME works on decompyle6
     [test_timeit.py]=1 # FIXME works on decompyle6
     [test_urllib2_localnet.py]=1 # FIXME works on decompyle6
     [test_urllib2.py]=1 # FIXME: works on uncompyle6
-    [test_generators.py]=1  # Investigate improper lamdba with bogus "False" added
+    [test_generators.py]=1  # FIXME: works on uncompyle6 - lambda parsing probably
-    [test_grammar.py]=1 # investigate: like above: semantic rule missing probably
+    [test_grammar.py]=1 # FIXME: works on uncompyle6 - lambda parsing probably
 
     [test___all__.py]=1 # it fails on its own
     [test_argparse.py]=1 #- it fails on its own
@@ -133,7 +148,8 @@ SKIP_TESTS=(
 )
 # 306 unit-test files in about 19 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
+    SKIP_TESTS[test_capi.py]=1 # more than 15 secs to run on POWER
     SKIP_TESTS[test_dbm_gnu.py]=1 # fails on its own on POWER
     SKIP_TESTS[test_distutils.py]=1
     SKIP_TESTS[test_fileio.py]=1

test/stdlib/3.8-exclude.sh

@@ -1,10 +1,14 @@
 SKIP_TESTS=(
-    [test_time.py]=1 # FIXME: works on uncompyle6?
+    [test_mimetypes.py]=1 # parse error. decompile3 works. Release 3.6.4 works?
+    [test_time.py]=1 # FIXME: parse eror. decompyle3 works. Release 3.6.4 works?
+    [test_aifc.py]=1 # parse error; decompile3 works
+
+    [test_doctest2.py]=1 # test failures release 3.6.4 works?
+    [test_finalization.py]=1  # test failures release 3.6.4 works?
     [test_urllib2.py]=1 # FIXME: works on uncompyle6?
     [test_zipimport.py]=1 # FIXME: works on uncompyle6
 
     [test___all__.py]=1 # it fails on its own
-    [test_aifc.py]=1 # parse error
     [test_argparse.py]=1 #- it fails on its own
     [test_array.py]=1 #- parse error
     [test_asdl_parser.py]=1 # it fails on its own
@@ -76,6 +80,7 @@ SKIP_TESTS=(
     [test_difflib.py]=1   # parse error
     [test_dis.py]=1   # We change line numbers - duh!
     [test_doctest.py]=1 # parse error
+    [test_doctest2.py]=1 # test faiures relesae 3.6.4 works?
     [test_docxmlrpc.py]=1
     [test_dtrace.py]=1 # parse error
     [test_dummy_thread.py]=1 # parse error
@@ -91,6 +96,7 @@ SKIP_TESTS=(
     [test_exceptions.py]=1   # parse error
 
     [test_faulthandler.py]=1   # takes too long
+    [test_file_eintr.py]=1 # too long to run test; works on 3.7.7
     [test_fcntl.py]=1
     [test_filecmp.py]=1   # parse error
     [test_fileinput.py]=1
@@ -329,7 +335,7 @@ SKIP_TESTS=(
 )
 # 114 test files, Elapsed time about 7 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_idle.py]=1 # Probably installation specific
     SKIP_TESTS[test_tix.py]=1 # fails on its own
     SKIP_TESTS[test_ttk_textonly.py]=1 # Installation dependent?

test/stdlib/run-and-email.sh

@@ -3,28 +3,50 @@
 function displaytime {
     printf "ran in "
     local T=$1
-    local D=$((T/60/60/24))
+    ((D=T/60/60/24))
-    local H=$((T/60/60%24))
+    ((H=T/60/60%24))
-    local M=$((T/60%60))
+    ((M=T/60%60))
-    local S=$((T%60))
+    ((S=T%60))
-    (( $D > 0 )) && printf '%d days ' $D
+    (( D > 0 )) && printf '%d days ' $D
-    (( $H > 0 )) && printf '%d hours ' $H
+    (( H > 0 )) && printf '%d hours ' $H
-    (( $M > 0 )) && printf '%d minutes ' $M
+    (( M > 0 )) && printf '%d minutes ' $M
-    (( $D > 0 || $H > 0 || $M > 0 )) && printf 'and '
+    (( D > 0 || H > 0 || M > 0 )) && printf 'and '
     printf '%d seconds\n' $S
 }
 
-. ../../admin-tools/pyenv-newer-versions
+bs=${BASH_SOURCE[0]}
+if [[ $0 != $bs ]] ; then
+    echo "This script should not be *sourced* but run through bash"
+    exit 1
+fi
+
+mydir=$(dirname $bs)
+cd $mydir
+
+branch=$(cat ../../.git/HEAD | cut -d'/' -f 3)
+if [[ $branch == 'python-2.4' ]]; then
+    . ../../admin-tools/pyenv-older-versions
+elif [[ $branch == 'master' ]]; then
+    . ../../admin-tools/pyenv-newer-versions
+else
+    echo &1>2 "Error git branch should either be 'master' or 'python-2.4'; got: '$branch'"
+    exit 1
+fi
+
+MAIN="runtests.sh"
+
 
 USER=${USER:-rocky}
 EMAIL=${EMAIL:-rb@dustyfeet.com}
-SUBJECT_PREFIX="stdlib unit testing for"
+WHAT="uncompyle6 ${MAIN}"
+export BATCH=1
 
 typeset -i RUN_STARTTIME=$(date +%s)
 
-actual_versions=""
 DEBUG=""  # -x
 
+MAILBODY=/tmp/${MAIN}-mailbody-$$.txt
+
 for VERSION in $PYVERSIONS ; do
     typeset -i rc=0
     LOGFILE=/tmp/runtests-$VERSION-$$.log
@@ -34,24 +56,37 @@ for VERSION in $PYVERSIONS ; do
 	    continue
 	    ;;
     esac
-    actual_versions="$actual_versions $VERSION"
 
     if ! pyenv local $VERSION ; then
 	rc=1
     else
-      STOP_ONERROR=1 /bin/bash $DEBUG ./runtests.sh  >$LOGFILE 2>&1
+	typeset -i ALL_FILES_STARTTIME=$(date +%s)
-      rc=$?
+	STOP_ONERROR=1 /bin/bash $DEBUG ./runtests.sh  >$LOGFILE 2>&1
+	rc=$?
+
+	echo Python Version $(pyenv local) >> $LOGFILE
+	echo "" >>$LOGFILE
+	typeset -i ALL_FILES_ENDTIME=$(date +%s)
+	(( time_diff =  ALL_FILES_ENDTIME - ALL_FILES_STARTTIME))
+	time_str=$(displaytime $time_diff)
+	echo ${time_str}. >> $LOGFILE
     fi
-    SUBJECT_PREFIX="runtests verify for"
+
+    SUBJECT_PREFIX="$WHAT for"
     if ((rc == 0)); then
+	mailbody_line="Python $VERSION ok; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION ok" ${USER}@localhost
     else
+	mailbody_line="Python $VERSION failed; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${USER}@localhost
-	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" $EMAIL
+	tail -v $LOGFILE | mail -s "$HOST $SUBJECT_PREFIX $VERSION not ok" $EMAIL
     fi
+    echo $mailbody_line >> $MAILBODY
 done
 
 typeset -i RUN_ENDTIME=$(date +%s)
 (( time_diff =  RUN_ENDTIME - RUN_STARTTIME))
 elapsed_time=$(displaytime $time_diff)
-echo "Run complete $elapsed_time for versions $actual_versions" | mail -s "runtests in $elapsed_time" ${EMAIL}
+echo "${WHAT} complete; ${elapsed_time}." >> $MAILBODY
+echo "Full results are in ${LOGFILE}." >> $MAILBODY
+cat $MAILBODY | mail -s "$HOST $WHAT $elapsed_time" ${EMAIL}

test/stdlib/runtests.sh

@@ -1,10 +1,12 @@
 #!/bin/bash
 me=${BASH_SOURCE[0]}
 
-typeset -i batch=1
+typeset -i BATCH=${BATCH:-0}
-isatty=$(/usr/bin/tty 2>/dev/null)
+if (( ! BATCH )) ; then
-if [[ -n $isatty ]] && [[ "$isatty" != 'not a tty' ]] ; then
+    isatty=$(/usr/bin/tty 2>/dev/null)
-    batch=0
+    if [[ -n $isatty ]] && [[ "$isatty" != 'not a tty' ]] ; then
+	BATCH=0
+    fi
 fi
 
 
@@ -72,11 +74,10 @@ case $PYVERSION in
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_fileio.py]=1
 	)
-	if (( batch )) ; then
+	if (( BATCH )) ; then
 	    # Fails in crontab environment?
 	    # Figure out what's up here
 	    SKIP_TESTS[test_exception_variations.py]=1
-	    SKIP_TESTS[test_quopri.py]=1
 	fi
 	;;
     3.1)
@@ -85,11 +86,10 @@ case $PYVERSION in
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_fileio.py]=1
 	)
-	if (( batch )) ; then
+	if (( BATCH )) ; then
 	    # Fails in crontab environment?
 	    # Figure out what's up here
 	    SKIP_TESTS[test_exception_variations.py]=1
-	    SKIP_TESTS[test_quopri.py]=1
 	fi
 	;;
     3.2)
@@ -129,6 +129,7 @@ fulldir=$(pwd)
 
 # DECOMPILER=uncompyle2
 DECOMPILER=${DECOMPILER:-"$fulldir/../../bin/uncompyle6"}
+OPTS=${OPTS:-""}
 TESTDIR=/tmp/test${PYVERSION}
 if [[ -e $TESTDIR ]] ; then
     rm -fr $TESTDIR
@@ -137,6 +138,8 @@ fi
 PYENV_ROOT=${PYENV_ROOT:-$HOME/.pyenv}
 pyenv_local=$(pyenv local)
 
+echo Python version is $pyenv_local
+
 # pyenv version update
 for dir in ../ ../../ ; do
     cp -v .python-version $dir
@@ -145,18 +148,25 @@ done
 
 mkdir $TESTDIR || exit $?
 cp -r ${PYENV_ROOT}/versions/${PYVERSION}.${MINOR}/lib/python${PYVERSION}/test $TESTDIR
-cd $TESTDIR/test
+if [[ $PYVERSION == 3.2 ]] ; then
+    cp ${PYENV_ROOT}/versions/${PYVERSION}.${MINOR}/lib/python${PYVERSION}/test/* $TESTDIR
+    cd $TESTDIR
+else
+    cd $TESTDIR/test
+fi
 pyenv local $FULLVERSION
 export PYTHONPATH=$TESTDIR
 export PATH=${PYENV_ROOT}/shims:${PATH}
 
+DONT_SKIP_TESTS=${DONT_SKIP_TESTS:-0}
+
 # Run tests
 typeset -i i=0
 typeset -i allerrs=0
 if [[ -n $1 ]] ; then
-    files=$1
+    files=$@
-    typeset -a files_ary=( $(echo $1) )
+    typeset -a files_ary=( $(echo $@) )
-    if (( ${#files_ary[@]} == 1 )) ; then
+    if (( ${#files_ary[@]} == 1 || DONT_SKIP_TESTS == 1 )) ; then
 	SKIP_TESTS=()
     fi
 else
@@ -166,9 +176,12 @@ fi
 typeset -i ALL_FILES_STARTTIME=$(date +%s)
 typeset -i skipped=0
 
+NOT_INVERTED_TESTS=${NOT_INVERTED_TESTS:-1}
+
 for file in $files; do
     # AIX bash doesn't grok [[ -v SKIP... ]]
-    if [[ ${SKIP_TESTS[$file]} == 1 ]] ; then
+    [[ -z ${SKIP_TESTS[$file]} ]] && SKIP_TESTS[$file]=0
+    if [[ ${SKIP_TESTS[$file]} == ${NOT_INVERTED_TESTS} ]] ; then
 	((skipped++))
 	continue
     fi
@@ -185,7 +198,7 @@ for file in $files; do
     typeset -i ENDTIME=$(date +%s)
     typeset -i time_diff
     (( time_diff =  ENDTIME - STARTTIME))
-    if (( time_diff > 10 )) ; then
+    if (( time_diff > $timeout )) ; then
 	echo "Skipping test $file -- test takes too long to run: $time_diff seconds"
 	continue
     fi
@@ -197,7 +210,7 @@ for file in $files; do
     $fulldir/compile-file.py $file && \
     mv $file{,.orig} && \
     echo ==========  $(date +%X) Decompiling $file ===========
-    $DECOMPILER $decompiled_file > $file
+    $DECOMPILER $OPTS $decompiled_file > $file
     rc=$?
     if (( rc == 0 )) ; then
 	echo ========== $(date +%X) Running $file ===========

uncompyle6/Makefile

@@ -1,7 +0,0 @@
-# Whatever it is you want to do, it should be forwarded to the 
-# to top-level irectories
-PHONY=check all
-all: check
-
-%: 
-	$(MAKE) -C .. $@

uncompyle6/disas.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016, 2818-2019 by Rocky Bernstein
+#  Copyright (c) 2015-2016, 2818-2020 by Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
@@ -34,8 +34,7 @@ from __future__ import print_function
 import sys
 from collections import deque
 
-from xdis.code import iscode
+from xdis import check_object_path, iscode, load_module
-from xdis.load import check_object_path, load_module
 from uncompyle6.scanner import get_scanner
 
 
@@ -100,7 +99,9 @@ def disassemble_file(filename, outstream=None):
     try to find the corresponding compiled object.
     """
     filename = check_object_path(filename)
-    (version, timestamp, magic_int, co, is_pypy, source_size) = load_module(filename)
+    (version, timestamp, magic_int, co, is_pypy, source_size, sip_hash) = load_module(
+        filename
+    )
     if type(co) == list:
         for con in co:
             disco(version, con, outstream)

uncompyle6/linenumbers.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016, 2818 by Rocky Bernstein
+#  Copyright (c) 2015-2016, 2818, 2020 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -15,14 +15,27 @@
 
 from collections import deque
 
-from xdis.code import iscode
+from xdis import (
-from xdis.load import load_file, load_module
+    Bytecode,
-from xdis.main import get_opcode
+    iscode,
-from xdis.bytecode import Bytecode, findlinestarts, offset2line
+    findlinestarts,
+    get_opcode,
+    offset2line,
+    load_file,
+    load_module,
+)
+
 
 def line_number_mapping(pyc_filename, src_filename):
-    (version, timestamp, magic_int, code1, is_pypy,
+    (
-     source_size) = load_module(pyc_filename)
+        version,
+        timestamp,
+        magic_int,
+        code1,
+        is_pypy,
+        source_size,
+        sip_hash,
+    ) = load_module(pyc_filename)
     try:
         code2 = load_file(src_filename)
     except SyntaxError as e:
@@ -44,7 +57,10 @@ def number_loop(queue, mappings, opc):
         assert code1.co_name == code2.co_name
         linestarts_orig = findlinestarts(code1)
         linestarts_uncompiled = list(findlinestarts(code2))
-        mappings += [[line, offset2line(offset, linestarts_uncompiled)] for offset, line in linestarts_orig]
+        mappings += [
+            [line, offset2line(offset, linestarts_uncompiled)]
+            for offset, line in linestarts_orig
+        ]
         bytecode1 = Bytecode(code1, opc)
         bytecode2 = Bytecode(code2, opc)
         instr2s = bytecode2.get_instructions(code2)

uncompyle6/main.py

@@ -16,8 +16,7 @@ from __future__ import print_function
 import datetime, py_compile, os, subprocess, sys, tempfile
 
 from uncompyle6 import verify, IS_PYPY, PYTHON_VERSION
-from xdis.code import iscode
+from xdis import iscode, sysinfo2float
-from xdis.magics import sysinfo2float
 from uncompyle6.disas import check_object_path
 from uncompyle6.semantics import pysource
 from uncompyle6.parser import ParserError
@@ -102,7 +101,7 @@ def decompile(
     )
     if PYTHON_VERSION < 3.0 and bytecode_version >= 3.0:
         write(
-            "# Warning: this version has problems handling the Python 3 byte type in contants properly.\n"
+            '# Warning: this version has problems handling the Python 3 "byte" type in constants properly.\n'
         )
 
     if co.co_filename:
@@ -183,7 +182,7 @@ def decompile_file(
 
     filename = check_object_path(filename)
     code_objects = {}
-    (version, timestamp, magic_int, co, is_pypy, source_size) = load_module(
+    (version, timestamp, magic_int, co, is_pypy, source_size, sip_hash) = load_module(
         filename, code_objects
     )
 

uncompyle6/parser.py

@@ -23,16 +23,16 @@ from __future__ import print_function
 
 import sys
 
-from xdis.code import iscode
+from xdis import iscode, py_str2float
-from xdis.magics import py_str2float
 from spark_parser import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.show import maybe_show_asm
 
 
 class ParserError(Exception):
-    def __init__(self, token, offset):
+    def __init__(self, token, offset, debug=PARSER_DEFAULT_DEBUG):
         self.token = token
         self.offset = offset
+        self.debug = debug
 
     def __str__(self):
         return "Parse error at or near `%r' instruction at offset %s\n" % (
@@ -206,9 +206,9 @@ class PythonParser(GenericASTBuilder):
                 else:
                     indent = "-> "
                 print("%s%s" % (indent, instructions[i]))
-            raise ParserError(err_token, err_token.offset)
+            raise ParserError(err_token, err_token.offset, self.debug["reduce"])
         else:
-            raise ParserError(None, -1)
+            raise ParserError(None, -1, self.debug["reduce"])
 
     def get_pos_kw(self, token):
         """Return then the number of positional parameters and
@@ -351,7 +351,7 @@ class PythonParser(GenericASTBuilder):
         stmt ::= try_except
         stmt ::= tryelsestmt
         stmt ::= tryfinallystmt
-        stmt ::= withstmt
+        stmt ::= with
         stmt ::= withasstmt
 
         stmt ::= del_stmt

uncompyle6/parsers/parse10.py

@@ -7,7 +7,7 @@ from uncompyle6.parsers.parse11 import Python11Parser
 
 class Python10Parser(Python11Parser):
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python11Parser, self).__init__(debug_parser)
+        super(Python10Parser, self).__init__(debug_parser)
         self.customized = {}
 
 

uncompyle6/parsers/parse11.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019 Rocky Bernstein
+#  Copyright (c) 2019-2020 Rocky Bernstein
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.parser import PythonParserSingle
@@ -7,7 +7,7 @@ from uncompyle6.parsers.parse12 import Python12Parser
 
 class Python11Parser(Python12Parser):
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python12Parser, self).__init__(debug_parser)
+        super(Python11Parser, self).__init__(debug_parser)
         self.customized = {}
 
 

uncompyle6/parsers/parse2.py

@@ -196,8 +196,9 @@ class Python2Parser(PythonParser):
         expr ::= slice3
         expr ::= unary_convert
 
-        and  ::= expr jmp_false expr come_from_opt
+        expr_jt  ::= expr jmp_true
-        or   ::= expr jmp_true  expr come_from_opt
+        or       ::= expr_jt  expr come_from_opt
+        and      ::= expr jmp_false expr come_from_opt
 
         unary_convert ::= expr UNARY_CONVERT
 

uncompyle6/parsers/parse21.py

@@ -8,7 +8,7 @@ from uncompyle6.parsers.parse22 import Python22Parser
 class Python21Parser(Python22Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python22Parser, self).__init__(debug_parser)
+        super(Python21Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_forstmt21(self, args):

uncompyle6/parsers/parse22.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2017 Rocky Bernstein
+#  Copyright (c) 2016-2017, 2020 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
@@ -8,7 +8,7 @@ from uncompyle6.parsers.parse23 import Python23Parser
 class Python22Parser(Python23Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python23Parser, self).__init__(debug_parser)
+        super(Python22Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_misc22(self, args):

uncompyle6/parsers/parse23.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2018 Rocky Bernstein
+#  Copyright (c) 2016-2018, 2020 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 #  Copyright (c) 1999 John Aycock
 
@@ -9,7 +9,7 @@ from uncompyle6.parsers.parse24 import Python24Parser
 class Python23Parser(Python24Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python24Parser, self).__init__(debug_parser)
+        super(Python23Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_misc23(self, args):
@@ -32,8 +32,21 @@ class Python23Parser(Python24Parser):
         while1stmt ::= _while1test l_stmts_opt JUMP_BACK
                        POP_TOP POP_BLOCK COME_FROM
 
-        while1stmt ::= _while1test l_stmts_opt JUMP_BACK
+        while1stmt ::= _while1test l_stmts_opt JUMP_BACK COME_FROM
-                       COME_FROM POP_TOP POP_BLOCK COME_FROM
+                       POP_TOP POP_BLOCK COME_FROM
+
+        # Python 2.3
+        # The following has no "JUMP_BACK" after l_stmts because
+        # l_stmts ends in a "break", "return", or "continue"
+        while1stmt ::= _while1test l_stmts
+                       POP_TOP POP_BLOCK
+
+        # The following has a "COME_FROM" at the end which comes from
+        # a "break" inside "l_stmts".
+        while1stmt ::= _while1test l_stmts COME_FROM JUMP_BACK
+                       POP_TOP POP_BLOCK COME_FROM
+        while1stmt ::= _while1test l_stmts JUMP_BACK
+                       POP_TOP POP_BLOCK
 
         list_comp  ::= BUILD_LIST_0 DUP_TOP LOAD_ATTR store list_iter del_stmt
         list_for   ::= expr for_iter store list_iter JUMP_BACK come_froms POP_TOP JUMP_BACK

uncompyle6/parsers/parse24.py

@@ -38,8 +38,23 @@ class Python24Parser(Python25Parser):
         _ifstmts_jump24 ::= c_stmts_opt JUMP_FORWARD POP_TOP
 
         # Python 2.5+ omits POP_TOP POP_BLOCK
-        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_TOP POP_BLOCK COME_FROM
+        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK
-        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_TOP POP_BLOCK
+                       POP_TOP POP_BLOCK COME_FROM
+        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK
+                       POP_TOP POP_BLOCK
+
+        continue   ::= JUMP_BACK JUMP_ABSOLUTE
+
+        # Python 2.4
+        # The following has no "JUMP_BACK" after l_stmts because
+        # l_stmts ends in a "break", "return", or "continue"
+        while1stmt ::= SETUP_LOOP l_stmts
+                       POP_TOP POP_BLOCK
+
+        # The following has a "COME_FROM" at the end which comes from
+        # a "break" inside "l_stmts".
+        while1stmt ::= SETUP_LOOP l_stmts COME_FROM JUMP_BACK
+                       POP_TOP POP_BLOCK COME_FROM
 
         # Python 2.5+:
         #  call_stmt ::= expr POP_TOP
@@ -73,8 +88,8 @@ class Python24Parser(Python25Parser):
         with_cleanup  ::= LOAD_FAST DELETE_FAST WITH_CLEANUP END_FINALLY
         with_cleanup  ::= LOAD_NAME DELETE_NAME WITH_CLEANUP END_FINALLY
         withasstmt    ::= expr setupwithas store suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM with_cleanup
-        withstmt      ::= expr setupwith SETUP_FINALLY suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM with_cleanup
+        with          ::= expr setupwith SETUP_FINALLY suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM with_cleanup
-        stmt ::= withstmt
+        stmt ::= with
         stmt ::= withasstmt
         """)
         super(Python24Parser, self).customize_grammar_rules(tokens, customize)

uncompyle6/parsers/parse25.py

@@ -27,7 +27,7 @@ class Python25Parser(Python26Parser):
                         setup_finally
         # opcode SETUP_WITH
         setupwith ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 POP_TOP
-        withstmt  ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+        with      ::= expr setupwith SETUP_FINALLY suite_stmts_opt
                       POP_BLOCK LOAD_CONST COME_FROM with_cleanup
 
         # Semantic actions want store to be at index 2
@@ -62,7 +62,7 @@ class Python25Parser(Python26Parser):
         # Remove grammar rules inherited from Python 2.6 or Python 2
         self.remove_rules("""
         setupwith  ::= DUP_TOP LOAD_ATTR ROT_TWO LOAD_ATTR CALL_FUNCTION_0 POP_TOP
-        withstmt   ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+        with       ::= expr setupwith SETUP_FINALLY suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY
         withasstmt ::= expr setupwithas store suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY

uncompyle6/parsers/parse26.py

@@ -119,8 +119,8 @@ class Python26Parser(Python2Parser):
         ifelsestmtc ::= testexpr c_stmts_opt ja_cf_pop    else_suitec
 
         # Semantic actions want suite_stmts_opt to be at index 3
-        withstmt ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+        with        ::= expr setupwith SETUP_FINALLY suite_stmts_opt
-                     POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY
+                        POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY
 
         # Semantic actions want store to be at index 2
         withasstmt ::= expr setupwithas store suite_stmts_opt

uncompyle6/parsers/parse27.py

@@ -7,6 +7,7 @@ from xdis import next_offset
 from uncompyle6.parser import PythonParserSingle, nop_func
 from uncompyle6.parsers.parse2 import Python2Parser
 from uncompyle6.parsers.reducecheck import (
+    or_check,
     ifelsestmt,
     tryelsestmt,
 )
@@ -101,8 +102,9 @@ class Python27Parser(Python2Parser):
         ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
         if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
-        or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
+        expr_jitop ::= expr JUMP_IF_TRUE_OR_POP
-        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
+        or         ::= expr_jitop expr COME_FROM
+        and        ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
 
         # compare_chained{1,2} is used exclusively in chained_compare
         compare_chained1 ::= expr DUP_TOP ROT_THREE COMPARE_OP JUMP_IF_FALSE_OR_POP
@@ -139,9 +141,11 @@ class Python27Parser(Python2Parser):
         # assert condition, expr
         assert2    ::= assert_expr jmp_true LOAD_ASSERT expr CALL_FUNCTION_1 RAISE_VARARGS_1
 
+        continue   ::= JUMP_BACK JUMP_ABSOLUTE
+
         for_block  ::= returns _come_froms
 
-        withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+        with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM_WITH
                        WITH_CLEANUP END_FINALLY
 
@@ -227,6 +231,7 @@ class Python27Parser(Python2Parser):
         # FIXME: Put more in this table
         self.reduce_check_table = {
             # "ifelsestmt": ifelsestmt,
+            "or": or_check,
             "tryelsestmt": tryelsestmt,
             "tryelsestmtl": tryelsestmt,
         }
@@ -237,7 +242,7 @@ class Python27Parser(Python2Parser):
         self.check_reduce["except_handler"] = "tokens"
         self.check_reduce["except_handler_else"] = "tokens"
 
-        # self.check_reduce["or"] = "AST"
+        self.check_reduce["or"] = "AST"
         self.check_reduce["raise_stmt1"] = "AST"
         self.check_reduce["iflaststmtl"] = "AST"
         self.check_reduce["ifelsestmt"] = "AST"

uncompyle6/parsers/parse3.py

@@ -74,6 +74,7 @@ class Python3Parser(PythonParser):
 
         jb_or_c ::= JUMP_BACK
         jb_or_c ::= CONTINUE
+        jb_cfs  ::= JUMP_BACK _come_froms
 
         stmt ::= set_comp_func
 
@@ -269,18 +270,24 @@ class Python3Parser(PythonParser):
         jmp_abs ::= JUMP_ABSOLUTE
         jmp_abs ::= JUMP_BACK
 
-        withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+        with    ::= expr SETUP_WITH POP_TOP suite_stmts_opt
-                POP_BLOCK LOAD_CONST COME_FROM_WITH
+                    POP_BLOCK LOAD_CONST COME_FROM_WITH
-                WITH_CLEANUP END_FINALLY
+                    WITH_CLEANUP END_FINALLY
 
         withasstmt ::= expr SETUP_WITH store suite_stmts_opt
                 POP_BLOCK LOAD_CONST COME_FROM_WITH
                 WITH_CLEANUP END_FINALLY
 
+        expr_jt     ::= expr jmp_true
+        expr_jitop  ::= expr JUMP_IF_TRUE_OR_POP
+
         ## FIXME: Right now we have erroneous jump targets
         ## This below is probably not correct when the COME_FROM is put in the right place
-        and ::= expr jmp_false expr COME_FROM
+        and  ::= expr jmp_false expr COME_FROM
-        or  ::= expr jmp_true  expr COME_FROM
+        or   ::= expr_jt  expr COME_FROM
+        or   ::= expr_jt expr
+        or   ::= expr_jitop expr COME_FROM
+        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
 
         # # something like the below is needed when the jump targets are fixed
         ## or  ::= expr JUMP_IF_TRUE_OR_POP COME_FROM expr
@@ -338,9 +345,6 @@ class Python3Parser(PythonParser):
         ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
         if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
-        or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
-        or   ::= expr jmp_true expr
-        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
 
         # compare_chained1 is used exclusively in chained_compare
         compare_chained1 ::= expr DUP_TOP ROT_THREE COMPARE_OP JUMP_IF_FALSE_OR_POP
@@ -435,10 +439,11 @@ class Python3Parser(PythonParser):
         while1elsestmt    ::= SETUP_LOOP          l_stmts     JUMP_BACK
                               else_suitel
 
-        whileelsestmt     ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK
+        whileelsestmt     ::= SETUP_LOOP testexpr l_stmts_opt jb_cfs POP_BLOCK
                               else_suitel COME_FROM_LOOP
 
-        whileelsestmt2     ::= SETUP_LOOP testexpr l_stmts_opt  JUMP_BACK POP_BLOCK
+
+        whileelsestmt2    ::= SETUP_LOOP testexpr l_stmts_opt  JUMP_BACK POP_BLOCK
                               else_suitel JUMP_BACK COME_FROM_LOOP
 
         whileTruestmt     ::= SETUP_LOOP l_stmts_opt          JUMP_BACK POP_BLOCK
@@ -674,6 +679,7 @@ class Python3Parser(PythonParser):
                 "RAISE",
                 "SETUP",
                 "UNPACK",
+                "WITH",
             )
         )
 

uncompyle6/parsers/parse30.py

@@ -134,8 +134,9 @@ class Python30Parser(Python31Parser):
 
         jump_except           ::= _jump COME_FROM POP_TOP
 
+        expr_jt               ::= expr jmp_true
         or                    ::= expr jmp_false expr jmp_true expr
-        or                    ::= expr jmp_true expr
+        or                    ::= expr_jt expr
 
         import_from ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist _come_froms POP_TOP
 
@@ -212,7 +213,7 @@ class Python30Parser(Python31Parser):
         whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK JUMP_BACK COME_FROM_LOOP
         whilestmt      ::= SETUP_LOOP testexpr returns POP_TOP POP_BLOCK COME_FROM_LOOP
         withasstmt     ::= expr SETUP_WITH store suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
-        withstmt       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
+        with           ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
 
         # lc_body ::= LOAD_FAST expr LIST_APPEND
         # lc_body ::= LOAD_NAME expr LIST_APPEND

uncompyle6/parsers/parse31.py

@@ -15,7 +15,7 @@ class Python31Parser(Python32Parser):
 
         setupwith      ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 POP_TOP
         setupwithas    ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 store
-        withstmt       ::= expr setupwith SETUP_FINALLY
+        with           ::= expr setupwith SETUP_FINALLY
                            suite_stmts_opt
                            POP_BLOCK LOAD_CONST COME_FROM_FINALLY
                            load del_stmt WITH_CLEANUP END_FINALLY

uncompyle6/parsers/parse35.py

@@ -50,7 +50,7 @@ class Python35Parser(Python34Parser):
 
         # Python 3.5+ has WITH_CLEANUP_START/FINISH
 
-        withstmt   ::= expr
+        with       ::= expr
                        SETUP_WITH POP_TOP suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM_WITH
                        WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
@@ -138,7 +138,7 @@ class Python35Parser(Python34Parser):
         self.remove_rules("""
           yield_from ::= expr GET_ITER LOAD_CONST YIELD_FROM
           yield_from ::= expr expr YIELD_FROM
-          withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+          with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt
                          POP_BLOCK LOAD_CONST COME_FROM_WITH
                          WITH_CLEANUP END_FINALLY
           withasstmt ::= expr SETUP_WITH store suite_stmts_opt
@@ -209,10 +209,10 @@ class Python35Parser(Python34Parser):
             elif opname == 'SETUP_WITH':
                 # Python 3.5+ has WITH_CLEANUP_START/FINISH
                 rules_str = """
-                  withstmt   ::= expr
+                  with ::= expr
-                       SETUP_WITH POP_TOP suite_stmts_opt
+                           SETUP_WITH POP_TOP suite_stmts_opt
-                       POP_BLOCK LOAD_CONST COME_FROM_WITH
+                           POP_BLOCK LOAD_CONST COME_FROM_WITH
-                       WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                           WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
                   withasstmt ::= expr
                        SETUP_WITH store suite_stmts_opt

uncompyle6/parsers/parse36.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2019 Rocky Bernstein
+#  Copyright (c) 2016-2020 Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -305,28 +305,25 @@ class Python36Parser(Python35Parser):
                 self.addRule(rule, nop_func)
                 # Check to combine assignment + annotation into one statement
                 self.check_reduce['assign'] = 'token'
+            elif opname == "WITH_CLEANUP_START":
+                rules_str = """
+                  stmt        ::= with_null
+                  with_null   ::= with_suffix
+                  with_suffix ::= WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                """
+                self.addRule(rules_str, nop_func)
             elif opname == 'SETUP_WITH':
                 rules_str = """
-                withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
+                  with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
-                               WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 with_suffix
 
-                # Removes POP_BLOCK LOAD_CONST from 3.6-
+                  # Removes POP_BLOCK LOAD_CONST from 3.6-
-                withasstmt ::= expr SETUP_WITH store suite_stmts_opt COME_FROM_WITH
+                  withasstmt ::= expr SETUP_WITH store suite_stmts_opt COME_FROM_WITH
-                               WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 with_suffix
+                  with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                                 BEGIN_FINALLY COME_FROM_WITH
+                                 with_suffix
                 """
-                if self.version < 3.8:
-                    rules_str += """
-                    withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
-                                   LOAD_CONST
-                                   WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
-                    """
-                else:
-                    rules_str += """
-                    withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
-                                   BEGIN_FINALLY COME_FROM_WITH
-                                   WITH_CLEANUP_START WITH_CLEANUP_FINISH
-                                   END_FINALLY
-                    """
                 self.addRule(rules_str, nop_func)
                 pass
             pass

uncompyle6/parsers/parse37.py

@@ -94,6 +94,9 @@ class Python37Parser(Python37BaseParser):
         else_suitec ::= c_stmts
         else_suitec ::= returns
 
+        else_suite_opt ::= else_suite
+        else_suite_opt ::= pass
+
         stmt ::= classdef
         stmt ::= call_stmt
 
@@ -632,10 +635,19 @@ class Python37Parser(Python37BaseParser):
     def p_37conditionals(self, args):
         """
         expr                       ::= if_exp37
-        if_exp37              ::= expr expr jf_cfs expr COME_FROM
+        if_exp37                   ::= expr expr jf_cfs expr COME_FROM
         jf_cfs                     ::= JUMP_FORWARD _come_froms
         ifelsestmt                 ::= testexpr c_stmts_opt jf_cfs else_suite opt_come_from_except
 
+        # This is probably more realistically an "ifstmt" (with a null else)
+        # see _cmp() of python3.8/distutils/__pycache__/version.cpython-38.opt-1.pyc
+        ifelsestmt                 ::= testexpr stmts jf_cfs else_suite_opt opt_come_from_except
+
+
+        expr_pjit                  ::= expr POP_JUMP_IF_TRUE
+        expr_jit                   ::= expr JUMP_IF_TRUE
+        expr_jt                    ::= expr jmp_true
+
         jmp_false37                ::= POP_JUMP_IF_FALSE COME_FROM
         list_if                    ::= expr jmp_false37 list_iter
         list_iter                  ::= list_if37
@@ -925,12 +937,14 @@ class Python37Parser(Python37BaseParser):
         ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
         if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
-        jitop_come_from ::= JUMP_IF_TRUE_OR_POP come_froms
+        jitop_come_from_expr ::= JUMP_IF_TRUE_OR_POP come_froms expr
         jifop_come_from ::= JUMP_IF_FALSE_OR_POP come_froms
-        or        ::= and jitop_come_from expr COME_FROM
+        expr_jitop      ::= expr JUMP_IF_TRUE_OR_POP
-        or        ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
+
-        or        ::= expr JUMP_IF_TRUE expr COME_FROM
+        or        ::= and jitop_come_from_expr COME_FROM
-        or        ::= expr POP_JUMP_IF_TRUE expr POP_JUMP_IF_FALSE COME_FROM
+        or        ::= expr_jitop  expr COME_FROM
+        or        ::= expr_jit expr COME_FROM
+        or        ::= expr_pjit expr POP_JUMP_IF_FALSE COME_FROM
 
         testfalse_not_or   ::= expr jmp_false expr jmp_false COME_FROM
         testfalse_not_and ::= and jmp_true come_froms
@@ -946,17 +960,17 @@ class Python37Parser(Python37BaseParser):
         testexprl   ::= testfalsel
         testfalsel  ::= expr jmp_true
 
-        or          ::= expr jmp_true expr
+        or          ::= expr_jt expr
 
         and  ::= expr JUMP_IF_FALSE_OR_POP expr come_from_opt
         and  ::= expr jifop_come_from expr
 
-        pjit_come_from ::= POP_JUMP_IF_TRUE COME_FROM
+        expr_pjit_come_from ::= expr POP_JUMP_IF_TRUE COME_FROM
-        or  ::= expr pjit_come_from expr
+        or  ::= expr_pjit_come_from expr
 
         ## Note that "jmp_false" is what we check on in the "and" reduce rule.
         and ::= expr jmp_false expr COME_FROM
-        or  ::= expr jmp_true  expr COME_FROM
+        or  ::= expr_jt  expr COME_FROM
 
         # compare_chained1 is used exclusively in chained_compare
         compare_chained1 ::= expr DUP_TOP ROT_THREE COMPARE_OP JUMP_IF_FALSE_OR_POP
@@ -1271,7 +1285,7 @@ class Python37Parser(Python37BaseParser):
                 self.addRule(rule, nop_func)
             elif opname == "SETUP_WITH":
                 rules_str = """
-                withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
+                with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
                                WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
                 # Removes POP_BLOCK LOAD_CONST from 3.6-
@@ -1280,13 +1294,13 @@ class Python37Parser(Python37BaseParser):
                 """
                 if self.version < 3.8:
                     rules_str += """
-                    withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                    with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                    LOAD_CONST
                                    WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
                     """
                 else:
                     rules_str += """
-                    withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                    with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                    BEGIN_FINALLY COME_FROM_WITH
                                    WITH_CLEANUP_START WITH_CLEANUP_FINISH
                                    END_FINALLY

uncompyle6/parsers/parse37base.py

@@ -2,10 +2,10 @@
 """
 Python 3.7 base code. We keep non-custom-generated grammar rules out of this file.
 """
-from uncompyle6.scanners.tok import Token
+from uncompyle6.parser import ParserError, PythonParser, nop_func
-from uncompyle6.parser import PythonParser, PythonParserSingle, nop_func
 from uncompyle6.parsers.treenode import SyntaxTree
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser.spark import rule2str
 
 from uncompyle6.parsers.reducecheck import (
     and_check,
@@ -127,6 +127,7 @@ class Python37BaseParser(PythonParser):
                 "RAISE",
                 "SETUP",
                 "UNPACK",
+                "WITH",
             )
         )
 
@@ -210,41 +211,86 @@ class Python37BaseParser(PythonParser):
 
                 if self.version < 3.8:
                     rules_str += """
-                      stmt                ::= async_with_stmt SETUP_ASYNC_WITH
+                      stmt                 ::= async_with_stmt SETUP_ASYNC_WITH
-                      async_with_stmt     ::= expr
+                      c_stmt               ::= c_async_with_stmt SETUP_ASYNC_WITH
+                      async_with_stmt      ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               suite_stmts_opt
+                                               POP_BLOCK LOAD_CONST
+                                               async_with_post
+                      c_async_with_stmt    ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               c_suite_stmts_opt
+                                               POP_BLOCK LOAD_CONST
+                                               async_with_post
+                      async_with_stmt      ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               suite_stmts_opt
+                                               async_with_post
+                      c_async_with_stmt    ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               c_suite_stmts_opt
+                                               async_with_post
+                      async_with_as_stmt   ::= expr
+                                               async_with_pre
+                                               store
+                                               suite_stmts_opt
+                                               POP_BLOCK LOAD_CONST
+                                               async_with_post
+                      c_async_with_as_stmt ::= expr
                                               async_with_pre
-                                              POP_TOP
+                                              store
-                                              suite_stmts_opt
+                                              c_suite_stmts_opt
                                               POP_BLOCK LOAD_CONST
                                               async_with_post
-                      async_with_stmt     ::= expr
+                      async_with_as_stmt   ::= expr
-                                              async_with_pre
+                                              async_with_pre
-                                              POP_TOP
+                                              store
-                                              suite_stmts_opt
+                                              suite_stmts_opt
-                                              async_with_post
+                                              async_with_post
-                      async_with_as_stmt  ::= expr
+                      c_async_with_as_stmt ::= expr
                                               async_with_pre
                                               store
                                               suite_stmts_opt
-                                              POP_BLOCK LOAD_CONST
                                               async_with_post
                     """
                 else:
                     rules_str += """
-                      async_with_pre      ::= BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM SETUP_ASYNC_WITH
+                      async_with_pre       ::= BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM SETUP_ASYNC_WITH
-                      async_with_post     ::= BEGIN_FINALLY COME_FROM_ASYNC_WITH
+                      async_with_post      ::= BEGIN_FINALLY COME_FROM_ASYNC_WITH
-                                              WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                               WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
-                                              WITH_CLEANUP_FINISH END_FINALLY
+                                               WITH_CLEANUP_FINISH END_FINALLY
-                      async_with_stmt     ::= expr
+                      async_with_stmt      ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               suite_stmts
+                                               POP_TOP POP_BLOCK
+                                               async_with_post
+                      c_async_with_stmt    ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               c_suite_stmts
+                                               POP_TOP POP_BLOCK
+                                               async_with_post
+                      async_with_stmt      ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               suite_stmts
+                                               POP_BLOCK
+                                               BEGIN_FINALLY
+                                               WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                               WITH_CLEANUP_FINISH POP_FINALLY LOAD_CONST RETURN_VALUE
+                                               COME_FROM_ASYNC_WITH
+                                               WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                               WITH_CLEANUP_FINISH END_FINALLY
+                      c_async_with_stmt   ::= expr
                                               async_with_pre
                                               POP_TOP
-                                              suite_stmts
+                                              c_suite_stmts
-                                              POP_TOP POP_BLOCK
-                                              async_with_post
-                      async_with_stmt     ::= expr
-                                              async_with_pre
-                                              POP_TOP
-                                              suite_stmts
                                               POP_BLOCK
                                               BEGIN_FINALLY
                                               WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
@@ -252,15 +298,24 @@ class Python37BaseParser(PythonParser):
                                               COME_FROM_ASYNC_WITH
                                               WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
                                               WITH_CLEANUP_FINISH END_FINALLY
-                      async_with_as_stmt  ::= expr
+                      async_with_as_stmt   ::= expr
-                                              async_with_pre
+                                               async_with_pre
-                                              store suite_stmts
+                                               store suite_stmts
-                                              POP_TOP POP_BLOCK
+                                               POP_TOP POP_BLOCK
-                                              async_with_post
+                                               async_with_post
-                      async_with_as_stmt  ::= expr
+                      c_async_with_as_stmt ::= expr
-                                              async_with_pre
+                                               async_with_pre
-                                              store suite_stmts
+                                               store suite_stmts
-                                              POP_BLOCK async_with_post
+                                               POP_TOP POP_BLOCK
+                                               async_with_post
+                      async_with_as_stmt   ::= expr
+                                               async_with_pre
+                                               store suite_stmts
+                                               POP_BLOCK async_with_post
+                      c_async_with_as_stmt ::= expr
+                                               async_with_pre
+                                               store suite_stmts
+                                               POP_BLOCK async_with_post
                     """
                 self.addRule(rules_str, nop_func)
 
@@ -539,7 +594,7 @@ class Python37BaseParser(PythonParser):
 
                     stmt                ::= genexpr_func_async
 
-                    func_async_prefix   ::= SETUP_EXCEPT GET_ANEXT LOAD_CONST YIELD_FROM
+                    func_async_prefix   ::= _come_froms SETUP_EXCEPT GET_ANEXT LOAD_CONST YIELD_FROM
                     func_async_middle   ::= POP_BLOCK JUMP_FORWARD COME_FROM_EXCEPT
                                             DUP_TOP LOAD_GLOBAL COMPARE_OP POP_JUMP_IF_TRUE
                                             END_FINALLY COME_FROM
@@ -548,22 +603,24 @@ class Python37BaseParser(PythonParser):
                                             JUMP_BACK COME_FROM
                                             POP_TOP POP_TOP POP_TOP POP_EXCEPT POP_TOP
 
-                    expr                ::= listcomp_async
+                    expr                ::= list_comp_async
-                    listcomp_async      ::= LOAD_LISTCOMP LOAD_STR MAKE_FUNCTION_0
+                    list_comp_async     ::= LOAD_LISTCOMP LOAD_STR MAKE_FUNCTION_0
                                             expr GET_AITER CALL_FUNCTION_1
                                             GET_AWAITABLE LOAD_CONST
                                             YIELD_FROM
 
-                    expr                 ::= listcomp_async
+                    expr                ::= list_comp_async
-                    listcomp_async       ::= BUILD_LIST_0 LOAD_FAST func_async_prefix
+                    list_afor2          ::= func_async_prefix
                                             store func_async_middle list_iter
                                             JUMP_BACK COME_FROM
                                             POP_TOP POP_TOP POP_TOP POP_EXCEPT POP_TOP
-
+                    list_comp_async     ::= BUILD_LIST_0 LOAD_FAST list_afor2
+                    get_aiter           ::= expr GET_AITER
+                    list_afor           ::= get_aiter list_afor2
+                    list_iter           ::= list_afor
                    """,
                     nop_func,
                 )
-                custom_ops_processed.add(opname)
             elif opname == "JUMP_IF_NOT_DEBUG":
                 v = token.attr
                 self.addRule(
@@ -937,55 +994,70 @@ class Python37BaseParser(PythonParser):
                 )
                 custom_ops_processed.add(opname)
 
+            elif opname == "WITH_CLEANUP_START":
+                rules_str = """
+                  stmt        ::= with_null
+                  with_null   ::= with_suffix
+                  with_suffix ::= WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                """
+                self.addRule(rules_str, nop_func)
             elif opname == "SETUP_WITH":
                 rules_str = """
-                  stmt       ::= withstmt
+                  stmt       ::= with
                   stmt       ::= withasstmt
 
-                  withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
+                  with       ::= expr
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 SETUP_WITH POP_TOP
+                                 suite_stmts_opt
+                                 COME_FROM_WITH
+                                 with_suffix
                   withasstmt ::= expr SETUP_WITH store suite_stmts_opt COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 with_suffix
 
-                  withstmt   ::= expr
+                  with       ::= expr
-                                 SETUP_WITH POP_TOP suite_stmts_opt
+                                 SETUP_WITH POP_TOP
+                                 suite_stmts_opt
                                  POP_BLOCK LOAD_CONST COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 with_suffix
+
                   withasstmt ::= expr
                                  SETUP_WITH store suite_stmts_opt
                                  POP_BLOCK LOAD_CONST COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 with_suffix
 
-                  withstmt   ::= expr
+                  with       ::= expr
                                  SETUP_WITH POP_TOP suite_stmts_opt
                                  POP_BLOCK LOAD_CONST COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 with_suffix
                   withasstmt ::= expr
                                  SETUP_WITH store suite_stmts_opt
                                  POP_BLOCK LOAD_CONST COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 with_suffix
                 """
                 if self.version < 3.8:
                     rules_str += """
-                    withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                    with     ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                    LOAD_CONST
-                                   WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                   with_suffix
                     """
                 else:
                     rules_str += """
-                      withstmt   ::= expr
+                      with       ::= expr
                                      SETUP_WITH POP_TOP suite_stmts_opt
                                      POP_BLOCK LOAD_CONST COME_FROM_WITH
-                                     WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                     with_suffix
 
                       withasstmt ::= expr
                                      SETUP_WITH store suite_stmts_opt
                                      POP_BLOCK LOAD_CONST COME_FROM_WITH
 
-                       withstmt  ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                      withasstmt ::= expr
+                                     SETUP_WITH store suite_stmts
+                                     POP_BLOCK BEGIN_FINALLY COME_FROM_WITH with_suffix
+
+                      with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                      BEGIN_FINALLY COME_FROM_WITH
-                                     WITH_CLEANUP_START WITH_CLEANUP_FINISH
+                                     with_suffix
-                                     END_FINALLY
                     """
                 self.addRule(rules_str, nop_func)
 
@@ -1091,8 +1163,8 @@ class Python37BaseParser(PythonParser):
                 + token.kind
             )
 
-            # Note: semantic actions make use of the fact of wheter  "args_pos"
+            # Note: Semantic actions make use of whether or not "args_pos"
-            # zero or not in creating a template rule.
+            # is zero when creating a template rule.
             self.add_unique_rule(rule, token.kind, args_pos, customize)
         else:
             token.kind = self.call_fn_name(token)
@@ -1126,10 +1198,28 @@ class Python37BaseParser(PythonParser):
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
         lhs = rule[0]
         n = len(tokens)
-        last = min(last, n-1)
+        last = min(last, n - 1)
         fn = self.reduce_check_table.get(lhs, None)
-        if fn:
+        try:
-            return fn(self, lhs, n, rule, ast, tokens, first, last)
+            if fn:
+                return fn(self, lhs, n, rule, ast, tokens, first, last)
+        except:
+            import sys, traceback
+
+            print(
+                ("Exception in %s %s\n"
+                 + "rule: %s\n"
+                 + "offsets %s .. %s")
+                % (
+                    fn.__name__,
+                    sys.exc_info()[1],
+                    rule2str(rule),
+                    tokens[first].offset,
+                    tokens[last].offset,
+                )
+            )
+            print(traceback.print_tb(sys.exc_info()[2], -1))
+            raise ParserError(tokens[last], tokens[last].off2int(), self.debug["rules"])
 
         if lhs in ("aug_assign1", "aug_assign2") and ast[0][0] == "and":
             return True

uncompyle6/parsers/parse38.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2017-2019 Rocky Bernstein
+#  Copyright (c) 2017-2020 Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -39,14 +39,17 @@ class Python38Parser(Python37Parser):
         stmt               ::= forelselaststmtl38
         stmt               ::= tryfinally38stmt
         stmt               ::= tryfinally38rstmt
+        stmt               ::= tryfinally38rstmt2
+        stmt               ::= tryfinally38rstmt3
         stmt               ::= tryfinally38astmt
         stmt               ::= try_elsestmtl38
         stmt               ::= try_except_ret38
         stmt               ::= try_except38
         stmt               ::= whilestmt38
         stmt               ::= whileTruestmt38
-        stmt               ::= call
+        stmt               ::= call_stmt
 
+        call_stmt          ::= call
         break ::= POP_BLOCK BREAK_LOOP
         break ::= POP_BLOCK POP_TOP BREAK_LOOP
         break ::= POP_TOP BREAK_LOOP
@@ -54,15 +57,12 @@ class Python38Parser(Python37Parser):
 
         # FIXME: this should be restricted to being inside a try block
         stmt               ::= except_ret38
+        stmt               ::= except_ret38a
 
         # FIXME: this should be added only when seeing GET_AITER or YIELD_FROM
-        async_for_stmt38   ::= expr
+        async_for          ::= GET_AITER _come_froms
-                               GET_AITER
+                               SETUP_FINALLY GET_ANEXT LOAD_CONST YIELD_FROM POP_BLOCK
-                               SETUP_FINALLY
+        async_for_stmt38   ::= expr async_for
-                               GET_ANEXT
-                               LOAD_CONST
-                               YIELD_FROM
-                               POP_BLOCK
                                store for_block
                                COME_FROM_FINALLY
                                END_ASYNC_FOR
@@ -81,7 +81,20 @@ class Python38Parser(Python37Parser):
                                END_ASYNC_FOR
                                else_suite
 
-        return             ::= ret_expr ROT_TWO POP_TOP RETURN_VALUE
+        # Seems to be used to discard values before a return in a "for" loop
+        discard_top        ::= ROT_TWO POP_TOP
+        discard_tops       ::= discard_top+
+
+        return             ::= ret_expr
+                               discard_tops RETURN_VALUE
+
+        return             ::= popb_return
+        return             ::= pop_return
+        return             ::= pop_ex_return
+        except_stmt        ::= pop_ex_return
+        pop_return         ::= POP_TOP ret_expr RETURN_VALUE
+        popb_return        ::= ret_expr POP_BLOCK RETURN_VALUE
+        pop_ex_return      ::= ret_expr ROT_FOUR POP_EXCEPT RETURN_VALUE
 
         # 3.8 can push a looping JUMP_BACK into into a JUMP_ from a statement that jumps to it
         lastl_stmt         ::= ifpoplaststmtl
@@ -128,8 +141,14 @@ class Python38Parser(Python37Parser):
                                except_handler38
         try_except38       ::= SETUP_FINALLY POP_BLOCK POP_TOP suite_stmts_opt
                                except_handler38a
-        try_except_ret38   ::= SETUP_FINALLY expr POP_BLOCK
+
-                               RETURN_VALUE except_ret38a
+        # suite_stmts has a return
+        try_except38       ::= SETUP_FINALLY POP_BLOCK suite_stmts
+                               except_handler38b
+
+        try_except_ret38   ::= SETUP_FINALLY returns except_ret38a
+        try_except_ret38a  ::= SETUP_FINALLY returns except_handler38c
+                               END_FINALLY come_from_opt
 
         # Note: there is a suite_stmts_opt which seems
         # to be bookkeeping which is not expressed in source code
@@ -149,19 +168,42 @@ class Python38Parser(Python37Parser):
         tryfinallystmt     ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                BEGIN_FINALLY COME_FROM_FINALLY suite_stmts_opt
                                END_FINALLY
-        tryfinally38rstmt  ::= SETUP_FINALLY POP_BLOCK CALL_FINALLY
+
+
+        lc_setup_finally   ::= LOAD_CONST SETUP_FINALLY
+        call_finally_pt    ::= CALL_FINALLY POP_TOP
+        cf_cf_finally      ::= come_from_opt COME_FROM_FINALLY
+        pop_finally_pt     ::= POP_FINALLY POP_TOP
+        ss_end_finally     ::= suite_stmts END_FINALLY
+        sf_pb_call_returns ::= SETUP_FINALLY POP_BLOCK CALL_FINALLY returns
+
+
+        # FIXME: DRY rules below
+        tryfinally38rstmt  ::= sf_pb_call_returns
+                               cf_cf_finally
+                               ss_end_finally
+        tryfinally38rstmt  ::= sf_pb_call_returns
+                               cf_cf_finally END_FINALLY
+                               suite_stmts
+        tryfinally38rstmt  ::= sf_pb_call_returns
+                               cf_cf_finally POP_FINALLY
+                               ss_end_finally
+        tryfinally38rstmt  ::= sf_bp_call_returns
+                               COME_FROM_FINALLY POP_FINALLY
+                               ss_end_finally
+
+        tryfinally38rstmt2 ::= lc_setup_finally POP_BLOCK call_finally_pt
                                returns
-                               COME_FROM_FINALLY END_FINALLY suite_stmts
+                               cf_cf_finally pop_finally_pt
-        tryfinally38rstmt  ::= SETUP_FINALLY POP_BLOCK CALL_FINALLY
+                               ss_end_finally POP_TOP
-                               returns
+        tryfinally38rstmt3 ::= SETUP_FINALLY expr POP_BLOCK CALL_FINALLY RETURN_VALUE
-                               COME_FROM_FINALLY POP_FINALLY returns
+                               COME_FROM COME_FROM_FINALLY
-                               END_FINALLY
+                               ss_end_finally
-        tryfinally38stmt   ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
+
-                               BEGIN_FINALLY COME_FROM_FINALLY
-                               POP_FINALLY suite_stmts_opt END_FINALLY
         tryfinally38stmt   ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                BEGIN_FINALLY COME_FROM_FINALLY
                                POP_FINALLY suite_stmts_opt END_FINALLY
+
         tryfinally38astmt  ::= LOAD_CONST SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                BEGIN_FINALLY COME_FROM_FINALLY
                                POP_FINALLY POP_TOP suite_stmts_opt END_FINALLY POP_TOP