Some typos

correct "return None" semantic action

.circleci/config.yml

@@ -36,7 +36,6 @@ jobs:
           # fallback to using the latest cache if no exact match is found
           - v2-dependencies-
 
-    # This is based on your 1.0 configuration file or project settings
     - run: pip install --user --upgrade setuptools
     - run: pip install --user -e .
     - run: pip install --user -r requirements-dev.txt

.github/FUNDING.yml

@@ -0,0 +1,12 @@
+# These are supported funding model platforms
+
+github: [rocky]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

.gitignore

@@ -10,6 +10,7 @@
 /.pytest_cache
 /.python-version
 /.tox
+.mypy_cache
 /.venv*
 /README
 /__pkginfo__.pyc

.travis.yml

@@ -1,9 +1,9 @@
 language: python
 
 python:
-  - '3.5'
-  - '2.7'
-  - '3.4'
+#  - '3.5'
+#  - '2.7'
+#  - '3.4'
   - '3.6'
   - '3.8'
 

NEWS.md

@@ -1,3 +1,41 @@
+3.7.0: 2020-5-19 Primidi 1st Prairial - Alfalfa - HF
+====================================================
+
+The main impetus for this release is to pull in the recent changes from xdis.
+We simplify imports using xdis 4.6.0.
+
+There were some bugfixes to Python 3.4-3.8. See the ChangeLog for details
+
+
+3.6.7: 2020-4-27 xdis again
+===========================
+
+More upheaval in xdis which we need to track here.
+
+3.6.6: 2020-4-20 Love in the time of Cholera
+============================================
+
+The main reason for this release is an incompatablity bump in xdis which handles
+3.7 SipHash better.
+
+* Go over "yield" as an expression precidence
+* Some small alignment with code in decompyle3 for "or" and "and" was done
+
+
+3.6.5: 2020-4-1 April Fool
+==========================
+
+Back port some of the changes in decompile3 here which mostly helps 3.7 and 3.8 decompilation, although this may also help 3.6ish versions too.
+
+- Handle nested `async for in for...`  and better async comprehension detection via `xdis`.  Still more work is needed.
+- include token number in listings when `-g` and there is a parser error
+- remove unneeded `Makefile`s now that remake 4.3+1.5dbg is a thing that has `-c`
+- Bug in finding annotations in functions with docstrings
+- Fix bug found by 2.4 sre_parse.py testing
+- Fix `transform` module's  `ifelseif` bugs
+- Fix bug in 3.0 name module detection
+- Fix docstring detection
+
 3.6.4: 2020-2-9 Plateau
 =======================
 

README.rst

@@ -211,7 +211,7 @@ however that the magic of a released version is usually the same as
 the *last* candidate version prior to release.
 
 There are also customized Python interpreters, notably Dropbox,
-which use their own magic and encrypt bytcode. With the exception of
+which use their own magic and encrypt bytecode. With the exception of
 the Dropbox's old Python 2.5 interpreter this kind of thing is not
 handled.
 
@@ -230,7 +230,7 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back ehre.
+* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back here.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
 * https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details

__pkginfo__.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2018 Rocky Bernstein <rocky@gnu.org>
+# Copyright (C) 2018, 2020 Rocky Bernstein <rocky@gnu.org>
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -21,9 +21,20 @@
 # less elegant than having it here with reduced code, albeit there
 # still is some room for improvement.
 
+# Python-version | package | last-version |
+# -----------------------------------------
+# 2.5            | pip     |  1.1         |
+# 2.6            | pip     |  1.5.6       |
+# 2.7            | pip     | 19.2.3       |
+# 2.7            | pip     |  1.2.1       |
+# 3.1            | pip     |  1.5.6       |
+# 3.2            | pip     |  7.1.2       |
+# 3.3            | pip     | 10.0.1       |
+# 3.4            | pip     | 19.1.1       |
+
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015-2019 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015-2020 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
 classifiers =  ["Development Status :: 5 - Production/Stable",
@@ -58,7 +69,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ["spark-parser >= 1.8.9, < 1.9.0",
-                      "xdis >= 4.2.2, < 4.3.0"]
+                      "xdis >= 4.6.0, < 4.7.0"]
 
 license            = "GPL3"
 mailing_list       = "python-debugger@googlegroups.com"

admin-tools/how-to-make-a-release.md

@@ -55,7 +55,7 @@
 # Make packages and tag
 
     $ . ./admin-tools/make-dist-older.sh
-    $ git tag release-python-2.4-$VERSION
+	$ pyenv local 3.8.3
 	$ twine check dist/uncompyle6-$VERSION*
     $ . ./admin-tools/make-dist-newer.sh
 	$ twine check dist/uncompyle6-$VERSION*

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.17 3.2.6 3.1.5 3.4.10 3.7.6 3.8.1'
+export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.18 3.2.6 3.1.5 3.4.10 3.7.7 3.8.3'

admin-tools/setup-master.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-PYTHON_VERSION=3.7.6
+PYTHON_VERSION=3.7.7
 
 # FIXME put some of the below in a common routine
 function finish {

appveyor.yml

@@ -53,6 +53,7 @@ install:
   # compiled extensions and are not provided as pre-built wheel packages,
   # pip will build them from source using the MSVC compiler matching the
   # target Python version and architecture
+  - "%CMD_IN_ENV% pip install git+git://github.com/rocky/python-uncompyle6.git#egg=uncompyle6-3.6.6"
   - "%CMD_IN_ENV% pip install -r requirements.txt"
 
 build_script:

pytest/test_token.py

@@ -1,14 +1,13 @@
-from uncompyle6 import PYTHON_VERSION
 from uncompyle6.scanners.tok import Token
 
 
 def test_token():
     # Test token formatting of: LOAD_CONST None
     t = Token("LOAD_CONST", offset=0, attr=None, pattr=None, has_arg=True)
-    expect = "             0  LOAD_CONST               None"
+    expect = "0  LOAD_CONST               None"
     # print(t.format())
     assert t
-    assert t.format() == expect
+    assert t.format().strip() == expect.strip()
 
     # Make sure equality testing of tokens ignores offset
     t2 = Token("LOAD_CONST", offset=2, attr=None, pattr=None, has_arg=True)
@@ -17,8 +16,8 @@ def test_token():
     # Make sure formatting of: LOAD_CONST False. We assume False is the 0th index
     # of co_consts.
     t = Token("LOAD_CONST", offset=1, attr=False, pattr=False, has_arg=True)
-    expect = "             1  LOAD_CONST               False"
-    assert t.format() == expect
+    expect = "1  LOAD_CONST               False"
+    assert t.format().strip() == expect.strip()
 
 
 if __name__ == "__main__":

pytest/testdata/if-2.7.right

@@ -1,12 +1,12 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_if.py
 
-   6         0  LOAD_NAME             0  'True'
-             3  POP_JUMP_IF_FALSE    15  'to 15'
+ L.   6         0  LOAD_NAME             0  'True'
+                3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   7         6  LOAD_NAME             1  'False'
-             9  STORE_NAME            2  'b'
-            12  JUMP_FORWARD          0  'to 15'
-          15_0  COME_FROM            12  '12'
-            15  LOAD_CONST               None
-            18  RETURN_VALUE     
+ L.   7         6  LOAD_NAME             1  'False'
+                9  STORE_NAME            2  'b'
+               12  JUMP_FORWARD          0  'to 15'
+             15_0  COME_FROM            12  '12'
+               15  LOAD_CONST               None
+               18  RETURN_VALUE     

pytest/testdata/ifelse-2.7.right

@@ -1,15 +1,15 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_ifelse.py
 
-   3         0  LOAD_NAME             0  'True'
-             3  POP_JUMP_IF_FALSE    15  'to 15'
+ L.   3         0  LOAD_NAME             0  'True'
+                3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   4         6  LOAD_CONST               1
-             9  STORE_NAME            1  'b'
-            12  JUMP_FORWARD          6  'to 21'
+ L.   4         6  LOAD_CONST               1
+                9  STORE_NAME            1  'b'
+               12  JUMP_FORWARD          6  'to 21'
 
-   6        15  LOAD_CONST               2
-            18  STORE_NAME            2  'd'
-          21_0  COME_FROM            12  '12'
-            21  LOAD_CONST               None
-            24  RETURN_VALUE     
+ L.   6        15  LOAD_CONST               2
+               18  STORE_NAME            2  'd'
+             21_0  COME_FROM            12  '12'
+               21  LOAD_CONST               None
+               24  RETURN_VALUE     

pytest/validate.py

@@ -12,8 +12,7 @@ import functools
 from uncompyle6 import PYTHON_VERSION, PYTHON3, IS_PYPY, code_deparse
 
 # TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
-from xdis.bytecode import Bytecode
-from xdis.main import get_opcode
+from xdis import Bytecode, get_opcode
 
 opc = get_opcode(PYTHON_VERSION, IS_PYPY)
 Bytecode = functools.partial(Bytecode, opc=opc)

setup.py

@@ -4,40 +4,54 @@ import sys
 """Setup script for the 'uncompyle6' distribution."""
 
 SYS_VERSION = sys.version_info[0:2]
-if not ((2, 6) <= SYS_VERSION  <= (3, 9)):
+if not ((2, 6) <= SYS_VERSION <= (3, 9)):
     mess = "Python Release 2.6 .. 3.9 are supported in this code branch."
-    if ((2, 4) <= SYS_VERSION <= (2, 7)):
-        mess += ("\nFor your Python, version %s, use the python-2.4 code/branch." %
-                 sys.version[0:3])
+    if (2, 4) <= SYS_VERSION <= (2, 7):
+        mess += (
+            "\nFor your Python, version %s, use the python-2.4 code/branch."
+            % sys.version[0:3]
+        )
     elif SYS_VERSION < (2, 4):
-        mess += ("\nThis package is not supported for Python version %s."
-                 % sys.version[0:3])
+        mess += (
+            "\nThis package is not supported for Python version %s." % sys.version[0:3]
+        )
     print(mess)
     raise Exception(mess)
 
-from __pkginfo__ import \
-    author,           author_email,       install_requires,          \
-    license,          long_description,   classifiers,               \
-    entry_points,     modname,            py_modules,                \
-    short_desc,       VERSION,            web,                       \
-    zip_safe
+from __pkginfo__ import (
+    author,
+    author_email,
+    install_requires,
+    license,
+    long_description,
+    classifiers,
+    entry_points,
+    modname,
+    py_modules,
+    short_desc,
+    VERSION,
+    web,
+    zip_safe,
+)
 
 from setuptools import setup, find_packages
+
 setup(
-       author             = author,
-       author_email       = author_email,
-       classifiers        = classifiers,
-       description        = short_desc,
-       entry_points       = entry_points,
-       install_requires   = install_requires,
-       license            = license,
-       long_description   = long_description,
-       long_description_content_type = "text/x-rst",
-       name               = modname,
-       packages           = find_packages(),
-       py_modules         = py_modules,
-       test_suite         = 'nose.collector',
-       url                = web,
-       tests_require      = ['nose>=1.0'],
-       version            = VERSION,
-       zip_safe           = zip_safe)
+    author=author,
+    author_email=author_email,
+    classifiers=classifiers,
+    description=short_desc,
+    entry_points=entry_points,
+    install_requires=install_requires,
+    license=license,
+    long_description=long_description,
+    long_description_content_type="text/x-rst",
+    name=modname,
+    packages=find_packages(),
+    py_modules=py_modules,
+    test_suite="nose.collector",
+    url=web,
+    tests_require=["nose>=1.0"],
+    version=VERSION,
+    zip_safe=zip_safe,
+)

test/.gitignore

@@ -1,3 +1,8 @@
 /.coverage
 /.python-version
 /nohup.out
+/pycdc
+/test_pycdc_tests.sh
+/test_uncompyle2.py
+/test_unpy33.py
+/test_unpy37.py

test/Makefile

@@ -78,7 +78,7 @@ check-3.8: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.8 --syntax-verify $(COMPILE)
 
 # FIXME
-#: this is called when running under pypy3.5-5.8.0 or pypy2-5.6.0
+#: this is called when running under pypy3.5-5.8.0, pypy2-5.6.0, or pypy3.6-7.3.0
 5.8 5.6:
 
 #: Check deparsing only, but from a different Python version
@@ -95,12 +95,19 @@ check-bytecode-2:
         --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
 
 #: Check deparsing bytecode 3.x only
+# check-bytecode-3:
+# 	$(PYTHON) test_pythonlib.py \
+#         --bytecode-3.0 \
+# 	--bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
+# 	--bytecode-3.4 --bytecode-3.5 --bytecode-3.6 \
+#         --bytecode-3.7  \
+# 	--bytecode-pypy3.2 --bytecode-pypy3.6 --bytecode-3.8
+
+# FIXME: Until we shaked out problems with xdis...
 check-bytecode-3:
-	$(PYTHON) test_pythonlib.py --bytecode-3.0 \
-	--bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
+	$(PYTHON) test_pythonlib.py \
 	--bytecode-3.4 --bytecode-3.5 --bytecode-3.6 \
-        --bytecode-3.7  \
-	--bytecode-pypy3.2 --bytecode-pypy3.6 --bytecode-3.8
+        --bytecode-3.7  --bytecode-3.8
 
 #: Check deparsing on selected bytecode 3.x
 check-bytecode-3-short:
@@ -339,8 +346,8 @@ check-bytecode-pypy3.6: 7.1
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
 
 #: PyPy 5.0.x with Python 3.6.9
-check-bytecode-pypy3.6: 7.2
-7.2:
+check-bytecode-pypy3.6: 7.2 7.3
+7.3 7.2:
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6-run --verify-run
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
 

test/run-and-email.sh

@@ -3,32 +3,55 @@
 function displaytime {
     printf "ran in "
     local T=$1
-    local D=$((T/60/60/24))
-    local H=$((T/60/60%24))
-    local M=$((T/60%60))
-    local S=$((T%60))
-    (( $D > 0 )) && printf '%d days ' $D
-    (( $H > 0 )) && printf '%d hours ' $H
-    (( $M > 0 )) && printf '%d minutes ' $M
-    (( $D > 0 || $H > 0 || $M > 0 )) && printf 'and '
+    ((D=T/60/60/24))
+    ((H=T/60/60%24))
+    ((M=T/60%60))
+    ((S=T%60))
+    (( D > 0 )) && printf '%d days ' $D
+    (( H > 0 )) && printf '%d hours ' $H
+    (( M > 0 )) && printf '%d minutes ' $M
+    (( D > 0 || H > 0 || M > 0 )) && printf 'and '
     printf '%d seconds\n' $S
 }
 
-. ../admin-tools/pyenv-newer-versions
+bs=${BASH_SOURCE[0]}
+if [[ $0 != $bs ]] ; then
+    echo "This script should not be *sourced* but run through bash"
+    exit 1
+fi
+
+mydir=$(dirname $bs)
+cd $mydir
+
+branch=$(cat ../.git/HEAD | cut -d'/' -f 3)
+if [[ $branch == 'python-2.4' ]]; then
+    . ../admin-tools/pyenv-older-versions
+elif [[ $branch == 'master' ]]; then
+    . ../admin-tools/pyenv-newer-versions
+else
+    echo &1>2 "Error git branch should either be 'master' or 'python-2.4'; got: '$branch'"
+    exit 1
+fi
+
+MAIN="test_pyenvlib.py"
 
 USER=${USER:-rocky}
 EMAIL=${EMAIL:-rb@dustyfeet.com}
+WHAT="uncompyle6 ${MAIN}"
 MAX_TESTS=${MAX_TESTS:-800}
+export BATCH=1
+
 typeset -i RUN_STARTTIME=$(date +%s)
 
 # PYVERSIONS="3.5.6"
-actual_versions=""
+MAILBODY=/tmp/${MAIN}-mailbody-$$.txt
+# for VERSION in 3.3.7 ; do
 for VERSION in $PYVERSIONS ; do
     typeset -i rc=0
-    LOGFILE=/tmp/pyenvlib-$VERSION-$$.log
+    LOGFILE=/tmp/${MAIN}-$VERSION-$$.log
 
     case "$VERSION" in
-	3.7.6 | 3.8.1 | 3.1.5 | 3.0.1 )
+	3.7.7 | 3.8.2 | 3.1.5 | 3.0.1 )
 	    continue
 	    ;;
 	3.5.9 )
@@ -44,7 +67,15 @@ for VERSION in $PYVERSIONS ; do
 	    MAX_TESTS=800
 	    ;;
 	3.6.10 )
-	    MAX_TESTS=1300  # about 2139 exist
+	    # MAX_TESTS=1300  # about 2139 exist
+	    # fails on _pyio.cpython-36.opt-1.pyc
+	    MAX_TESTS=34
+	    ;;
+	2.4.6 )
+	    MAX_TESTS=600
+	    ;;
+	2.5.6 )
+	    MAX_TESTS=600
 	    ;;
 	2.6.9 )
 	    MAX_TESTS=1300
@@ -58,11 +89,13 @@ for VERSION in $PYVERSIONS ; do
 
     if ! pyenv local $VERSION ; then
 	rc=1
+	mailbody_line="pyenv local $VERSION not installed"
+	echo $mailbody_line >> $MAILBODY
     else
       echo Python Version $(pyenv local) > $LOGFILE
       echo "" >> $LOGFILE
       typeset -i ALL_FILES_STARTTIME=$(date +%s)
-      cmd="python ./test_pyenvlib.py --max ${MAX_TESTS} --syntax-verify --$VERSION"
+      cmd="python ./${MAIN} --max ${MAX_TESTS} --syntax-verify --$VERSION"
       echo "$cmd" >>$LOGFILE 2>&1
       $cmd >>$LOGFILE 2>&1
       rc=$?
@@ -72,20 +105,25 @@ for VERSION in $PYVERSIONS ; do
 
       typeset -i ALL_FILES_ENDTIME=$(date +%s)
       (( time_diff =  ALL_FILES_ENDTIME - ALL_FILES_STARTTIME))
-      displaytime $time_diff >> $LOGFILE
+      time_str=$(displaytime $time_diff)
+      echo ${time_str}. >> $LOGFILE
     fi
 
-    SUBJECT_PREFIX="pyenv weak verify (max $MAX_TESTS) for"
+    SUBJECT_PREFIX="$WHAT (max $MAX_TESTS) for"
     if ((rc == 0)); then
+	mailbody_line="Python $VERSION ok; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION ok" ${USER}@localhost
     else
+	mailbody_line="Python $VERSION failed; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${USER}@localhost
-	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${EMAIL}
+	tail -v $LOGFILE | mail -s "$HOST $SUBJECT_PREFIX $VERSION not ok" ${EMAIL}
     fi
+    echo $mailbody_line >> $MAILBODY
     rm .python-version
 done
 
 typeset -i RUN_ENDTIME=$(date +%s)
 (( time_diff =  RUN_ENDTIME - RUN_STARTTIME))
 elapsed_time=$(displaytime $time_diff)
-echo "Run complete $elapsed_time for versions $actual_versions" | mail -s "pyenv weak verify in $elapsed_time" ${EMAIL}
+echo "${WHAT} complete; ${elapsed_time}." >> $MAILBODY
+cat $MAILBODY | mail -s "$HOST $WHAT ${elapsed_time}." ${EMAIL}

test/simple-decompile-code-test.py

@@ -3,7 +3,7 @@
 from __future__ import print_function
 
 from uncompyle6.main import decompile
-from xdis.magics import sysinfo2float
+from xdis import sysinfo2float
 import sys, inspect
 
 def uncompyle_test():

test/simple_source/bug31/04_def_annotate.py

@@ -66,6 +66,14 @@ def div(a: dict(type=float, help='the dividend'),
     """Divide a by b"""
     return a / b
 
+# From 3.7.6 functools.py
+# Bug is in picking up the annotation.
+def f(a:"This is a new annotation"):
+    """This is a test"""
+    assert f.__annotations__['a'] == "This is a new annotation"
+
+f(5)
+
 class TestSignatureObject1():
     def test_signature_on_wkwonly(self):
         def test(*, a:float, b:str, c:str = 'test', **kwargs: int) -> int:

test/simple_source/bug33/02_pos_args.py

@@ -1,6 +1,8 @@
+# Python 3.3+
+#
 # From Python 3.3.6 hmac.py
 # Problem was getting wrong placement of positional args.
-# In 3.6+ paramter handling changes
+# In 3.6+ parameter handling changes
 
 # RUNNABLE!
 

test/simple_source/bug36/02_var_annotate.py

@@ -1,6 +1,9 @@
 # 3.6+ type annotations on variables
 from typing import List
 
+# This test program is part of the uncompyle6 test suite
+# tests STORE_ANNOTATION and SETUP_ANOTATIONS
+
 # RUNNABLE!
 y = 2
 x: bool

test/simple_source/bug37/02_async_for_generator.py

@@ -6,3 +6,11 @@ def make_arange(n):
 
 async def run(m):
     return [i async for i in m]
+
+# From 3.7.6 test_coroutines.py
+async def run_list(pair, f):
+    return [i for pair in p async for i in f]
+
+# FIXME: add this. It works in decompyle3
+# async def run_gen():
+#     return (i async for i in f if 0 < i < 4)

test/simple_source/bug37/04_async.py

@@ -1,23 +0,0 @@
-# from 3.7 test_contextlib_async.py
-# Bugs were not adding "async" when a function is a decorator,
-# and a misaligment when using "async with as".
-@_async_test
-async def test_enter(self):
-    self.assertIs(await manager.__aenter__(), manager)
-
-    async with manager as context:
-        async with woohoo() as x:
-            x = 1
-            y = 2
-        assert manager is context
-
-# From 3.7.6 test_coroutines.py
-# Bug was different form of code for "async with" below
-class CoroutineTest():
-    def test_with_8(self):
-        CNT = 0
-        async def foo():
-            nonlocal CNT
-            async with CM():
-                CNT += 1
-                return

test/simple_source/bug37/04_grammar.py

@@ -0,0 +1,87 @@
+# From test_grammar.py
+# RUNNABLE!
+def check_syntax_error(statement):
+    try:
+        compile(statement, '<test string>', 'exec')
+    except SyntaxError:
+        return
+    assert False
+
+def test_yield():
+    # Requires parentheses as call argument
+    def g():
+        f((yield 1), 1)
+    def g():
+        f((yield from ()))
+    def g():
+        f((yield from ()), 1)
+    def g():
+        f((yield 1))
+
+    # Allowed as standalone statement
+    def g():
+        yield 1
+
+    def g():
+        yield from ()
+
+    # Allowed as RHS of assignment
+    def g():
+        x = yield 1
+
+    def g():
+        x = yield from ()
+
+    # Ordinary yield accepts implicit tuples
+    def g():
+        yield 1, 1
+
+    def g():
+        x = yield 1, 1
+
+    # 'yield from' does not
+    check_syntax_error("def g(): yield from (), 1")
+    check_syntax_error("def g(): x = yield from (), 1")
+    # Requires parentheses as subexpression
+    def g():
+        1, (yield 1)
+
+    def g():
+        1, (yield from ())
+
+    check_syntax_error("def g(): 1, yield 1")
+    check_syntax_error("def g(): 1, yield from ()")
+    # Requires parentheses as call argument
+    def g():
+        f((yield 1))
+
+    def g():
+        f((yield 1), 1)
+
+    def g():
+        f((yield from ()))
+
+    def g():
+        f((yield from ()), 1)
+
+    check_syntax_error("def g(): f(yield 1)")
+    check_syntax_error("def g(): f(yield 1, 1)")
+    check_syntax_error("def g(): f(yield from ())")
+    check_syntax_error("def g(): f(yield from (), 1)")
+    # Not allowed at top level
+    check_syntax_error("yield")
+    check_syntax_error("yield from")
+    # Not allowed at class scope
+    check_syntax_error("class foo:yield 1")
+    check_syntax_error("class foo:yield from ()")
+    # Check annotation refleak on SyntaxError
+    check_syntax_error("def g(a:(yield)): pass")
+
+test_yield()
+
+# From test_types.py
+# Bug was needing parens around (yield 2)
+def gen_func():
+    yield 1
+    return (yield 2)
+gen = gen_func()

test/simple_source/bug37/10_async.py

@@ -0,0 +1,70 @@
+# from 3.7 test_contextlib_async.py
+# Bugs were not adding "async" when a function is a decorator,
+# and a misaligment when using "async with ... as".
+
+"""This program is self-checking!"""
+import asyncio
+from contextlib import asynccontextmanager, AbstractAsyncContextManager
+import functools
+
+
+def _async_test(func):
+    """Decorator to turn an async function into a test case."""
+
+    @functools.wraps(func)
+    def wrapper(*args, **kwargs):
+        coro = func(*args, **kwargs)
+        loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(loop)
+        try:
+            return loop.run_until_complete(coro)
+        finally:
+            loop.close()
+            asyncio.set_event_loop(None)
+
+    return wrapper
+
+
+state = []
+
+
+@asynccontextmanager
+async def woohoo():
+    state.append(1)
+    yield 42
+    state.append(999)
+
+
+@_async_test
+async def test_enter():
+    class DefaultEnter(AbstractAsyncContextManager):
+        async def __aexit__(*args):
+            return
+            # await super().__aexit__(*args)
+
+    manager = DefaultEnter()
+    got_manager = await manager.__aenter__()
+    # print(got_manager, manager)
+    assert got_manager is manager
+
+    async with manager as context:
+        async with woohoo() as x:
+            x = 1
+            y = 2
+        assert manager is context
+
+
+# From 3.7.6 test_coroutines.py
+# Bug was different form of code for "async with" below
+class CoroutineTest:
+    def test_with_8(self):
+        CNT = 0
+
+        async def foo():
+            nonlocal CNT
+            async with CM():
+                CNT += 1
+                return
+
+
+test_enter()

test/simple_source/expression/09_yield_from.py

@@ -17,9 +17,9 @@ def ybug(g):
 # From 3.5.1 _wakrefset.py
 #
 #  3.5:
-#  withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt
-#              POP_BLOCK LOAD_CONST COME_FROM
-#              WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+#  with ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+#           POP_BLOCK LOAD_CONST COME_FROM
+#           WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
 
 def __iter__(self, IterationGuard):

test/simple_source/looping/08_while1_if_continue.py

@@ -7,3 +7,40 @@ def readline (self):
                 continue
 
         return
+
+# From 2.4.6 sre.py
+# Bug has to do with "break" not being recognized
+# and is a JUMP_FORWARD.
+def _parse(a, b, source, state):
+    while 1:
+        if b:
+            while 1:
+                break
+        else:
+            raise
+
+def _parse2(source, state):
+    while 1:
+        if a:
+            if b:
+                while 1:
+                    this = 1
+                    break
+                continue
+
+        while 1:
+            if b:
+                break
+
+        x = 3
+
+# Bug was in 2.3 decompilation
+def _parse3(source, state):
+    while 1:
+        if a:
+            if b:
+                x = 1
+                while 1:
+                    if a:
+                        break
+                    raise

test/simple_source/stmts/03_if_elif.py

@@ -1,11 +1,13 @@
 # 2.6.9 symbols.py
 # Bug in 2.6 is having multple COME_FROMs due to the
 # "and" in the "if" clause
+
+# RUNNABLE
 if __name__:
     if __file__ and __name__:
         pass
-    elif __name__:
-        pass
+    elif not __name__:
+        assert False
 
 # 2.6.9 transformer.py
 # Bug in 2.6 is multple COME_FROMs as a result
@@ -21,3 +23,20 @@ elif __file__:
     assert __name__ or __file__
 else:
     pass
+
+# From 3.3.7 test_binop.py
+# Bug was in ifelsestmt(c) ensuring b+=5 is not in "else"
+# Also note: ifelsetmtc should not have been used since this
+# this is not in a loop!
+def __floordiv__(a, b):
+    if a:
+        b += 1
+    elif not b:
+        return a
+    b += 5
+    return b
+
+assert __floordiv__(1, 1) == 7
+assert __floordiv__(1, 0) == 6
+assert __floordiv__(0, 3) == 8
+assert __floordiv__(0, 0) == 0

test/stdlib/.gitignore

@@ -1 +1,3 @@
 /.python-version
+/runun33.sh
+/runun7.sh

test/stdlib/2.5-exclude.sh

@@ -81,6 +81,6 @@ SKIP_TESTS=(
 )
 # About 265 tests in 14 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_doctest.py]=1 # Fails on ppc64le
 fi

test/stdlib/2.6-exclude.sh

@@ -1,4 +1,12 @@
 SKIP_TESTS=(
+    # ifelsestmt is borked in:
+    #   if filename == 'srcfile':
+    #      return srcfile
+    #   if filename == 'destfile':
+    #      return destfile
+    #   assert 0  # shouldn't reach here.
+    [test_shutil.py]=1
+
 
     [test___all__.py]=1  # it fails on its own
     [test___all__.py]=1 # it fails on its own
@@ -59,6 +67,7 @@ SKIP_TESTS=(
 
     [test_scriptpackages.py]=1 # it fails on its own
     [test_select.py]=1 # test takes too long to run: 11 seconds
+
     [test_socket.py]=1 # test takes too long to run: 12 seconds
     [test_startfile.py]=1 # it fails on its own
     [test_structmembers.py]=1 # it fails on its own
@@ -85,7 +94,7 @@ SKIP_TESTS=(
 )
 # About 305 unit-test files in about 12 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     # Fails in crontab environment?
     # Figure out what's up here
     SKIP_TESTS[test_aifc.py]=1

test/stdlib/2.7-exclude.sh

@@ -1,4 +1,6 @@
 SKIP_TESTS=(
+    [test_cgi.py]=1 # FIXME: Works on c90ff51
+
     [test_bsddb3.py]=1 # test takes too long to run: 110 seconds
     [test_compile.py]=1  # Code introspects on co_consts in a non-decompilable way
     [test_curses.py]=1  # Possibly fails on its own but not detected
@@ -25,6 +27,7 @@ SKIP_TESTS=(
     [test_ssl.py]=1  #
     [test_subprocess.py]=1 # Runs ok but takes 22 seconds
     [test_sys_settrace.py]=1 # Line numbers are expected to be different
+
     [test_tokenize.py]=1 # test takes too long to run: 19 seconds
     [test_traceback.py]=1 # Line numbers change - duh.
     [test_unicode.py]=1  # Too long to run 11 seconds
@@ -34,7 +37,7 @@ SKIP_TESTS=(
 )
 # 334 unit-test files in about 15 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     # Fails in crontab environment?
     # Figure out what's up here
     SKIP_TESTS[test_array.py]=1
@@ -43,6 +46,7 @@ if (( batch )) ; then
     SKIP_TESTS[test_doctest2.py]=1 # a POWER thing?
     SKIP_TESTS[test_httplib.py]=1  # Ok, but POWER has problems with this
     SKIP_TESTS[test_pdb.py]=1 # Ok, but POWER has problems with this
+    SKIP_TESTS[test_tarfile.py]=1 # test can take over 15 seconds to run on an overloaded POWER7 system
 
     # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
     SKIP_TESTS[test_base64.py]=1

test/stdlib/3.2-exclude.sh

@@ -31,7 +31,7 @@ SKIP_TESTS=(
 
 )
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     # Fails in crontab environment?
     # Figure out what's up here
     SKIP_TESTS[test_exception_variations.py]=1

test/stdlib/3.3-exclude.sh

@@ -1,7 +1,8 @@
 SKIP_TESTS=(
-    [test_atexit.py]=1  # The atexit test starting at 3.3 looks for specific comments in error lines
+    [test_buffer.py]=1  # FIXME: Works on c90ff51
+    [test_cmath.py]=1  # FIXME: Works on c90ff51
 
-    [test_buffer.py]=1  # parse error
+    [test_atexit.py]=1  # The atexit test starting at 3.3 looks for specific comments in error lines
 
     [test_cmd_line.py]=1 # too long?
     [test_concurrent_futures.py]=1  # too long?
@@ -30,6 +31,7 @@ SKIP_TESTS=(
 
     [test_nntplib.py]=1
 
+    [test_pep352.py]=1  # test failures
     [test_peepholer.py]=1
     [test_poll.py]=1  # test takes too long to run: 11 seconds
     [test_pty.py]=1  # FIXME: Needs grammar loop isolation separation
@@ -57,7 +59,7 @@ SKIP_TESTS=(
 )
 # About 300 unit-test files in about 20 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_ftplib.py]=1  # Runs too long on POWER; over 15 seconds
     SKIP_TESTS[test_idle.py]=1  # No tk
     SKIP_TESTS[test_pep352.py]=1  # UnicodeDecodeError may be funny on weird environments

test/stdlib/3.4-exclude.sh

@@ -1,11 +1,18 @@
 SKIP_TESTS=(
+    [test_buffer.py]=1  # FIXME: Works on c90ff51
+    [test_cmath.py]=1  # FIXME: Works on c90ff51
+    [test_strftime.py]=1  # FIXME: Works on c90ff51
+
     [test___all__.py]=1  # it fails on its own
     [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
+
     [test_cmd_line.py]=1 # takes too long to run
     [test_concurrent_futures.py]=1  # too long?
+
     [test_configparser.py]=1  # Doesn't terminate
     [test_ctypes.py]=1 # it fails on its own
     [test_curses.py]=1 # Investigate
+
     [test_dbm_gnu.py]=1   # fails on its own
     [test_devpoll.py]=1 # it fails on its own
     [test_descr.py]=1   # test assertion errors
@@ -14,33 +21,44 @@ SKIP_TESTS=(
     [test_doctest2.py]=1
     [test_doctest.py]=1  # test assert failures
     [test_docxmlrpc.py]=1
+
     [test_enum.py]=1 # compile syntax?
     [test_exceptions.py]=1
+
     [test_faulthandler.py]=1
     [test_file_eintr.py]=1   # parse error
     [test_fork1.py]=1 # too long
+
     [test_gdb.py]=1 # it fails on its own
     [test_grammar.py]=1 # parse error
+
     [test_httplib.py]=1 # it fails on its own
+
     [test_import.py]=1 # it fails on its own
     [test_io.py]=1
     [test_ioctl.py]=1 # it fails on its own
     [test_inspect.py]=1 # Syntax error Investigate
+
     [test_logging.py]=1 # Too long to run
     [test_long.py]=1 # FIXME: Works on c90ff51
+
     [test_modulefinder.py]=1  # test assertion error
     [test_multiprocessing_fork.py]=1 # doesn't terminate
     [test_multiprocessing_forkserver.py]=1 # doesn't terminate
     [test_multiprocessing_main_handling.py]=1  # doesn't terminate
     [test_multiprocessing_spawn.py]=1 # doesn't terminate
+
     [test_nntplib.py]=1 # too long to run
+
     [test_peepholer.py]=1 # control flow?
     [test_pep352.py]=1 # test assert failures
     [test_pickle.py]=1 # test assert failures
     [test_pkgimport.py]=1 # long
     [test_poll.py]=1 # Too long to run: 11 seconds
     [test_pydoc.py]=1 # test assertion failures
+
     [test_runpy.py]=1 # Too long:
+
     [test_select.py]=1 # Too long: 11 seconds
     [test_selectors.py]=1  # Too long: 11 seconds
     [test_signal.py]=1 # Too long: 22 seconds
@@ -50,12 +68,15 @@ SKIP_TESTS=(
     [test_subprocess.py]=1 # Too long
     [test_symtable.py]=1 # Investigate bad output
     [test_sys_settrace.py]=1 # test assert failures
+
     [test_tcl.py]=1 # May be implementation specific. On POWER though it fails
     [test_threading.py]=1 # Too long
     [test_threadsignals.py]=1 # Too long to run: 12 seconds
     [test_timeout.py]=1 # Too long to run: 19 seconds
     [test_traceback.py]=1 # introspects on code
+
     [test_urllib2net.py]=1 # Doesn't terminate
+
     [test_zipfile64.py]=1
     [test_zlib.py]=1
 )

test/stdlib/3.5-exclude.sh

@@ -1,11 +1,7 @@
 SKIP_TESTS=(
     [test_buffer.py]=1  # FIXME: Works on c90ff51
-    [test_decorators.py]=1  # FIXME: Works on c90ff51
-    [test_platform.py]=1 # FIXME: Works on c90ff51
-    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
-    [test_tempfile.py]=1 # FIXME: Works on c90ff51
-    [test_uu.py]=1 # FIXME: Works on c90ff51
-    [test_ftplib.py]=1 # FIXME: Works on c90ff51
+    [test_platform.py]=1  # FIXME: Works on c90ff51
+    [test_pyclbr.py]=1  # FIXME: Works on c90ff51
 
     [test___all__.py]=1 # it fails on its own
     [test_aifc.py]=1  #
@@ -100,6 +96,7 @@ SKIP_TESTS=(
     [test_sys_settrace.py]=1 # test assert fail
 
     [test_tcl.py]=1  # it fails on its own
+    [test_tempfile.py]=1  # test assertion failures
     [test_thread.py]=1
     [test_threading.py]=1
     [test_timeout.py]=1
@@ -115,6 +112,7 @@ SKIP_TESTS=(
     [test_urllib2net.py]=1 # it fails on its own
     [test_urllibnet.py]=1 # it fails on its own
     [test_urlparse.py]=1 # test assert error
+    [test_uu.py]=1 # test assert error
 
     [test_winreg.py]=1 # it fails on its own
     [test_winsound.py]=1 # it fails on its own
@@ -129,7 +127,7 @@ SKIP_TESTS=(
 )
 # About 260 unit-test in about 16 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_asyncore.py]=1 # Ok, but takes more than 15 seconds to run
     SKIP_TESTS[test_bisect.py]=1
     SKIP_TESTS[test_compileall.py]=1  # Something weird on POWER
@@ -137,10 +135,11 @@ if (( batch )) ; then
     SKIP_TESTS[test_distutils.py]=1
 
     SKIP_TESTS[test_exception_variations.py]=1
+    SKIP_TESTS[test_ioctl.py]=1 # it fails on its own
     SKIP_TESTS[test_poplib.py]=1 # May be a result of POWER installation
 
     SKIP_TESTS[test_quopri.py]=1
-    SKIP_TESTS[test_ioctl.py]=1 # it fails on its own
+    SKIP_TESTS[test_sysconfig.py]=1 # POWER extension fails
     SKIP_TESTS[test_tarfile.py]=1 # too long to run on POWER 15 secs
     SKIP_TESTS[test_venv.py]=1 # takes too long 11 seconds
 fi

test/stdlib/3.6-exclude.sh

@@ -1,13 +1,12 @@
 SKIP_TESTS=(
     [test_ast.py]=1 # FIXME: Works on c90ff51
-    [test_binop.py]=1 # FIXME: Works on c90ff51
-    [test_complex.py]=1 # FIXME: Works on c90ff51
+    [test_cmath.py]=1  # FIXME: Works on c90ff51
     [test_format.py]=1 # FIXME: Works on c90ff51
     [test_ftplib.py]=1 # FIXME: Works on c90ff51
     [test_slice.py]=1 # FIXME: Works on c90ff51
     [test_sort.py]=1 # FIXME: Works on c90ff51
     [test_timeit.py]=1 # FIXME: Works on c90ff51
-    [test_os.py]=1 # FIXME: Works on c90ff51
+    [test_os.py]=1 # parse error FIXME: Works on c90ff51
 
     [test___all__.py]=1  # it fails on its own
     [test_aifc.py]=1  #
@@ -49,6 +48,7 @@ SKIP_TESTS=(
     [test_datetime.py]=1 # it fails on its own
     [test_dbm_ndbm.py]=1 # it fails on its own
     [test_decimal.py]=1
+    [test_decorators.py]=1 # control-flow failures
     [test_descr.py]=1   # syntax error: Investigate
     [test_devpoll.py]=1 # it fails on its own
     [test_dict.py]=1 # it fails on its own
@@ -106,7 +106,8 @@ SKIP_TESTS=(
     [test_nntplib.py]=1 # test takes too long to run: 31 seconds
     [test_normalization.py]=1 # it fails on its own
 
-    [test_ordered_dict.py]= # it fails on its own
+    [test_optparse.py]=1 # test fails
+    [test_ordered_dict.py]=1 # it fails on its own
     [test_ossaudiodev.py]=1 # it fails on its own
 
     [test_pdb.py]=1 # Probably introspection
@@ -147,6 +148,7 @@ SKIP_TESTS=(
     [test_startfile.py]=1 # it fails on its own
     [test_statistics.py]=1 # it fails on its own
     [test_string_literals.py]=1
+    [test_strftime.py]=1 # test assertion failures
     [test_strtod.py]=1 # it fails on its own
     [test_struct.py]=1  # test assertion errors
     [test_subprocess.py]=1
@@ -200,8 +202,10 @@ SKIP_TESTS=(
 )
 # 236 unit-test files in about 13 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_codeccallbacks.py]=1
+    SKIP_TESTS[test_complex.py]=1 # Something funky with POWER8
+
     # locale on test machine is probably customized
     SKIP_TESTS[test__locale.py]=1
 fi

test/stdlib/3.7-exclude.sh

@@ -1,6 +1,7 @@
 SKIP_TESTS=(
     [test_builtin.py]=1 # FIXME works on decompyle6
     [test_context.py]=1 # FIXME works on decompyle6
+    [test_doctest2.py]=1 # FIXME works on decompyle6
     [test_format.py]=1 # FIXME works on decompyle6
     [test_marshal.py]=1 # FIXME works on decompyle6
     [test_normalization.py]=1 # FIXME works on decompyle6
@@ -9,12 +10,11 @@ SKIP_TESTS=(
     [test_slice.py]=1 # FIXME works on decompyle6
     [test_sort.py]=1 # FIXME works on decompyle6
     [test_statistics.py]=1 # FIXME works on decompyle6
-    [test_string_literals.py]=1 # FIXME works on decompyle6
     [test_timeit.py]=1 # FIXME works on decompyle6
     [test_urllib2_localnet.py]=1 # FIXME works on decompyle6
     [test_urllib2.py]=1 # FIXME: works on uncompyle6
-    [test_generators.py]=1  # Investigate improper lamdba with bogus "False" added
-    [test_grammar.py]=1 # investigate: like above: semantic rule missing probably
+    [test_generators.py]=1  # FIXME: works on uncompyle6 - lambda parsing probably
+    [test_grammar.py]=1 # FIXME: works on uncompyle6 - lambda parsing probably
 
     [test___all__.py]=1 # it fails on its own
     [test_argparse.py]=1 #- it fails on its own
@@ -133,7 +133,8 @@ SKIP_TESTS=(
 )
 # 306 unit-test files in about 19 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
+    SKIP_TESTS[test_capi.py]=1 # more than 15 secs to run on POWER
     SKIP_TESTS[test_dbm_gnu.py]=1 # fails on its own on POWER
     SKIP_TESTS[test_distutils.py]=1
     SKIP_TESTS[test_fileio.py]=1

test/stdlib/3.8-exclude.sh

@@ -1,10 +1,14 @@
 SKIP_TESTS=(
-    [test_time.py]=1 # FIXME: works on uncompyle6?
+    [test_mimetypes.py]=1 # parse error. decompile3 works. Release 3.6.4 works?
+    [test_time.py]=1 # FIXME: parse eror. decompyle3 works. Release 3.6.4 works?
+    [test_aifc.py]=1 # parse error; decompile3 works
+
+    [test_doctest2.py]=1 # test failures release 3.6.4 works?
+    [test_finalization.py]=1  # test failures release 3.6.4 works?
     [test_urllib2.py]=1 # FIXME: works on uncompyle6?
     [test_zipimport.py]=1 # FIXME: works on uncompyle6
 
     [test___all__.py]=1 # it fails on its own
-    [test_aifc.py]=1 # parse error
     [test_argparse.py]=1 #- it fails on its own
     [test_array.py]=1 #- parse error
     [test_asdl_parser.py]=1 # it fails on its own
@@ -76,6 +80,7 @@ SKIP_TESTS=(
     [test_difflib.py]=1   # parse error
     [test_dis.py]=1   # We change line numbers - duh!
     [test_doctest.py]=1 # parse error
+    [test_doctest2.py]=1 # test faiures relesae 3.6.4 works?
     [test_docxmlrpc.py]=1
     [test_dtrace.py]=1 # parse error
     [test_dummy_thread.py]=1 # parse error
@@ -91,6 +96,7 @@ SKIP_TESTS=(
     [test_exceptions.py]=1   # parse error
 
     [test_faulthandler.py]=1   # takes too long
+    [test_file_eintr.py]=1 # too long to run test; works on 3.7.7
     [test_fcntl.py]=1
     [test_filecmp.py]=1   # parse error
     [test_fileinput.py]=1
@@ -329,7 +335,7 @@ SKIP_TESTS=(
 )
 # 114 test files, Elapsed time about 7 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_idle.py]=1 # Probably installation specific
     SKIP_TESTS[test_tix.py]=1 # fails on its own
     SKIP_TESTS[test_ttk_textonly.py]=1 # Installation dependent?

test/stdlib/run-and-email.sh

@@ -3,28 +3,50 @@
 function displaytime {
     printf "ran in "
     local T=$1
-    local D=$((T/60/60/24))
-    local H=$((T/60/60%24))
-    local M=$((T/60%60))
-    local S=$((T%60))
-    (( $D > 0 )) && printf '%d days ' $D
-    (( $H > 0 )) && printf '%d hours ' $H
-    (( $M > 0 )) && printf '%d minutes ' $M
-    (( $D > 0 || $H > 0 || $M > 0 )) && printf 'and '
+    ((D=T/60/60/24))
+    ((H=T/60/60%24))
+    ((M=T/60%60))
+    ((S=T%60))
+    (( D > 0 )) && printf '%d days ' $D
+    (( H > 0 )) && printf '%d hours ' $H
+    (( M > 0 )) && printf '%d minutes ' $M
+    (( D > 0 || H > 0 || M > 0 )) && printf 'and '
     printf '%d seconds\n' $S
 }
 
-. ../../admin-tools/pyenv-newer-versions
+bs=${BASH_SOURCE[0]}
+if [[ $0 != $bs ]] ; then
+    echo "This script should not be *sourced* but run through bash"
+    exit 1
+fi
+
+mydir=$(dirname $bs)
+cd $mydir
+
+branch=$(cat ../../.git/HEAD | cut -d'/' -f 3)
+if [[ $branch == 'python-2.4' ]]; then
+    . ../../admin-tools/pyenv-older-versions
+elif [[ $branch == 'master' ]]; then
+    . ../../admin-tools/pyenv-newer-versions
+else
+    echo &1>2 "Error git branch should either be 'master' or 'python-2.4'; got: '$branch'"
+    exit 1
+fi
+
+MAIN="runtests.sh"
+
 
 USER=${USER:-rocky}
 EMAIL=${EMAIL:-rb@dustyfeet.com}
-SUBJECT_PREFIX="stdlib unit testing for"
+WHAT="uncompyle6 ${MAIN}"
+export BATCH=1
 
 typeset -i RUN_STARTTIME=$(date +%s)
 
-actual_versions=""
 DEBUG=""  # -x
 
+MAILBODY=/tmp/${MAIN}-mailbody-$$.txt
+
 for VERSION in $PYVERSIONS ; do
     typeset -i rc=0
     LOGFILE=/tmp/runtests-$VERSION-$$.log
@@ -34,24 +56,37 @@ for VERSION in $PYVERSIONS ; do
 	    continue
 	    ;;
     esac
-    actual_versions="$actual_versions $VERSION"
 
     if ! pyenv local $VERSION ; then
 	rc=1
     else
-      STOP_ONERROR=1 /bin/bash $DEBUG ./runtests.sh  >$LOGFILE 2>&1
-      rc=$?
+	typeset -i ALL_FILES_STARTTIME=$(date +%s)
+	STOP_ONERROR=1 /bin/bash $DEBUG ./runtests.sh  >$LOGFILE 2>&1
+	rc=$?
+
+	echo Python Version $(pyenv local) >> $LOGFILE
+	echo "" >>$LOGFILE
+	typeset -i ALL_FILES_ENDTIME=$(date +%s)
+	(( time_diff =  ALL_FILES_ENDTIME - ALL_FILES_STARTTIME))
+	time_str=$(displaytime $time_diff)
+	echo ${time_str}. >> $LOGFILE
     fi
-    SUBJECT_PREFIX="runtests verify for"
+
+    SUBJECT_PREFIX="$WHAT for"
     if ((rc == 0)); then
+	mailbody_line="Python $VERSION ok; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION ok" ${USER}@localhost
     else
+	mailbody_line="Python $VERSION failed; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${USER}@localhost
-	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" $EMAIL
+	tail -v $LOGFILE | mail -s "$HOST $SUBJECT_PREFIX $VERSION not ok" $EMAIL
     fi
+    echo $mailbody_line >> $MAILBODY
 done
 
 typeset -i RUN_ENDTIME=$(date +%s)
 (( time_diff =  RUN_ENDTIME - RUN_STARTTIME))
 elapsed_time=$(displaytime $time_diff)
-echo "Run complete $elapsed_time for versions $actual_versions" | mail -s "runtests in $elapsed_time" ${EMAIL}
+echo "${WHAT} complete; ${elapsed_time}." >> $MAILBODY
+echo "Full results are in ${LOGFILE}." >> $MAILBODY
+cat $MAILBODY | mail -s "$HOST $WHAT $elapsed_time" ${EMAIL}

test/stdlib/runtests.sh

@@ -1,10 +1,12 @@
 #!/bin/bash
 me=${BASH_SOURCE[0]}
 
-typeset -i batch=1
-isatty=$(/usr/bin/tty 2>/dev/null)
-if [[ -n $isatty ]] && [[ "$isatty" != 'not a tty' ]] ; then
-    batch=0
+typeset -i BATCH=${BATCH:-0}
+if (( ! BATCH )) ; then
+    isatty=$(/usr/bin/tty 2>/dev/null)
+    if [[ -n $isatty ]] && [[ "$isatty" != 'not a tty' ]] ; then
+	BATCH=0
+    fi
 fi
 
 
@@ -72,7 +74,7 @@ case $PYVERSION in
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_fileio.py]=1
 	)
-	if (( batch )) ; then
+	if (( BATCH )) ; then
 	    # Fails in crontab environment?
 	    # Figure out what's up here
 	    SKIP_TESTS[test_exception_variations.py]=1
@@ -85,7 +87,7 @@ case $PYVERSION in
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_fileio.py]=1
 	)
-	if (( batch )) ; then
+	if (( BATCH )) ; then
 	    # Fails in crontab environment?
 	    # Figure out what's up here
 	    SKIP_TESTS[test_exception_variations.py]=1
@@ -129,6 +131,7 @@ fulldir=$(pwd)
 
 # DECOMPILER=uncompyle2
 DECOMPILER=${DECOMPILER:-"$fulldir/../../bin/uncompyle6"}
+OPTS=${OPTS:-""}
 TESTDIR=/tmp/test${PYVERSION}
 if [[ -e $TESTDIR ]] ; then
     rm -fr $TESTDIR
@@ -137,6 +140,8 @@ fi
 PYENV_ROOT=${PYENV_ROOT:-$HOME/.pyenv}
 pyenv_local=$(pyenv local)
 
+echo Python version is $pyenv_local
+
 # pyenv version update
 for dir in ../ ../../ ; do
     cp -v .python-version $dir
@@ -145,18 +150,25 @@ done
 
 mkdir $TESTDIR || exit $?
 cp -r ${PYENV_ROOT}/versions/${PYVERSION}.${MINOR}/lib/python${PYVERSION}/test $TESTDIR
-cd $TESTDIR/test
+if [[ $PYVERSION == 3.2 ]] ; then
+    cp ${PYENV_ROOT}/versions/${PYVERSION}.${MINOR}/lib/python${PYVERSION}/test/* $TESTDIR
+    cd $TESTDIR
+else
+    cd $TESTDIR/test
+fi
 pyenv local $FULLVERSION
 export PYTHONPATH=$TESTDIR
 export PATH=${PYENV_ROOT}/shims:${PATH}
 
+DONT_SKIP_TESTS=${DONT_SKIP_TESTS:-0}
+
 # Run tests
 typeset -i i=0
 typeset -i allerrs=0
 if [[ -n $1 ]] ; then
-    files=$1
-    typeset -a files_ary=( $(echo $1) )
-    if (( ${#files_ary[@]} == 1 )) ; then
+    files=$@
+    typeset -a files_ary=( $(echo $@) )
+    if (( ${#files_ary[@]} == 1 || DONT_SKIP_TESTS == 1 )) ; then
 	SKIP_TESTS=()
     fi
 else
@@ -166,9 +178,12 @@ fi
 typeset -i ALL_FILES_STARTTIME=$(date +%s)
 typeset -i skipped=0
 
+NOT_INVERTED_TESTS=${NOT_INVERTED_TESTS:-1}
+
 for file in $files; do
     # AIX bash doesn't grok [[ -v SKIP... ]]
-    if [[ ${SKIP_TESTS[$file]} == 1 ]] ; then
+    [[ -z ${SKIP_TESTS[$file]} ]] && SKIP_TESTS[$file]=0
+    if [[ ${SKIP_TESTS[$file]} == ${NOT_INVERTED_TESTS} ]] ; then
 	((skipped++))
 	continue
     fi
@@ -185,7 +200,7 @@ for file in $files; do
     typeset -i ENDTIME=$(date +%s)
     typeset -i time_diff
     (( time_diff =  ENDTIME - STARTTIME))
-    if (( time_diff > 10 )) ; then
+    if (( time_diff > $timeout )) ; then
 	echo "Skipping test $file -- test takes too long to run: $time_diff seconds"
 	continue
     fi
@@ -197,7 +212,7 @@ for file in $files; do
     $fulldir/compile-file.py $file && \
     mv $file{,.orig} && \
     echo ==========  $(date +%X) Decompiling $file ===========
-    $DECOMPILER $decompiled_file > $file
+    $DECOMPILER $OPTS $decompiled_file > $file
     rc=$?
     if (( rc == 0 )) ; then
 	echo ========== $(date +%X) Running $file ===========

uncompyle6/Makefile

@@ -1,7 +0,0 @@
-# Whatever it is you want to do, it should be forwarded to the 
-# to top-level irectories
-PHONY=check all
-all: check
-
-%: 
-	$(MAKE) -C .. $@

uncompyle6/disas.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016, 2818-2019 by Rocky Bernstein
+#  Copyright (c) 2015-2016, 2818-2020 by Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
@@ -34,8 +34,7 @@ from __future__ import print_function
 import sys
 from collections import deque
 
-from xdis.code import iscode
-from xdis.load import check_object_path, load_module
+from xdis import check_object_path, iscode, load_module
 from uncompyle6.scanner import get_scanner
 
 
@@ -100,7 +99,9 @@ def disassemble_file(filename, outstream=None):
     try to find the corresponding compiled object.
     """
     filename = check_object_path(filename)
-    (version, timestamp, magic_int, co, is_pypy, source_size) = load_module(filename)
+    (version, timestamp, magic_int, co, is_pypy, source_size, sip_hash) = load_module(
+        filename
+    )
     if type(co) == list:
         for con in co:
             disco(version, con, outstream)

uncompyle6/linenumbers.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016, 2818 by Rocky Bernstein
+#  Copyright (c) 2015-2016, 2818, 2020 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -15,14 +15,27 @@
 
 from collections import deque
 
-from xdis.code import iscode
-from xdis.load import load_file, load_module
-from xdis.main import get_opcode
-from xdis.bytecode import Bytecode, findlinestarts, offset2line
+from xdis import (
+    Bytecode,
+    iscode,
+    findlinestarts,
+    get_opcode,
+    offset2line,
+    load_file,
+    load_module,
+)
+
 
 def line_number_mapping(pyc_filename, src_filename):
-    (version, timestamp, magic_int, code1, is_pypy,
-     source_size) = load_module(pyc_filename)
+    (
+        version,
+        timestamp,
+        magic_int,
+        code1,
+        is_pypy,
+        source_size,
+        sip_hash,
+    ) = load_module(pyc_filename)
     try:
         code2 = load_file(src_filename)
     except SyntaxError as e:
@@ -44,7 +57,10 @@ def number_loop(queue, mappings, opc):
         assert code1.co_name == code2.co_name
         linestarts_orig = findlinestarts(code1)
         linestarts_uncompiled = list(findlinestarts(code2))
-        mappings += [[line, offset2line(offset, linestarts_uncompiled)] for offset, line in linestarts_orig]
+        mappings += [
+            [line, offset2line(offset, linestarts_uncompiled)]
+            for offset, line in linestarts_orig
+        ]
         bytecode1 = Bytecode(code1, opc)
         bytecode2 = Bytecode(code2, opc)
         instr2s = bytecode2.get_instructions(code2)

uncompyle6/main.py

@@ -16,8 +16,7 @@ from __future__ import print_function
 import datetime, py_compile, os, subprocess, sys, tempfile
 
 from uncompyle6 import verify, IS_PYPY, PYTHON_VERSION
-from xdis.code import iscode
-from xdis.magics import sysinfo2float
+from xdis import iscode, sysinfo2float
 from uncompyle6.disas import check_object_path
 from uncompyle6.semantics import pysource
 from uncompyle6.parser import ParserError
@@ -102,7 +101,7 @@ def decompile(
     )
     if PYTHON_VERSION < 3.0 and bytecode_version >= 3.0:
         write(
-            "# Warning: this version has problems handling the Python 3 byte type in contants properly.\n"
+            '# Warning: this version has problems handling the Python 3 "byte" type in constants properly.\n'
         )
 
     if co.co_filename:
@@ -183,7 +182,7 @@ def decompile_file(
 
     filename = check_object_path(filename)
     code_objects = {}
-    (version, timestamp, magic_int, co, is_pypy, source_size) = load_module(
+    (version, timestamp, magic_int, co, is_pypy, source_size, sip_hash) = load_module(
         filename, code_objects
     )
 

uncompyle6/parser.py

@@ -23,16 +23,16 @@ from __future__ import print_function
 
 import sys
 
-from xdis.code import iscode
-from xdis.magics import py_str2float
+from xdis import iscode, py_str2float
 from spark_parser import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.show import maybe_show_asm
 
 
 class ParserError(Exception):
-    def __init__(self, token, offset):
+    def __init__(self, token, offset, debug=PARSER_DEFAULT_DEBUG):
         self.token = token
         self.offset = offset
+        self.debug = debug
 
     def __str__(self):
         return "Parse error at or near `%r' instruction at offset %s\n" % (
@@ -206,9 +206,9 @@ class PythonParser(GenericASTBuilder):
                 else:
                     indent = "-> "
                 print("%s%s" % (indent, instructions[i]))
-            raise ParserError(err_token, err_token.offset)
+            raise ParserError(err_token, err_token.offset, self.debug["reduce"])
         else:
-            raise ParserError(None, -1)
+            raise ParserError(None, -1, self.debug["reduce"])
 
     def get_pos_kw(self, token):
         """Return then the number of positional parameters and
@@ -351,7 +351,7 @@ class PythonParser(GenericASTBuilder):
         stmt ::= try_except
         stmt ::= tryelsestmt
         stmt ::= tryfinallystmt
-        stmt ::= withstmt
+        stmt ::= with
         stmt ::= withasstmt
 
         stmt ::= del_stmt

uncompyle6/parsers/parse10.py

@@ -7,7 +7,7 @@ from uncompyle6.parsers.parse11 import Python11Parser
 
 class Python10Parser(Python11Parser):
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python11Parser, self).__init__(debug_parser)
+        super(Python10Parser, self).__init__(debug_parser)
         self.customized = {}
 
 

uncompyle6/parsers/parse11.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019 Rocky Bernstein
+#  Copyright (c) 2019-2020 Rocky Bernstein
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.parser import PythonParserSingle
@@ -7,7 +7,7 @@ from uncompyle6.parsers.parse12 import Python12Parser
 
 class Python11Parser(Python12Parser):
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python12Parser, self).__init__(debug_parser)
+        super(Python11Parser, self).__init__(debug_parser)
         self.customized = {}
 
 

uncompyle6/parsers/parse2.py

@@ -196,8 +196,9 @@ class Python2Parser(PythonParser):
         expr ::= slice3
         expr ::= unary_convert
 
-        and  ::= expr jmp_false expr come_from_opt
-        or   ::= expr jmp_true  expr come_from_opt
+        expr_jt  ::= expr jmp_true
+        or       ::= expr_jt  expr come_from_opt
+        and      ::= expr jmp_false expr come_from_opt
 
         unary_convert ::= expr UNARY_CONVERT
 

uncompyle6/parsers/parse21.py

@@ -8,7 +8,7 @@ from uncompyle6.parsers.parse22 import Python22Parser
 class Python21Parser(Python22Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python22Parser, self).__init__(debug_parser)
+        super(Python21Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_forstmt21(self, args):

uncompyle6/parsers/parse22.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2017 Rocky Bernstein
+#  Copyright (c) 2016-2017, 2020 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
@@ -8,7 +8,7 @@ from uncompyle6.parsers.parse23 import Python23Parser
 class Python22Parser(Python23Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python23Parser, self).__init__(debug_parser)
+        super(Python22Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_misc22(self, args):

uncompyle6/parsers/parse23.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2018 Rocky Bernstein
+#  Copyright (c) 2016-2018, 2020 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 #  Copyright (c) 1999 John Aycock
 
@@ -9,7 +9,7 @@ from uncompyle6.parsers.parse24 import Python24Parser
 class Python23Parser(Python24Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python24Parser, self).__init__(debug_parser)
+        super(Python23Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_misc23(self, args):
@@ -32,8 +32,21 @@ class Python23Parser(Python24Parser):
         while1stmt ::= _while1test l_stmts_opt JUMP_BACK
                        POP_TOP POP_BLOCK COME_FROM
 
-        while1stmt ::= _while1test l_stmts_opt JUMP_BACK
-                       COME_FROM POP_TOP POP_BLOCK COME_FROM
+        while1stmt ::= _while1test l_stmts_opt JUMP_BACK COME_FROM
+                       POP_TOP POP_BLOCK COME_FROM
+
+        # Python 2.3
+        # The following has no "JUMP_BACK" after l_stmts because
+        # l_stmts ends in a "break", "return", or "continue"
+        while1stmt ::= _while1test l_stmts
+                       POP_TOP POP_BLOCK
+
+        # The following has a "COME_FROM" at the end which comes from
+        # a "break" inside "l_stmts".
+        while1stmt ::= _while1test l_stmts COME_FROM JUMP_BACK
+                       POP_TOP POP_BLOCK COME_FROM
+        while1stmt ::= _while1test l_stmts JUMP_BACK
+                       POP_TOP POP_BLOCK
 
         list_comp  ::= BUILD_LIST_0 DUP_TOP LOAD_ATTR store list_iter del_stmt
         list_for   ::= expr for_iter store list_iter JUMP_BACK come_froms POP_TOP JUMP_BACK

uncompyle6/parsers/parse24.py

@@ -38,8 +38,23 @@ class Python24Parser(Python25Parser):
         _ifstmts_jump24 ::= c_stmts_opt JUMP_FORWARD POP_TOP
 
         # Python 2.5+ omits POP_TOP POP_BLOCK
-        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_TOP POP_BLOCK COME_FROM
-        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_TOP POP_BLOCK
+        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK
+                       POP_TOP POP_BLOCK COME_FROM
+        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK
+                       POP_TOP POP_BLOCK
+
+        continue   ::= JUMP_BACK JUMP_ABSOLUTE
+
+        # Python 2.4
+        # The following has no "JUMP_BACK" after l_stmts because
+        # l_stmts ends in a "break", "return", or "continue"
+        while1stmt ::= SETUP_LOOP l_stmts
+                       POP_TOP POP_BLOCK
+
+        # The following has a "COME_FROM" at the end which comes from
+        # a "break" inside "l_stmts".
+        while1stmt ::= SETUP_LOOP l_stmts COME_FROM JUMP_BACK
+                       POP_TOP POP_BLOCK COME_FROM
 
         # Python 2.5+:
         #  call_stmt ::= expr POP_TOP
@@ -73,8 +88,8 @@ class Python24Parser(Python25Parser):
         with_cleanup  ::= LOAD_FAST DELETE_FAST WITH_CLEANUP END_FINALLY
         with_cleanup  ::= LOAD_NAME DELETE_NAME WITH_CLEANUP END_FINALLY
         withasstmt    ::= expr setupwithas store suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM with_cleanup
-        withstmt      ::= expr setupwith SETUP_FINALLY suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM with_cleanup
-        stmt ::= withstmt
+        with          ::= expr setupwith SETUP_FINALLY suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM with_cleanup
+        stmt ::= with
         stmt ::= withasstmt
         """)
         super(Python24Parser, self).customize_grammar_rules(tokens, customize)

uncompyle6/parsers/parse25.py

@@ -27,7 +27,7 @@ class Python25Parser(Python26Parser):
                         setup_finally
         # opcode SETUP_WITH
         setupwith ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 POP_TOP
-        withstmt  ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+        with      ::= expr setupwith SETUP_FINALLY suite_stmts_opt
                       POP_BLOCK LOAD_CONST COME_FROM with_cleanup
 
         # Semantic actions want store to be at index 2
@@ -62,7 +62,7 @@ class Python25Parser(Python26Parser):
         # Remove grammar rules inherited from Python 2.6 or Python 2
         self.remove_rules("""
         setupwith  ::= DUP_TOP LOAD_ATTR ROT_TWO LOAD_ATTR CALL_FUNCTION_0 POP_TOP
-        withstmt   ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+        with       ::= expr setupwith SETUP_FINALLY suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY
         withasstmt ::= expr setupwithas store suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY

uncompyle6/parsers/parse26.py

@@ -119,8 +119,8 @@ class Python26Parser(Python2Parser):
         ifelsestmtc ::= testexpr c_stmts_opt ja_cf_pop    else_suitec
 
         # Semantic actions want suite_stmts_opt to be at index 3
-        withstmt ::= expr setupwith SETUP_FINALLY suite_stmts_opt
-                     POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY
+        with        ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+                        POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY
 
         # Semantic actions want store to be at index 2
         withasstmt ::= expr setupwithas store suite_stmts_opt

uncompyle6/parsers/parse27.py

@@ -7,6 +7,7 @@ from xdis import next_offset
 from uncompyle6.parser import PythonParserSingle, nop_func
 from uncompyle6.parsers.parse2 import Python2Parser
 from uncompyle6.parsers.reducecheck import (
+    or_check,
     ifelsestmt,
     tryelsestmt,
 )
@@ -101,8 +102,9 @@ class Python27Parser(Python2Parser):
         ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
         if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
-        or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
-        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
+        expr_jitop ::= expr JUMP_IF_TRUE_OR_POP
+        or         ::= expr_jitop expr COME_FROM
+        and        ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
 
         # compare_chained{1,2} is used exclusively in chained_compare
         compare_chained1 ::= expr DUP_TOP ROT_THREE COMPARE_OP JUMP_IF_FALSE_OR_POP
@@ -139,9 +141,11 @@ class Python27Parser(Python2Parser):
         # assert condition, expr
         assert2    ::= assert_expr jmp_true LOAD_ASSERT expr CALL_FUNCTION_1 RAISE_VARARGS_1
 
+        continue   ::= JUMP_BACK JUMP_ABSOLUTE
+
         for_block  ::= returns _come_froms
 
-        withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+        with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM_WITH
                        WITH_CLEANUP END_FINALLY
 
@@ -227,6 +231,7 @@ class Python27Parser(Python2Parser):
         # FIXME: Put more in this table
         self.reduce_check_table = {
             # "ifelsestmt": ifelsestmt,
+            "or": or_check,
             "tryelsestmt": tryelsestmt,
             "tryelsestmtl": tryelsestmt,
         }
@@ -237,7 +242,7 @@ class Python27Parser(Python2Parser):
         self.check_reduce["except_handler"] = "tokens"
         self.check_reduce["except_handler_else"] = "tokens"
 
-        # self.check_reduce["or"] = "AST"
+        self.check_reduce["or"] = "AST"
         self.check_reduce["raise_stmt1"] = "AST"
         self.check_reduce["iflaststmtl"] = "AST"
         self.check_reduce["ifelsestmt"] = "AST"

uncompyle6/parsers/parse3.py

@@ -74,6 +74,7 @@ class Python3Parser(PythonParser):
 
         jb_or_c ::= JUMP_BACK
         jb_or_c ::= CONTINUE
+        jb_cfs  ::= JUMP_BACK _come_froms
 
         stmt ::= set_comp_func
 
@@ -269,18 +270,24 @@ class Python3Parser(PythonParser):
         jmp_abs ::= JUMP_ABSOLUTE
         jmp_abs ::= JUMP_BACK
 
-        withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt
-                POP_BLOCK LOAD_CONST COME_FROM_WITH
-                WITH_CLEANUP END_FINALLY
+        with    ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+                    POP_BLOCK LOAD_CONST COME_FROM_WITH
+                    WITH_CLEANUP END_FINALLY
 
         withasstmt ::= expr SETUP_WITH store suite_stmts_opt
                 POP_BLOCK LOAD_CONST COME_FROM_WITH
                 WITH_CLEANUP END_FINALLY
 
+        expr_jt     ::= expr jmp_true
+        expr_jitop  ::= expr JUMP_IF_TRUE_OR_POP
+
         ## FIXME: Right now we have erroneous jump targets
         ## This below is probably not correct when the COME_FROM is put in the right place
-        and ::= expr jmp_false expr COME_FROM
-        or  ::= expr jmp_true  expr COME_FROM
+        and  ::= expr jmp_false expr COME_FROM
+        or   ::= expr_jt  expr COME_FROM
+        or   ::= expr_jt expr
+        or   ::= expr_jitop expr COME_FROM
+        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
 
         # # something like the below is needed when the jump targets are fixed
         ## or  ::= expr JUMP_IF_TRUE_OR_POP COME_FROM expr
@@ -338,9 +345,6 @@ class Python3Parser(PythonParser):
         ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
         if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
-        or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
-        or   ::= expr jmp_true expr
-        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
 
         # compare_chained1 is used exclusively in chained_compare
         compare_chained1 ::= expr DUP_TOP ROT_THREE COMPARE_OP JUMP_IF_FALSE_OR_POP
@@ -435,10 +439,11 @@ class Python3Parser(PythonParser):
         while1elsestmt    ::= SETUP_LOOP          l_stmts     JUMP_BACK
                               else_suitel
 
-        whileelsestmt     ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK
+        whileelsestmt     ::= SETUP_LOOP testexpr l_stmts_opt jb_cfs POP_BLOCK
                               else_suitel COME_FROM_LOOP
 
-        whileelsestmt2     ::= SETUP_LOOP testexpr l_stmts_opt  JUMP_BACK POP_BLOCK
+
+        whileelsestmt2    ::= SETUP_LOOP testexpr l_stmts_opt  JUMP_BACK POP_BLOCK
                               else_suitel JUMP_BACK COME_FROM_LOOP
 
         whileTruestmt     ::= SETUP_LOOP l_stmts_opt          JUMP_BACK POP_BLOCK

uncompyle6/parsers/parse30.py

@@ -134,8 +134,9 @@ class Python30Parser(Python31Parser):
 
         jump_except           ::= _jump COME_FROM POP_TOP
 
+        expr_jt               ::= expr jmp_true
         or                    ::= expr jmp_false expr jmp_true expr
-        or                    ::= expr jmp_true expr
+        or                    ::= expr_jt expr
 
         import_from ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist _come_froms POP_TOP
 
@@ -212,7 +213,7 @@ class Python30Parser(Python31Parser):
         whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK JUMP_BACK COME_FROM_LOOP
         whilestmt      ::= SETUP_LOOP testexpr returns POP_TOP POP_BLOCK COME_FROM_LOOP
         withasstmt     ::= expr SETUP_WITH store suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
-        withstmt       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
+        with           ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
 
         # lc_body ::= LOAD_FAST expr LIST_APPEND
         # lc_body ::= LOAD_NAME expr LIST_APPEND

uncompyle6/parsers/parse31.py

@@ -15,7 +15,7 @@ class Python31Parser(Python32Parser):
 
         setupwith      ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 POP_TOP
         setupwithas    ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 store
-        withstmt       ::= expr setupwith SETUP_FINALLY
+        with           ::= expr setupwith SETUP_FINALLY
                            suite_stmts_opt
                            POP_BLOCK LOAD_CONST COME_FROM_FINALLY
                            load del_stmt WITH_CLEANUP END_FINALLY

uncompyle6/parsers/parse35.py

@@ -50,7 +50,7 @@ class Python35Parser(Python34Parser):
 
         # Python 3.5+ has WITH_CLEANUP_START/FINISH
 
-        withstmt   ::= expr
+        with       ::= expr
                        SETUP_WITH POP_TOP suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM_WITH
                        WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
@@ -138,7 +138,7 @@ class Python35Parser(Python34Parser):
         self.remove_rules("""
           yield_from ::= expr GET_ITER LOAD_CONST YIELD_FROM
           yield_from ::= expr expr YIELD_FROM
-          withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+          with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt
                          POP_BLOCK LOAD_CONST COME_FROM_WITH
                          WITH_CLEANUP END_FINALLY
           withasstmt ::= expr SETUP_WITH store suite_stmts_opt
@@ -209,10 +209,10 @@ class Python35Parser(Python34Parser):
             elif opname == 'SETUP_WITH':
                 # Python 3.5+ has WITH_CLEANUP_START/FINISH
                 rules_str = """
-                  withstmt   ::= expr
-                       SETUP_WITH POP_TOP suite_stmts_opt
-                       POP_BLOCK LOAD_CONST COME_FROM_WITH
-                       WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                  with ::= expr
+                           SETUP_WITH POP_TOP suite_stmts_opt
+                           POP_BLOCK LOAD_CONST COME_FROM_WITH
+                           WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
                   withasstmt ::= expr
                        SETUP_WITH store suite_stmts_opt

uncompyle6/parsers/parse36.py

@@ -307,7 +307,7 @@ class Python36Parser(Python35Parser):
                 self.check_reduce['assign'] = 'token'
             elif opname == 'SETUP_WITH':
                 rules_str = """
-                withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
+                with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
                                WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
                 # Removes POP_BLOCK LOAD_CONST from 3.6-
@@ -316,13 +316,13 @@ class Python36Parser(Python35Parser):
                 """
                 if self.version < 3.8:
                     rules_str += """
-                    withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                    with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                    LOAD_CONST
                                    WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
                     """
                 else:
                     rules_str += """
-                    withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                    with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                    BEGIN_FINALLY COME_FROM_WITH
                                    WITH_CLEANUP_START WITH_CLEANUP_FINISH
                                    END_FINALLY

uncompyle6/parsers/parse37.py

@@ -632,9 +632,12 @@ class Python37Parser(Python37BaseParser):
     def p_37conditionals(self, args):
         """
         expr                       ::= if_exp37
-        if_exp37              ::= expr expr jf_cfs expr COME_FROM
+        if_exp37                   ::= expr expr jf_cfs expr COME_FROM
         jf_cfs                     ::= JUMP_FORWARD _come_froms
         ifelsestmt                 ::= testexpr c_stmts_opt jf_cfs else_suite opt_come_from_except
+        expr_pjit                  ::= expr POP_JUMP_IF_TRUE
+        expr_jit                   ::= expr JUMP_IF_TRUE
+        expr_jt                    ::= expr jmp_true
 
         jmp_false37                ::= POP_JUMP_IF_FALSE COME_FROM
         list_if                    ::= expr jmp_false37 list_iter
@@ -925,12 +928,14 @@ class Python37Parser(Python37BaseParser):
         ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
         if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
-        jitop_come_from ::= JUMP_IF_TRUE_OR_POP come_froms
+        jitop_come_from_expr ::= JUMP_IF_TRUE_OR_POP come_froms expr
         jifop_come_from ::= JUMP_IF_FALSE_OR_POP come_froms
-        or        ::= and jitop_come_from expr COME_FROM
-        or        ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
-        or        ::= expr JUMP_IF_TRUE expr COME_FROM
-        or        ::= expr POP_JUMP_IF_TRUE expr POP_JUMP_IF_FALSE COME_FROM
+        expr_jitop      ::= expr JUMP_IF_TRUE_OR_POP
+
+        or        ::= and jitop_come_from_expr COME_FROM
+        or        ::= expr_jitop  expr COME_FROM
+        or        ::= expr_jit expr COME_FROM
+        or        ::= expr_pjit expr POP_JUMP_IF_FALSE COME_FROM
 
         testfalse_not_or   ::= expr jmp_false expr jmp_false COME_FROM
         testfalse_not_and ::= and jmp_true come_froms
@@ -946,17 +951,17 @@ class Python37Parser(Python37BaseParser):
         testexprl   ::= testfalsel
         testfalsel  ::= expr jmp_true
 
-        or          ::= expr jmp_true expr
+        or          ::= expr_jt expr
 
         and  ::= expr JUMP_IF_FALSE_OR_POP expr come_from_opt
         and  ::= expr jifop_come_from expr
 
-        pjit_come_from ::= POP_JUMP_IF_TRUE COME_FROM
-        or  ::= expr pjit_come_from expr
+        expr_pjit_come_from ::= expr POP_JUMP_IF_TRUE COME_FROM
+        or  ::= expr_pjit_come_from expr
 
         ## Note that "jmp_false" is what we check on in the "and" reduce rule.
         and ::= expr jmp_false expr COME_FROM
-        or  ::= expr jmp_true  expr COME_FROM
+        or  ::= expr_jt  expr COME_FROM
 
         # compare_chained1 is used exclusively in chained_compare
         compare_chained1 ::= expr DUP_TOP ROT_THREE COMPARE_OP JUMP_IF_FALSE_OR_POP
@@ -1271,7 +1276,7 @@ class Python37Parser(Python37BaseParser):
                 self.addRule(rule, nop_func)
             elif opname == "SETUP_WITH":
                 rules_str = """
-                withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
+                with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
                                WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
                 # Removes POP_BLOCK LOAD_CONST from 3.6-
@@ -1280,13 +1285,13 @@ class Python37Parser(Python37BaseParser):
                 """
                 if self.version < 3.8:
                     rules_str += """
-                    withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                    with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                    LOAD_CONST
                                    WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
                     """
                 else:
                     rules_str += """
-                    withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                    with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                    BEGIN_FINALLY COME_FROM_WITH
                                    WITH_CLEANUP_START WITH_CLEANUP_FINISH
                                    END_FINALLY

uncompyle6/parsers/parse37base.py

@@ -2,10 +2,10 @@
 """
 Python 3.7 base code. We keep non-custom-generated grammar rules out of this file.
 """
-from uncompyle6.scanners.tok import Token
-from uncompyle6.parser import PythonParser, PythonParserSingle, nop_func
+from uncompyle6.parser import ParserError, PythonParser, nop_func
 from uncompyle6.parsers.treenode import SyntaxTree
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser.spark import rule2str
 
 from uncompyle6.parsers.reducecheck import (
     and_check,
@@ -210,41 +210,86 @@ class Python37BaseParser(PythonParser):
 
                 if self.version < 3.8:
                     rules_str += """
-                      stmt                ::= async_with_stmt SETUP_ASYNC_WITH
-                      async_with_stmt     ::= expr
+                      stmt                 ::= async_with_stmt SETUP_ASYNC_WITH
+                      c_stmt               ::= c_async_with_stmt SETUP_ASYNC_WITH
+                      async_with_stmt      ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               suite_stmts_opt
+                                               POP_BLOCK LOAD_CONST
+                                               async_with_post
+                      c_async_with_stmt    ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               c_suite_stmts_opt
+                                               POP_BLOCK LOAD_CONST
+                                               async_with_post
+                      async_with_stmt      ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               suite_stmts_opt
+                                               async_with_post
+                      c_async_with_stmt    ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               c_suite_stmts_opt
+                                               async_with_post
+                      async_with_as_stmt   ::= expr
+                                               async_with_pre
+                                               store
+                                               suite_stmts_opt
+                                               POP_BLOCK LOAD_CONST
+                                               async_with_post
+                      c_async_with_as_stmt ::= expr
                                               async_with_pre
-                                              POP_TOP
-                                              suite_stmts_opt
+                                              store
+                                              c_suite_stmts_opt
                                               POP_BLOCK LOAD_CONST
                                               async_with_post
-                      async_with_stmt     ::= expr
-                                              async_with_pre
-                                              POP_TOP
-                                              suite_stmts_opt
-                                              async_with_post
-                      async_with_as_stmt  ::= expr
+                      async_with_as_stmt   ::= expr
+                                              async_with_pre
+                                              store
+                                              suite_stmts_opt
+                                              async_with_post
+                      c_async_with_as_stmt ::= expr
                                               async_with_pre
                                               store
                                               suite_stmts_opt
-                                              POP_BLOCK LOAD_CONST
                                               async_with_post
                     """
                 else:
                     rules_str += """
-                      async_with_pre      ::= BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM SETUP_ASYNC_WITH
-                      async_with_post     ::= BEGIN_FINALLY COME_FROM_ASYNC_WITH
-                                              WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
-                                              WITH_CLEANUP_FINISH END_FINALLY
-                      async_with_stmt     ::= expr
+                      async_with_pre       ::= BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM SETUP_ASYNC_WITH
+                      async_with_post      ::= BEGIN_FINALLY COME_FROM_ASYNC_WITH
+                                               WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                               WITH_CLEANUP_FINISH END_FINALLY
+                      async_with_stmt      ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               suite_stmts
+                                               POP_TOP POP_BLOCK
+                                               async_with_post
+                      c_async_with_stmt    ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               c_suite_stmts
+                                               POP_TOP POP_BLOCK
+                                               async_with_post
+                      async_with_stmt      ::= expr
+                                               async_with_pre
+                                               POP_TOP
+                                               suite_stmts
+                                               POP_BLOCK
+                                               BEGIN_FINALLY
+                                               WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                               WITH_CLEANUP_FINISH POP_FINALLY LOAD_CONST RETURN_VALUE
+                                               COME_FROM_ASYNC_WITH
+                                               WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                               WITH_CLEANUP_FINISH END_FINALLY
+                      c_async_with_stmt   ::= expr
                                               async_with_pre
                                               POP_TOP
-                                              suite_stmts
-                                              POP_TOP POP_BLOCK
-                                              async_with_post
-                      async_with_stmt     ::= expr
-                                              async_with_pre
-                                              POP_TOP
-                                              suite_stmts
+                                              c_suite_stmts
                                               POP_BLOCK
                                               BEGIN_FINALLY
                                               WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
@@ -252,15 +297,24 @@ class Python37BaseParser(PythonParser):
                                               COME_FROM_ASYNC_WITH
                                               WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
                                               WITH_CLEANUP_FINISH END_FINALLY
-                      async_with_as_stmt  ::= expr
-                                              async_with_pre
-                                              store suite_stmts
-                                              POP_TOP POP_BLOCK
-                                              async_with_post
-                      async_with_as_stmt  ::= expr
-                                              async_with_pre
-                                              store suite_stmts
-                                              POP_BLOCK async_with_post
+                      async_with_as_stmt   ::= expr
+                                               async_with_pre
+                                               store suite_stmts
+                                               POP_TOP POP_BLOCK
+                                               async_with_post
+                      c_async_with_as_stmt ::= expr
+                                               async_with_pre
+                                               store suite_stmts
+                                               POP_TOP POP_BLOCK
+                                               async_with_post
+                      async_with_as_stmt   ::= expr
+                                               async_with_pre
+                                               store suite_stmts
+                                               POP_BLOCK async_with_post
+                      c_async_with_as_stmt ::= expr
+                                               async_with_pre
+                                               store suite_stmts
+                                               POP_BLOCK async_with_post
                     """
                 self.addRule(rules_str, nop_func)
 
@@ -539,7 +593,7 @@ class Python37BaseParser(PythonParser):
 
                     stmt                ::= genexpr_func_async
 
-                    func_async_prefix   ::= SETUP_EXCEPT GET_ANEXT LOAD_CONST YIELD_FROM
+                    func_async_prefix   ::= _come_froms SETUP_EXCEPT GET_ANEXT LOAD_CONST YIELD_FROM
                     func_async_middle   ::= POP_BLOCK JUMP_FORWARD COME_FROM_EXCEPT
                                             DUP_TOP LOAD_GLOBAL COMPARE_OP POP_JUMP_IF_TRUE
                                             END_FINALLY COME_FROM
@@ -548,22 +602,24 @@ class Python37BaseParser(PythonParser):
                                             JUMP_BACK COME_FROM
                                             POP_TOP POP_TOP POP_TOP POP_EXCEPT POP_TOP
 
-                    expr                ::= listcomp_async
-                    listcomp_async      ::= LOAD_LISTCOMP LOAD_STR MAKE_FUNCTION_0
+                    expr                ::= list_comp_async
+                    list_comp_async     ::= LOAD_LISTCOMP LOAD_STR MAKE_FUNCTION_0
                                             expr GET_AITER CALL_FUNCTION_1
                                             GET_AWAITABLE LOAD_CONST
                                             YIELD_FROM
 
-                    expr                 ::= listcomp_async
-                    listcomp_async       ::= BUILD_LIST_0 LOAD_FAST func_async_prefix
+                    expr                ::= list_comp_async
+                    list_afor2          ::= func_async_prefix
                                             store func_async_middle list_iter
                                             JUMP_BACK COME_FROM
                                             POP_TOP POP_TOP POP_TOP POP_EXCEPT POP_TOP
-
+                    list_comp_async     ::= BUILD_LIST_0 LOAD_FAST list_afor2
+                    get_aiter           ::= expr GET_AITER
+                    list_afor           ::= get_aiter list_afor2
+                    list_iter           ::= list_afor
                    """,
                     nop_func,
                 )
-                custom_ops_processed.add(opname)
             elif opname == "JUMP_IF_NOT_DEBUG":
                 v = token.attr
                 self.addRule(
@@ -939,15 +995,15 @@ class Python37BaseParser(PythonParser):
 
             elif opname == "SETUP_WITH":
                 rules_str = """
-                  stmt       ::= withstmt
+                  stmt       ::= with
                   stmt       ::= withasstmt
 
-                  withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
+                  with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
                                  WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
                   withasstmt ::= expr SETUP_WITH store suite_stmts_opt COME_FROM_WITH
                                  WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
-                  withstmt   ::= expr
+                  with       ::= expr
                                  SETUP_WITH POP_TOP suite_stmts_opt
                                  POP_BLOCK LOAD_CONST COME_FROM_WITH
                                  WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
@@ -956,7 +1012,7 @@ class Python37BaseParser(PythonParser):
                                  POP_BLOCK LOAD_CONST COME_FROM_WITH
                                  WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
-                  withstmt   ::= expr
+                  with       ::= expr
                                  SETUP_WITH POP_TOP suite_stmts_opt
                                  POP_BLOCK LOAD_CONST COME_FROM_WITH
                                  WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
@@ -967,13 +1023,13 @@ class Python37BaseParser(PythonParser):
                 """
                 if self.version < 3.8:
                     rules_str += """
-                    withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                    with     ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                    LOAD_CONST
                                    WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
                     """
                 else:
                     rules_str += """
-                      withstmt   ::= expr
+                      with    ::= expr
                                      SETUP_WITH POP_TOP suite_stmts_opt
                                      POP_BLOCK LOAD_CONST COME_FROM_WITH
                                      WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
@@ -982,7 +1038,7 @@ class Python37BaseParser(PythonParser):
                                      SETUP_WITH store suite_stmts_opt
                                      POP_BLOCK LOAD_CONST COME_FROM_WITH
 
-                       withstmt  ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                       with    ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                      BEGIN_FINALLY COME_FROM_WITH
                                      WITH_CLEANUP_START WITH_CLEANUP_FINISH
                                      END_FINALLY
@@ -1091,8 +1147,8 @@ class Python37BaseParser(PythonParser):
                 + token.kind
             )
 
-            # Note: semantic actions make use of the fact of wheter  "args_pos"
-            # zero or not in creating a template rule.
+            # Note: Semantic actions make use of whether or not "args_pos"
+            # is zero when creating a template rule.
             self.add_unique_rule(rule, token.kind, args_pos, customize)
         else:
             token.kind = self.call_fn_name(token)
@@ -1126,10 +1182,28 @@ class Python37BaseParser(PythonParser):
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
         lhs = rule[0]
         n = len(tokens)
-        last = min(last, n-1)
+        last = min(last, n - 1)
         fn = self.reduce_check_table.get(lhs, None)
-        if fn:
-            return fn(self, lhs, n, rule, ast, tokens, first, last)
+        try:
+            if fn:
+                return fn(self, lhs, n, rule, ast, tokens, first, last)
+        except:
+            import sys, traceback
+
+            print(
+                ("Exception in %s %s\n"
+                 + "rule: %s\n"
+                 + "offsets %s .. %s")
+                % (
+                    fn.__name__,
+                    sys.exc_info()[1],
+                    rule2str(rule),
+                    tokens[first].offset,
+                    tokens[last].offset,
+                )
+            )
+            print(traceback.print_tb(sys.exc_info()[2], -1))
+            raise ParserError(tokens[last], tokens[last].off2int(), self.debug["rules"])
 
         if lhs in ("aug_assign1", "aug_assign2") and ast[0][0] == "and":
             return True

uncompyle6/parsers/parse38.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2017-2019 Rocky Bernstein
+#  Copyright (c) 2017-2020 Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -39,14 +39,17 @@ class Python38Parser(Python37Parser):
         stmt               ::= forelselaststmtl38
         stmt               ::= tryfinally38stmt
         stmt               ::= tryfinally38rstmt
+        stmt               ::= tryfinally38rstmt2
+        stmt               ::= tryfinally38rstmt3
         stmt               ::= tryfinally38astmt
         stmt               ::= try_elsestmtl38
         stmt               ::= try_except_ret38
         stmt               ::= try_except38
         stmt               ::= whilestmt38
         stmt               ::= whileTruestmt38
-        stmt               ::= call
+        stmt               ::= call_stmt
 
+        call_stmt          ::= call
         break ::= POP_BLOCK BREAK_LOOP
         break ::= POP_BLOCK POP_TOP BREAK_LOOP
         break ::= POP_TOP BREAK_LOOP
@@ -54,15 +57,12 @@ class Python38Parser(Python37Parser):
 
         # FIXME: this should be restricted to being inside a try block
         stmt               ::= except_ret38
+        stmt               ::= except_ret38a
 
         # FIXME: this should be added only when seeing GET_AITER or YIELD_FROM
-        async_for_stmt38   ::= expr
-                               GET_AITER
-                               SETUP_FINALLY
-                               GET_ANEXT
-                               LOAD_CONST
-                               YIELD_FROM
-                               POP_BLOCK
+        async_for          ::= GET_AITER _come_froms
+                               SETUP_FINALLY GET_ANEXT LOAD_CONST YIELD_FROM POP_BLOCK
+        async_for_stmt38   ::= expr async_for
                                store for_block
                                COME_FROM_FINALLY
                                END_ASYNC_FOR
@@ -81,7 +81,20 @@ class Python38Parser(Python37Parser):
                                END_ASYNC_FOR
                                else_suite
 
-        return             ::= ret_expr ROT_TWO POP_TOP RETURN_VALUE
+        # Seems to be used to discard values before a return in a "for" loop
+        discard_top        ::= ROT_TWO POP_TOP
+        discard_tops       ::= discard_top+
+
+        return             ::= ret_expr
+                               discard_tops RETURN_VALUE
+
+        return             ::= popb_return
+        return             ::= pop_return
+        return             ::= pop_ex_return
+        except_stmt        ::= pop_ex_return
+        pop_return         ::= POP_TOP ret_expr RETURN_VALUE
+        popb_return        ::= ret_expr POP_BLOCK RETURN_VALUE
+        pop_ex_return      ::= ret_expr ROT_FOUR POP_EXCEPT RETURN_VALUE
 
         # 3.8 can push a looping JUMP_BACK into into a JUMP_ from a statement that jumps to it
         lastl_stmt         ::= ifpoplaststmtl
@@ -128,8 +141,14 @@ class Python38Parser(Python37Parser):
                                except_handler38
         try_except38       ::= SETUP_FINALLY POP_BLOCK POP_TOP suite_stmts_opt
                                except_handler38a
-        try_except_ret38   ::= SETUP_FINALLY expr POP_BLOCK
-                               RETURN_VALUE except_ret38a
+
+        # suite_stmts has a return
+        try_except38       ::= SETUP_FINALLY POP_BLOCK suite_stmts
+                               except_handler38b
+
+        try_except_ret38   ::= SETUP_FINALLY returns except_ret38a
+        try_except_ret38a  ::= SETUP_FINALLY returns except_handler38c
+                               END_FINALLY come_from_opt
 
         # Note: there is a suite_stmts_opt which seems
         # to be bookkeeping which is not expressed in source code
@@ -149,19 +168,42 @@ class Python38Parser(Python37Parser):
         tryfinallystmt     ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                BEGIN_FINALLY COME_FROM_FINALLY suite_stmts_opt
                                END_FINALLY
-        tryfinally38rstmt  ::= SETUP_FINALLY POP_BLOCK CALL_FINALLY
+
+
+        lc_setup_finally   ::= LOAD_CONST SETUP_FINALLY
+        call_finally_pt    ::= CALL_FINALLY POP_TOP
+        cf_cf_finally      ::= come_from_opt COME_FROM_FINALLY
+        pop_finally_pt     ::= POP_FINALLY POP_TOP
+        ss_end_finally     ::= suite_stmts END_FINALLY
+        sf_pb_call_returns ::= SETUP_FINALLY POP_BLOCK CALL_FINALLY returns
+
+
+        # FIXME: DRY rules below
+        tryfinally38rstmt  ::= sf_pb_call_returns
+                               cf_cf_finally
+                               ss_end_finally
+        tryfinally38rstmt  ::= sf_pb_call_returns
+                               cf_cf_finally END_FINALLY
+                               suite_stmts
+        tryfinally38rstmt  ::= sf_pb_call_returns
+                               cf_cf_finally POP_FINALLY
+                               ss_end_finally
+        tryfinally38rstmt  ::= sf_bp_call_returns
+                               COME_FROM_FINALLY POP_FINALLY
+                               ss_end_finally
+
+        tryfinally38rstmt2 ::= lc_setup_finally POP_BLOCK call_finally_pt
                                returns
-                               COME_FROM_FINALLY END_FINALLY suite_stmts
-        tryfinally38rstmt  ::= SETUP_FINALLY POP_BLOCK CALL_FINALLY
-                               returns
-                               COME_FROM_FINALLY POP_FINALLY returns
-                               END_FINALLY
-        tryfinally38stmt   ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
-                               BEGIN_FINALLY COME_FROM_FINALLY
-                               POP_FINALLY suite_stmts_opt END_FINALLY
+                               cf_cf_finally pop_finally_pt
+                               ss_end_finally POP_TOP
+        tryfinally38rstmt3 ::= SETUP_FINALLY expr POP_BLOCK CALL_FINALLY RETURN_VALUE
+                               COME_FROM COME_FROM_FINALLY
+                               ss_end_finally
+
         tryfinally38stmt   ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                BEGIN_FINALLY COME_FROM_FINALLY
                                POP_FINALLY suite_stmts_opt END_FINALLY
+
         tryfinally38astmt  ::= LOAD_CONST SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                BEGIN_FINALLY COME_FROM_FINALLY
                                POP_FINALLY POP_TOP suite_stmts_opt END_FINALLY POP_TOP

uncompyle6/parsers/reducecheck/ifelsestmt.py

@@ -33,13 +33,23 @@ IFELSE_STMT_RULES = frozenset(
                 "else_suitec",
             ),
         ),
+        (
+            "ifelsestmtc",
+            (
+                "testexpr",
+                "c_stmts_opt",
+                "jump_forward_else",
+                "else_suitec",
+                "\\e__come_froms",
+            ),
+        ),
         (
             "ifelsestmtc",
             (
                 "testexpr",
                 "c_stmts_opt",
                 "jump_absolute_else",
-                "else_suitec"
+                "else_suitec",
             ),
         ),
         (
@@ -94,6 +104,7 @@ def ifelsestmt(self, lhs, n, rule, ast, tokens, first, last):
         return True
 
     if rule not in IFELSE_STMT_RULES:
+        # print("XXX", rule)
         return False
 
     # Avoid if/else where the "then" is a "raise_stmt1" for an

uncompyle6/parsers/reducecheck/or_check.py

@@ -2,7 +2,15 @@
 
 ASSERT_OPS = frozenset(["LOAD_ASSERT", "RAISE_VARARGS_1"])
 def or_check(self, lhs, n, rule, ast, tokens, first, last):
-    if rule == ("or", ("expr", "jmp_true", "expr")):
+    rhs = rule[1]
+
+    # print("XXX", first, last, rule)
+    # for t in range(first, last): print(tokens[t])
+    # print("="*40)
+
+    if rhs[0:2] in (("expr_jt", "expr"),
+                    ("expr_jitop", "expr"),
+                    ("expr_jit", "expr")):
         if tokens[last] in  ASSERT_OPS or tokens[last-1] in ASSERT_OPS:
             return True
 
@@ -18,21 +26,40 @@ def or_check(self, lhs, n, rule, ast, tokens, first, last):
             return True
 
         first_offset = tokens[first].off2int()
-        jmp_true_target = ast[1][0].attr
+        expr_jt = ast[0]
+
+        if expr_jt == "expr_jitop":
+            jump_true = expr_jt[1]
+        else:
+            jump_true = expr_jt[1][0]
+
+        jmp_true_target = jump_true.attr
+
+        last_token = tokens[last]
+        last_token_offset = last_token.off2int()
+
+        # FIXME: use instructions for all of this
         if jmp_true_target < first_offset:
             return False
+        elif jmp_true_target < last_token_offset:
+            return True
 
-        jmp_false = tokens[last]
         # If the jmp is backwards
-        if jmp_false == "POP_JUMP_IF_FALSE":
-            jmp_false_offset = jmp_false.off2int()
-            if jmp_false.attr < jmp_false_offset:
+        if last_token == "POP_JUMP_IF_FALSE" and not self.version in (2.7, 3.5, 3.6):
+            if last_token.attr < last_token_offset:
                 # For a backwards loop, well compare to the instruction *after*
                 # then POP_JUMP...
-                jmp_false = tokens[last + 1]
+                last_token = tokens[last + 1]
+            # HACK alert 3 is the Python < 3.6ish thing.
+            # Convert to using instructions
             return not (
-                (jmp_false_offset <= jmp_true_target <= jmp_false_offset + 2)
+                (last_token_offset <= jmp_true_target <= last_token_offset + 3)
                 or jmp_true_target < tokens[first].off2int()
             )
+        elif last_token == "JUMP_FORWARD" and expr_jt.kind != "expr_jitop":
+            # "or" has to fall through to the next statement
+            # FIXME: use instructions for all of this
+            return True
+
 
     return False

uncompyle6/parsers/treenode.py

@@ -6,6 +6,7 @@ from spark_parser.ast import AST as spark_AST
 if PYTHON3:
     intern = sys.intern
 
+
 class SyntaxTree(spark_AST):
     def __init__(self, *args, **kwargs):
         super(SyntaxTree, self).__init__(*args, **kwargs)
@@ -17,7 +18,7 @@ class SyntaxTree(spark_AST):
         return len(self.data) == 1 and NoneToken == self.data[0]
 
     def __repr__(self):
-        return self.__repr1__('', None)
+        return self.__repr1__("", None)
 
     def __repr1__(self, indent, sibNum=None):
         rv = str(self.kind)
@@ -33,16 +34,16 @@ class SyntaxTree(spark_AST):
             else:
                 rv += " (transformed by %s)" % self.transformed_by
         rv = indent + rv
-        indent += '    '
+        indent += "    "
         i = 0
         for node in self:
-            if hasattr(node, '__repr1__'):
+            if hasattr(node, "__repr1__"):
                 if enumerate_children:
-                    child =  node.__repr1__(indent, i)
+                    child = node.__repr1__(indent, i)
                 else:
                     child = node.__repr1__(indent, None)
             else:
-                inst = node.format(line_prefix='L.')
+                inst = node.format(line_prefix="")
                 if inst.startswith("\n"):
                     # Nuke leading \n
                     inst = inst[1:]

uncompyle6/scanner.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016, 2018-2019 by Rocky Bernstein
+#  Copyright (c) 2016, 2018-2020 by Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
@@ -30,9 +30,7 @@ import sys
 from uncompyle6 import PYTHON3, IS_PYPY
 from uncompyle6.scanners.tok import Token
 import xdis
-from xdis.bytecode import Bytecode, instruction_size, extended_arg_val, next_offset
-from xdis.magics import canonic_python_version
-from xdis.util import code2num
+from xdis import Bytecode, canonic_python_version, code2num, instruction_size, extended_arg_val, next_offset
 
 # The byte code versions we support.
 # Note: these all have to be floats

uncompyle6/scanners/pypy33.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019 by Rocky Bernstein
+#  Copyright (c) 2019-2020 by Rocky Bernstein
 """
 Python PyPy 3.3 decompiler scanner.
 
@@ -10,6 +10,7 @@ import uncompyle6.scanners.scanner33 as scan
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_33pypy as opc
+
 JUMP_OPs = map(lambda op: opc.opname[op], opc.hasjrel + opc.hasjabs)
 
 # We base this off of 3.3

uncompyle6/scanners/scanner26.py

@@ -279,7 +279,7 @@ class Scanner26(scan.Scanner2):
 
         if show_asm in ('both', 'after'):
             for t in tokens:
-                print(t.format(line_prefix='L.'))
+                print(t.format(line_prefix=""))
             print()
         return tokens, customize
 

uncompyle6/scanners/scanner3.py

@@ -35,8 +35,8 @@ Finally we save token information.
 
 from __future__ import print_function
 
-from xdis.code import iscode
-from xdis.bytecode import instruction_size, _get_const_info
+from xdis import iscode, instruction_size
+from xdis.bytecode import _get_const_info
 
 from uncompyle6.scanner import Token, parse_fn_counts
 import xdis
@@ -379,7 +379,7 @@ class Scanner3(Scanner):
                     # pattr = 'code_object @ 0x%x %s->%s' %\
                     # (id(const), const.co_filename, const.co_name)
                     pattr = "<code_object " + const.co_name + ">"
-                elif isinstance(const, str):
+                elif isinstance(const, str) or xdis.PYTHON_VERSION <= 2.7 and isinstance(const, unicode):
                     opname = "LOAD_STR"
                 else:
                     if isinstance(inst.arg, int) and inst.arg < len(co.co_consts):
@@ -539,7 +539,7 @@ class Scanner3(Scanner):
 
         if show_asm in ("both", "after"):
             for t in tokens:
-                print(t.format(line_prefix="L."))
+                print(t.format(line_prefix=""))
             print()
         return tokens, customize
 
@@ -888,6 +888,7 @@ class Scanner3(Scanner):
                     start, self.next_stmt[offset], self.opc.POP_JUMP_IF_FALSE, target
                 )
 
+                # FIXME: Remove this whole "if" block
                 # If we still have any offsets in set, start working on it
                 if match:
                     is_jump_forward = self.is_jump_forward(pre_rtarget)
@@ -956,7 +957,7 @@ class Scanner3(Scanner):
                             )
                         ):
                             pass
-                        else:
+                        elif self.version <= 3.2:
                             fix = None
                             jump_ifs = self.inst_matches(
                                 start,

uncompyle6/scanners/scanner30.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016, 2017 by Rocky Bernstein
+#  Copyright (c) 2016-2017, 2020 by Rocky Bernstein
 """
 Python 3.0 bytecode scanner/deparser
 
@@ -10,17 +10,19 @@ from __future__ import print_function
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_30 as opc
-from xdis.bytecode import instruction_size
+from xdis import instruction_size
 import xdis
 
 JUMP_TF = frozenset([opc.JUMP_IF_FALSE, opc.JUMP_IF_TRUE])
 
 from uncompyle6.scanners.scanner3 import Scanner3
-class Scanner30(Scanner3):
 
+
+class Scanner30(Scanner3):
     def __init__(self, show_asm=None, is_pypy=False):
         Scanner3.__init__(self, 3.0, show_asm, is_pypy)
         return
+
     pass
 
     def detect_control_flow(self, offset, targets, inst_index):
@@ -35,17 +37,18 @@ class Scanner30(Scanner3):
 
         # Detect parent structure
         parent = self.structs[0]
-        start  = parent['start']
-        end    = parent['end']
+        start = parent["start"]
+        end = parent["end"]
 
         # Pick inner-most parent for our offset
         for struct in self.structs:
-            current_start = struct['start']
-            current_end   = struct['end']
-            if ((current_start <= offset < current_end)
-                and (current_start >= start and current_end <= end)):
-                start  = current_start
-                end    = current_end
+            current_start = struct["start"]
+            current_end = struct["end"]
+            if (current_start <= offset < current_end) and (
+                current_start >= start and current_end <= end
+            ):
+                start = current_start
+                end = current_end
                 parent = struct
 
         if op == self.opc.SETUP_LOOP:
@@ -56,28 +59,35 @@ class Scanner30(Scanner3):
 
             start += instruction_size(op, self.opc)
             target = self.get_target(offset)
-            end    = self.restrict_to_parent(target, parent)
+            end = self.restrict_to_parent(target, parent)
             self.setup_loops[target] = offset
 
             if target != end:
                 self.fixed_jumps[offset] = end
 
             (line_no, next_line_byte) = self.lines[offset]
-            jump_back = self.last_instr(start, end, self.opc.JUMP_ABSOLUTE,
-                                            next_line_byte, False)
+            jump_back = self.last_instr(
+                start, end, self.opc.JUMP_ABSOLUTE, next_line_byte, False
+            )
 
             if jump_back:
-                jump_forward_offset = xdis.next_offset(code[jump_back], self.opc, jump_back)
+                jump_forward_offset = xdis.next_offset(
+                    code[jump_back], self.opc, jump_back
+                )
             else:
                 jump_forward_offset = None
 
             return_val_offset1 = self.prev[self.prev[end]]
 
-            if (jump_back and jump_back != self.prev_op[end]
-                and self.is_jump_forward(jump_forward_offset)):
-                if (code[self.prev_op[end]] == self.opc.RETURN_VALUE or
-                    (code[self.prev_op[end]] == self.opc.POP_BLOCK
-                     and code[return_val_offset1] == self.opc.RETURN_VALUE)):
+            if (
+                jump_back
+                and jump_back != self.prev_op[end]
+                and self.is_jump_forward(jump_forward_offset)
+            ):
+                if code[self.prev_op[end]] == self.opc.RETURN_VALUE or (
+                    code[self.prev_op[end]] == self.opc.POP_BLOCK
+                    and code[return_val_offset1] == self.opc.RETURN_VALUE
+                ):
                     jump_back = None
             if not jump_back:
                 # loop suite ends in return
@@ -92,56 +102,63 @@ class Scanner30(Scanner3):
                 if code[self.prev_op[next_line_byte]] not in JUMP_TF:
                     if_offset = self.prev[next_line_byte]
                 if if_offset:
-                    loop_type = 'while'
+                    loop_type = "while"
                     self.ignore_if.add(if_offset)
                 else:
-                    loop_type = 'for'
+                    loop_type = "for"
                 target = next_line_byte
                 end = jump_back + 3
             else:
                 if self.get_target(jump_back) >= next_line_byte:
-                    jump_back = self.last_instr(start, end, self.opc.JUMP_ABSOLUTE, start, False)
+                    jump_back = self.last_instr(
+                        start, end, self.opc.JUMP_ABSOLUTE, start, False
+                    )
 
                 jb_inst = self.get_inst(jump_back)
 
                 jb_next_offset = self.next_offset(jb_inst.opcode, jump_back)
                 if end > jb_next_offset and self.is_jump_forward(end):
                     if self.is_jump_forward(jb_next_offset):
-                        if self.get_target(jump_back+4) == self.get_target(end):
-                            self.fixed_jumps[offset] = jump_back+4
+                        if self.get_target(jump_back + 4) == self.get_target(end):
+                            self.fixed_jumps[offset] = jump_back + 4
                             end = jb_next_offset
                 elif target < offset:
-                    self.fixed_jumps[offset] = jump_back+4
+                    self.fixed_jumps[offset] = jump_back + 4
                     end = jb_next_offset
 
                 target = self.get_target(jump_back)
 
                 if code[target] in (self.opc.FOR_ITER, self.opc.GET_ITER):
-                    loop_type = 'for'
+                    loop_type = "for"
                 else:
-                    loop_type = 'while'
+                    loop_type = "while"
                     test = self.prev_op[next_line_byte]
 
                     if test == offset:
-                        loop_type = 'while 1'
+                        loop_type = "while 1"
                     elif self.code[test] in self.opc.JUMP_OPs:
                         self.ignore_if.add(test)
                         test_target = self.get_target(test)
-                        if test_target > (jump_back+3):
+                        if test_target > (jump_back + 3):
                             jump_back = test_target
                 self.not_continue.add(jump_back)
             self.loops.append(target)
-            self.structs.append({'type': loop_type + '-loop',
-                                 'start': target,
-                                 'end':   jump_back})
+            self.structs.append(
+                {"type": loop_type + "-loop", "start": target, "end": jump_back}
+            )
             after_jump_offset = xdis.next_offset(code[jump_back], self.opc, jump_back)
-            if (self.get_inst(after_jump_offset).opname == 'POP_TOP'):
-                after_jump_offset = xdis.next_offset(code[after_jump_offset], self.opc,
-                                                     after_jump_offset)
+            if self.get_inst(after_jump_offset).opname == "POP_TOP":
+                after_jump_offset = xdis.next_offset(
+                    code[after_jump_offset], self.opc, after_jump_offset
+                )
             if after_jump_offset != end:
-                self.structs.append({'type': loop_type + '-else',
-                                     'start': after_jump_offset,
-                                     'end':   end})
+                self.structs.append(
+                    {
+                        "type": loop_type + "-else",
+                        "start": after_jump_offset,
+                        "end": end,
+                    }
+                )
         elif op in self.pop_jump_tf:
             start = offset + instruction_size(op, self.opc)
             target = self.get_target(offset)
@@ -149,7 +166,7 @@ class Scanner30(Scanner3):
             prev_op = self.prev_op
 
             # Do not let jump to go out of parent struct bounds
-            if target != rtarget and parent['type'] == 'and/or':
+            if target != rtarget and parent["type"] == "and/or":
                 self.fixed_jumps[offset] = rtarget
                 return
 
@@ -158,12 +175,15 @@ class Scanner30(Scanner3):
             # rocky: if we have a conditional jump to the next instruction, then
             # possibly I am "skipping over" a "pass" or null statement.
 
-            if ((code[prev_op[target]] in self.pop_jump_if_pop) and
-                (target > offset) and prev_op[target] != offset):
+            if (
+                (code[prev_op[target]] in self.pop_jump_if_pop)
+                and (target > offset)
+                and prev_op[target] != offset
+            ):
                 self.fixed_jumps[offset] = prev_op[target]
-                self.structs.append({'type': 'and/or',
-                                     'start': start,
-                                     'end': prev_op[target]})
+                self.structs.append(
+                    {"type": "and/or", "start": start, "end": prev_op[target]}
+                )
                 return
 
             # The op offset just before the target jump offset is important
@@ -176,35 +196,80 @@ class Scanner30(Scanner3):
                 # Search for another JUMP_IF_FALSE targetting the same op,
                 # in current statement, starting from current offset, and filter
                 # everything inside inner 'or' jumps and midline ifs
-                match = self.rem_or(start, self.next_stmt[offset],
-                                    opc.JUMP_IF_FALSE, target)
+                match = self.rem_or(
+                    start, self.next_stmt[offset], opc.JUMP_IF_FALSE, target
+                )
 
                 # If we still have any offsets in set, start working on it
                 if match:
                     is_jump_forward = self.is_jump_forward(pre_rtarget)
-                    if (is_jump_forward and pre_rtarget not in self.stmts and
-                        self.restrict_to_parent(self.get_target(pre_rtarget), parent) == rtarget):
-                        if (code[prev_op[pre_rtarget]] == self.opc.JUMP_ABSOLUTE
-                            and self.remove_mid_line_ifs([offset]) and
-                            target == self.get_target(prev_op[pre_rtarget]) and
-                            (prev_op[pre_rtarget] not in self.stmts or
-                             self.get_target(prev_op[pre_rtarget]) > prev_op[pre_rtarget]) and
-                            1 == len(self.remove_mid_line_ifs(self.rem_or(start, prev_op[pre_rtarget], JUMP_TF, target)))):
+                    if (
+                        is_jump_forward
+                        and pre_rtarget not in self.stmts
+                        and self.restrict_to_parent(
+                            self.get_target(pre_rtarget), parent
+                        )
+                        == rtarget
+                    ):
+                        if (
+                            code[prev_op[pre_rtarget]] == self.opc.JUMP_ABSOLUTE
+                            and self.remove_mid_line_ifs([offset])
+                            and target == self.get_target(prev_op[pre_rtarget])
+                            and (
+                                prev_op[pre_rtarget] not in self.stmts
+                                or self.get_target(prev_op[pre_rtarget])
+                                > prev_op[pre_rtarget]
+                            )
+                            and 1
+                            == len(
+                                self.remove_mid_line_ifs(
+                                    self.rem_or(
+                                        start, prev_op[pre_rtarget], JUMP_TF, target
+                                    )
+                                )
+                            )
+                        ):
                             pass
-                        elif (code[prev_op[pre_rtarget]] == self.opc.RETURN_VALUE
-                              and self.remove_mid_line_ifs([offset]) and
-                              1 == (len(set(self.remove_mid_line_ifs(self.rem_or(start, prev_op[pre_rtarget],
-                                                                                 JUMP_TF, target))) |
-                                    set(self.remove_mid_line_ifs(self.rem_or(start, prev_op[pre_rtarget],
-                                                                             (opc.JUMP_IF_FALSE,
-                                                                              opc.JUMP_IF_TRUE,
-                                                                              opc.JUMP_ABSOLUTE),
-                                                                             pre_rtarget, True)))))):
+                        elif (
+                            code[prev_op[pre_rtarget]] == self.opc.RETURN_VALUE
+                            and self.remove_mid_line_ifs([offset])
+                            and 1
+                            == (
+                                len(
+                                    set(
+                                        self.remove_mid_line_ifs(
+                                            self.rem_or(
+                                                start,
+                                                prev_op[pre_rtarget],
+                                                JUMP_TF,
+                                                target,
+                                            )
+                                        )
+                                    )
+                                    | set(
+                                        self.remove_mid_line_ifs(
+                                            self.rem_or(
+                                                start,
+                                                prev_op[pre_rtarget],
+                                                (
+                                                    opc.JUMP_IF_FALSE,
+                                                    opc.JUMP_IF_TRUE,
+                                                    opc.JUMP_ABSOLUTE,
+                                                ),
+                                                pre_rtarget,
+                                                True,
+                                            )
+                                        )
+                                    )
+                                )
+                            )
+                        ):
                             pass
                         else:
                             fix = None
-                            jump_ifs = self.inst_matches(start, self.next_stmt[offset],
-                                                         opc.JUMP_IF_FALSE)
+                            jump_ifs = self.inst_matches(
+                                start, self.next_stmt[offset], opc.JUMP_IF_FALSE
+                            )
                             last_jump_good = True
                             for j in jump_ifs:
                                 if target == self.get_target(j):
@@ -226,14 +291,19 @@ class Scanner30(Scanner3):
                     pass
                 elif self.is_jump_forward(next) and target == self.get_target(next):
                     if code[prev_op[next]] == opc.JUMP_IF_FALSE:
-                        if (code[next] == self.opc.JUMP_FORWARD
+                        if (
+                            code[next] == self.opc.JUMP_FORWARD
                             or target != rtarget
-                            or code[prev_op[pre_rtarget]] not in
-                            (self.opc.JUMP_ABSOLUTE, self.opc.RETURN_VALUE)):
+                            or code[prev_op[pre_rtarget]]
+                            not in (self.opc.JUMP_ABSOLUTE, self.opc.RETURN_VALUE)
+                        ):
                             self.fixed_jumps[offset] = prev_op[next]
                             return
-                elif (code[next] == self.opc.JUMP_ABSOLUTE and self.is_jump_forward(target) and
-                      self.get_target(target) == self.get_target(next)):
+                elif (
+                    code[next] == self.opc.JUMP_ABSOLUTE
+                    and self.is_jump_forward(target)
+                    and self.get_target(target) == self.get_target(next)
+                ):
                     self.fixed_jumps[offset] = prev_op[next]
                     return
 
@@ -241,13 +311,17 @@ class Scanner30(Scanner3):
             if offset in self.ignore_if:
                 return
 
-            if (code[pre_rtarget] == self.opc.JUMP_ABSOLUTE and
-                pre_rtarget in self.stmts and
-                pre_rtarget != offset and
-                prev_op[pre_rtarget] != offset and
-                not (code[rtarget] == self.opc.JUMP_ABSOLUTE and
-                     code[rtarget+3] == self.opc.POP_BLOCK and
-                     code[prev_op[pre_rtarget]] != self.opc.JUMP_ABSOLUTE)):
+            if (
+                code[pre_rtarget] == self.opc.JUMP_ABSOLUTE
+                and pre_rtarget in self.stmts
+                and pre_rtarget != offset
+                and prev_op[pre_rtarget] != offset
+                and not (
+                    code[rtarget] == self.opc.JUMP_ABSOLUTE
+                    and code[rtarget + 3] == self.opc.POP_BLOCK
+                    and code[prev_op[pre_rtarget]] != self.opc.JUMP_ABSOLUTE
+                )
+            ):
                 rtarget = pre_rtarget
 
             # Does the "jump if" jump beyond a jump op?
@@ -268,16 +342,17 @@ class Scanner30(Scanner3):
                 if_end = self.get_target(pre_rtarget, 0)
 
                 # If the jump target is back, we are looping
-                if (if_end < pre_rtarget and
-                    (code[prev_op[if_end]] == self.opc.SETUP_LOOP)):
-                    if (if_end > start):
+                if if_end < pre_rtarget and (
+                    code[prev_op[if_end]] == self.opc.SETUP_LOOP
+                ):
+                    if if_end > start:
                         return
 
                 end = self.restrict_to_parent(if_end, parent)
 
-                self.structs.append({'type': 'if-then',
-                                     'start': start,
-                                     'end': pre_rtarget})
+                self.structs.append(
+                    {"type": "if-then", "start": start, "end": pre_rtarget}
+                )
                 self.not_continue.add(pre_rtarget)
 
                 # if rtarget < end and (
@@ -291,20 +366,17 @@ class Scanner30(Scanner3):
                 #     self.else_start[rtarget] = end
             elif self.is_jump_back(pre_rtarget, 0):
                 if_end = rtarget
-                self.structs.append({'type': 'if-then',
-                                     'start': start,
-                                     'end': pre_rtarget})
+                self.structs.append(
+                    {"type": "if-then", "start": start, "end": pre_rtarget}
+                )
                 self.not_continue.add(pre_rtarget)
-            elif code[pre_rtarget] in (self.opc.RETURN_VALUE,
-                                       self.opc.BREAK_LOOP):
-                self.structs.append({'type': 'if-then',
-                                     'start': start,
-                                     'end': rtarget})
+            elif code[pre_rtarget] in (self.opc.RETURN_VALUE, self.opc.BREAK_LOOP):
+                self.structs.append({"type": "if-then", "start": start, "end": rtarget})
                 # It is important to distingish if this return is inside some sort
                 # except block return
                 jump_prev = prev_op[offset]
                 if self.is_pypy and code[jump_prev] == self.opc.COMPARE_OP:
-                    if self.opc.cmp_op[code[jump_prev+1]] == 'exception-match':
+                    if self.opc.cmp_op[code[jump_prev + 1]] == "exception-match":
                         return
                 if self.version >= 3.5:
                     # Python 3.5 may remove as dead code a JUMP
@@ -332,7 +404,10 @@ class Scanner30(Scanner3):
                         if code[next_op] == self.opc.POP_TOP:
                             next_op = rtarget
                         for block in self.structs:
-                            if block['type'] == 'while-loop' and block['end'] == next_op:
+                            if (
+                                block["type"] == "while-loop"
+                                and block["end"] == next_op
+                            ):
                                 return
                         next_op += instruction_size(self.code[next_op], self.opc)
                         if code[next_op] == self.opc.POP_BLOCK:
@@ -342,20 +417,21 @@ class Scanner30(Scanner3):
                     self.fixed_jumps[offset] = rtarget
                     self.not_continue.add(pre_rtarget)
 
-
         elif op == self.opc.SETUP_EXCEPT:
             target = self.get_target(offset)
-            end    = self.restrict_to_parent(target, parent)
+            end = self.restrict_to_parent(target, parent)
             self.fixed_jumps[offset] = end
         elif op == self.opc.SETUP_FINALLY:
             target = self.get_target(offset)
-            end    = self.restrict_to_parent(target, parent)
+            end = self.restrict_to_parent(target, parent)
             self.fixed_jumps[offset] = end
         elif op in self.jump_if_pop:
             target = self.get_target(offset)
             if target > offset:
-                unop_target = self.last_instr(offset, target, self.opc.JUMP_FORWARD, target)
-                if unop_target and code[unop_target+3] != self.opc.ROT_TWO:
+                unop_target = self.last_instr(
+                    offset, target, self.opc.JUMP_FORWARD, target
+                )
+                if unop_target and code[unop_target + 3] != self.opc.ROT_TWO:
                     self.fixed_jumps[offset] = unop_target
                 else:
                     self.fixed_jumps[offset] = self.restrict_to_parent(target, parent)
@@ -366,8 +442,11 @@ class Scanner30(Scanner3):
             # misclassified as RETURN_END_IF. Handle that here.
             # In RETURN_VALUE, JUMP_ABSOLUTE, RETURN_VALUE is never RETURN_END_IF
             if op == self.opc.RETURN_VALUE:
-                if (offset+1 < len(code) and code[offset+1] == self.opc.JUMP_ABSOLUTE and
-                    offset in self.return_end_ifs):
+                if (
+                    offset + 1 < len(code)
+                    and code[offset + 1] == self.opc.JUMP_ABSOLUTE
+                    and offset in self.return_end_ifs
+                ):
                     self.return_end_ifs.remove(offset)
                     pass
                 pass
@@ -377,8 +456,10 @@ class Scanner30(Scanner3):
                 # then RETURN_VALUE is not RETURN_END_IF
                 rtarget = self.get_target(offset)
                 rtarget_prev = self.prev[rtarget]
-                if (code[rtarget_prev] == self.opc.RETURN_VALUE and
-                    rtarget_prev in self.return_end_ifs):
+                if (
+                    code[rtarget_prev] == self.opc.RETURN_VALUE
+                    and rtarget_prev in self.return_end_ifs
+                ):
                     i = rtarget_prev
                     while i != offset:
                         if code[i] in [opc.JUMP_FORWARD, opc.JUMP_ABSOLUTE]:
@@ -388,15 +469,17 @@ class Scanner30(Scanner3):
                 pass
         return
 
+
 if __name__ == "__main__":
     from uncompyle6 import PYTHON_VERSION
+
     if PYTHON_VERSION == 3.0:
         import inspect
+
         co = inspect.currentframe().f_code
         tokens, customize = Scanner30().ingest(co)
         for t in tokens:
             print(t)
         pass
     else:
-        print("Need to be Python 3.0 to demo; I am %s." %
-              PYTHON_VERSION)
+        print("Need to be Python 3.0 to demo; I am %s." % PYTHON_VERSION)

uncompyle6/scanners/scanner37base.py

@@ -29,8 +29,8 @@ For example:
 Finally we save token information.
 """
 
-from xdis.code import iscode
-from xdis.bytecode import instruction_size, _get_const_info, Instruction
+from xdis import iscode, instruction_size, Instruction
+from xdis.bytecode import _get_const_info
 
 from uncompyle6.scanner import Token
 import xdis
@@ -518,7 +518,7 @@ class Scanner37Base(Scanner):
 
         if show_asm in ("both", "after"):
             for t in tokens:
-                print(t.format(line_prefix="L."))
+                print(t.format(line_prefix=""))
             print()
         return tokens, customize
 

uncompyle6/scanners/tok.py

@@ -120,12 +120,19 @@ class Token:
     def __str__(self):
         return self.format(line_prefix="")
 
-    def format(self, line_prefix=""):
-        prefix = (
-            "\n%s%4d  " % (line_prefix, self.linestart)
-            if self.linestart
-            else (" " * (6 + len(line_prefix)))
-        )
+    def format(self, line_prefix="", token_num=None):
+        if token_num is not None:
+            prefix = (
+                "\n(%03d)%s L.%4d  " % (token_num, line_prefix, self.linestart)
+                if self.linestart
+                else ("(%03d)%s" % (token_num, " " * (9 + len(line_prefix))))
+            )
+        else:
+            prefix = (
+                "\n%s L.%4d  " % (line_prefix, self.linestart)
+                if self.linestart
+                else (" " * (9 + len(line_prefix)))
+            )
         offset_opname = "%8s  %-17s" % (self.offset, self.kind)
 
         if not self.has_arg:

uncompyle6/semantics/Makefile

@@ -1,7 +0,0 @@
-# Whatever it is you want to do, it should be forwarded to the
-# to top-level irectories
-PHONY=check all
-all: check
-
-%:
-	$(MAKE) -C ../.. $@