Merge branch 'master' into python-2.4

As seen in _cmp() of python3.8/distutils/version.py with optimization -O2

.circleci/config.yml

@@ -31,7 +31,6 @@ jobs:
           # fallback to using the latest cache if no exact match is found
           - v2-dependencies-
 
-    # This is based on your 1.0 configuration file or project settings
     - run:
         command: sudo easy_install xdis spark-parser && sudo pip install -e . && sudo pip install -r requirements-dev.txt
 

.github/FUNDING.yml

@@ -0,0 +1,12 @@
+# These are supported funding model platforms
+
+github: [rocky]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

.gitignore

@@ -10,6 +10,7 @@
 /.pytest_cache
 /.python-version
 /.tox
+.mypy_cache
 /.venv*
 /README
 /__pkginfo__.pyc

NEWS.md

@@ -1,3 +1,50 @@
+3.7.1: 2020-6-12 Fleetwood66
+====================================================
+
+Released to pick up new xdis version which has fixes to read bytestings better on 3.x
+
+* Handle 3.7+ "else" branch removal adAs seen in `_cmp()` of `python3.8/distutils/version.py` with optimization `-O2`
+* 3.6+ "with" and "with .. as" grammar improvements
+* ast-check for "for" loop was missing some grammar rules
+
+3.7.0: 2020-5-19 Primidi 1st Prairial - Alfalfa - HF
+====================================================
+
+The main impetus for this release is to pull in the recent changes from xdis.
+We simplify imports using xdis 4.6.0.
+
+There were some bugfixes to Python 3.4-3.8. See the ChangeLog for details
+
+
+3.6.7: 2020-4-27 xdis again
+===========================
+
+More upheaval in xdis which we need to track here.
+
+3.6.6: 2020-4-20 Love in the time of Cholera
+============================================
+
+The main reason for this release is an incompatablity bump in xdis which handles
+3.7 SipHash better.
+
+* Go over "yield" as an expression precidence
+* Some small alignment with code in decompyle3 for "or" and "and" was done
+
+
+3.6.5: 2020-4-1 April Fool
+==========================
+
+Back port some of the changes in decompile3 here which mostly helps 3.7 and 3.8 decompilation, although this may also help 3.6ish versions too.
+
+- Handle nested `async for in for...`  and better async comprehension detection via `xdis`.  Still more work is needed.
+- include token number in listings when `-g` and there is a parser error
+- remove unneeded `Makefile`s now that remake 4.3+1.5dbg is a thing that has `-c`
+- Bug in finding annotations in functions with docstrings
+- Fix bug found by 2.4 sre_parse.py testing
+- Fix `transform` module's  `ifelseif` bugs
+- Fix bug in 3.0 name module detection
+- Fix docstring detection
+
 3.6.4: 2020-2-9 Plateau
 =======================
 

README.rst

@@ -54,7 +54,7 @@ only; another patched that and handled only 3.3.  You get the
 idea. This code pulls all of these forks together and *moves
 forward*. There is some serious refactoring and cleanup in this code
 base over those old forks. Even more experimental refactoring is going
-on in decompile3_.
+on in decompyle3_.
 
 This demonstrably does the best in decompiling Python across all
 Python versions. And even when there is another project that only
@@ -210,7 +210,7 @@ however that the magic of a released version is usually the same as
 the *last* candidate version prior to release.
 
 There are also customized Python interpreters, notably Dropbox,
-which use their own magic and encrypt bytcode. With the exception of
+which use their own magic and encrypt bytecode. With the exception of
 the Dropbox's old Python 2.5 interpreter this kind of thing is not
 handled.
 
@@ -229,7 +229,7 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back ehre.
+* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back here.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
 * https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details

__pkginfo__.py

@@ -21,6 +21,17 @@
 # less elegant than having it here with reduced code, albeit there
 # still is some room for improvement.
 
+# Python-version | package | last-version |
+# -----------------------------------------
+# 2.5            | pip     |  1.1         |
+# 2.6            | pip     |  1.5.6       |
+# 2.7            | pip     | 19.2.3       |
+# 2.7            | pip     |  1.2.1       |
+# 3.1            | pip     |  1.5.6       |
+# 3.2            | pip     |  7.1.2       |
+# 3.3            | pip     | 10.0.1       |
+# 3.4            | pip     | 19.1.1       |
+
 # Things that change more often go here.
 copyright   = """
 Copyright (C) 2015-2020 Rocky Bernstein <rb@dustyfeet.com>.
@@ -58,7 +69,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ["spark-parser >= 1.8.9, < 1.9.0",
-                      "xdis >= 4.2.2, < 4.3.0"]
+                      "xdis >= 4.6.1, < 4.8.0"]
 
 license            = "GPL3"
 mailing_list       = "python-debugger@googlegroups.com"

admin-tools/how-to-make-a-release.md

@@ -55,11 +55,10 @@
 # Make packages and tag
 
     $ . ./admin-tools/make-dist-older.sh
-	$ pyenv local 3.8.1
+	$ pyenv local 3.8.3
 	$ twine check dist/uncompyle6-$VERSION*
     $ git tag release-python-2.4-$VERSION
     $ . ./admin-tools/make-dist-newer.sh
-	$ twine check dist/uncompyle6-$VERSION*
 
 # Upload single package and look at Rst Formating
 

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.17 3.2.6 3.1.5 3.4.10 3.7.6 3.8.1'
+export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.18 3.2.6 3.1.5 3.4.10 3.7.7 3.8.3'

admin-tools/setup-master.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-PYTHON_VERSION=3.7.6
+PYTHON_VERSION=3.7.7
 
 # FIXME put some of the below in a common routine
 function finish {

appveyor.yml

@@ -53,6 +53,7 @@ install:
   # compiled extensions and are not provided as pre-built wheel packages,
   # pip will build them from source using the MSVC compiler matching the
   # target Python version and architecture
+  - "%CMD_IN_ENV% pip install git+git://github.com/rocky/python-uncompyle6.git#egg=uncompyle6-3.6.6"
   - "%CMD_IN_ENV% pip install -r requirements.txt"
 
 build_script:

pytest/test_token.py

@@ -1,14 +1,13 @@
-from uncompyle6 import PYTHON_VERSION
 from uncompyle6.scanners.tok import Token
 
 
 def test_token():
     # Test token formatting of: LOAD_CONST None
     t = Token("LOAD_CONST", offset=0, attr=None, pattr=None, has_arg=True)
-    expect = "             0  LOAD_CONST               None"
+    expect = "0  LOAD_CONST               None"
     # print(t.format())
     assert t
-    assert t.format() == expect
+    assert t.format().strip() == expect.strip()
 
     # Make sure equality testing of tokens ignores offset
     t2 = Token("LOAD_CONST", offset=2, attr=None, pattr=None, has_arg=True)
@@ -17,8 +16,8 @@ def test_token():
     # Make sure formatting of: LOAD_CONST False. We assume False is the 0th index
     # of co_consts.
     t = Token("LOAD_CONST", offset=1, attr=False, pattr=False, has_arg=True)
-    expect = "             1  LOAD_CONST               False"
-    assert t.format() == expect
+    expect = "1  LOAD_CONST               False"
+    assert t.format().strip() == expect.strip()
 
 
 if __name__ == "__main__":

pytest/testdata/if-2.7.right

@@ -1,12 +1,12 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_if.py
 
-   6         0  LOAD_NAME             0  'True'
-             3  POP_JUMP_IF_FALSE    15  'to 15'
+ L.   6         0  LOAD_NAME             0  'True'
+                3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   7         6  LOAD_NAME             1  'False'
-             9  STORE_NAME            2  'b'
-            12  JUMP_FORWARD          0  'to 15'
-          15_0  COME_FROM            12  '12'
-            15  LOAD_CONST               None
-            18  RETURN_VALUE
+ L.   7         6  LOAD_NAME             1  'False'
+                9  STORE_NAME            2  'b'
+               12  JUMP_FORWARD          0  'to 15'
+             15_0  COME_FROM            12  '12'
+               15  LOAD_CONST               None
+               18  RETURN_VALUE

pytest/testdata/ifelse-2.7.right

@@ -1,15 +1,15 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_ifelse.py
 
-   3         0  LOAD_NAME             0  'True'
-             3  POP_JUMP_IF_FALSE    15  'to 15'
+ L.   3         0  LOAD_NAME             0  'True'
+                3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   4         6  LOAD_CONST               1
-             9  STORE_NAME            1  'b'
-            12  JUMP_FORWARD          6  'to 21'
+ L.   4         6  LOAD_CONST               1
+                9  STORE_NAME            1  'b'
+               12  JUMP_FORWARD          6  'to 21'
 
-   6        15  LOAD_CONST               2
-            18  STORE_NAME            2  'd'
-          21_0  COME_FROM            12  '12'
-            21  LOAD_CONST               None
-            24  RETURN_VALUE
+ L.   6        15  LOAD_CONST               2
+               18  STORE_NAME            2  'd'
+             21_0  COME_FROM            12  '12'
+               21  LOAD_CONST               None
+               24  RETURN_VALUE

pytest/validate.py

@@ -12,8 +12,7 @@ import functools
 from uncompyle6 import PYTHON_VERSION, PYTHON3, IS_PYPY, code_deparse
 
 # TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
-from xdis.bytecode import Bytecode
-from xdis.main import get_opcode
+from xdis import Bytecode, get_opcode
 
 opc = get_opcode(PYTHON_VERSION, IS_PYPY)
 Bytecode = functools.partial(Bytecode, opc=opc)

setup.py

@@ -15,29 +15,40 @@ if not ((2, 4) <= SYS_VERSION  <= (2, 7)):
     print(mess)
     raise Exception(mess)
 
-from __pkginfo__ import \
-    author,           author_email,       install_requires,          \
-    license,          long_description,   classifiers,               \
-    entry_points,     modname,            py_modules,                \
-    short_desc,       VERSION,            web,                       \
-    zip_safe
+from __pkginfo__ import (
+    author,
+    author_email,
+    install_requires,
+    license,
+    long_description,
+    classifiers,
+    entry_points,
+    modname,
+    py_modules,
+    short_desc,
+    VERSION,
+    web,
+    zip_safe,
+)
 
 from setuptools import setup, find_packages
+
 setup(
-       author             = author,
-       author_email       = author_email,
-       classifiers        = classifiers,
-       description        = short_desc,
-       entry_points       = entry_points,
-       install_requires   = install_requires,
-       license            = license,
-       long_description   = long_description,
-       long_description_content_type = "text/x-rst",
-       name               = modname,
-       packages           = find_packages(),
-       py_modules         = py_modules,
-       test_suite         = 'nose.collector',
-       url                = web,
-       tests_require      = ['nose>=1.0'],
-       version            = VERSION,
-       zip_safe           = zip_safe)
+    author=author,
+    author_email=author_email,
+    classifiers=classifiers,
+    description=short_desc,
+    entry_points=entry_points,
+    install_requires=install_requires,
+    license=license,
+    long_description=long_description,
+    long_description_content_type="text/x-rst",
+    name=modname,
+    packages=find_packages(),
+    py_modules=py_modules,
+    test_suite="nose.collector",
+    url=web,
+    tests_require=["nose>=1.0"],
+    version=VERSION,
+    zip_safe=zip_safe,
+)

test/.gitignore

@@ -1,3 +1,8 @@
 /.coverage
 /.python-version
 /nohup.out
+/pycdc
+/test_pycdc_tests.sh
+/test_uncompyle2.py
+/test_unpy33.py
+/test_unpy37.py

test/Makefile

@@ -78,7 +78,7 @@ check-3.7: check-bytecode
 # 	$(PYTHON) test_pythonlib.py --bytecode-3.8 --syntax-verify $(COMPILE)
 
 # FIXME
-#: this is called when running under pypy3.5-5.8.0 or pypy2-5.6.0
+#: this is called when running under pypy3.5-5.8.0, pypy2-5.6.0, or pypy3.6-7.3.0
 5.8 5.6:
 
 #: Check deparsing only, but from a different Python version
@@ -95,12 +95,19 @@ check-bytecode-2:
         --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
 
 #: Check deparsing bytecode 3.x only
+# check-bytecode-3:
+# 	$(PYTHON) test_pythonlib.py \
+#         --bytecode-3.0 \
+# 	--bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
+# 	--bytecode-3.4 --bytecode-3.5 --bytecode-3.6 \
+#         --bytecode-3.7  \
+# 	--bytecode-pypy3.2 --bytecode-pypy3.6 --bytecode-3.8
+
+# FIXME: Until we shaked out problems with xdis...
 check-bytecode-3:
-	$(PYTHON) test_pythonlib.py --bytecode-3.0 \
-	--bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
+	$(PYTHON) test_pythonlib.py \
 	--bytecode-3.4 --bytecode-3.5 --bytecode-3.6 \
-        --bytecode-3.7 \
-	--bytecode-pypy3.2 --bytecode-pypy3.6
+        --bytecode-3.7  --bytecode-3.8
 
 #: Check deparsing on selected bytecode 3.x
 check-bytecode-3-short:
@@ -344,8 +351,8 @@ check-bytecode-pypy3.6: 7.1
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
 
 #: PyPy 5.0.x with Python 3.6.9
-check-bytecode-pypy3.6: 7.2
-7.2:
+check-bytecode-pypy3.6: 7.2 7.3
+7.3 7.2:
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6-run --verify-run
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
 

test/add-test.py

@@ -19,6 +19,7 @@ for path in py_source:
     else:
         cfile = "bytecode_%s%s/%s" % (version, suffix, short) + "c"
     print("byte-compiling %s to %s" % (path, cfile))
-    py_compile.compile(path, cfile)
+    optimize = 2
+    py_compile.compile(path, cfile, optimize=optimize)
     if isinstance(version, str) or version >= (2, 6, 0):
         os.system("../bin/uncompyle6 -a -T %s" % cfile)

test/run-and-email.sh

@@ -3,32 +3,54 @@
 function displaytime {
     printf "ran in "
     local T=$1
-    local D=$((T/60/60/24))
-    local H=$((T/60/60%24))
-    local M=$((T/60%60))
-    local S=$((T%60))
-    (( $D > 0 )) && printf '%d days ' $D
-    (( $H > 0 )) && printf '%d hours ' $H
-    (( $M > 0 )) && printf '%d minutes ' $M
-    (( $D > 0 || $H > 0 || $M > 0 )) && printf 'and '
+    ((D=T/60/60/24))
+    ((H=T/60/60%24))
+    ((M=T/60%60))
+    ((S=T%60))
+    (( D > 0 )) && printf '%d days ' $D
+    (( H > 0 )) && printf '%d hours ' $H
+    (( M > 0 )) && printf '%d minutes ' $M
+    (( D > 0 || H > 0 || M > 0 )) && printf 'and '
     printf '%d seconds\n' $S
 }
 
-. ../admin-tools/pyenv-newer-versions
+bs=${BASH_SOURCE[0]}
+if [[ $0 != $bs ]] ; then
+    echo "This script should not be *sourced* but run through bash"
+    exit 1
+fi
 
+mydir=$(dirname $bs)
+cd $mydir
+
+branch=$(cat ../.git/HEAD | cut -d'/' -f 3)
+if [[ $branch == 'python-2.4' ]]; then
+    . ../admin-tools/pyenv-older-versions
+elif [[ $branch == 'master' ]]; then
+    . ../admin-tools/pyenv-newer-versions
+else
+    echo &1>2 "Error git branch should either be 'master' or 'python-2.4'; got: '$branch'"
+    exit 1
+fi
+
+MAIN="test_pyenvlib.py"
 USER=${USER:-rocky}
 EMAIL=${EMAIL:-rb@dustyfeet.com}
+WHAT="uncompyle6 2.4 ${MAIN}"
 MAX_TESTS=${MAX_TESTS:-800}
+export BATCH=1
+
 typeset -i RUN_STARTTIME=$(date +%s)
 
 # PYVERSIONS="3.5.6"
-actual_versions=""
+MAILBODY=/tmp/${MAIN}-mailbody-$$.txt
+# for VERSION in 2.4.6 2.5.9 ; do
 for VERSION in $PYVERSIONS ; do
     typeset -i rc=0
-    LOGFILE=/tmp/pyenvlib-$VERSION-$$.log
+    LOGFILE=/tmp/${MAIN}-$VERSION-$$.log
 
     case "$VERSION" in
-	3.7.6 | 3.8.1 | 3.1.5 | 3.0.1 )
+	3.7.7 | 3.8.2 | 3.1.5 | 3.0.1 )
 	    continue
 	    ;;
 	3.5.9 )
@@ -44,7 +66,15 @@ for VERSION in $PYVERSIONS ; do
 	    MAX_TESTS=800
 	    ;;
 	3.6.10 )
-	    MAX_TESTS=1300  # about 2139 exist
+	    # MAX_TESTS=1300  # about 2139 exist
+	    # fails on _pyio.cpython-36.opt-1.pyc
+	    MAX_TESTS=34
+	    ;;
+	2.4.6 )
+	    MAX_TESTS=600
+	    ;;
+	2.5.6 )
+	    MAX_TESTS=600
 	    ;;
 	2.6.9 )
 	    MAX_TESTS=1300
@@ -58,11 +88,13 @@ for VERSION in $PYVERSIONS ; do
 
     if ! pyenv local $VERSION ; then
 	rc=1
+	mailbody_line="pyenv local $VERSION not installed"
+	echo $mailbody_line >> $MAILBODY
     else
       echo Python Version $(pyenv local) > $LOGFILE
       echo "" >> $LOGFILE
       typeset -i ALL_FILES_STARTTIME=$(date +%s)
-      cmd="python ./test_pyenvlib.py --max ${MAX_TESTS} --syntax-verify --$VERSION"
+      cmd="python ./${MAIN} --max ${MAX_TESTS} --syntax-verify --$VERSION"
       echo "$cmd" >>$LOGFILE 2>&1
       $cmd >>$LOGFILE 2>&1
       rc=$?
@@ -72,20 +104,25 @@ for VERSION in $PYVERSIONS ; do
 
       typeset -i ALL_FILES_ENDTIME=$(date +%s)
       (( time_diff =  ALL_FILES_ENDTIME - ALL_FILES_STARTTIME))
-      displaytime $time_diff >> $LOGFILE
+      time_str=$(displaytime $time_diff)
+      echo ${time_str}. >> $LOGFILE
     fi
 
-    SUBJECT_PREFIX="pyenv weak verify (max $MAX_TESTS) for"
+    SUBJECT_PREFIX="$WHAT (max $MAX_TESTS) for"
     if ((rc == 0)); then
+	mailbody_line="Python $VERSION ok; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION ok" ${USER}@localhost
     else
+	mailbody_line="Python $VERSION failed; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${USER}@localhost
-	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${EMAIL}
+	tail -v $LOGFILE | mail -s "$HOST $SUBJECT_PREFIX $VERSION not ok" ${EMAIL}
     fi
+    echo $mailbody_line >> $MAILBODY
     rm .python-version
 done
 
 typeset -i RUN_ENDTIME=$(date +%s)
 (( time_diff =  RUN_ENDTIME - RUN_STARTTIME))
 elapsed_time=$(displaytime $time_diff)
-echo "Run complete $elapsed_time for versions $actual_versions" | mail -s "pyenv weak verify in $elapsed_time" ${EMAIL}
+echo "${WHAT} complete; ${elapsed_time}." >> $MAILBODY
+cat $MAILBODY | mail -s "$HOST $WHAT ${elapsed_time}." ${EMAIL}

test/simple-decompile-code-test.py

@@ -2,7 +2,7 @@
 
 from uncompyle6 import uncompyle
 from uncompyle6.main import decompile
-from xdis.magics import sysinfo2float
+from xdis import sysinfo2float
 import sys, inspect
 
 def uncompyle_test():

test/simple_source/bug31/04_def_annotate.py

@@ -66,6 +66,14 @@ def div(a: dict(type=float, help='the dividend'),
     """Divide a by b"""
     return a / b
 
+# From 3.7.6 functools.py
+# Bug is in picking up the annotation.
+def f(a:"This is a new annotation"):
+    """This is a test"""
+    assert f.__annotations__['a'] == "This is a new annotation"
+
+f(5)
+
 class TestSignatureObject1():
     def test_signature_on_wkwonly(self):
         def test(*, a:float, b:str, c:str = 'test', **kwargs: int) -> int:

test/simple_source/bug33/02_pos_args.py

@@ -1,6 +1,8 @@
+# Python 3.3+
+#
 # From Python 3.3.6 hmac.py
 # Problem was getting wrong placement of positional args.
-# In 3.6+ paramter handling changes
+# In 3.6+ parameter handling changes
 
 # RUNNABLE!
 

test/simple_source/bug35/04_call_function.py

@@ -2,6 +2,7 @@
 # Bug was code not knowing which Python versions
 # have kwargs coming before positional args in code.
 
+"""This program is self-checking!"""
 # RUNNABLE!
 
 def tometadata(self, metadata, schema, Table, args, name=None):

test/simple_source/bug36/02_var_annotate.py

@@ -1,6 +1,9 @@
 # 3.6+ type annotations on variables
 from typing import List
 
+# This test program is part of the uncompyle6 test suite
+# tests STORE_ANNOTATION and SETUP_ANOTATIONS
+
 # RUNNABLE!
 y = 2
 x: bool

test/simple_source/bug37/02_async_for_generator.py

@@ -6,3 +6,11 @@ def make_arange(n):
 
 async def run(m):
     return [i async for i in m]
+
+# From 3.7.6 test_coroutines.py
+async def run_list(pair, f):
+    return [i for pair in p async for i in f]
+
+# FIXME: add this. It works in decompyle3
+# async def run_gen():
+#     return (i async for i in f if 0 < i < 4)

test/simple_source/bug37/03_else_removal.py

@@ -0,0 +1,12 @@
+# From python3.8/distutils/version.py with optimization -O2
+# The bug was that the other "else" constant propagated removed.
+
+# NOTE: this program needs to be compile with optimization
+def _cmp (b, c):
+    if b:
+        if c:
+            return 0
+        else:
+            return 1
+    else:
+        assert False, "never get here"

test/simple_source/bug37/04_async.py

@@ -1,23 +0,0 @@
-# from 3.7 test_contextlib_async.py
-# Bugs were not adding "async" when a function is a decorator,
-# and a misaligment when using "async with as".
-@_async_test
-async def test_enter(self):
-    self.assertIs(await manager.__aenter__(), manager)
-
-    async with manager as context:
-        async with woohoo() as x:
-            x = 1
-            y = 2
-        assert manager is context
-
-# From 3.7.6 test_coroutines.py
-# Bug was different form of code for "async with" below
-class CoroutineTest():
-    def test_with_8(self):
-        CNT = 0
-        async def foo():
-            nonlocal CNT
-            async with CM():
-                CNT += 1
-                return

test/simple_source/bug37/04_grammar.py

@@ -0,0 +1,87 @@
+# From test_grammar.py
+# RUNNABLE!
+def check_syntax_error(statement):
+    try:
+        compile(statement, '<test string>', 'exec')
+    except SyntaxError:
+        return
+    assert False
+
+def test_yield():
+    # Requires parentheses as call argument
+    def g():
+        f((yield 1), 1)
+    def g():
+        f((yield from ()))
+    def g():
+        f((yield from ()), 1)
+    def g():
+        f((yield 1))
+
+    # Allowed as standalone statement
+    def g():
+        yield 1
+
+    def g():
+        yield from ()
+
+    # Allowed as RHS of assignment
+    def g():
+        x = yield 1
+
+    def g():
+        x = yield from ()
+
+    # Ordinary yield accepts implicit tuples
+    def g():
+        yield 1, 1
+
+    def g():
+        x = yield 1, 1
+
+    # 'yield from' does not
+    check_syntax_error("def g(): yield from (), 1")
+    check_syntax_error("def g(): x = yield from (), 1")
+    # Requires parentheses as subexpression
+    def g():
+        1, (yield 1)
+
+    def g():
+        1, (yield from ())
+
+    check_syntax_error("def g(): 1, yield 1")
+    check_syntax_error("def g(): 1, yield from ()")
+    # Requires parentheses as call argument
+    def g():
+        f((yield 1))
+
+    def g():
+        f((yield 1), 1)
+
+    def g():
+        f((yield from ()))
+
+    def g():
+        f((yield from ()), 1)
+
+    check_syntax_error("def g(): f(yield 1)")
+    check_syntax_error("def g(): f(yield 1, 1)")
+    check_syntax_error("def g(): f(yield from ())")
+    check_syntax_error("def g(): f(yield from (), 1)")
+    # Not allowed at top level
+    check_syntax_error("yield")
+    check_syntax_error("yield from")
+    # Not allowed at class scope
+    check_syntax_error("class foo:yield 1")
+    check_syntax_error("class foo:yield from ()")
+    # Check annotation refleak on SyntaxError
+    check_syntax_error("def g(a:(yield)): pass")
+
+test_yield()
+
+# From test_types.py
+# Bug was needing parens around (yield 2)
+def gen_func():
+    yield 1
+    return (yield 2)
+gen = gen_func()

test/simple_source/bug37/10_async.py

@@ -0,0 +1,70 @@
+# from 3.7 test_contextlib_async.py
+# Bugs were not adding "async" when a function is a decorator,
+# and a misaligment when using "async with ... as".
+
+"""This program is self-checking!"""
+import asyncio
+from contextlib import asynccontextmanager, AbstractAsyncContextManager
+import functools
+
+
+def _async_test(func):
+    """Decorator to turn an async function into a test case."""
+
+    @functools.wraps(func)
+    def wrapper(*args, **kwargs):
+        coro = func(*args, **kwargs)
+        loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(loop)
+        try:
+            return loop.run_until_complete(coro)
+        finally:
+            loop.close()
+            asyncio.set_event_loop(None)
+
+    return wrapper
+
+
+state = []
+
+
+@asynccontextmanager
+async def woohoo():
+    state.append(1)
+    yield 42
+    state.append(999)
+
+
+@_async_test
+async def test_enter():
+    class DefaultEnter(AbstractAsyncContextManager):
+        async def __aexit__(*args):
+            return
+            # await super().__aexit__(*args)
+
+    manager = DefaultEnter()
+    got_manager = await manager.__aenter__()
+    # print(got_manager, manager)
+    assert got_manager is manager
+
+    async with manager as context:
+        async with woohoo() as x:
+            x = 1
+            y = 2
+        assert manager is context
+
+
+# From 3.7.6 test_coroutines.py
+# Bug was different form of code for "async with" below
+class CoroutineTest:
+    def test_with_8(self):
+        CNT = 0
+
+        async def foo():
+            nonlocal CNT
+            async with CM():
+                CNT += 1
+                return
+
+
+test_enter()

test/simple_source/expression/09_yield_from.py

@@ -17,9 +17,9 @@ def ybug(g):
 # From 3.5.1 _wakrefset.py
 #
 #  3.5:
-#  withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt
-#              POP_BLOCK LOAD_CONST COME_FROM
-#              WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+#  with ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+#           POP_BLOCK LOAD_CONST COME_FROM
+#           WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
 
 def __iter__(self, IterationGuard):

test/simple_source/looping/08_while1_if_continue.py

@@ -7,3 +7,40 @@ def readline (self):
                 continue
 
         return
+
+# From 2.4.6 sre.py
+# Bug has to do with "break" not being recognized
+# and is a JUMP_FORWARD.
+def _parse(a, b, source, state):
+    while 1:
+        if b:
+            while 1:
+                break
+        else:
+            raise
+
+def _parse2(source, state):
+    while 1:
+        if a:
+            if b:
+                while 1:
+                    this = 1
+                    break
+                continue
+
+        while 1:
+            if b:
+                break
+
+        x = 3
+
+# Bug was in 2.3 decompilation
+def _parse3(source, state):
+    while 1:
+        if a:
+            if b:
+                x = 1
+                while 1:
+                    if a:
+                        break
+                    raise

test/simple_source/stmts/03_if_elif.py

@@ -1,11 +1,13 @@
 # 2.6.9 symbols.py
 # Bug in 2.6 is having multple COME_FROMs due to the
 # "and" in the "if" clause
+
+# RUNNABLE
 if __name__:
     if __file__ and __name__:
         pass
-    elif __name__:
-        pass
+    elif not __name__:
+        assert False
 
 # 2.6.9 transformer.py
 # Bug in 2.6 is multple COME_FROMs as a result
@@ -21,3 +23,20 @@ elif __file__:
     assert __name__ or __file__
 else:
     pass
+
+# From 3.3.7 test_binop.py
+# Bug was in ifelsestmt(c) ensuring b+=5 is not in "else"
+# Also note: ifelsetmtc should not have been used since this
+# this is not in a loop!
+def __floordiv__(a, b):
+    if a:
+        b += 1
+    elif not b:
+        return a
+    b += 5
+    return b
+
+assert __floordiv__(1, 1) == 7
+assert __floordiv__(1, 0) == 6
+assert __floordiv__(0, 3) == 8
+assert __floordiv__(0, 0) == 0

test/simple_source/stmts/04_withas.py

@@ -16,3 +16,25 @@ def withas_bug(self, nested, a, b):
     with self.assertRaises(ZeroDivisionError):
         with nested(a(), b()) as (x, y):
                 1 // 0
+
+# From 3.7.7 test_functools.py
+# Bug is a unreachable code after "return"
+def test_invalid_registrations(x):
+    return
+    with x:
+        x = 1
+
+# From 3.7.7 test_re.py
+# Bug was hooking in c_with.
+def test_re_tests(tests):
+    for t in tests:
+        with a:
+            continue
+
+# Adapted from 3.8 distutils/command/config.py
+# In 3.8 the problem was in handling "with .. as" code
+def _gen_temp_sourcefile(x, a, headers, lang):
+    with x as y:
+        if a:
+            y = 2
+    return 5

test/stdlib/.gitignore

@@ -1 +1,3 @@
 /.python-version
+/runun33.sh
+/runun7.sh

test/stdlib/2.5-exclude.sh

@@ -81,6 +81,6 @@ SKIP_TESTS=(
 )
 # About 265 tests in 14 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_doctest.py]=1 # Fails on ppc64le
 fi

test/stdlib/2.6-exclude.sh

@@ -1,4 +1,12 @@
 SKIP_TESTS=(
+    # ifelsestmt is borked in:
+    #   if filename == 'srcfile':
+    #      return srcfile
+    #   if filename == 'destfile':
+    #      return destfile
+    #   assert 0  # shouldn't reach here.
+    [test_shutil.py]=1
+
 
     [test___all__.py]=1  # it fails on its own
     [test___all__.py]=1 # it fails on its own
@@ -59,6 +67,7 @@ SKIP_TESTS=(
 
     [test_scriptpackages.py]=1 # it fails on its own
     [test_select.py]=1 # test takes too long to run: 11 seconds
+
     [test_socket.py]=1 # test takes too long to run: 12 seconds
     [test_startfile.py]=1 # it fails on its own
     [test_structmembers.py]=1 # it fails on its own
@@ -85,7 +94,7 @@ SKIP_TESTS=(
 )
 # About 305 unit-test files in about 12 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     # Fails in crontab environment?
     # Figure out what's up here
     SKIP_TESTS[test_aifc.py]=1

test/stdlib/2.7-exclude.sh

@@ -1,4 +1,6 @@
 SKIP_TESTS=(
+    [test_cgi.py]=1 # FIXME: Works on c90ff51
+
     [test_bsddb3.py]=1 # test takes too long to run: 110 seconds
     [test_compile.py]=1  # Code introspects on co_consts in a non-decompilable way
     [test_curses.py]=1  # Possibly fails on its own but not detected
@@ -25,6 +27,7 @@ SKIP_TESTS=(
     [test_ssl.py]=1  #
     [test_subprocess.py]=1 # Runs ok but takes 22 seconds
     [test_sys_settrace.py]=1 # Line numbers are expected to be different
+
     [test_tokenize.py]=1 # test takes too long to run: 19 seconds
     [test_traceback.py]=1 # Line numbers change - duh.
     [test_unicode.py]=1  # Too long to run 11 seconds
@@ -34,7 +37,7 @@ SKIP_TESTS=(
 )
 # 334 unit-test files in about 15 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     # Fails in crontab environment?
     # Figure out what's up here
     SKIP_TESTS[test_array.py]=1
@@ -43,6 +46,7 @@ if (( batch )) ; then
     SKIP_TESTS[test_doctest2.py]=1 # a POWER thing?
     SKIP_TESTS[test_httplib.py]=1  # Ok, but POWER has problems with this
     SKIP_TESTS[test_pdb.py]=1 # Ok, but POWER has problems with this
+    SKIP_TESTS[test_tarfile.py]=1 # test can take over 15 seconds to run on an overloaded POWER7 system
 
     # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
     SKIP_TESTS[test_base64.py]=1

test/stdlib/3.2-exclude.sh

@@ -17,8 +17,6 @@ SKIP_TESTS=(
     [test_peepholer.py]=1
     [test_pep352.py]=1
 
-    [test_quopri.py]=1 # TypeError: Can't convert 'bytes' object to str implicitly
-
     [test_runpy.py]=1
 
     [test_ssl.py]=1 # too installation specific
@@ -31,9 +29,8 @@ SKIP_TESTS=(
 
 )
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     # Fails in crontab environment?
     # Figure out what's up here
     SKIP_TESTS[test_exception_variations.py]=1
-    SKIP_TESTS[test_quopri.py]=1
 fi

test/stdlib/3.3-exclude.sh

@@ -1,7 +1,19 @@
 SKIP_TESTS=(
-    [test_atexit.py]=1  # The atexit test starting at 3.3 looks for specific comments in error lines
+    # FIXME: Did this work sometime in the past ?
+    #   for elem in g(s):
+    #     if not tgt and isOdd(elem): continue
+    # is erroneously:
+    #   for elem in g(s):
+    #    if tgt or isOdd(elem):
+    #       pass
+    #    else:
+    #       tgt.append(elem)
+    [test_itertools.py]=1
 
-    [test_buffer.py]=1  # parse error
+    [test_buffer.py]=1  # FIXME: Works on c90ff51
+    [test_cmath.py]=1  # FIXME: Works on c90ff51
+
+    [test_atexit.py]=1  # The atexit test starting at 3.3 looks for specific comments in error lines
 
     [test_cmd_line.py]=1 # too long?
     [test_concurrent_futures.py]=1  # too long?
@@ -30,6 +42,7 @@ SKIP_TESTS=(
 
     [test_nntplib.py]=1
 
+    [test_pep352.py]=1  # test failures
     [test_peepholer.py]=1
     [test_poll.py]=1  # test takes too long to run: 11 seconds
     [test_pty.py]=1  # FIXME: Needs grammar loop isolation separation
@@ -57,7 +70,7 @@ SKIP_TESTS=(
 )
 # About 300 unit-test files in about 20 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_ftplib.py]=1  # Runs too long on POWER; over 15 seconds
     SKIP_TESTS[test_idle.py]=1  # No tk
     SKIP_TESTS[test_pep352.py]=1  # UnicodeDecodeError may be funny on weird environments

test/stdlib/3.4-exclude.sh

@@ -1,11 +1,29 @@
 SKIP_TESTS=(
+    # FIXME: Did this work sometime in the past ?
+    #   for elem in g(s):
+    #     if not tgt and isOdd(elem): continue
+    # is erroneously:
+    #   for elem in g(s):
+    #    if tgt or isOdd(elem):
+    #       pass
+    #    else:
+    #       tgt.append(elem)
+    [test_itertools.py]=1
+
+    [test_buffer.py]=1  # FIXME: Works on c90ff51
+    [test_cmath.py]=1  # FIXME: Works on c90ff51
+    [test_strftime.py]=1  # FIXME: Works on c90ff51
+
     [test___all__.py]=1  # it fails on its own
     [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
+
     [test_cmd_line.py]=1 # takes too long to run
     [test_concurrent_futures.py]=1  # too long?
+
     [test_configparser.py]=1  # Doesn't terminate
     [test_ctypes.py]=1 # it fails on its own
     [test_curses.py]=1 # Investigate
+
     [test_dbm_gnu.py]=1   # fails on its own
     [test_devpoll.py]=1 # it fails on its own
     [test_descr.py]=1   # test assertion errors
@@ -14,33 +32,44 @@ SKIP_TESTS=(
     [test_doctest2.py]=1
     [test_doctest.py]=1  # test assert failures
     [test_docxmlrpc.py]=1
+
     [test_enum.py]=1 # compile syntax?
     [test_exceptions.py]=1
+
     [test_faulthandler.py]=1
     [test_file_eintr.py]=1   # parse error
     [test_fork1.py]=1 # too long
+
     [test_gdb.py]=1 # it fails on its own
     [test_grammar.py]=1 # parse error
+
     [test_httplib.py]=1 # it fails on its own
+
     [test_import.py]=1 # it fails on its own
     [test_io.py]=1
     [test_ioctl.py]=1 # it fails on its own
     [test_inspect.py]=1 # Syntax error Investigate
+
     [test_logging.py]=1 # Too long to run
     [test_long.py]=1 # FIXME: Works on c90ff51
+
     [test_modulefinder.py]=1  # test assertion error
     [test_multiprocessing_fork.py]=1 # doesn't terminate
     [test_multiprocessing_forkserver.py]=1 # doesn't terminate
     [test_multiprocessing_main_handling.py]=1  # doesn't terminate
     [test_multiprocessing_spawn.py]=1 # doesn't terminate
+
     [test_nntplib.py]=1 # too long to run
+
     [test_peepholer.py]=1 # control flow?
     [test_pep352.py]=1 # test assert failures
     [test_pickle.py]=1 # test assert failures
     [test_pkgimport.py]=1 # long
     [test_poll.py]=1 # Too long to run: 11 seconds
     [test_pydoc.py]=1 # test assertion failures
+
     [test_runpy.py]=1 # Too long:
+
     [test_select.py]=1 # Too long: 11 seconds
     [test_selectors.py]=1  # Too long: 11 seconds
     [test_signal.py]=1 # Too long: 22 seconds
@@ -50,12 +79,15 @@ SKIP_TESTS=(
     [test_subprocess.py]=1 # Too long
     [test_symtable.py]=1 # Investigate bad output
     [test_sys_settrace.py]=1 # test assert failures
+
     [test_tcl.py]=1 # May be implementation specific. On POWER though it fails
     [test_threading.py]=1 # Too long
     [test_threadsignals.py]=1 # Too long to run: 12 seconds
     [test_timeout.py]=1 # Too long to run: 19 seconds
     [test_traceback.py]=1 # introspects on code
+
     [test_urllib2net.py]=1 # Doesn't terminate
+
     [test_zipfile64.py]=1
     [test_zlib.py]=1
 )
@@ -66,5 +98,4 @@ if (( batch )) ; then
     # Figure out what's up here
     SKIP_TESTS[test_exception_variations.py]=1
     SKIP_TESTS[test_mailbox.py]=1 # Takes to long on POWER; over 15 secs
-    SKIP_TESTS[test_quopri.py]=1
 fi

test/stdlib/3.5-exclude.sh

@@ -1,11 +1,7 @@
 SKIP_TESTS=(
     [test_buffer.py]=1  # FIXME: Works on c90ff51
-    [test_decorators.py]=1  # FIXME: Works on c90ff51
-    [test_platform.py]=1 # FIXME: Works on c90ff51
-    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
-    [test_tempfile.py]=1 # FIXME: Works on c90ff51
-    [test_uu.py]=1 # FIXME: Works on c90ff51
-    [test_ftplib.py]=1 # FIXME: Works on c90ff51
+    [test_platform.py]=1  # FIXME: Works on c90ff51
+    [test_pyclbr.py]=1  # FIXME: Works on c90ff51
 
     [test___all__.py]=1 # it fails on its own
     [test_aifc.py]=1  #
@@ -100,6 +96,7 @@ SKIP_TESTS=(
     [test_sys_settrace.py]=1 # test assert fail
 
     [test_tcl.py]=1  # it fails on its own
+    [test_tempfile.py]=1  # test assertion failures
     [test_thread.py]=1
     [test_threading.py]=1
     [test_timeout.py]=1
@@ -115,6 +112,7 @@ SKIP_TESTS=(
     [test_urllib2net.py]=1 # it fails on its own
     [test_urllibnet.py]=1 # it fails on its own
     [test_urlparse.py]=1 # test assert error
+    [test_uu.py]=1 # test assert error
 
     [test_winreg.py]=1 # it fails on its own
     [test_winsound.py]=1 # it fails on its own
@@ -129,7 +127,7 @@ SKIP_TESTS=(
 )
 # About 260 unit-test in about 16 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_asyncore.py]=1 # Ok, but takes more than 15 seconds to run
     SKIP_TESTS[test_bisect.py]=1
     SKIP_TESTS[test_compileall.py]=1  # Something weird on POWER
@@ -137,10 +135,10 @@ if (( batch )) ; then
     SKIP_TESTS[test_distutils.py]=1
 
     SKIP_TESTS[test_exception_variations.py]=1
+    SKIP_TESTS[test_ioctl.py]=1 # it fails on its own
     SKIP_TESTS[test_poplib.py]=1 # May be a result of POWER installation
 
-    SKIP_TESTS[test_quopri.py]=1
-    SKIP_TESTS[test_ioctl.py]=1 # it fails on its own
+    SKIP_TESTS[test_sysconfig.py]=1 # POWER extension fails
     SKIP_TESTS[test_tarfile.py]=1 # too long to run on POWER 15 secs
     SKIP_TESTS[test_venv.py]=1 # takes too long 11 seconds
 fi

test/stdlib/3.6-exclude.sh

@@ -1,15 +1,13 @@
 SKIP_TESTS=(
     [test_ast.py]=1 # FIXME: Works on c90ff51
-    [test_binop.py]=1 # FIXME: Works on c90ff51
-    [test_complex.py]=1 # FIXME: Works on c90ff51
-    [test_decorators.py]=1 # FIXME: Works on c90ff51
+    [test_cmath.py]=1  # FIXME: Works on c90ff51
     [test_format.py]=1 # FIXME: Works on c90ff51
     [test_ftplib.py]=1 # FIXME: Works on c90ff51
     [test_slice.py]=1 # FIXME: Works on c90ff51
     [test_sort.py]=1 # FIXME: Works on c90ff51
     [test_strftime.py]=1 # FIXME: Works on c90ff51
     [test_timeit.py]=1 # FIXME: Works on c90ff51
-    [test_os.py]=1 # FIXME: Works on c90ff51
+    [test_os.py]=1 # parse error FIXME: Works on c90ff51
 
     [test___all__.py]=1  # it fails on its own
     [test_aifc.py]=1  #
@@ -51,6 +49,7 @@ SKIP_TESTS=(
     [test_datetime.py]=1 # it fails on its own
     [test_dbm_ndbm.py]=1 # it fails on its own
     [test_decimal.py]=1
+    [test_decorators.py]=1 # control-flow failures
     [test_descr.py]=1   # syntax error: Investigate
     [test_devpoll.py]=1 # it fails on its own
     [test_dict.py]=1 # it fails on its own
@@ -108,7 +107,8 @@ SKIP_TESTS=(
     [test_nntplib.py]=1 # test takes too long to run: 31 seconds
     [test_normalization.py]=1 # it fails on its own
 
-    [test_ordered_dict.py]= # it fails on its own
+    [test_optparse.py]=1 # test fails
+    [test_ordered_dict.py]=1 # it fails on its own
     [test_ossaudiodev.py]=1 # it fails on its own
 
     [test_pdb.py]=1 # Probably introspection
@@ -124,8 +124,6 @@ SKIP_TESTS=(
     [test_pyclbr.py]=1 # it fails on its own
     [test_pydoc.py]=1 # it fails on its own
 
-    [test_quopri.py]=1      # AssertionError: b'123=four' != '123=four'
-
     [test_random.py]=1 # it fails on its own
     [test_range.py]=1
     [test_regrtest.py]=1 # test takes too long to run: 12 seconds
@@ -149,6 +147,7 @@ SKIP_TESTS=(
     [test_startfile.py]=1 # it fails on its own
     [test_statistics.py]=1 # it fails on its own
     [test_string_literals.py]=1
+    [test_strftime.py]=1 # test assertion failures
     [test_strtod.py]=1 # it fails on its own
     [test_struct.py]=1  # test assertion errors
     [test_subprocess.py]=1
@@ -202,8 +201,10 @@ SKIP_TESTS=(
 )
 # 236 unit-test files in about 13 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_codeccallbacks.py]=1
+    SKIP_TESTS[test_complex.py]=1 # Something funky with POWER8
+
     # locale on test machine is probably customized
     SKIP_TESTS[test__locale.py]=1
 fi

test/stdlib/3.7-exclude.sh

@@ -1,6 +1,22 @@
 SKIP_TESTS=(
+    # FIXME: Did this work sometime in the past ?
+    #   for elem in g(s):
+    #     if not tgt and isOdd(elem): continue
+    # is erroneously:
+    #   for elem in g(s):
+    #    if tgt or isOdd(elem):
+    #       pass
+    #    else:
+    #       tgt.append(elem)
+    [test_itertools.py]=1
+
+    # Fails on decompyle3 as well.
+    # complicated control flow and "and/or" expressions
+    [test_pickle.py]=1
+
     [test_builtin.py]=1 # FIXME works on decompyle6
     [test_context.py]=1 # FIXME works on decompyle6
+    [test_doctest2.py]=1 # FIXME works on decompyle6
     [test_format.py]=1 # FIXME works on decompyle6
     [test_marshal.py]=1 # FIXME works on decompyle6
     [test_normalization.py]=1 # FIXME works on decompyle6
@@ -9,12 +25,11 @@ SKIP_TESTS=(
     [test_slice.py]=1 # FIXME works on decompyle6
     [test_sort.py]=1 # FIXME works on decompyle6
     [test_statistics.py]=1 # FIXME works on decompyle6
-    [test_string_literals.py]=1 # FIXME works on decompyle6
     [test_timeit.py]=1 # FIXME works on decompyle6
     [test_urllib2_localnet.py]=1 # FIXME works on decompyle6
     [test_urllib2.py]=1 # FIXME: works on uncompyle6
-    [test_generators.py]=1  # Investigate improper lamdba with bogus "False" added
-    [test_grammar.py]=1 # investigate: like above: semantic rule missing probably
+    [test_generators.py]=1  # FIXME: works on uncompyle6 - lambda parsing probably
+    [test_grammar.py]=1 # FIXME: works on uncompyle6 - lambda parsing probably
 
     [test___all__.py]=1 # it fails on its own
     [test_argparse.py]=1 #- it fails on its own
@@ -133,7 +148,8 @@ SKIP_TESTS=(
 )
 # 306 unit-test files in about 19 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
+    SKIP_TESTS[test_capi.py]=1 # more than 15 secs to run on POWER
     SKIP_TESTS[test_dbm_gnu.py]=1 # fails on its own on POWER
     SKIP_TESTS[test_distutils.py]=1
     SKIP_TESTS[test_fileio.py]=1

test/stdlib/3.8-exclude.sh

@@ -1,10 +1,14 @@
 SKIP_TESTS=(
-    [test_time.py]=1 # FIXME: works on uncompyle6?
+    [test_mimetypes.py]=1 # parse error. decompile3 works. Release 3.6.4 works?
+    [test_time.py]=1 # FIXME: parse eror. decompyle3 works. Release 3.6.4 works?
+    [test_aifc.py]=1 # parse error; decompile3 works
+
+    [test_doctest2.py]=1 # test failures release 3.6.4 works?
+    [test_finalization.py]=1  # test failures release 3.6.4 works?
     [test_urllib2.py]=1 # FIXME: works on uncompyle6?
     [test_zipimport.py]=1 # FIXME: works on uncompyle6
 
     [test___all__.py]=1 # it fails on its own
-    [test_aifc.py]=1 # parse error
     [test_argparse.py]=1 #- it fails on its own
     [test_array.py]=1 #- parse error
     [test_asdl_parser.py]=1 # it fails on its own
@@ -76,6 +80,7 @@ SKIP_TESTS=(
     [test_difflib.py]=1   # parse error
     [test_dis.py]=1   # We change line numbers - duh!
     [test_doctest.py]=1 # parse error
+    [test_doctest2.py]=1 # test faiures relesae 3.6.4 works?
     [test_docxmlrpc.py]=1
     [test_dtrace.py]=1 # parse error
     [test_dummy_thread.py]=1 # parse error
@@ -91,6 +96,7 @@ SKIP_TESTS=(
     [test_exceptions.py]=1   # parse error
 
     [test_faulthandler.py]=1   # takes too long
+    [test_file_eintr.py]=1 # too long to run test; works on 3.7.7
     [test_fcntl.py]=1
     [test_filecmp.py]=1   # parse error
     [test_fileinput.py]=1
@@ -329,7 +335,7 @@ SKIP_TESTS=(
 )
 # 114 test files, Elapsed time about 7 minutes
 
-if (( batch )) ; then
+if (( BATCH )) ; then
     SKIP_TESTS[test_idle.py]=1 # Probably installation specific
     SKIP_TESTS[test_tix.py]=1 # fails on its own
     SKIP_TESTS[test_ttk_textonly.py]=1 # Installation dependent?

test/stdlib/run-and-email.sh

@@ -3,28 +3,50 @@
 function displaytime {
     printf "ran in "
     local T=$1
-    local D=$((T/60/60/24))
-    local H=$((T/60/60%24))
-    local M=$((T/60%60))
-    local S=$((T%60))
-    (( $D > 0 )) && printf '%d days ' $D
-    (( $H > 0 )) && printf '%d hours ' $H
-    (( $M > 0 )) && printf '%d minutes ' $M
-    (( $D > 0 || $H > 0 || $M > 0 )) && printf 'and '
+    ((D=T/60/60/24))
+    ((H=T/60/60%24))
+    ((M=T/60%60))
+    ((S=T%60))
+    (( D > 0 )) && printf '%d days ' $D
+    (( H > 0 )) && printf '%d hours ' $H
+    (( M > 0 )) && printf '%d minutes ' $M
+    (( D > 0 || H > 0 || M > 0 )) && printf 'and '
     printf '%d seconds\n' $S
 }
 
-. ../../admin-tools/pyenv-newer-versions
+bs=${BASH_SOURCE[0]}
+if [[ $0 != $bs ]] ; then
+    echo "This script should not be *sourced* but run through bash"
+    exit 1
+fi
+
+mydir=$(dirname $bs)
+cd $mydir
+
+branch=$(cat ../../.git/HEAD | cut -d'/' -f 3)
+if [[ $branch == 'python-2.4' ]]; then
+    . ../../admin-tools/pyenv-older-versions
+elif [[ $branch == 'master' ]]; then
+    . ../../admin-tools/pyenv-newer-versions
+else
+    echo &1>2 "Error git branch should either be 'master' or 'python-2.4'; got: '$branch'"
+    exit 1
+fi
+
+MAIN="runtests.sh"
+
 
 USER=${USER:-rocky}
 EMAIL=${EMAIL:-rb@dustyfeet.com}
-SUBJECT_PREFIX="stdlib unit testing for"
+WHAT="uncompyle6 ${MAIN}"
+export BATCH=1
 
 typeset -i RUN_STARTTIME=$(date +%s)
 
-actual_versions=""
 DEBUG=""  # -x
 
+MAILBODY=/tmp/${MAIN}-mailbody-$$.txt
+
 for VERSION in $PYVERSIONS ; do
     typeset -i rc=0
     LOGFILE=/tmp/runtests-$VERSION-$$.log
@@ -34,24 +56,37 @@ for VERSION in $PYVERSIONS ; do
 	    continue
 	    ;;
     esac
-    actual_versions="$actual_versions $VERSION"
 
     if ! pyenv local $VERSION ; then
 	rc=1
     else
-      STOP_ONERROR=1 /bin/bash $DEBUG ./runtests.sh  >$LOGFILE 2>&1
-      rc=$?
+	typeset -i ALL_FILES_STARTTIME=$(date +%s)
+	STOP_ONERROR=1 /bin/bash $DEBUG ./runtests.sh  >$LOGFILE 2>&1
+	rc=$?
+
+	echo Python Version $(pyenv local) >> $LOGFILE
+	echo "" >>$LOGFILE
+	typeset -i ALL_FILES_ENDTIME=$(date +%s)
+	(( time_diff =  ALL_FILES_ENDTIME - ALL_FILES_STARTTIME))
+	time_str=$(displaytime $time_diff)
+	echo ${time_str}. >> $LOGFILE
     fi
-    SUBJECT_PREFIX="runtests verify for"
+
+    SUBJECT_PREFIX="$WHAT for"
     if ((rc == 0)); then
+	mailbody_line="Python $VERSION ok; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION ok" ${USER}@localhost
     else
+	mailbody_line="Python $VERSION failed; ${time_str}."
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${USER}@localhost
-	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" $EMAIL
+	tail -v $LOGFILE | mail -s "$HOST $SUBJECT_PREFIX $VERSION not ok" $EMAIL
     fi
+    echo $mailbody_line >> $MAILBODY
 done
 
 typeset -i RUN_ENDTIME=$(date +%s)
 (( time_diff =  RUN_ENDTIME - RUN_STARTTIME))
 elapsed_time=$(displaytime $time_diff)
-echo "Run complete $elapsed_time for versions $actual_versions" | mail -s "runtests in $elapsed_time" ${EMAIL}
+echo "${WHAT} complete; ${elapsed_time}." >> $MAILBODY
+echo "Full results are in ${LOGFILE}." >> $MAILBODY
+cat $MAILBODY | mail -s "$HOST $WHAT $elapsed_time" ${EMAIL}

test/stdlib/runtests.sh

@@ -1,10 +1,12 @@
 #!/bin/bash
 me=${BASH_SOURCE[0]}
 
-typeset -i batch=1
-isatty=$(/usr/bin/tty 2>/dev/null)
-if [[ -n $isatty ]] && [[ "$isatty" != 'not a tty' ]] ; then
-    batch=0
+typeset -i BATCH=${BATCH:-0}
+if (( ! BATCH )) ; then
+    isatty=$(/usr/bin/tty 2>/dev/null)
+    if [[ -n $isatty ]] && [[ "$isatty" != 'not a tty' ]] ; then
+	BATCH=0
+    fi
 fi
 
 
@@ -72,11 +74,10 @@ case $PYVERSION in
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_fileio.py]=1
 	)
-	if (( batch )) ; then
+	if (( BATCH )) ; then
 	    # Fails in crontab environment?
 	    # Figure out what's up here
 	    SKIP_TESTS[test_exception_variations.py]=1
-	    SKIP_TESTS[test_quopri.py]=1
 	fi
 	;;
     3.1)
@@ -85,11 +86,10 @@ case $PYVERSION in
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_fileio.py]=1
 	)
-	if (( batch )) ; then
+	if (( BATCH )) ; then
 	    # Fails in crontab environment?
 	    # Figure out what's up here
 	    SKIP_TESTS[test_exception_variations.py]=1
-	    SKIP_TESTS[test_quopri.py]=1
 	fi
 	;;
     3.2)
@@ -129,6 +129,7 @@ fulldir=$(pwd)
 
 # DECOMPILER=uncompyle2
 DECOMPILER=${DECOMPILER:-"$fulldir/../../bin/uncompyle6"}
+OPTS=${OPTS:-""}
 TESTDIR=/tmp/test${PYVERSION}
 if [[ -e $TESTDIR ]] ; then
     rm -fr $TESTDIR
@@ -137,6 +138,8 @@ fi
 PYENV_ROOT=${PYENV_ROOT:-$HOME/.pyenv}
 pyenv_local=$(pyenv local)
 
+echo Python version is $pyenv_local
+
 # pyenv version update
 for dir in ../ ../../ ; do
     cp -v .python-version $dir
@@ -145,18 +148,25 @@ done
 
 mkdir $TESTDIR || exit $?
 cp -r ${PYENV_ROOT}/versions/${PYVERSION}.${MINOR}/lib/python${PYVERSION}/test $TESTDIR
-cd $TESTDIR/test
+if [[ $PYVERSION == 3.2 ]] ; then
+    cp ${PYENV_ROOT}/versions/${PYVERSION}.${MINOR}/lib/python${PYVERSION}/test/* $TESTDIR
+    cd $TESTDIR
+else
+    cd $TESTDIR/test
+fi
 pyenv local $FULLVERSION
 export PYTHONPATH=$TESTDIR
 export PATH=${PYENV_ROOT}/shims:${PATH}
 
+DONT_SKIP_TESTS=${DONT_SKIP_TESTS:-0}
+
 # Run tests
 typeset -i i=0
 typeset -i allerrs=0
 if [[ -n $1 ]] ; then
-    files=$1
-    typeset -a files_ary=( $(echo $1) )
-    if (( ${#files_ary[@]} == 1 )) ; then
+    files=$@
+    typeset -a files_ary=( $(echo $@) )
+    if (( ${#files_ary[@]} == 1 || DONT_SKIP_TESTS == 1 )) ; then
 	SKIP_TESTS=()
     fi
 else
@@ -166,9 +176,12 @@ fi
 typeset -i ALL_FILES_STARTTIME=$(date +%s)
 typeset -i skipped=0
 
+NOT_INVERTED_TESTS=${NOT_INVERTED_TESTS:-1}
+
 for file in $files; do
     # AIX bash doesn't grok [[ -v SKIP... ]]
-    if [[ ${SKIP_TESTS[$file]} == 1 ]] ; then
+    [[ -z ${SKIP_TESTS[$file]} ]] && SKIP_TESTS[$file]=0
+    if [[ ${SKIP_TESTS[$file]} == ${NOT_INVERTED_TESTS} ]] ; then
 	((skipped++))
 	continue
     fi
@@ -185,7 +198,7 @@ for file in $files; do
     typeset -i ENDTIME=$(date +%s)
     typeset -i time_diff
     (( time_diff =  ENDTIME - STARTTIME))
-    if (( time_diff > 10 )) ; then
+    if (( time_diff > $timeout )) ; then
 	echo "Skipping test $file -- test takes too long to run: $time_diff seconds"
 	continue
     fi
@@ -197,7 +210,7 @@ for file in $files; do
     $fulldir/compile-file.py $file && \
     mv $file{,.orig} && \
     echo ==========  $(date +%X) Decompiling $file ===========
-    $DECOMPILER $decompiled_file > $file
+    $DECOMPILER $OPTS $decompiled_file > $file
     rc=$?
     if (( rc == 0 )) ; then
 	echo ========== $(date +%X) Running $file ===========

uncompyle6/Makefile

@@ -1,7 +0,0 @@
-# Whatever it is you want to do, it should be forwarded to the 
-# to top-level irectories
-PHONY=check all
-all: check
-
-%: 
-	$(MAKE) -C .. $@

uncompyle6/disas.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016, 2818-2019 by Rocky Bernstein
+#  Copyright (c) 2015-2016, 2818-2020 by Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
@@ -32,8 +32,7 @@ want to run on earlier Python versions.
 import sys
 from collections import deque
 
-from xdis.code import iscode
-from xdis.load import check_object_path, load_module
+from xdis import check_object_path, iscode, load_module
 from uncompyle6.scanner import get_scanner
 
 
@@ -95,7 +94,9 @@ def disassemble_file(filename, outstream=None):
     try to find the corresponding compiled object.
     """
     filename = check_object_path(filename)
-    (version, timestamp, magic_int, co, is_pypy, source_size) = load_module(filename)
+    (version, timestamp, magic_int, co, is_pypy, source_size, sip_hash) = load_module(
+        filename
+    )
     if type(co) == list:
         for con in co:
             disco(version, con, outstream)

uncompyle6/linenumbers.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016, 2818 by Rocky Bernstein
+#  Copyright (c) 2015-2016, 2818, 2020 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -15,14 +15,27 @@
 
 from collections import deque
 
-from xdis.code import iscode
-from xdis.load import load_file, load_module
-from xdis.main import get_opcode
-from xdis.bytecode import Bytecode, findlinestarts, offset2line
+from xdis import (
+    Bytecode,
+    iscode,
+    findlinestarts,
+    get_opcode,
+    offset2line,
+    load_file,
+    load_module,
+)
+
 
 def line_number_mapping(pyc_filename, src_filename):
-    (version, timestamp, magic_int, code1, is_pypy,
-     source_size) = load_module(pyc_filename)
+    (
+        version,
+        timestamp,
+        magic_int,
+        code1,
+        is_pypy,
+        source_size,
+        sip_hash,
+    ) = load_module(pyc_filename)
     try:
         code2 = load_file(src_filename)
     except SyntaxError, e:
@@ -44,7 +57,10 @@ def number_loop(queue, mappings, opc):
         assert code1.co_name == code2.co_name
         linestarts_orig = findlinestarts(code1)
         linestarts_uncompiled = list(findlinestarts(code2))
-        mappings += [[line, offset2line(offset, linestarts_uncompiled)] for offset, line in linestarts_orig]
+        mappings += [
+            [line, offset2line(offset, linestarts_uncompiled)]
+            for offset, line in linestarts_orig
+        ]
         bytecode1 = Bytecode(code1, opc)
         bytecode2 = Bytecode(code2, opc)
         instr2s = bytecode2.get_instructions(code2)

uncompyle6/main.py

@@ -15,8 +15,7 @@
 import datetime, os, subprocess, sys
 
 from uncompyle6 import verify, IS_PYPY, PYTHON_VERSION
-from xdis.code import iscode
-from xdis.magics import sysinfo2float
+from xdis import iscode, sysinfo2float
 from uncompyle6.disas import check_object_path
 from uncompyle6.semantics import pysource
 from uncompyle6.parser import ParserError
@@ -107,7 +106,7 @@ def decompile(
           )
     )
     if bytecode_version >= 3.0:
-        write("# Warning: this version has problems handling the Python 3 byte type in contants properly.\n")
+        write("# Warning: this version has problems handling the Python 3 byte type in constants properly.\n")
     if co.co_filename:
         write("# Embedded file name: %s" % co.co_filename,)
     if timestamp:
@@ -188,7 +187,7 @@ def decompile_file(
 
     filename = check_object_path(filename)
     code_objects = {}
-    (version, timestamp, magic_int, co, is_pypy, source_size) = load_module(
+    (version, timestamp, magic_int, co, is_pypy, source_size, sip_hash) = load_module(
         filename, code_objects
     )
 

uncompyle6/parser.py

@@ -21,16 +21,16 @@ Common uncompyle6 parser routines.
 
 import sys
 
-from xdis.code import iscode
-from xdis.magics import py_str2float
+from xdis import iscode, py_str2float
 from spark_parser import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.show import maybe_show_asm
 
 
 class ParserError(Exception):
-    def __init__(self, token, offset):
+    def __init__(self, token, offset, debug=PARSER_DEFAULT_DEBUG):
         self.token = token
         self.offset = offset
+        self.debug = debug
 
     def __str__(self):
         return "Parse error at or near `%r' instruction at offset %s\n" % (
@@ -203,10 +203,10 @@ class PythonParser(GenericASTBuilder):
                     indent = "   "
                 else:
                     indent = "-> "
-                print "%s%s" % (indent, instructions[i])
-            raise ParserError(err_token, err_token.offset)
+                print("%s%s" % (indent, instructions[i]))
+            raise ParserError(err_token, err_token.offset, self.debug["reduce"])
         else:
-            raise ParserError(None, -1)
+            raise ParserError(None, -1, self.debug["reduce"])
 
     def get_pos_kw(self, token):
         """Return then the number of positional parameters and
@@ -349,7 +349,7 @@ class PythonParser(GenericASTBuilder):
         stmt ::= try_except
         stmt ::= tryelsestmt
         stmt ::= tryfinallystmt
-        stmt ::= withstmt
+        stmt ::= with
         stmt ::= withasstmt
 
         stmt ::= del_stmt

uncompyle6/parsers/parse10.py

@@ -7,7 +7,7 @@ from uncompyle6.parsers.parse11 import Python11Parser
 
 class Python10Parser(Python11Parser):
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python11Parser, self).__init__(debug_parser)
+        super(Python10Parser, self).__init__(debug_parser)
         self.customized = {}
 
 

uncompyle6/parsers/parse11.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019 Rocky Bernstein
+#  Copyright (c) 2019-2020 Rocky Bernstein
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.parser import PythonParserSingle
@@ -7,7 +7,7 @@ from uncompyle6.parsers.parse12 import Python12Parser
 
 class Python11Parser(Python12Parser):
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python12Parser, self).__init__(debug_parser)
+        super(Python11Parser, self).__init__(debug_parser)
         self.customized = {}
 
 

uncompyle6/parsers/parse2.py

@@ -194,8 +194,9 @@ class Python2Parser(PythonParser):
         expr ::= slice3
         expr ::= unary_convert
 
-        and  ::= expr jmp_false expr come_from_opt
-        or   ::= expr jmp_true  expr come_from_opt
+        expr_jt  ::= expr jmp_true
+        or       ::= expr_jt  expr come_from_opt
+        and      ::= expr jmp_false expr come_from_opt
 
         unary_convert ::= expr UNARY_CONVERT
 
@@ -712,7 +713,7 @@ class Python2Parser(PythonParser):
         elif lhs in ("raise_stmt1",):
             # We will assume 'LOAD_ASSERT' will be handled by an assert grammar rule
             return tokens[first] == "LOAD_ASSERT" and (last >= len(tokens))
-        elif rule == ("or", ("expr", "jmp_true", "expr", "\\e_come_from_opt")):
+        elif rule == ("or", ("expr_jit", "expr", "\\e_come_from_opt")):
             expr2 = ast[2]
             return expr2 == "expr" and expr2[0] == "LOAD_ASSERT"
         elif lhs in ("delete_subscript", "del_expr"):

uncompyle6/parsers/parse21.py

@@ -8,7 +8,7 @@ from uncompyle6.parsers.parse22 import Python22Parser
 class Python21Parser(Python22Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python22Parser, self).__init__(debug_parser)
+        super(Python21Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_forstmt21(self, args):

uncompyle6/parsers/parse22.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2017 Rocky Bernstein
+#  Copyright (c) 2016-2017, 2020 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
@@ -8,7 +8,7 @@ from uncompyle6.parsers.parse23 import Python23Parser
 class Python22Parser(Python23Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python23Parser, self).__init__(debug_parser)
+        super(Python22Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_misc22(self, args):

uncompyle6/parsers/parse23.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2018 Rocky Bernstein
+#  Copyright (c) 2016-2018, 2020 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 #  Copyright (c) 1999 John Aycock
 
@@ -9,7 +9,7 @@ from uncompyle6.parsers.parse24 import Python24Parser
 class Python23Parser(Python24Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python24Parser, self).__init__(debug_parser)
+        super(Python23Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_misc23(self, args):
@@ -32,8 +32,21 @@ class Python23Parser(Python24Parser):
         while1stmt ::= _while1test l_stmts_opt JUMP_BACK
                        POP_TOP POP_BLOCK COME_FROM
 
-        while1stmt ::= _while1test l_stmts_opt JUMP_BACK
-                       COME_FROM POP_TOP POP_BLOCK COME_FROM
+        while1stmt ::= _while1test l_stmts_opt JUMP_BACK COME_FROM
+                       POP_TOP POP_BLOCK COME_FROM
+
+        # Python 2.3
+        # The following has no "JUMP_BACK" after l_stmts because
+        # l_stmts ends in a "break", "return", or "continue"
+        while1stmt ::= _while1test l_stmts
+                       POP_TOP POP_BLOCK
+
+        # The following has a "COME_FROM" at the end which comes from
+        # a "break" inside "l_stmts".
+        while1stmt ::= _while1test l_stmts COME_FROM JUMP_BACK
+                       POP_TOP POP_BLOCK COME_FROM
+        while1stmt ::= _while1test l_stmts JUMP_BACK
+                       POP_TOP POP_BLOCK
 
         list_comp  ::= BUILD_LIST_0 DUP_TOP LOAD_ATTR store list_iter del_stmt
         list_for   ::= expr for_iter store list_iter JUMP_BACK come_froms POP_TOP JUMP_BACK

uncompyle6/parsers/parse24.py

@@ -38,8 +38,20 @@ class Python24Parser(Python25Parser):
         _ifstmts_jump24 ::= c_stmts_opt JUMP_FORWARD POP_TOP
 
         # Python 2.5+ omits POP_TOP POP_BLOCK
-        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_TOP POP_BLOCK COME_FROM
-        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_TOP POP_BLOCK
+        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK
+                       POP_TOP POP_BLOCK COME_FROM
+        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK
+                       POP_TOP POP_BLOCK
+        while1stmt ::= SETUP_LOOP l_stmts COME_FROM JUMP_BACK
+                       POP_TOP POP_BLOCK COME_FROM
+
+        continue   ::= JUMP_BACK JUMP_ABSOLUTE
+
+        # Python 2.4
+        # The following has no "JUMP_BACK" after l_stmts because
+        # l_stmts ends in a "break", "return", or "continue"
+        while1stmt ::= SETUP_LOOP l_stmts
+                       POP_TOP POP_BLOCK
 
         # Python 2.5+:
         #  call_stmt ::= expr POP_TOP
@@ -73,8 +85,8 @@ class Python24Parser(Python25Parser):
         with_cleanup  ::= LOAD_FAST DELETE_FAST WITH_CLEANUP END_FINALLY
         with_cleanup  ::= LOAD_NAME DELETE_NAME WITH_CLEANUP END_FINALLY
         withasstmt    ::= expr setupwithas store suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM with_cleanup
-        withstmt      ::= expr setupwith SETUP_FINALLY suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM with_cleanup
-        stmt ::= withstmt
+        with          ::= expr setupwith SETUP_FINALLY suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM with_cleanup
+        stmt ::= with
         stmt ::= withasstmt
         """)
         super(Python24Parser, self).customize_grammar_rules(tokens, customize)

uncompyle6/parsers/parse25.py

@@ -30,7 +30,7 @@ class Python25Parser(Python26Parser):
                         setup_finally
         # opcode SETUP_WITH
         setupwith ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 POP_TOP
-        withstmt  ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+        with      ::= expr setupwith SETUP_FINALLY suite_stmts_opt
                       POP_BLOCK LOAD_CONST COME_FROM with_cleanup
 
         # Semantic actions want store to be at index 2
@@ -70,7 +70,7 @@ class Python25Parser(Python26Parser):
         ifelsestmt    ::= testexpr c_stmts_opt jf_cf_pop else_suite come_froms
 
         setupwith  ::= DUP_TOP LOAD_ATTR ROT_TWO LOAD_ATTR CALL_FUNCTION_0 POP_TOP
-        withstmt   ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+        with       ::= expr setupwith SETUP_FINALLY suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY
         withasstmt ::= expr setupwithas store suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY

uncompyle6/parsers/parse26.py

@@ -119,8 +119,8 @@ class Python26Parser(Python2Parser):
         ifelsestmtc ::= testexpr c_stmts_opt ja_cf_pop    else_suitec
 
         # Semantic actions want suite_stmts_opt to be at index 3
-        withstmt ::= expr setupwith SETUP_FINALLY suite_stmts_opt
-                     POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY
+        with        ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+                        POP_BLOCK LOAD_CONST COME_FROM WITH_CLEANUP END_FINALLY
 
         # Semantic actions want store to be at index 2
         withasstmt ::= expr setupwithas store suite_stmts_opt

uncompyle6/parsers/parse27.py

@@ -7,6 +7,7 @@ from xdis import next_offset
 from uncompyle6.parser import PythonParserSingle, nop_func
 from uncompyle6.parsers.parse2 import Python2Parser
 from uncompyle6.parsers.reducecheck import (
+    or_check,
     ifelsestmt,
     tryelsestmt,
 )
@@ -101,8 +102,9 @@ class Python27Parser(Python2Parser):
         ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
         if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
-        or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
-        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
+        expr_jitop ::= expr JUMP_IF_TRUE_OR_POP
+        or         ::= expr_jitop expr COME_FROM
+        and        ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
 
         # compare_chained{1,2} is used exclusively in chained_compare
         compare_chained1 ::= expr DUP_TOP ROT_THREE COMPARE_OP JUMP_IF_FALSE_OR_POP
@@ -139,9 +141,11 @@ class Python27Parser(Python2Parser):
         # assert condition, expr
         assert2    ::= assert_expr jmp_true LOAD_ASSERT expr CALL_FUNCTION_1 RAISE_VARARGS_1
 
+        continue   ::= JUMP_BACK JUMP_ABSOLUTE
+
         for_block  ::= returns _come_froms
 
-        withstmt   ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+        with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM_WITH
                        WITH_CLEANUP END_FINALLY
 
@@ -227,6 +231,7 @@ class Python27Parser(Python2Parser):
         # FIXME: Put more in this table
         self.reduce_check_table = {
             # "ifelsestmt": ifelsestmt,
+            "or": or_check,
             "tryelsestmt": tryelsestmt,
             "tryelsestmtl": tryelsestmt,
         }
@@ -237,7 +242,7 @@ class Python27Parser(Python2Parser):
         self.check_reduce["except_handler"] = "tokens"
         self.check_reduce["except_handler_else"] = "tokens"
 
-        # self.check_reduce["or"] = "AST"
+        self.check_reduce["or"] = "AST"
         self.check_reduce["raise_stmt1"] = "AST"
         self.check_reduce["iflaststmtl"] = "AST"
         self.check_reduce["ifelsestmt"] = "AST"

uncompyle6/parsers/parse3.py

@@ -74,6 +74,7 @@ class Python3Parser(PythonParser):
 
         jb_or_c ::= JUMP_BACK
         jb_or_c ::= CONTINUE
+        jb_cfs  ::= JUMP_BACK _come_froms
 
         stmt ::= set_comp_func
 
@@ -269,18 +270,24 @@ class Python3Parser(PythonParser):
         jmp_abs ::= JUMP_ABSOLUTE
         jmp_abs ::= JUMP_BACK
 
-        withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt
-                POP_BLOCK LOAD_CONST COME_FROM_WITH
-                WITH_CLEANUP END_FINALLY
+        with    ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+                    POP_BLOCK LOAD_CONST COME_FROM_WITH
+                    WITH_CLEANUP END_FINALLY
 
         withasstmt ::= expr SETUP_WITH store suite_stmts_opt
                 POP_BLOCK LOAD_CONST COME_FROM_WITH
                 WITH_CLEANUP END_FINALLY
 
+        expr_jt     ::= expr jmp_true
+        expr_jitop  ::= expr JUMP_IF_TRUE_OR_POP
+
         ## FIXME: Right now we have erroneous jump targets
         ## This below is probably not correct when the COME_FROM is put in the right place
-        and ::= expr jmp_false expr COME_FROM
-        or  ::= expr jmp_true  expr COME_FROM
+        and  ::= expr jmp_false expr COME_FROM
+        or   ::= expr_jt  expr COME_FROM
+        or   ::= expr_jt expr
+        or   ::= expr_jitop expr COME_FROM
+        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
 
         # # something like the below is needed when the jump targets are fixed
         ## or  ::= expr JUMP_IF_TRUE_OR_POP COME_FROM expr
@@ -338,9 +345,6 @@ class Python3Parser(PythonParser):
         ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
         if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
-        or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
-        or   ::= expr jmp_true expr
-        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
 
         # compare_chained1 is used exclusively in chained_compare
         compare_chained1 ::= expr DUP_TOP ROT_THREE COMPARE_OP JUMP_IF_FALSE_OR_POP
@@ -435,10 +439,11 @@ class Python3Parser(PythonParser):
         while1elsestmt    ::= SETUP_LOOP          l_stmts     JUMP_BACK
                               else_suitel
 
-        whileelsestmt     ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK
+        whileelsestmt     ::= SETUP_LOOP testexpr l_stmts_opt jb_cfs POP_BLOCK
                               else_suitel COME_FROM_LOOP
 
-        whileelsestmt2     ::= SETUP_LOOP testexpr l_stmts_opt  JUMP_BACK POP_BLOCK
+
+        whileelsestmt2    ::= SETUP_LOOP testexpr l_stmts_opt  JUMP_BACK POP_BLOCK
                               else_suitel JUMP_BACK COME_FROM_LOOP
 
         whileTruestmt     ::= SETUP_LOOP l_stmts_opt          JUMP_BACK POP_BLOCK
@@ -676,6 +681,7 @@ class Python3Parser(PythonParser):
                 "RAISE",
                 "SETUP",
                 "UNPACK",
+                "WITH",
             )
         )
 

uncompyle6/parsers/parse30.py

@@ -133,8 +133,9 @@ class Python30Parser(Python31Parser):
 
         jump_except           ::= _jump COME_FROM POP_TOP
 
+        expr_jt               ::= expr jmp_true
         or                    ::= expr jmp_false expr jmp_true expr
-        or                    ::= expr jmp_true expr
+        or                    ::= expr_jt expr
 
         import_from ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist _come_froms POP_TOP
 
@@ -211,7 +212,7 @@ class Python30Parser(Python31Parser):
         whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK JUMP_BACK COME_FROM_LOOP
         whilestmt      ::= SETUP_LOOP testexpr returns POP_TOP POP_BLOCK COME_FROM_LOOP
         withasstmt     ::= expr SETUP_WITH store suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
-        withstmt       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
+        with           ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
 
         # lc_body ::= LOAD_FAST expr LIST_APPEND
         # lc_body ::= LOAD_NAME expr LIST_APPEND

uncompyle6/parsers/parse31.py

@@ -14,7 +14,7 @@ class Python31Parser(Python32Parser):
 
         setupwith      ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 POP_TOP
         setupwithas    ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 store
-        withstmt       ::= expr setupwith SETUP_FINALLY
+        with           ::= expr setupwith SETUP_FINALLY
                            suite_stmts_opt
                            POP_BLOCK LOAD_CONST COME_FROM_FINALLY
                            load del_stmt WITH_CLEANUP END_FINALLY