Readme link typo

until we can more fully untangle if stmts in loops.
Current tests break urllib2.pyc and cgi.pyc

.circleci/config.yml

@@ -12,9 +12,8 @@ jobs:
       COMPILE: --compile
     # To see the list of pre-built images that CircleCI provides for most common languages see
     # https://circleci.com/docs/2.0/circleci-images/
-    machine:
+    docker:
-      python:
+      - image: circleci/python:3.6.9
-        version: 2.7.14
     steps:
     # Machine Setup
     #   If you break your build into multiple jobs with workflows, you will probably want to do the parts of this that are relevant in each
@@ -27,7 +26,7 @@ jobs:
     # This is based on your 1.0 configuration file or project settings
     - run:
         working_directory: ~/rocky/python-uncompyle6
-        command: pip install virtualenv && pip install nose && pip install pep8 && pyenv rehash
+        command: pip install --user virtualenv && pip install --user nose && pip install --user pep8
     # Dependencies
     #   This would typically go in either a build or a build-and-test job when using workflows
     # Restore the dependency cache
@@ -38,9 +37,9 @@ jobs:
           - v2-dependencies-
 
     # This is based on your 1.0 configuration file or project settings
-    - run: pip install --upgrade setuptools
+    - run: pip install --user --upgrade setuptools
-    - run: pip install -e .
+    - run: pip install --user -e .
-    - run: pip install -r requirements-dev.txt
+    - run: pip install --user -r requirements-dev.txt
 
     # Save dependency cache
     - save_cache:
@@ -58,7 +57,7 @@ jobs:
     # Test
     #   This would typically be a build job when using workflows, possibly combined with build
     # This is based on your 1.0 configuration file or project settings
-    - run: python ./setup.py develop && make check-2.7
+    - run: sudo python ./setup.py develop && make check-3.6
     - run: cd ./test/stdlib && bash ./runtests.sh 'test_[p-z]*.py'
     # Teardown
     #   If you break your build into multiple jobs with workflows, you will probably want to do the parts of this that are relevant in each

NEWS.md

@@ -1,3 +1,53 @@
+3.6.4: 2020-2-9 Plateau
+=======================
+
+The main focus in this release was fix some of the more glaring problems creapt in from the last release due to that refactor.
+
+`uncompyle6` code is at a plateau where what is most needed is a code refactoring. In doing this, until everything refactored and replaced, decomplation may get worse.
+Therefore, this release largely serves as a checkpoint before more major upheaval.
+
+The upheaval, in  started last release, I believe the pinnicle was around c90ff51 which wasn't a release. I suppose I should tag that.
+
+After c90ff5, I started down the road of redoing control flow in a more comprehensible, debuggable, and scalable way. See [The Control Flow Mess](https://github.com/rocky/python-uncompyle6/wiki/The-Control-Flow-Mess)
+
+The bulk of the refactoring going on in the [decompyle3](https://github.com/rocky/python-decompil3) project, but I try to trickle down the changes.
+
+It is tricky because the changes are large and I have to figure decompose things so that little testable pieces can be done. And there is also the problem that what is in decompyle3 is incomplete as well.
+
+Other than control flow, another change that will probably happen in the next release is to redo the grammar for lambda expressions. Right now, we treat them as Python statements, you know, things with compound statements in them. But lambda aren't that. And so there is hackery to paper over difference making a statement out of an expression the wrong thing to do. For example, a return of an "and" expression can be expressed as nested "if" statements with return inside them, but the "if" variant of the bytecode is not valid in a lambda.
+
+In the decompyle3 code, I've gone down the road making the grammar goal symbol be an expression. This also offers the opportunity to split the grammar making parsing inside lambda not only more reliable because the wrong choices don't exist, but also simpler and faster because all those rules just need don't need to exist in parsing.
+
+I cringe in thinking about how the code has lived for so long without noticing such a simple stupidity, and lapse of sufficient thought.
+
+Some stats from testing. The below give numbers of decompiled tests from Python's test suite which succesfully ran
+
+```
+   Version  test-suites passing
+   -------  -------------------
+   2.4.6     243
+   2.5.6     265
+   2.6.9     305
+   3.3.7     300
+   3.4.10    304
+   3.5.9     260
+   3.6.10    236
+   3.7.6     306
+   3.8.1     114
+```
+
+Decompiled bytecode files distributed with Python (syntax check only):
+
+```
+2.7.17  647 files:   0 failed
+3.2.6   900 files:   0 failed
+3.3.7  1256 files:   0 failed
+3.4.10  800 files:   0 failed
+3.5.9   900 files:   0 failed
+3.6.10 1300 files:  28 failed
+```
+
+
 3.6.3: 2020-1-26 Martin and Susanne
 ===================================
 

README.rst

@@ -54,7 +54,7 @@ only; another patched that and handled only 3.3.  You get the
 idea. This code pulls all of these forks together and *moves
 forward*. There is some serious refactoring and cleanup in this code
 base over those old forks. Even more experimental refactoring is going
-on in decompile3_.
+on in decompyle3_.
 
 This demonstrably does the best in decompiling Python across all
 Python versions. And even when there is another project that only
@@ -140,9 +140,16 @@ Python syntax changes, you should use this option if the bytecode is
 the right bytecode for the Python interpreter that will be checking
 the syntax.
 
-You can also cross compare the results with another python decompiler
+You can also cross compare the results with either another version of
-like pycdc_ . Since they work differently, bugs here often aren't in
+`uncompyle6` since there are are sometimes regressions in decompiling
-that, and vice versa.
+specific bytecode as the overall quality improves.
+
+For Python 3.7 and above, the code in decompyle3_ is generally
+better.
+
+Or try specific another python decompiler like uncompyle2_, unpyc37_,
+or pycdc_.  Since the later two work differently, bugs here often
+aren't in that, and vice versa.
 
 There is an interesting class of these programs that is readily
 available give stronger verification: those programs that when run
@@ -241,7 +248,9 @@ See Also
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake
 .. _pycdc: https://github.com/zrax/pycdc
-.. _decompile3: https://github.com/rocky/python-decompile3
+.. _decompyle3: https://github.com/rocky/python-decompile3
+.. _uncompyle2: https://github.com/wibiti/uncompyle2
+.. _unpyc37: https://github.com/andrew-tavera/unpyc37
 .. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
 .. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
 		 :target: https://travis-ci.org/rocky/python-uncompyle6

test/.gitignore

@@ -1,2 +1,3 @@
 /.coverage
+/.python-version
 /nohup.out

test/decompyle/test_loops2.py

@@ -8,7 +8,7 @@ See http://www.goebel-consult.de/decompyle/ for download and
 for further information
 """
 
-# This is a seperate test pattern, since 'continue' within 'try'
+# This is a separate test pattern, since 'continue' within 'try'
 # was not allowed till Python 2.1
 
 for term in args:

test/simple_source/branching/02_ifelse_lambda.py

@@ -22,7 +22,7 @@ assert i[0]('a') == True
 assert i[0]('A') == False
 
 # Issue #170. Bug is needing an "conditional_not_lambda" grammar rule
-# in addition the the "if_expr_lambda" rule
+# in addition the the "if_exp_lambda" rule
 j = lambda a: False if not a else True
 assert j(True) == True
 assert j(False) == False

test/simple_source/branching/10_if_else_ternary.py

@@ -1,8 +1,8 @@
 # Tests:
 
 # ret_expr_or_cond ::= ret_expr
-# ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
+# if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
-# ret_expr_or_cond ::= ret_cond
+# ret_expr_or_cond ::= if_exp_ret
 # ret_or ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
 
 # See https://github.com/rocky/python-uncompyle6/issues/5

test/simple_source/bug26/00_future_divide.py

@@ -0,0 +1,4 @@
+# From 2.7.17 fractions
+"""Rational, infinite-precision, real numbers."""
+
+from __future__ import division

test/simple_source/bug26/03_weird26.py

@@ -8,7 +8,7 @@ list(x for x in range(10) if x % 2 if x % 3)
 
 # expresion which evaluates True unconditionally,
 # but leave dead code or junk around that we have to match on.
-# Tests "if_expr_true" rule
+# Tests "if_exp_true" rule
 5 if 1 else 2
 
 0 or max(5, 3) if 0 else 3

test/simple_source/bug27+/03_not_dead_code.py

@@ -1,6 +1,6 @@
 # Bug found in 2.7 test_itertools.py
-# Bug was erroneously using reduction to if_expr_true
+# Bug was erroneously using reduction to if_exp_true
-# A proper fix would be to use if_expr_true only when we
+# A proper fix would be to use if_exp_true only when we
 # can determine there is or was dead code.
 from itertools import izip_longest
 for args in [['abc', range(6)]]:

test/simple_source/bug30/05_block_fallback.py

@@ -0,0 +1,32 @@
+# Adapted from 3.7.6 test_contains
+# The bug was in reconstructing something equivalent to
+# "while False: yelid None" which is *needed* in __iter__().
+# Sheeh!
+
+# RUNNABLE!
+def test_block_fallback():
+    # blocking fallback with __contains__ = None
+    class ByContains(object):
+        def __contains__(self, other):
+            return False
+    c = ByContains()
+    class BlockContains(ByContains):
+        """Is not a container
+
+        This class is a perfectly good iterable (as tested by
+        list(bc)), as well as inheriting from a perfectly good
+        container, but __contains__ = None prevents the usual
+        fallback to iteration in the container protocol. That
+        is, normally, 0 in bc would fall back to the equivalent
+        of any(x==0 for x in bc), but here it's blocked from
+        doing so.
+        """
+        def __iter__(self):
+            while False:
+                yield None
+        __contains__ = None
+    bc = BlockContains()
+    assert not (0 in c)
+    assert not (0 in list(bc))
+
+test_block_fallback()

test/simple_source/bug35/02_async_for.py

@@ -1,3 +1,12 @@
 async def a(b, c):
     async for b in c:
         pass
+
+# From 3.7 test_generators.py
+# Bug was getting indentation correct for multiple async's
+async def foo(X):
+    async for i in X:
+        pass
+    async for i in X:
+        pass
+    raise Done

test/simple_source/bug37/04_async.py

@@ -10,3 +10,14 @@ async def test_enter(self):
             x = 1
             y = 2
         assert manager is context
+
+# From 3.7.6 test_coroutines.py
+# Bug was different form of code for "async with" below
+class CoroutineTest():
+    def test_with_8(self):
+        CNT = 0
+        async def foo():
+            nonlocal CNT
+            async with CM():
+                CNT += 1
+                return

test/simple_source/def/05_abc_class.py

@@ -12,3 +12,9 @@ class abstractclassmethod(classmethod):
     def __init__(self, callable):
         callable.__isabstractmethod__ = True
         super().__init__(callable)
+
+# From 2.7.17 test_abc.py
+# Bug was handling OldstyleClass correctly without
+# a "return locals() but with a "pass"
+class OldstyleClass:
+    pass

test/simple_source/operation_logic/05_control_flow_bugs.py

@@ -0,0 +1,76 @@
+# From 3.6.10 test_binascii.py
+# Bug was getting "while c and noise" parsed correclty
+# and not put into the "ifelsesmt"
+
+# RUNNABLE!
+def addnoise(c, noise):
+    while c and noise:
+        if c < 3:
+            c = 2
+        else:
+            c = 3
+        noise = False
+    return c
+
+assert addnoise(0, True) == 0
+assert addnoise(1, False) == 1
+assert addnoise(2, True) == 2
+assert addnoise(3, True) == 3
+assert addnoise(4, True) == 3
+assert addnoise(5, False) == 5
+
+# From 3.6.10 test_dbm_dumb.py
+# Bug was getting attaching "else" to the right "if" in the
+# presense of a loop.
+def test_random(a, r):
+    x = 0
+    for dummy in r:
+        if dummy:
+            if a:
+                x += 2
+        else:
+            x += 1
+    return x
+
+assert test_random(True, [1]) == 2
+assert test_random(True, [1, 1]) == 4
+assert test_random(False, [1]) == 0
+assert test_random(False, [1, 1]) == 0
+
+# From 2.7.17 test_frozen.py
+# Bug was getting making sure we have "try" not
+# "try"/"else"
+def test_frozen(a, b):
+    try:
+        x = 1 / a
+    except:
+        x = 2
+
+    try:
+        x += 3 / b
+    except:
+        x += 4
+
+    return x
+
+assert test_frozen(1, 1) == 4.0
+assert test_frozen(0, 1) == 5.0
+assert test_frozen(0.5, 0) == 6.0
+assert test_frozen(0, 0.5) == 8.0
+
+# From 3.6.10 test_binop.py
+# Bug was getting "other += 3" outside of "if"/"else.
+def __floordiv__(a, b):
+    other = 0
+    if a:
+        other = 1
+    else:
+        if not b:
+            return 2
+    other += 3
+    return other
+
+assert __floordiv__(True, True) == 4
+assert __floordiv__(True, False) == 4
+assert __floordiv__(False, True) == 3
+assert __floordiv__(False, False) == 2

test/stdlib/2.4-exclude.sh

@@ -0,0 +1,55 @@
+SKIP_TESTS=(
+    [test_aepack.py]=1 # it fails on its own
+    [test_al.py]=1 # it fails on its own
+    [test_applesingle.py]=1 # it fails on its own
+    [test_bsddb185.py]=1 # it fails on its own
+    [test_bsddb3.py]=1 # it fails on its own
+    [test_bsddb.py]=1 # it fails on its own
+    [test_cd.py]=1 # it fails on its own
+    [test_cl.py]=1 # it fails on its own
+    [test_codecmaps_cn.py]=1 # it fails on its own
+    [test_codecmaps_hk.py]=1 # it fails on its own
+    [test_codecmaps_jp.py]=1 # it fails on its own
+    [test_codecmaps_kr.py]=1 # it fails on its own
+    [test_codecmaps_tw.py]=1 # it fails on its own
+    [test_curses.py]=1 # it fails on its own
+    [test_dbm.py]=1 # it fails on its own
+    [test_dl.py]=1 # it fails on its own
+    [test_gdbm.py]=1 # it fails on its own
+    [test_gl.py]=1 # it fails on its own
+    [test_imageop.py]=1 # it fails on its own
+    [test_imgfile.py]=1 # it fails on its own
+    [test_linuxaudiodev.py]=1 # it fails on its own
+    [test_macfs.py]=1 # it fails on its own
+    [test_macostools.py]=1 # it fails on its own
+    [test_nis.py]=1 # it fails on its own
+    [test_normalization.py]=1 # it fails on its own
+    [test_ossaudiodev.py]=1 # it fails on its own
+    [test_pep277.py]=1 # it fails on its own
+    [test_plistlib.py]=1 # it fails on its own
+    [test_rgbimg.py]=1 # it fails on its own
+    [test_scriptpackages.py]=1 # it fails on its own
+    [test_socket_ssl.py]=1 # it fails on its own
+    [test_sunaudiodev.py]=1 # it fails on its own
+    [test_support.py]=1 # it fails on its own
+    [test_tcl.py]=1 # it fails on its own
+    [test_urllib2net.py]=1 # it fails on its own
+    [test_urllibnet.py]=1 # it fails on its own
+    [test_winreg.py]=1 # it fails on its own
+    [test_winsound.py]=1 # it fails on its own
+    [test_zlib.py]=1 # it fails on its own
+
+    [test_decimal.py]=1  #
+    [test_dis.py]=1   # We change line numbers - duh!
+    [test_generators.py]=1  # Investigate
+    [test_grammar.py]=1    # Too many stmts. Handle large stmts
+    [test_grp.py]=1      # Long test - might work Control flow?
+    [test_pep247.py]=1 # Long test - might work? Control flow?
+    [test_pwd.py]=1 # Long test - might work? Control flow?
+    [test_socketserver.py]=1 # -- test takes too long to run: 40 seconds
+    [test_threading.py]=1 # test takes too long to run: 11 seconds
+    [test_thread.py]=1 # test takes too long to run: 36 seconds
+    [test_trace.py]=1 # Long test - works
+    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
+)
+ # About 243 files, 0 in 19 minutes

test/stdlib/2.5-exclude.sh

@@ -1,4 +1,44 @@
 SKIP_TESTS=(
+    [test_aepack.py]=1 # it fails on its own
+    [test_al.py]=1 # it fails on its own
+    [test_applesingle.py]=1 # it fails on its own
+    [test_bsddb185.py]=1 # it fails on its own
+    [test_bsddb3.py]=1 # it fails on its own
+    [test_bsddb.py]=1 # it fails on its own
+    [test_cd.py]=1 # it fails on its own
+    [test_cl.py]=1 # it fails on its own
+    [test_codecmaps_cn.py]=1 # it fails on its own
+    [test_codecmaps_hk.py]=1 # it fails on its own
+    [test_codecmaps_jp.py]=1 # it fails on its own
+    [test_codecmaps_kr.py]=1 # it fails on its own
+    [test_codecmaps_tw.py]=1 # it fails on its own
+    [test_curses.py]=1 # it fails on its own
+    [test_dbm.py]=1 # it fails on its own
+    [test_dl.py]=1 # it fails on its own
+    [test_gdbm.py]=1 # it fails on its own
+    [test_gl.py]=1 # it fails on its own
+    [test_imageop.py]=1 # it fails on its own
+    [test_imgfile.py]=1 # it fails on its own
+    [test_linuxaudiodev.py]=1 # it fails on its own
+    [test_macfs.py]=1 # it fails on its own
+    [test_macostools.py]=1 # it fails on its own
+    [test_nis.py]=1 # it fails on its own
+    [test_normalization.py]=1 # it fails on its own
+    [test_ossaudiodev.py]=1 # it fails on its own
+    [test_pep277.py]=1 # it fails on its own
+    [test_plistlib.py]=1 # it fails on its own
+    [test_rgbimg.py]=1 # it fails on its own
+    [test_scriptpackages.py]=1 # it fails on its own
+    [test_sunaudiodev.py]=1 # it fails on its own
+    [test_support.py]=1 # it fails on its own
+    [test_tcl.py]=1 # it fails on its own
+    [test_urllib2net.py]=1 # it fails on its own
+    [test_urllibnet.py]=1 # it fails on its own
+    [test_winreg.py]=1 # it fails on its own
+    [test_winsound.py]=1 # it fails on its own
+    [test_zlib.py]=1 # it fails on its own
+
+
     [test_coercion.py]=1
     [test_decimal.py]=1
     [test_dis.py]=1        # We change line numbers - duh!
@@ -28,14 +68,19 @@ SKIP_TESTS=(
     [test_struct.py]=1 # "if and" confused for if .. assert and
     [test_sunaudiodev.py]=1 # it fails on its own
     [test_support.py]=1 # it fails on its own
-    [test_tcl.py=1] # it fails on its own
+    [test_tcl.py]=1 # it fails on its own
     [test_threading.py]=1 # test takes too long to run: 11 seconds
     [test_thread.py]=1 # test takes too long to run: 36 seconds
     [test_trace.py]=1  # Line numbers are expected to be different
     [test_urllib2net.py]=1 # is interactive?
     [test_urllibnet.py]=1 # it fails on its own
     [test_winreg.py]=1 # it fails on its own
-    [test_winsound.py[=1 # it fails on its own
+    [test_winsound.py]=1 # it fails on its own
     [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
     [test_zlib]=1  # fails on its own
 )
+# About 265 tests in 14 minutes
+
+if (( batch )) ; then
+    SKIP_TESTS[test_doctest.py]=1 # Fails on ppc64le
+fi

test/stdlib/3.2-exclude.sh

@@ -1,14 +1,5 @@
 SKIP_TESTS=(
-    [test_decorators.py]=1 # FIXME: Works on c90ff51
-    [test_optparse.py]=1 # FIXME: Works on c90ff51
-    [test_os.py]=1 # FIXME: Works on c90ff51
-    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
-    [test_strftime.py]=1 # FIXME: Works on c90ff51
-
-    [test_binop.py]=1 # FIXME: Works on c90ff51?
-    [test_cgi.py]=1 # FIXME: Works on c90ff51?
     [test_descr.py]=1 # FIXME: Works on c90ff51?
-    [test_doctest2.py]=1 # FIXME: Works on c90ff51?
 
     [test_cmd_line.py]=1
     [test_collections.py]=1

test/stdlib/3.3-exclude.sh

@@ -1,16 +1,4 @@
 SKIP_TESTS=(
-    [test_binop.py]=1 # FIXME: Works on c90ff51
-    [test_cgi.py]=1 # FIXME: Works on c90ff51
-    [test_decorators.py]=1 # FIXME: Works on c90ff51
-    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
-    [test_optparse.py]=1 # FIXME: Works on c90ff51
-    [test_os.py]=1 # FIXME: Works on c90ff51
-    [test_pep352.py]=1 # FIXME: Works on c90ff51
-    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
-    [test_shutil.py]=1 # FIXME: Works on c90ff51
-    [test_strftime.py]=1 # FIXME: Works on c90ff51
-    [test_symtable.py]=1 # FIXME: Works on c90ff51
-
     [test_atexit.py]=1  # The atexit test starting at 3.3 looks for specific comments in error lines
 
     [test_buffer.py]=1  # parse error

test/stdlib/3.4-exclude.sh

@@ -1,13 +1,4 @@
 SKIP_TESTS=(
-    [test_buffer.py]=1 # FIXME: Works on c90ff51
-    [test_decorators.py]=1 # FIXME: Works on c90ff51
-    [test_faulthandler.py]=1 # FIXME: too long to run 20 seconds. Works on c90ff51
-    [test_decimal.py]=1 # FIXME: Works on c90ff51
-    [test_optparse.py]=1 # FIXME: Works on c90ff51
-    [test_os.py]=1 # FIXME: Works on c90ff51
-    [test_shutil.py]=1 # FIXME: Works on c90ff51
-    [test_strftime.py]=1 # FIXME: Works on c90ff51
-
     [test___all__.py]=1  # it fails on its own
     [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
     [test_cmd_line.py]=1 # takes too long to run

test/stdlib/3.5-exclude.sh

@@ -1,20 +1,11 @@
 SKIP_TESTS=(
-    [test_buffer.py]=1 # FIXME: Works on c90ff51
+    [test_buffer.py]=1  # FIXME: Works on c90ff51
-    [test_decorators.py]=1 # FIXME: Works on c90ff51
+    [test_decorators.py]=1  # FIXME: Works on c90ff51
-    [test_faulthandler.py]=1 # FIXME: too long to run 20 seconds. Works on c90ff51
-    [test_ftplib.py]=1 # Works on c90ff51
-    [test_marshal.py]=1 # FIXME: Works on c90ff51
-    [test_optparse.py]=1 # FIXME: Works on c90ff51
-    [test_os.py]=1 # FIXME: Works on c90ff51
     [test_platform.py]=1 # FIXME: Works on c90ff51
-    [test_poplib.py]=1 # FIXME: Works on c90ff51
     [test_pyclbr.py]=1 # FIXME: Works on c90ff51
-    [test_smtplib.py]=1 # FIXME: Works on c90ff51
-    [test_strftime.py]=1 # FIXME: Works on c90ff51
-    [test_shutil.py]=1 # FIXME: Works on c90ff51
-    [test_sysconfig.py]=1 # FIXME: Works on c90ff51
     [test_tempfile.py]=1 # FIXME: Works on c90ff51
     [test_uu.py]=1 # FIXME: Works on c90ff51
+    [test_ftplib.py]=1 # FIXME: Works on c90ff51
 
     [test___all__.py]=1 # it fails on its own
     [test_aifc.py]=1  #
@@ -33,7 +24,6 @@ SKIP_TESTS=(
     [test_collections.py]=1
     [test_compile.py]=1  # Code introspects on co_consts in a non-decompilable way
     [test_concurrent_futures.py]=1 # Takes long to run
-    [test_coroutines.py]=1 # Syntax error Investigate
     [test_curses.py]=1 #
 
     [test_devpoll.py]=1 # it fails on its own
@@ -50,6 +40,7 @@ SKIP_TESTS=(
     [test_exceptions.py]=1 # parse error
 
     [test_format.py]=1
+    [test_ftplib.py]=1 # Test assertion failures
 
     [test_gdb.py]=1 # it fails on its own
     [test_glob.py]=1 #
@@ -65,6 +56,7 @@ SKIP_TESTS=(
     [test_logging.py]=1 #
     [test_long.py]=1 # too long run time: 20 seconds
 
+    [test_marshal.py]=1 # test assertion errors
     [test_math.py]=1 # test assertion errors TypeError: a float is required
     [test_modulefinder.py]=1  # test assertion error
     [test_msilib.py]=1 # it fails on its own
@@ -99,7 +91,7 @@ SKIP_TESTS=(
     [test_selectors.py]=1 # Takes too long 17 seconds
     [test_set.py]=1 # # test assert failure and doesn't terminate
     [test_signal.py]=1 # too long?
-    [test_smtpd.py]=1 # test failures
+    [test_smtplib.py]=1 # probably control flow
     [test_socket.py]=1 # long
     [test_socketserver.py]=1
     [test_strtod.py]=1 # Test assert failure
@@ -138,15 +130,15 @@ SKIP_TESTS=(
 # About 260 unit-test in about 16 minutes
 
 if (( batch )) ; then
-    # Fails in crontab environment?
+    SKIP_TESTS[test_asyncore.py]=1 # Ok, but takes more than 15 seconds to run
-    # Figure out what's up here
     SKIP_TESTS[test_bisect.py]=1
-    SKIP_TESTS[test_buffer.py]=1  # too long
     SKIP_TESTS[test_compileall.py]=1  # Something weird on POWER
     SKIP_TESTS[test_codeccallbacks.py]=1 # Something differenet in locale?
     SKIP_TESTS[test_distutils.py]=1
 
     SKIP_TESTS[test_exception_variations.py]=1
+    SKIP_TESTS[test_poplib.py]=1 # May be a result of POWER installation
+
     SKIP_TESTS[test_quopri.py]=1
     SKIP_TESTS[test_ioctl.py]=1 # it fails on its own
     SKIP_TESTS[test_tarfile.py]=1 # too long to run on POWER 15 secs

test/stdlib/3.6-exclude.sh

@@ -1,14 +1,13 @@
 SKIP_TESTS=(
+    [test_ast.py]=1 # FIXME: Works on c90ff51
     [test_binop.py]=1 # FIXME: Works on c90ff51
-    [test_cmath.py]=1 # FIXME: Works on c90ff51
+    [test_complex.py]=1 # FIXME: Works on c90ff51
-    [test_decorators.py]=1 # FIXME: Works on c90ff51
+    [test_format.py]=1 # FIXME: Works on c90ff51
-    [test_format.py]=1  # FIXME: works on c90ff51
+    [test_ftplib.py]=1 # FIXME: Works on c90ff51
-    [test_locale.py]=1 # FIXME: Works on c90ff51
+    [test_slice.py]=1 # FIXME: Works on c90ff51
-    [test_optparse.py]=1 # FIXME: Works on c90ff51
-    [test_os.py]=1 # FIXME: Works on c90ff51
-    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
     [test_sort.py]=1 # FIXME: Works on c90ff51
-    [test_strftime.py]=1 # FIXME: Works on c90ff51
+    [test_timeit.py]=1 # FIXME: Works on c90ff51
+    [test_os.py]=1 # FIXME: Works on c90ff51
 
     [test___all__.py]=1  # it fails on its own
     [test_aifc.py]=1  #
@@ -22,9 +21,10 @@ SKIP_TESTS=(
     [test_bisect.py]=1  # it fails on its own
     [test_buffer.py]=1  # parse error
     [test_builtin.py]=1  # Fails on its own
+
     [test test_capi.py]=1 # it fails on its own
     [test_cmd_line.py]=1 # Interactive?
-
+    [test_codeccallbacks.py]=1 # TypeError: ... must return (str/bytes, int) tuple
     [test_codecencodings_cn.py]=1 # it fails on its own
     [test_codecencodings_hk.py]=1 # it fails on its own
     [test_codecencodings_iso2022.py]=1 # it fails on its own
@@ -42,7 +42,7 @@ SKIP_TESTS=(
     [test_concurrent_futures.py]=1 # Takes long
     [test_contextlib.py]=1 # test assertion failure
     [test_contextlib_async.py]=1 # Investigate
-    [test_coroutines.py]=1 # Parse error
+    [test_coroutines.py]=1 # parse error
     [test_curses.py]=1 # Parse error
     [test_ctypes.py]=1 # it fails on its own
 
@@ -152,6 +152,7 @@ SKIP_TESTS=(
     [test_subprocess.py]=1
     [test_sys.py]=1 # Investigate confusing "and" with nested "if" when there is an "else
     [test_sys_settrace.py]=1 # parse error
+    [test_sysconfig.py]=1 # if confused for ifelse in "test_triplet_in_ext_suffix"
 
     [test_tarfile.py]=1 # it fails on its own
     [test_tcl.py]=1 # Test assert failures
@@ -159,6 +160,9 @@ SKIP_TESTS=(
     [test_thread.py]=1 # it fails on its own
     [test_threading.py]=1
     [test_threadsignals.py]=1
+
+    [test_time.py]=1 # Works but not on POWER: Rounding error?
+
     [test_timeout.py]=1
     [test_tix.py]=1 # it fails on its own
     [test_tk.py]=1 # it fails on its own
@@ -177,6 +181,7 @@ SKIP_TESTS=(
     [test_urllib2.py]=1 # it fails on its own
     [test_urllibnet.py]=1 # it fails on its own
     [test_urllib.py]=1 # it fails on its own
+    [test_urlparse.py]=1 # test failure
 
     [test_venv.py]=1 # test takes too long to run: 13 seconds
 

test/stdlib/3.7-exclude.sh

@@ -1,136 +1,148 @@
 SKIP_TESTS=(
+    [test_builtin.py]=1 # FIXME works on decompyle6
+    [test_context.py]=1 # FIXME works on decompyle6
+    [test_format.py]=1 # FIXME works on decompyle6
+    [test_marshal.py]=1 # FIXME works on decompyle6
+    [test_normalization.py]=1 # FIXME works on decompyle6
+    [test_os.py]=1 # FIXME works on decompyle6
+    [test_pow.py]=1 # FIXME works on decompyle6
+    [test_slice.py]=1 # FIXME works on decompyle6
+    [test_sort.py]=1 # FIXME works on decompyle6
+    [test_statistics.py]=1 # FIXME works on decompyle6
+    [test_string_literals.py]=1 # FIXME works on decompyle6
+    [test_timeit.py]=1 # FIXME works on decompyle6
+    [test_urllib2_localnet.py]=1 # FIXME works on decompyle6
+    [test_urllib2.py]=1 # FIXME: works on uncompyle6
+    [test_generators.py]=1  # Investigate improper lamdba with bogus "False" added
+    [test_grammar.py]=1 # investigate: like above: semantic rule missing probably
+
     [test___all__.py]=1 # it fails on its own
     [test_argparse.py]=1 #- it fails on its own
     [test_asdl_parser.py]=1 # it fails on its own
-    [test_ast.py]=1  # Depends on comments in code
     [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
     [test_baseexception.py]=1  # UnboundLocalError: local variable 'exc' referenced before assignment
     [test_bdb.py]=1  #
-    [test_buffer.py]=1  # test assertion errors
+    [test_buffer.py]=1  # parse error
-    [test_builtin.py]=1  # parse error, but decompyle3 doesn't have this. (It has test assert failures though)
     [test_clinic.py]=1 # it fails on its own
     [test_cmath.py]=1 # test assertion failure
     [test_cmd_line.py]=1  # Interactive?
     [test_cmd_line_script.py]=1
-    [test_compare.py]=1 # Weird test assert faiure AssertionError: [1] == [1]
     [test_compileall.py]=1 # fails on its own
     [test_compile.py]=1  # Code introspects on co_consts in a non-decompilable way
     [test_concurrent_futures.py]=1 # too long
-    [test_context.py]=1
+    [test_coroutines.py]=1 # parse error
-    [test_coroutines.py]=1 # Investigate: Parse error - async/yield  stuff?
     [test_codecs.py]=1 # test assert failures; encoding/decoding stuff
     [test_ctypes.py]=1 # it fails on its own
     [test_curses.py]=1 # probably byte string not handled properly
     [test_dataclasses.py]=1   # FIXME: control flow probably: AssertionError: unknown result 'exception'
     [test_datetime.py]=1   # Takes too long
-    [test_dbm_gnu.py]=1   # Takes too long
     [test_dbm_ndbm.py]=1 # it fails on its own
-    [test_decimal.py]=1   # test assertion failures
+    [test_decimal.py]=1   # parse error
     [test_descr.py]=1   # test assertion failures
     [test_devpoll.py]=1 # it fails on its own
-    [test_dis.py]=1   # We change line numbers - duh!
+    [test_dis.py]=1   # Investigate async out of place. Then We change line numbers - duh!
-    [test_doctest2.py]=1 # FIXME: assert failure - works on decompyle3
+    [test_doctest.py]=1   # test failures
     [test_docxmlrpc.py]=1
+
     [test_enum.py]=1   # probably bad control flow
-    [test_faulthandler.py]=1   # takes too long
+
-    [test_fcntl.py]=1
+    [test_faulthandler.py]=1   # test takes too long before decompiling
     [test_fileinput.py]=1 # Test assertion failures
-    [test_format.py]=1 # Probably not handling bytestrings properly
     [test_frame.py]=1 # test assertion errors
     [test_ftplib.py]=1 # parse error
-    [test_functools.py]=1 # parse error
     [test_fstring.py]=1 # need to disambiguate leading fstrings from docstrings
+    [test_functools.py]=1 # parse error
+
     [test_gdb.py]=1 # it fails on its own
-    [test_generators.py]=1  # Investigate improper lamdba with bogus "False" added
     [test_glob.py]=1  # TypeError: join() argument must be str or bytes, not 'tuple'
-    [test_grammar.py]=1 # investigate: index out of range in decompiler (template_engine?)
     [test_grp.py]=1 # Doesn't terminate (killed)
-    [test_hashlib.py]=1 # test assert failures
+
-    [test_imaplib-3.7.py]=1  # test assert failures
+    [test_imaplib.py]=1  # test run loops before decompiling? More than 15 seconds to run
-    [test_idle.py]=1 # Probably installation specific
     [test_io.py]=1 # test takes too long to run: 37 seconds
-    [test_imaplib.py]=1 # test assert failures
+    [test_imaplib.py]=1 # decompiled test loops - killing after 15 seconds
     [test_inspect.py]=1 # Investigate test failures involving lambda
+
     [test_kqueue.py]=1 # it fails on its own
+
     [test_lib2to3.py]=1 # it fails on its own
     [test_long.py]=1 # FIX: if boundaries wrong in Rat __init__
     [test_logging.py]=1 # test takes too long to run: 20 seconds
-    [test_mailbox.py]=1
+
-    [test_marshal.py]=1
+    [test_mailbox.py]=1 # probably control flow
     [test_math.py]=1  # test assert failures
-    [test_modulefinder.py]=1
+    [test_msilib.py]=1 # it fails on its own
-    [test_msilib.py]=1
     [test_multiprocessing_fork.py]=1 # test takes too long to run: 62 seconds
     [test_multiprocessing_forkserver.py]=1
     [test_multiprocessing_spawn.py]=1
-    [test_normalization.py]=1 # probably control flow (uninitialized variable)
+
-    [test_nntplib.py]=1
+    [test_nntplib.py]=1 # Too long in running before decomplation takes 25 seconds
+
     [test_optparse.py]=1 # doesn't terminate at test_consume_separator_stop_at_option
-    [test_os.py]=1 # probably control flow (uninitialized variable)
     [test_ossaudiodev.py]=1 # it fails on its own
+
     [test_pdb.py]=1 # Probably relies on comments
     [test_peepholer.py]=1 # test assert error
-    [test_pickle.py]=1 # Probably relies on comments
-    [test_poll.py]=1
-    [test_poplib.py]=1
-    [test_pydoc.py]=1 # it fails on its own
-    [test_runpy.py]=1  #
     [test_pkg.py]=1 # Investigate: lists differ
     [test_pkgutil.py]=1 # Investigate:
-    [test_platform.py]=1 # probably control flow: uninitialized variable
+    [test_poll.py]=1 # Takes too long to run before decompiling 11 seconds
-    [test_pow.py]=1 # probably control flow: test assertion failure
     [test_pwd.py]=1 # killing - doesn't terminate
+    [test_pydoc.py]=1 # it fails on its own
+
     [test_regrtest.py]=1 # lists differ
-    [test_re.py]=1 # test assertion error
     [test_richcmp.py]=1 # parse error
+    [test_runpy.py]=1  # Too long to run before decompiling
+
     [test_select.py]=1 # test takes too long to run: 11 seconds
-    [test_selectors.py]=1
+    [test_selectors.py]=1 # Takes too long to run before decompling: 17 seconds
     [test_shutil.py]=1 # fails on its own
-    [test_signal.py]=1 #
+    [test_signal.py]=1 # Takes too long to run before decompiling: 22 seconds
-    [test_slice.py]=1 # test assert error in data; Investigate
+    [test_smtplib.py]=1 # test failures
-    [test_smtplib.py]=1 #
+    [test_socket.py]=1 # Takes too long to run before decompiling
-    [test_socket.py]=1
+    [test_ssl.py]=1 # Takes too long to run more than 15 seconds. Probably control flow; unintialized variable
-    [test_socketserver.py]=1
-    [test_sort.py]=1 # Probably control flow; unintialized varaible
-    [test_ssl.py]=1 # Probably control flow; unintialized varaible
     [test_startfile.py]=1 # it fails on its own
-    [test_statistics.py]=1 # Probably control flow; unintialized varaible
+    [test_strptime.py]=1 # parfse error
-    [test_string_literals.py]=1 # Investigate boolean parsing
-    [test_strptime.py]=1 # test assertions failed
     [test_strtod.py]=1 # test assertions failed
-    [test_structmembers.py]=1 # test assertions failed
+    [test_struct.py]=1 # probably control flow
-    [test_struct.py]=1 # test assertions failed
+    [test_subprocess.py]=1 # Takes too long to run before decompile: 25 seconds
-    [test_subprocess.py]=1
-    [test_sys_setprofile.py]=1 # test assertions failed
     [test_sys_settrace.py]=1 # parse error
+
     [test_tarfile.py]=1 # test assertions failed
-    [test_threading.py]=1 #
+    [test_threading.py]=1 # test assertion failers
-    [test_timeit.py]=1 # probably control flow uninitialized variable
     [test_tk.py]=1  # test takes too long to run: 13 seconds
-    [test_tokenize.py]=1
+    [test_tokenize.py]=1 # test takes too long to run before decompilation: 43 seconds
     [test_trace.py]=1  # it fails on its own
     [test_traceback.py]=1 # Probably uses comment for testing
-    [test_tracemalloc.py]=1 #
+    [test_tracemalloc.py]=1 # test assert failres
+    [test_ttk_guionly.py]=1  # implementation specfic and test takes too long to run: 19 seconds
     [test_ttk_guionly.py]=1  # implementation specfic and test takes too long to run: 19 seconds
     [test_typing.py]=1 # parse error
     [test_types.py]=1 # parse error
+
     [test_unicode.py]=1 # unicode thing
-    [test_urllib2_localnet.py]=1 #
     [test_urllibnet.py]=1 # probably control flow - uninitialized variable
+
     [test_weakref.py]=1 # probably control flow - uninitialized variable
     [test_with.py]=1 # probably control flow - uninitialized variable
-    [test_xml_dom_minicompat.py]=1 # FIXME: parse error; imports again
+
     [test_winconsoleio.py]=1 # it fails on its own
     [test_winreg.py]=1 # it fails on its own
     [test_winsound.py]=1 # it fails on its own
+
     [test_zipfile.py]=1 # it fails on its own
     [test_zipfile64.py]=1 # Too long to run
 )
-# 278 unit-test files in about 15 minutes
+# 306 unit-test files in about 19 minutes
 
 if (( batch )) ; then
+    SKIP_TESTS[test_dbm_gnu.py]=1 # fails on its own on POWER
     SKIP_TESTS[test_distutils.py]=1
     SKIP_TESTS[test_fileio.py]=1
     SKIP_TESTS[test_gc.py]=1
+    SKIP_TESTS[test_idle.py]=1 # Probably installation specific
+    SKIP_TESTS[test_sqlite.py]=1 # fails on its own on POWER
+    SKIP_TESTS[test_tix.py]=1 # it fails on its own
+    SKIP_TESTS[test_ttk_textonly.py]=1 # Installation dependent?
+    SKIP_TESTS[test_venv.py]=1 # Too long to run: 11 seconds
     SKIP_TESTS[test_zipimport_support.py]=1
+
 fi

test/stdlib/3.8-exclude.sh

@@ -1,152 +1,337 @@
 SKIP_TESTS=(
+    [test_time.py]=1 # FIXME: works on uncompyle6?
+    [test_urllib2.py]=1 # FIXME: works on uncompyle6?
+    [test_zipimport.py]=1 # FIXME: works on uncompyle6
+
     [test___all__.py]=1 # it fails on its own
+    [test_aifc.py]=1 # parse error
     [test_argparse.py]=1 #- it fails on its own
+    [test_array.py]=1 #- parse error
     [test_asdl_parser.py]=1 # it fails on its own
     [test_ast.py]=1  # Depends on comments in code
+    [test_asyncgen.py]=1 # parse error
+    [test_asynchat.py]=1 # parse error
+    [test_asyncore.py]=1 # parse error
     [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
+    [test_audioop.py]=1  # test failure
+    [test_audit.py]=1  # parse error
 
+    [test_base64.py]=1  # parse error
     [test_baseexception.py]=1  #
+    [test_bigaddrspace.py]=1  # parse error
+    [test_bigmem.py]=1  # parse error
     [test_bdb.py]=1  #
+    [test_binascii.py]=1  # test failure
+    [test_binhex.py]=1  # parse error
+    [test_binop.py]=1  # parse error
+    [test_bool.py]=1  # parse error
     [test_buffer.py]=1  # parse error
     [test_builtin.py]=1  # parse error
+    [test_bytes.py]=1  # parse error
+    [test_bz2.py]=1  # parse error
 
+    [test_calendar.py]=1  # parse error
+    [test_cgi.py]=1  # parse error
+    [test_cgitb.py]=1  # parse error
     [test_clinic.py]=1 # it fails on its own
     [test_cmath.py]=1 # test assertion failure
+    [test_cmd.py]=1  # parse error
     [test_cmd_line.py]=1  # Interactive?
     [test_cmd_line_script.py]=1
+    [test_code_module.py]=1 # test failure
+    [test_codecmaps_cn.py]=1 # test before decompile takes too long to run 135 secs
+    [test_codecmaps_hk.py]=1 # test before decompile takes too long to run 46 secs
+    [test_codecs.py]=1
     [test_collections.py]=1
     [test_compare.py]=1
-    [test_compileall.py]=1 # fails on its own
     [test_compile.py]=1
+    [test_compileall.py]=1 # fails on its own
+    [test_complex.py]=1 # Investigate
     [test_concurrent_futures.py]=1 # too long
     [test_configparser.py]=1
     [test_context.py]=1
+    [test_contextlib.py]=1 # parse error
+    [test_contextlib_async.py]=1 # parse error
     [test_coroutines.py]=1 # Parse error
-    [test_codecs.py]=1
+    [test_cprofile.py]=1 # parse error
-    [test_compile.py]=1  # Parse error, but after that, the code introspects on co_consts in a non-decompilable way
-    [test_complex.py]=1 # Investigate
     [test_crypt.py]=1 # Parse error
+    [test_csv.py]=1 # Parse error
     [test_ctypes.py]=1 # it fails on its own
     [test_curses.py]=1 # Parse error
 
     [test_dataclasses.py]=1  # test assertion errors
     [test_datetime.py]=1   # Takes too long
+    [test_dbm.py]=1   # parse error
+    [test_dbm_dumb.py]=1   # parse error
     [test_dbm_gnu.py]=1   # Takes too long
     [test_dbm_ndbm.py]=1 # it fails on its own
     [test_decimal.py]=1   # Parse error
+    [test_decorators.py]=1   # parse error
+    [test_deque.py]=1   # parse error
     [test_descr.py]=1   # Parse error
+    [test_descrtut.py]=1   # parse error
     [test_devpoll.py]=1 # it fails on its own
+    [test_dict.py]=1   # parse error
     [test_dictcomps.py]=1 # Bad semantics - Investigate
+    [test_difflib.py]=1   # parse error
     [test_dis.py]=1   # We change line numbers - duh!
+    [test_doctest.py]=1 # parse error
     [test_docxmlrpc.py]=1
+    [test_dtrace.py]=1 # parse error
+    [test_dummy_thread.py]=1 # parse error
 
-    [test_exceptions.py]=1   # parse error
+    [test_embed.py]=1   # parse error
-    [test_enumerate.py]=1   #
+    [test_ensureip.py]=1   #
+    [test_ensurepip.py]=1 # parse error
     [test_enum.py]=1   #
+    [test_enumerate.py]=1   #
+    [test_eof.py]=1   # parse error
+    [test_epoll.py]=1   # parse error
+    [test_exception_hierarchy.py]=1 # control flow?
+    [test_exceptions.py]=1   # parse error
 
     [test_faulthandler.py]=1   # takes too long
     [test_fcntl.py]=1
+    [test_filecmp.py]=1   # parse error
     [test_fileinput.py]=1
-    [test_float.py]=1
+    [test_fileio.py]=1
+    [test_float.py]=1  # Takes a long time to decompile
+    [test_flufl.py]=1  # parse error
     [test_format.py]=1
     [test_frame.py]=1
+    [test_frozen.py]=1 # parse error
     [test_fstring.py]=1 # Investigate
     [test_ftplib.py]=1
     [test_functools.py]=1
+    [test_future.py]=1 # parse error
+    [test___future__.py]=1 # test failure
+    [test_future5.py]=1 # parse error
 
+    [test_gc.py]=1 # parse error
     [test_gdb.py]=1 # it fails on its own
+    [test_genericpath.py]=1 # parse error
     [test_generators.py]=1  # improper decompile of assert i < n and (n-i) % 3 == 0
+    [test_getpass.py]=1  # parse error
+    [test_gettext.py]=1  # parse error
     [test_glob.py]=1  #
     [test_grammar.py]=1
     [test_grp.py]=1 # Doesn't terminate (killed)
     [test_gzip.py]=1 # parse error
 
     [test_hashlib.py]=1 # test assert failures
+    [test_heapq.py]=1  # test failure
+    [test_hmac.py]=1  # parse error
     [test_httplib.py]=1 # parse error
     [test_http_cookiejar.py]=1
-    [test_imaplib-3.7.py]=1
+    [test_httpservers.py]=1 # parse error
-    [test_idle.py]=1 # Probably installation specific
+
+    [test_imghdr.py]=1 # parse error
+    [test_imp.py]=1 # parse error
+    [test_int.py]=1 # parse error
     [test_io.py]=1 # test takes too long to run: 37 seconds
+    [test_ioctl.py]=1 # parse error
     [test_imaplib.py]=1
+    [test_ipaddress.py]=1 # parse error
     [test_index.py]=1
     [test_inspect.py]=1
+    [test_iter.py]=1 # parse error
     [test_itertools.py]=1 # parse error
+
     [test_keywordonlyarg.py]=1 # parse error
     [test_kqueue.py]=1 # it fails on its own
+
+    [test__locale.py]=1 # parse error
+    [test_largefile.py]=1 # parse error
     [test_lib2to3.py]=1 # it fails on its own
-    [test_long.py]=1 # investigate
+    [test_linecache.py]=1 # parse error
+    [test_lltrace.py]=1 # parse error
+    [test_locale.py]=1 # parse error
     [test_logging.py]=1 # test takes too long to run: 20 seconds
-    [test_mailbox.py]=1
+    [test_long.py]=1 # investigate
+    [test_lzma.py]=1 # it fails on its own
+
+    [test_mailbox.py]=1 # parse error
+    [test_mailcap.py]=1 # parse error
     [test_marshal.py]=1
     [test_math.py]=1
+    [test_memoryio.py]=1 # test failure
+    [test_memoryview.py]=1 # parse error
+    [test_minidom.py]=1 # test failure
+    [test_mmap.py]=1 # parse error
     [test_modulefinder.py]=1
     [test_msilib.py]=1
     [test_multiprocessing_fork.py]=1 # test takes too long to run: 62 seconds
     [test_multiprocessing_forkserver.py]=1
+    [test_multiprocessing_main_handling.py]=1 # parse error
     [test_multiprocessing_spawn.py]=1
+
+    [test_named_expressions.py]=1 # parse error
+    [test_netrc.py]=1 # parse error
+    [test_nis.py]=1 # break outside of loop
     [test_normalization.py]=1 # probably control flow (uninitialized variable)
     [test_nntplib.py]=1
+    [test_ntpath.py]=1
+
+    [test__osx_support.py]=1 # parse error
+    [test_opcodes.py]=1 # parse error
+    [test_operator.py]=1 # parse error
     [test_optparse.py]=1 # doesn't terminate (killed)
+    [test_ordered_dict.py]=1 # parse error
     [test_os.py]=1 # probably control flow (uninitialized variable)
     [test_ossaudiodev.py]=1 # it fails on its own
+    [test_osx_env.py]=1 # parse error
+
     [test_pathlib.py]=1 # parse error
     [test_pdb.py]=1 # Probably relies on comments
     [test_peepholer.py]=1 # decompile takes a long time; then test assert error
     [test_pickle.py]=1 # Probably relies on comments
-    [test_poll.py]=1
+    [test_picklebuffer.py]=1 # parse error
-    [test_poplib.py]=1
+    [test_pipes.py]=1 # parse error
-    [test_pydoc.py]=1 # it fails on its own
+    [test_pkg.py]=1 # Investigate: lists differ
-    [test_runpy.py]=1  #
+    [test_pkgimport.py]=1 # parse error
-    [test_pkg.py]=1 # parse error; Investigate: lists differ
+    [test_pkgutil.py]=1 # Investigate:
-    [test_pkgutil.py]=1 # parse error
     [test_platform.py]=1 # parse error
+    [test_plistlib.py]=1 # parse error
+    [test_poll.py]=1
+    [test_popen.py]=1 # parse error
+    [test_poplib.py]=1 # Parse error
+    [test_positional_only_arg.py]=1 # test failures
+    [test_posixpath.py]=1 # parse error
+    [test_posix.py]=1 # parse error
+    [test_print.py]=1 # parse error
+    [test_profile.py]=1 # parse error
     [test_pwd.py]=1 # killing - doesn't terminate
-    [test_regrtest.py]=1 # parse error; test assertion error: lists differ
+    [test_pulldom.py]=1 # killing - doesn't terminate
-    [test_re.py]=1 # parse error; test assertion error
+    [test_py_compile.py]=1 # parse error
-    [test_richcmp.py]=1 # Investigate: data[i] index error in semantic handling
+    [test_pyexpat.py]=1 # parse error
+    [test_pyclbr.py]=1 # test failure
+    [test_pydoc.py]=1 # it fails on its own
+
+    [test_queue.py]=1  # parse error
+
+    [test_raise.py]=1  # parse error
+    [test_random.py]=1  # parse error
+    [test_range.py]=1  # parse error
+    [test_rcompleter.py]=1  # parse error
+    [test_re.py]=1 # test assertion error
+    [test_readline.py]=1  # parse error
+    [test_robotparser.py]=1  # too long to run before decompiling: 31 secs
+    [test_regrtest.py]=1 # lists differ
+    [test_reprlib.py]=1 # parse error
+    [test_resource.py]=1  # parse error
+    [test_richcmp.py]=1 # parse error
+    [test_runpy.py]=1  #
+
+    [test_sax.py]=1 # parse error
+    [test_sched.py]=1 # parse error
+    [test_scope.py]=1 # parse error
+    [test_script_helper.py]=1 # parse error
     [test_select.py]=1 # test takes too long to run: 11 seconds
     [test_selectors.py]=1
+    [test_set.py]=1 # parse error
+    [test_shelve.py]=1 # parse error
+    [test_shlex.py]=1 # probably control flow
     [test_shutil.py]=1 # fails on its own
     [test_signal.py]=1 #
-    [test_slice.py]=1 # Investigate: test assertion error
+    [test_site.py]=1 # parse error
+    [test_slice.py]=1 # Investigate
+    [test_smtpd.py]=1 # parse error
     [test_smtplib.py]=1 #
+    [test_smtpnet.py]=1 # parse error
     [test_socket.py]=1
     [test_socketserver.py]=1
-    [test_sort.py]=1 # parse error;
+    [test_sort.py]=1 # Probably control flow; unintialized varaible
-    [test_ssl.py]=1 # parse error
+    [test_source_encoding.py]=1 # parse error
+    [test_spwd.py]=1 # parse error
+    [test_ssl.py]=1 # Probably control flow; unintialized varaible
     [test_startfile.py]=1 # it fails on its own
-    [test_statistics.py]=1 # Takes more than 15 secs to run. Assert failures
+    [test_stat.py]=1 # test assertions failed
-    [test_stat.py]=1 # parse error; test assertions failed
+    [test_statistics.py]=1 # Probably control flow; unintialized varaible
-    [test_string_literals.py]=1 # parse error; Investigate boolean parsing
+    [test_strftime.py]=1 # parse error
+    [test_string.py]=1 # parse error
+    [test_string_literals.py]=1 # parse error
     [test_strptime.py]=1 # test assertions failed
     [test_strtod.py]=1 # test assertions failed
-    [test_structmembers.py]=1 # test assertions failed
     [test_struct.py]=1 # test assertions failed
+    [test_structmembers.py]=1 # test assertions failed
+    [test_subclassinit.py]=1 # parse error
     [test_subprocess.py]=1
+    [test_super.py]=1 # parse error
+    [test_support.py]=1 # parse error
+    [test_symbol.py]=1 # parse error
+    [test_sys.py]=1 # parse error
     [test_sys_setprofile.py]=1 # test assertions failed
     [test_sys_settrace.py]=1 # parse error
+    [test_sysconfig.py]=1 # parse error
+
+    [test_tabnanny.py]=1 # parse error
     [test_tarfile.py]=1 # parse error
+    [test_tcl.py]=1 # parse error
+    [test_telnetlib.py]=1 # parse error
+    [test_tempfile.py]=1 # parse error
+    [test_thread.py]=1 # parse error
+    [test_threaded_import.py]=1 # parse error
     [test_threading.py]=1 #
     [test_timeit.py]=1 # probably control flow uninitialized variable
+    [test_timeout.py]=1 # parse error
     [test_tk.py]=1  # test takes too long to run: 13 seconds
     [test_tokenize.py]=1
     [test_trace.py]=1  # it fails on its own
     [test_traceback.py]=1 # Probably uses comment for testing
     [test_tracemalloc.py]=1 #
     [test_ttk_guionly.py]=1  # implementation specfic and test takes too long to run: 19 seconds
-    [test_typing.py]=1 # parse error
+    [test_turtle_import.py]=1 # parse error
+    [test_turtle.py]=1 # parse error
     [test_types.py]=1 # parse error
+    [test_typing.py]=1 # parse error
+
+    [test_ucn.py]=1 # parse error
     [test_unicode.py]=1 # unicode thing
+    [test_unicode_file_functions.py]=1 # parse faiure
+    [test_unicodedata.py]=1 # test faiure
+    [test_univnewlines.py]=1 # parse error
+    [test_urllib.py]=1 # parse error
     [test_urllib2.py]=1 #
+    [test_urllib_response.py]=1 # parse error
     [test_urllib2_localnet.py]=1 #
+    [test_urllib2net.py]=1 # parse error
     [test_urllibnet.py]=1 # probably control flow - uninitialized variable
+    [test_urlparse.py]=1 # parse error
+    [test_userdict.py]=1 # test failures
+    [test_userstring.py]=1 # parse error
+    [test_utf8.py]=1 # parse error
+    [test_utf8_mode.py]=1 # parse error
+    [test_uu.py]=1 # parse error
+    [test_uuid.py]=1 # parse error
+
+    [test_venv.py]=1 # parse error
+
     [test_weakref.py]=1 # probably control flow - uninitialized variable
+    [test_weakset.py]=1 # parse error
+    [test_webbrowser.py]=1 # parse error
     [test_with.py]=1 # probably control flow - uninitialized variable
-    [test_xml_dom_minicompat.py]=1 # parse error
     [test_winconsoleio.py]=1 # it fails on its own
     [test_winreg.py]=1 # it fails on its own
     [test_winsound.py]=1 # it fails on its own
+    [test_wsgiref.py]=1 # parse error
+
+    [test_xml_etree.py]=1 # parse error
+    [test_xmlrpc.py]=1 # parse error
+    [test__xxsubinterpreters.py]=1 # parse error
+
+    [test_yield_from.py]=1 # parse error
+
+    [test_zlib.py]=1 # test looping take more than 15 seconds to run
+    [test_zipapp.py]=1 # parse error
+    [test_zipimport_support.py]=1 # parse error
     [test_zipfile.py]=1 # it fails on its own
     [test_zipfile64.py]=1 #
 )
-# 268 About unit-test files, in about 11 minutes
+# 114 test files, Elapsed time about 7 minutes
+
+if (( batch )) ; then
+    SKIP_TESTS[test_idle.py]=1 # Probably installation specific
+    SKIP_TESTS[test_tix.py]=1 # fails on its own
+    SKIP_TESTS[test_ttk_textonly.py]=1 # Installation dependent?
+
+fi

test/stdlib/compile-file.py

@@ -19,6 +19,6 @@ bytecode = "%s-%s.pyc" % (basename, PY_VERSION)
 
 import py_compile
 print("compiling %s to %s" % (source, bytecode))
-py_compile.compile(source, bytecode, 'exec')
+py_compile.compile(source, bytecode, source)
 # import os
 # os.system("../bin/uncompyle6 %s" % bytecode)

test/stdlib/runtests.sh

@@ -51,20 +51,7 @@ function timeout_cmd {
 typeset -A SKIP_TESTS
 case $PYVERSION in
     2.4)
-	SKIP_TESTS=(
+	. ./2.4-exclude.sh
-	    [test_decimal.py]=1  #
-	    [test_dis.py]=1   # We change line numbers - duh!
-	    [test_generators.py]=1  # Investigate
-	    [test_grammar.py]=1    # Too many stmts. Handle large stmts
-	    [test_grp.py]=1      # Long test - might work Control flow?
-	    [test_pep247.py]=1 # Long test - might work? Control flow?
-	    [test_pwd.py]=1 # Long test - might work? Control flow?
-	    [test_socketserver.py]=1 # -- test takes too long to run: 40 seconds
-	    [test_threading.py]=1 # test takes too long to run: 11 seconds
-	    [test_thread.py]=1 # test takes too long to run: 36 seconds
-	    [test_trace.py]=1 # Long test - works
-	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
-	)
 	;;
     2.5)
 	. ./2.5-exclude.sh

test/test_pyenvlib.py

@@ -229,6 +229,6 @@ if __name__ == "__main__":
 
 # Verification notes:
 # - xdrlib fails verification due the same lambda used twice
-#   (verification is successfull when using original .pyo as
+#   (verification is successful when using original .pyo as
 #   input)
 #

uncompyle6/bin/pydisassemble.py

@@ -19,7 +19,7 @@ Usage:
 Disassemble FILE with the instruction mangling that is done to
 assist uncompyle6 in parsing the instruction stream. For example
 instructions with variable-length arguments like CALL_FUNCTION and
-BUILD_LIST have arguement counts appended to the instruction name, and
+BUILD_LIST have argument counts appended to the instruction name, and
 COME_FROM instructions are inserted into the instruction stream.
 
 Examples:

uncompyle6/parser.py

@@ -570,14 +570,14 @@ class PythonParser(GenericASTBuilder):
 
         _mklambda ::= mklambda
 
-        expr ::= conditional
+        expr ::= if_exp
 
         ret_expr ::= expr
         ret_expr ::= ret_and
         ret_expr ::= ret_or
 
         ret_expr_or_cond ::= ret_expr
-        ret_expr_or_cond ::= ret_cond
+        ret_expr_or_cond ::= if_exp_ret
 
         stmt ::= return_lambda
 

uncompyle6/parsers/parse2.py

@@ -671,8 +671,9 @@ class Python2Parser(PythonParser):
 
         fn = self.reduce_check_table.get(lhs, None)
         if fn:
-            return fn(self, lhs, n, rule, ast, tokens, first, last)
+            if fn(self, lhs, n, rule, ast, tokens, first, last):
-
+                return True
+            pass
         if rule == ("and", ("expr", "jmp_false", "expr", "\\e_come_from_opt")):
             # If the instruction after the instructions forming the "and"  is an "YIELD_VALUE"
             # then this is probably an "if" inside a comprehension.

uncompyle6/parsers/parse23.py

@@ -49,7 +49,7 @@ class Python23Parser(Python24Parser):
         and2 ::= _jump jmp_false COME_FROM expr COME_FROM
 
         alias       ::= IMPORT_NAME attributes store
-        conditional ::= expr jmp_false expr JUMP_FORWARD expr COME_FROM
+        if_exp      ::= expr jmp_false expr JUMP_FORWARD expr COME_FROM
         '''
 
     def customize_grammar_rules(self, tokens, customize):

uncompyle6/parsers/parse24.py

@@ -58,7 +58,7 @@ class Python24Parser(Python25Parser):
 
     def remove_rules_24(self):
         self.remove_rules("""
-        expr ::= conditional
+        expr ::= if_exp
         """)
 
 

uncompyle6/parsers/parse25.py

@@ -72,7 +72,7 @@ class Python25Parser(Python26Parser):
         classdefdeco1 ::= expr classdefdeco2 CALL_FUNCTION_1
         classdefdeco2 ::= LOAD_CONST expr mkfunc CALL_FUNCTION_0 BUILD_CLASS
         kv3 ::= expr expr STORE_MAP
-        ret_cond ::= expr jmp_false_then expr RETURN_END_IF POP_TOP ret_expr_or_cond
+        if_exp_ret       ::= expr jmp_false_then expr RETURN_END_IF POP_TOP ret_expr_or_cond
         return_if_lambda ::= RETURN_END_IF_LAMBDA POP_TOP
         return_if_stmt   ::= ret_expr RETURN_END_IF POP_TOP
         return_if_stmts  ::= return_if_stmt
@@ -81,13 +81,12 @@ class Python25Parser(Python26Parser):
         return_stmt_lambda ::= ret_expr RETURN_VALUE_LAMBDA
         setupwithas      ::= DUP_TOP LOAD_ATTR ROT_TWO LOAD_ATTR CALL_FUNCTION_0 setup_finally
         stmt             ::= classdefdeco
-        stmt             ::= if_expr_lambda
+        stmt             ::= if_exp_lambda
-        stmt             ::= conditional_not_lambda
+        stmt             ::= if_exp_not_lambda
-        if_expr_lambda   ::= expr jmp_false_then expr return_if_lambda
+        if_exp_lambda    ::= expr jmp_false_then expr return_if_lambda
-                               return_stmt_lambda LAMBDA_MARKER
-        conditional_not_lambda
-                           ::= expr jmp_true_then expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
+        if_exp_not_lambda ::= expr jmp_true_then expr return_if_lambda
+                              return_stmt_lambda LAMBDA_MARKER
         """)
         super(Python25Parser, self).customize_grammar_rules(tokens, customize)
         if self.version == 2.5:

uncompyle6/parsers/parse26.py

@@ -262,8 +262,8 @@ class Python26Parser(Python2Parser):
         '''
         ret_and      ::= expr jmp_false ret_expr_or_cond COME_FROM
         ret_or       ::= expr jmp_true ret_expr_or_cond COME_FROM
-        ret_cond     ::= expr jmp_false_then expr RETURN_END_IF POP_TOP ret_expr_or_cond
+        if_exp_ret   ::= expr jmp_false_then expr RETURN_END_IF POP_TOP ret_expr_or_cond
-        ret_cond     ::= expr jmp_false_then expr ret_expr_or_cond
+        if_exp_ret   ::= expr jmp_false_then expr ret_expr_or_cond
 
         return_if_stmt ::= ret_expr RETURN_END_IF POP_TOP
         return ::= ret_expr RETURN_VALUE POP_TOP
@@ -283,11 +283,11 @@ class Python26Parser(Python2Parser):
         kvlist ::= kvlist kv3
 
         # Note: preserve positions 0 2 and 4 for semantic actions
-        conditional_not    ::= expr jmp_true  expr jf_cf_pop expr COME_FROM
+        if_exp_not         ::= expr jmp_true  expr jf_cf_pop expr COME_FROM
-        conditional        ::= expr jmp_false expr jf_cf_pop expr come_from_opt
+        if_exp             ::= expr jmp_false expr jf_cf_pop expr come_from_opt
-        conditional        ::= expr jmp_false expr ja_cf_pop expr
+        if_exp             ::= expr jmp_false expr ja_cf_pop expr
 
-        expr               ::= conditional_not
+        expr               ::= if_exp_not
 
         and                ::= expr JUMP_IF_FALSE POP_TOP expr JUMP_IF_FALSE POP_TOP
 
@@ -311,27 +311,27 @@ class Python26Parser(Python2Parser):
         compare_chained2   ::= expr COMPARE_OP return_lambda
 
         return_if_lambda   ::= RETURN_END_IF_LAMBDA POP_TOP
-        stmt               ::= if_expr_lambda
+        stmt               ::= if_exp_lambda
-        stmt               ::= conditional_not_lambda
+        stmt               ::= if_exp_not_lambda
-        if_expr_lambda     ::= expr jmp_false_then expr return_if_lambda
+        if_exp_lambda      ::= expr jmp_false_then expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
-        conditional_not_lambda ::=
+        if_exp_not_lambda ::=
                                expr jmp_true_then expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
 
-        # if_expr_true are for conditions which always evaluate true
+        # if_exp_true are for conditions which always evaluate true
         # There is dead or non-optional remnants of the condition code though,
         # and we use that to match on to reconstruct the source more accurately
-        expr               ::= if_expr_true
+        expr               ::= if_exp_true
-        if_expr_true       ::= expr jf_pop expr COME_FROM
+        if_exp_true        ::= expr jf_pop expr COME_FROM
 
         # This comes from
         #   0 or max(5, 3) if 0 else 3
         # where there seems to be an additional COME_FROM at the
         # end. Not sure if this is appropriately named or
         # is the best way to handle
-        expr               ::= conditional_false
+        expr               ::= if_exp_false
-        conditional_false  ::= conditional COME_FROM
+        if_exp_false  ::= if_exp COME_FROM
 
         """
 
@@ -366,10 +366,10 @@ class Python26Parser(Python2Parser):
             if ast[1] is None:
                 return False
 
-            # For now, we won't let the 2nd 'expr' be a "conditional_not"
+            # For now, we won't let the 2nd 'expr' be a "if_exp_not"
             # However in < 2.6 where we don't have if/else expression it *can*
             # be.
-            if self.version >= 2.6 and ast[2][0] == 'conditional_not':
+            if self.version >= 2.6 and ast[2][0] == "if_exp_not":
                 return True
 
             test_index = last

uncompyle6/parsers/parse27.py

@@ -6,6 +6,10 @@ from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from xdis import next_offset
 from uncompyle6.parser import PythonParserSingle, nop_func
 from uncompyle6.parsers.parse2 import Python2Parser
+from uncompyle6.parsers.reducecheck import (
+    ifelsestmt,
+    tryelsestmt,
+)
 
 class Python27Parser(Python2Parser):
 
@@ -93,9 +97,9 @@ class Python27Parser(Python2Parser):
         jmp_false ::= POP_JUMP_IF_FALSE
         jmp_true  ::= POP_JUMP_IF_TRUE
 
-        ret_and  ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
+        ret_and    ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
-        ret_or   ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+        ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
-        ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
+        if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
         or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
         and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
@@ -112,17 +116,17 @@ class Python27Parser(Python2Parser):
         compare_chained2 ::= expr COMPARE_OP return_lambda
         compare_chained2 ::= expr COMPARE_OP return_lambda
 
-        # if_expr_true are for conditions which always evaluate true
+        # if_exp_true are for conditions which always evaluate true
         # There is dead or non-optional remnants of the condition code though,
         # and we use that to match on to reconstruct the source more accurately.
         # FIXME: we should do analysis and reduce *only* if there is dead code?
         #        right now we check that expr is "or". Any other nodes types?
 
-        expr             ::= if_expr_true
+        expr             ::= if_exp_true
-        if_expr_true     ::= expr JUMP_FORWARD expr COME_FROM
+        if_exp_true      ::= expr JUMP_FORWARD expr COME_FROM
 
-        conditional      ::= expr jmp_false expr JUMP_FORWARD expr COME_FROM
+        if_exp           ::= expr jmp_false expr JUMP_FORWARD expr COME_FROM
-        conditional      ::= expr jmp_false expr JUMP_ABSOLUTE expr
+        if_exp           ::= expr jmp_false expr JUMP_ABSOLUTE expr
         """
 
     def p_stmt27(self, args):
@@ -176,26 +180,29 @@ class Python27Parser(Python2Parser):
 
         ifstmt            ::= testexpr return_stmts COME_FROM
         ifstmt            ::= testexpr return_if_stmts COME_FROM
-        ifelsestmt        ::= testexpr c_stmts_opt JUMP_FORWARD else_suite COME_FROM
+        ifelsestmt        ::= testexpr c_stmts_opt JUMP_FORWARD else_suite come_froms
         ifelsestmtc       ::= testexpr c_stmts_opt JUMP_ABSOLUTE else_suitec
         ifelsestmtl       ::= testexpr c_stmts_opt JUMP_BACK else_suitel
         ifelsestmtl       ::= testexpr c_stmts_opt CONTINUE else_suitel
 
+        # In the future when we have ifelsestmtl checking we should add something like:
+        # ifelsestmtl       ::= testexpr c_stmts_opt JUMP_FORWARD else_suite come_froms
+        # c_stmts           ::= ifelsestmtl
+
         # "if"/"else" statement that ends in a RETURN
         ifelsestmtr       ::= testexpr return_if_stmts COME_FROM returns
 
         # Common with 2.6
         return_if_lambda   ::= RETURN_END_IF_LAMBDA COME_FROM
-        stmt               ::= if_expr_lambda
+        stmt               ::= if_exp_lambda
-        stmt               ::= conditional_not_lambda
+        stmt               ::= if_exp_not_lambda
-        if_expr_lambda     ::= expr jmp_false expr return_if_lambda
+        if_exp_lambda      ::= expr jmp_false expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
-        conditional_not_lambda
+        if_exp_not_lambda  ::= expr jmp_true expr return_if_lambda
-                           ::= expr jmp_true expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
 
-        expr ::= conditional_not
+        expr               ::= if_exp_not
-        conditional_not ::= expr jmp_true expr _jump expr COME_FROM
+        if_exp_not         ::= expr jmp_true expr _jump expr COME_FROM
 
         kv3 ::= expr expr STORE_MAP
         """
@@ -216,8 +223,16 @@ class Python27Parser(Python2Parser):
 
 
         super(Python27Parser, self).customize_grammar_rules(tokens, customize)
+
+        # FIXME: Put more in this table
+        self.reduce_check_table = {
+            # "ifelsestmt": ifelsestmt,
+            "tryelsestmt": tryelsestmt,
+            "tryelsestmtl": tryelsestmt,
+        }
+
         self.check_reduce["and"] = "AST"
-        self.check_reduce["conditional"] = "AST"
+        self.check_reduce["if_exp"] = "AST"
 
         self.check_reduce["except_handler"] = "tokens"
         self.check_reduce["except_handler_else"] = "tokens"
@@ -225,10 +240,11 @@ class Python27Parser(Python2Parser):
         # self.check_reduce["or"] = "AST"
         self.check_reduce["raise_stmt1"] = "AST"
         self.check_reduce["iflaststmtl"] = "AST"
+        self.check_reduce["ifelsestmt"] = "AST"
         self.check_reduce["list_if_not"] = "AST"
         self.check_reduce["list_if"] = "AST"
         self.check_reduce["comp_if"] = "AST"
-        self.check_reduce["if_expr_true"] = "tokens"
+        self.check_reduce["if_exp_true"] = "tokens"
         self.check_reduce["whilestmt"] = "tokens"
 
         return
@@ -238,6 +254,12 @@ class Python27Parser(Python2Parser):
                         self).reduce_is_invalid(rule, ast,
                                                 tokens, first, last)
 
+        lhs = rule[0]
+        n = len(tokens)
+        fn = self.reduce_check_table.get(lhs, None)
+        if fn:
+            invalid = fn(self, lhs, n, rule, ast, tokens, first, last)
+        last = min(last, n-1)
         if invalid:
             return invalid
 
@@ -247,7 +269,7 @@ class Python27Parser(Python2Parser):
                 return tokens[first].offset < jmp_false[0].attr < tokens[last].offset
             pass
         elif (rule[0], rule[1][0:5]) == (
-                "conditional",
+                "if_exp",
                 ("expr", "jmp_false", "expr", "JUMP_ABSOLUTE", "expr")):
             jmp_false = ast[1]
             if jmp_false[0] == "POP_JUMP_IF_FALSE":
@@ -317,7 +339,7 @@ class Python27Parser(Python2Parser):
             while (tokens[i] != "JUMP_BACK"):
                 i -= 1
             return tokens[i].attr != tokens[i-1].attr
-        elif rule[0] == "if_expr_true":
+        elif rule[0] == "if_exp_true":
             return (first) > 0 and tokens[first-1] == "POP_JUMP_IF_FALSE"
 
         return False

uncompyle6/parsers/parse3.py

@@ -30,9 +30,12 @@ import re
 from uncompyle6.scanners.tok import Token
 from uncompyle6.parser import PythonParser, PythonParserSingle, nop_func
 from uncompyle6.parsers.reducecheck import (
+    and_check,
     except_handler_else,
+    ifelsestmt,
     ifstmt,
-    # iflaststmt,
+    iflaststmt,
+    or_check,
     testtrue,
     tryelsestmtl3,
     tryexcept,
@@ -331,9 +334,9 @@ class Python3Parser(PythonParser):
         jmp_true  ::= POP_JUMP_IF_TRUE
 
         # FIXME: Common with 2.7
-        ret_and  ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
+        ret_and    ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
-        ret_or   ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+        ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
-        ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
+        if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
         or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
         or   ::= expr jmp_true expr
@@ -348,13 +351,12 @@ class Python3Parser(PythonParser):
 
     def p_stmt3(self, args):
         """
-        stmt               ::= if_expr_lambda
+        stmt               ::= if_exp_lambda
 
-        stmt               ::= conditional_not_lambda
+        stmt               ::= if_exp_not_lambda
-        if_expr_lambda     ::= expr jmp_false expr return_if_lambda
+        if_exp_lambda      ::= expr jmp_false expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
-        conditional_not_lambda
+        if_exp_not_lambda  ::= expr jmp_true expr return_if_lambda
-                           ::= expr jmp_true expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
 
         return_stmt_lambda ::= ret_expr RETURN_VALUE_LAMBDA
@@ -468,18 +470,18 @@ class Python3Parser(PythonParser):
     def p_expr3(self, args):
         """
         expr           ::= LOAD_STR
-        expr           ::= conditionalnot
+        expr           ::= if_exp_not
-        conditionalnot ::= expr jmp_true  expr jump_forward_else expr COME_FROM
+        if_exp_not     ::= expr jmp_true  expr jump_forward_else expr COME_FROM
 
         # a JUMP_FORWARD to another JUMP_FORWARD can get turned into
         # a JUMP_ABSOLUTE with no COME_FROM
-        conditional    ::= expr jmp_false expr jump_absolute_else expr
+        if_exp         ::= expr jmp_false expr jump_absolute_else expr
 
-        # if_expr_true are for conditions which always evaluate true
+        # if_exp_true are for conditions which always evaluate true
         # There is dead or non-optional remnants of the condition code though,
         # and we use that to match on to reconstruct the source more accurately
-        expr           ::= if_expr_true
+        expr           ::= if_exp_true
-        if_expr_true   ::= expr JUMP_FORWARD expr COME_FROM
+        if_exp_true    ::= expr JUMP_FORWARD expr COME_FROM
         """
 
     @staticmethod
@@ -704,12 +706,11 @@ class Python3Parser(PythonParser):
               stmt ::= assign2_pypy
               assign3_pypy       ::= expr expr expr store store store
               assign2_pypy       ::= expr expr store store
-              stmt               ::= if_expr_lambda
+              stmt               ::= if_exp_lambda
-              stmt               ::= conditional_not_lambda
+              stmt               ::= if_exp_not_lambda
               if_expr_lambda     ::= expr jmp_false expr return_if_lambda
                                      return_lambda LAMBDA_MARKER
-              conditional_not_lambda
+              if_exp_not_lambda  ::= expr jmp_true expr return_if_lambda
-                                 ::= expr jmp_true expr return_if_lambda
                                      return_lambda LAMBDA_MARKER
               """,
                 nop_func,
@@ -1528,15 +1529,36 @@ class Python3Parser(PythonParser):
                 pass
             pass
 
+        # FIXME: Put more in this table
+        self.reduce_check_table = {
+            "except_handler_else": except_handler_else,
+            # "ifstmt": ifstmt,
+            "ifstmtl": ifstmt,
+            "ifelsestmtc": ifelsestmt,
+            "ifelsestmt": ifelsestmt,
+            "or": or_check,
+            "testtrue": testtrue,
+            "tryelsestmtl3": tryelsestmtl3,
+            "try_except": tryexcept,
+        }
+
+        if self.version == 3.6:
+            self.reduce_check_table["and"] =  and_check
+            self.check_reduce["and"] = "AST"
+
+        self.check_reduce["annotate_tuple"] = "noAST"
         self.check_reduce["aug_assign1"] = "AST"
         self.check_reduce["aug_assign2"] = "AST"
-        self.check_reduce["while1stmt"] = "noAST"
-        self.check_reduce["while1elsestmt"] = "noAST"
-        self.check_reduce["ifelsestmt"] = "AST"
-        self.check_reduce["ifstmt"] = "AST"
-        # self.check_reduce["iflaststmtl"] = "AST"
-        self.check_reduce["annotate_tuple"] = "noAST"
         self.check_reduce["except_handler_else"] = "tokens"
+        self.check_reduce["ifelsestmt"] = "AST"
+        self.check_reduce["ifelsestmtc"] = "AST"
+        self.check_reduce["ifstmt"] = "AST"
+        self.check_reduce["ifstmtl"] = "AST"
+        if self.version == 3.6:
+            self.reduce_check_table["iflaststmtl"] = iflaststmt
+            self.check_reduce["iflaststmt"] = "AST"
+            self.check_reduce["iflaststmtl"] = "AST"
+        self.check_reduce["or"] = "AST"
         self.check_reduce["testtrue"] = "tokens"
         if not PYTHON3:
             self.check_reduce["kwarg"] = "noAST"
@@ -1546,25 +1568,29 @@ class Python3Parser(PythonParser):
             self.check_reduce["try_except"] = "AST"
 
         self.check_reduce["tryelsestmtl3"] = "AST"
-        # FIXME: remove parser errors caused by the below
+        self.check_reduce["while1stmt"] = "noAST"
-        # self.check_reduce['while1elsestmt'] = 'noAST'
+        self.check_reduce["while1elsestmt"] = "noAST"
         return
 
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
         lhs = rule[0]
         n = len(tokens)
         last = min(last, n-1)
+        fn = self.reduce_check_table.get(lhs, None)
+        if fn:
+            if fn(self, lhs, n, rule, ast, tokens, first, last):
+                return True
+            pass
+        # FIXME: put more in reduce_check_table
         if lhs in ("aug_assign1", "aug_assign2") and ast[0][0] == "and":
             return True
         elif lhs == "annotate_tuple":
             return not isinstance(tokens[first].attr, tuple)
-        elif lhs in ("except_handler_else"):
-            return except_handler_else(self, lhs, n, rule, ast, tokens, first, last)
         elif lhs == "kwarg":
             arg = tokens[first].attr
             return not (isinstance(arg, str) or isinstance(arg, unicode))
-        # elif lhs == "iflaststmtl":
+        elif lhs in ("iflaststmt", "iflaststmtl") and self.version == 3.6:
-        #     return iflaststmt(self, lhs, n, rule, ast, tokens, first, last)
+            return ifstmt(self, lhs, n, rule, ast, tokens, first, last)
         elif rule == ("ifstmt", ("testexpr", "_ifstmts_jump")):
             # FIXME: go over what's up with 3.0. Evetually I'd like to remove RETURN_END_IF
             if self.version <= 3.0 or tokens[last] == "RETURN_END_IF":
@@ -1604,10 +1630,6 @@ class Python3Parser(PythonParser):
                 <= jump_forward_else[0].attr
                 < tokens[last].off2int()
             )
-        elif lhs == "testtrue":
-            return testtrue(self, lhs, n, rule, ast, tokens, first, last)
-        elif lhs == "tryelsestmtl3":
-            return tryelsestmtl3(self, lhs, n, rule, ast, tokens, first, last)
         elif lhs == "while1stmt":
 
             if while1stmt(self, lhs, n, rule, ast, tokens, first, last):
@@ -1656,8 +1678,6 @@ class Python3Parser(PythonParser):
                 return False
             # 3.8+ Doesn't have SETUP_LOOP
             return self.version < 3.8 and tokens[first].attr > tokens[last].offset
-        elif lhs == "try_except":
-            return tryexcept(self, lhs, n, rule, ast, tokens, first, last)
         elif rule == (
             "ifelsestmt",
             (

uncompyle6/parsers/parse30.py

@@ -263,9 +263,9 @@ class Python30Parser(Python31Parser):
                              compare_chained2 COME_FROM
         ret_or           ::= expr JUMP_IF_TRUE_OR_POP  ret_expr_or_cond COME_FROM
         ret_and          ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
-        ret_cond         ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF
+        if_exp_ret       ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF
                              COME_FROM ret_expr_or_cond
-        ret_expr_or_cond ::= ret_cond
+        ret_expr_or_cond ::= if_exp_ret
         or               ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
         and              ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
         and              ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM

uncompyle6/parsers/parse32.py

@@ -17,7 +17,7 @@ class Python32Parser(Python3Parser):
 
     def p_32to35(self, args):
         """
-        conditional ::= expr jmp_false expr jump_forward_else expr COME_FROM
+        if_exp            ::= expr jmp_false expr jump_forward_else expr COME_FROM
 
         # compare_chained2 is used in a "chained_compare": x <= y <= z
         # used exclusively in compare_chained

uncompyle6/parsers/parse35.py

@@ -171,26 +171,33 @@ class Python35Parser(Python34Parser):
             elif opname == 'BEFORE_ASYNC_WITH' and self.version < 3.8:
                 # Some Python 3.5+ async additions
                 rules_str = """
-                   async_with_stmt ::= expr
+                   async_with_stmt    ::= expr
-                   stmt ::= async_with_stmt
+                   stmt               ::= async_with_stmt
+                   async_with_pre     ::= BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM SETUP_ASYNC_WITH
+                   async_with_post    ::= COME_FROM_ASYNC_WITH
+                                          WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                          WITH_CLEANUP_FINISH END_FINALLY
 
-                   async_with_stmt ::= expr
+                   async_with_stmt    ::= expr
-                            BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                          async_with_pre
-                            SETUP_ASYNC_WITH POP_TOP suite_stmts_opt
+                                          POP_TOP
-                            POP_BLOCK LOAD_CONST COME_FROM_ASYNC_WITH
+                                          suite_stmts_opt
-                            WITH_CLEANUP_START
+                                          POP_BLOCK LOAD_CONST
-                            GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                          async_with_post
-                            WITH_CLEANUP_FINISH END_FINALLY
+                   async_with_stmt    ::= expr
+                                          async_with_pre
+                                          POP_TOP
+                                          suite_stmts_opt
+                                          async_with_post
 
-                   stmt ::= async_with_as_stmt
+                   stmt               ::= async_with_as_stmt
 
                    async_with_as_stmt ::= expr
-                               BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                          async_with_pre
-                               SETUP_ASYNC_WITH store suite_stmts_opt
+                                          store
-                               POP_BLOCK LOAD_CONST COME_FROM_ASYNC_WITH
+                                          suite_stmts_opt
-                               WITH_CLEANUP_START
+                                          POP_BLOCK LOAD_CONST
-                               GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                          async_with_post
-                               WITH_CLEANUP_FINISH END_FINALLY
                 """
                 self.addRule(rules_str, nop_func)
             elif opname == 'BUILD_MAP_UNPACK':

uncompyle6/parsers/parse36.py

@@ -65,7 +65,7 @@ class Python36Parser(Python35Parser):
         jf_cf       ::= JUMP_FORWARD COME_FROM
         cf_jf_else  ::= come_froms JUMP_FORWARD ELSE
 
-        conditional ::= expr jmp_false expr jf_cf expr COME_FROM
+        if_exp ::= expr jmp_false expr jf_cf expr COME_FROM
 
         async_for_stmt     ::= SETUP_LOOP expr
                                GET_AITER
@@ -113,9 +113,16 @@ class Python36Parser(Python35Parser):
         except_suite ::= c_stmts_opt COME_FROM POP_EXCEPT jump_except COME_FROM
 
         jb_cfs      ::= JUMP_BACK come_froms
+
+        # If statement inside a loop.
+        stmt                ::= ifstmtl
+        ifstmtl            ::= testexpr _ifstmts_jumpl
+        _ifstmts_jumpl     ::= c_stmts JUMP_BACK
+
         ifelsestmtl ::= testexpr c_stmts_opt jb_cfs else_suitel
         ifelsestmtl ::= testexpr c_stmts_opt cf_jf_else else_suitel
-        ifelsestmt ::= testexpr c_stmts_opt cf_jf_else else_suite _come_froms
+        ifelsestmt  ::= testexpr c_stmts_opt cf_jf_else else_suite _come_froms
+        ifelsestmt  ::= testexpr c_stmts come_froms else_suite come_froms
 
         # In 3.6+, A sequence of statements ending in a RETURN can cause
         # JUMP_FORWARD END_FINALLY to be omitted from try middle
@@ -158,8 +165,8 @@ class Python36Parser(Python35Parser):
     # that and then we can remove this.
     def p_37conditionals(self, args):
         """
-        expr                       ::= conditional37
+        expr                       ::= if_exp37
-        conditional37              ::= expr expr jf_cfs expr COME_FROM
+        if_exp37                   ::= expr expr jf_cfs expr COME_FROM
         jf_cfs                     ::= JUMP_FORWARD _come_froms
         ifelsestmt                 ::= testexpr c_stmts_opt jf_cfs else_suite opt_come_from_except
         """
@@ -169,6 +176,8 @@ class Python36Parser(Python35Parser):
         # """)
         super(Python36Parser, self).customize_grammar_rules(tokens, customize)
         self.remove_rules("""
+           _ifstmts_jumpl     ::= c_stmts_opt
+           _ifstmts_jumpl     ::= _ifstmts_jump
            except_handler     ::= JUMP_FORWARD COME_FROM_EXCEPT except_stmts END_FINALLY COME_FROM
            async_for_stmt     ::= SETUP_LOOP expr
                                   GET_AITER
@@ -231,24 +240,26 @@ class Python36Parser(Python35Parser):
             elif opname == 'BEFORE_ASYNC_WITH':
                 rules_str = """
                   stmt ::= async_with_stmt
+                  async_with_pre     ::= BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM SETUP_ASYNC_WITH
+                  async_with_post    ::= COME_FROM_ASYNC_WITH
+                                         WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                         WITH_CLEANUP_FINISH END_FINALLY
                   async_with_as_stmt ::= expr
-                               BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                               async_with_pre
-                               SETUP_ASYNC_WITH store
+                               store
                                suite_stmts_opt
                                POP_BLOCK LOAD_CONST
-                               COME_FROM_ASYNC_WITH
+                               async_with_post
-                               WITH_CLEANUP_START
-                               GET_AWAITABLE LOAD_CONST YIELD_FROM
-                               WITH_CLEANUP_FINISH END_FINALLY
                  stmt ::= async_with_as_stmt
                  async_with_stmt ::= expr
-                               BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                               POP_TOP
-                               SETUP_ASYNC_WITH POP_TOP suite_stmts_opt
+                               suite_stmts_opt
                                POP_BLOCK LOAD_CONST
-                               COME_FROM_ASYNC_WITH
+                               async_with_post
-                               WITH_CLEANUP_START
+                 async_with_stmt ::= expr
-                               GET_AWAITABLE LOAD_CONST YIELD_FROM
+                               POP_TOP
-                               WITH_CLEANUP_FINISH END_FINALLY
+                               suite_stmts_opt
+                               async_with_post
                 """
                 self.addRule(rules_str, nop_func)
 

uncompyle6/parsers/parse37.py

@@ -186,14 +186,14 @@ class Python37Parser(Python37BaseParser):
 
         _mklambda ::= mklambda
 
-        expr ::= conditional
+        expr      ::= if_exp
 
-        ret_expr ::= expr
+        ret_expr  ::= expr
-        ret_expr ::= ret_and
+        ret_expr  ::= ret_and
-        ret_expr ::= ret_or
+        ret_expr  ::= ret_or
 
         ret_expr_or_cond ::= ret_expr
-        ret_expr_or_cond ::= ret_cond
+        ret_expr_or_cond ::= if_exp_ret
 
         stmt ::= return_lambda
 
@@ -322,6 +322,7 @@ class Python37Parser(Python37BaseParser):
 
         import           ::= LOAD_CONST LOAD_CONST alias
         import_from_star ::= LOAD_CONST LOAD_CONST IMPORT_NAME IMPORT_STAR
+        import_from_star ::= LOAD_CONST LOAD_CONST IMPORT_NAME_ATTR IMPORT_STAR
         import_from      ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist POP_TOP
         importmultiple   ::= LOAD_CONST LOAD_CONST alias imports_cont
 
@@ -405,7 +406,7 @@ class Python37Parser(Python37BaseParser):
 
     def p_32on(self, args):
         """
-        conditional::= expr jmp_false expr jump_forward_else expr COME_FROM
+        if_exp::= expr jmp_false expr jump_forward_else expr COME_FROM
 
         # compare_chained2 is used in a "chained_compare": x <= y <= z
         # used exclusively in compare_chained
@@ -630,8 +631,8 @@ class Python37Parser(Python37BaseParser):
 
     def p_37conditionals(self, args):
         """
-        expr                       ::= conditional37
+        expr                       ::= if_exp37
-        conditional37              ::= expr expr jf_cfs expr COME_FROM
+        if_exp37              ::= expr expr jf_cfs expr COME_FROM
         jf_cfs                     ::= JUMP_FORWARD _come_froms
         ifelsestmt                 ::= testexpr c_stmts_opt jf_cfs else_suite opt_come_from_except
 
@@ -712,18 +713,18 @@ class Python37Parser(Python37BaseParser):
 
     def p_expr3(self, args):
         """
-        expr           ::= conditionalnot
+        expr           ::= if_exp_not
-        conditionalnot ::= expr jmp_true  expr jump_forward_else expr COME_FROM
+        if_exp_not     ::= expr jmp_true  expr jump_forward_else expr COME_FROM
 
         # a JUMP_FORWARD to another JUMP_FORWARD can get turned into
         # a JUMP_ABSOLUTE with no COME_FROM
-        conditional    ::= expr jmp_false expr jump_absolute_else expr
+        if_exp         ::= expr jmp_false expr jump_absolute_else expr
 
-        # if_expr_true are for conditions which always evaluate true
+        # if_exp_true are for conditions which always evaluate true
         # There is dead or non-optional remnants of the condition code though,
         # and we use that to match on to reconstruct the source more accurately
-        expr           ::= if_expr_true
+        expr           ::= if_exp_true
-        if_expr_true   ::= expr JUMP_FORWARD expr COME_FROM
+        if_exp_true    ::= expr JUMP_FORWARD expr COME_FROM
         """
 
     def p_generator_exp3(self, args):
@@ -920,9 +921,9 @@ class Python37Parser(Python37BaseParser):
         jmp_true  ::= POP_JUMP_IF_TRUE
 
         # FIXME: Common with 2.7
-        ret_and  ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
+        ret_and    ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
-        ret_or   ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+        ret_or     ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
-        ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
+        if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF COME_FROM ret_expr_or_cond
 
         jitop_come_from ::= JUMP_IF_TRUE_OR_POP come_froms
         jifop_come_from ::= JUMP_IF_FALSE_OR_POP come_froms
@@ -966,26 +967,27 @@ class Python37Parser(Python37BaseParser):
 
     def p_stmt3(self, args):
         """
-        stmt               ::= if_expr_lambda
+        stmt               ::= if_exp_lambda
-        stmt               ::= conditional_not_lambda
+        stmt               ::= if_exp_not_lambda
 
         # If statement inside a loop:
         stmt               ::= ifstmtl
 
-        if_expr_lambda     ::= expr jmp_false expr return_if_lambda
+        if_exp_lambda      ::= expr jmp_false expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
-        conditional_not_lambda
+        if_exp_not_lambda
                            ::= expr jmp_true expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
 
         return_stmt_lambda ::= ret_expr RETURN_VALUE_LAMBDA
         return_if_lambda   ::= RETURN_END_IF_LAMBDA
 
-        stmt ::= return_closure
+        stmt               ::= return_closure
-        return_closure ::= LOAD_CLOSURE RETURN_VALUE RETURN_LAST
+        return_closure     ::= LOAD_CLOSURE RETURN_VALUE RETURN_LAST
 
-        stmt ::= whileTruestmt
+        stmt               ::= whileTruestmt
-        ifelsestmt ::= testexpr c_stmts_opt JUMP_FORWARD else_suite _come_froms
+        ifelsestmt         ::= testexpr c_stmts_opt JUMP_FORWARD else_suite _come_froms
+        ifelsestmtl        ::= testexpr c_stmts_opt jump_forward_else else_suitec
 
         ifstmtl            ::= testexpr _ifstmts_jumpl
 
@@ -1091,7 +1093,7 @@ class Python37Parser(Python37BaseParser):
         jf_cf       ::= JUMP_FORWARD COME_FROM
         cf_jf_else  ::= come_froms JUMP_FORWARD ELSE
 
-        conditional ::= expr jmp_false expr jf_cf expr COME_FROM
+        if_exp ::= expr jmp_false expr jf_cf expr COME_FROM
 
         async_for_stmt     ::= setup_loop expr
                                GET_AITER
@@ -1207,25 +1209,31 @@ class Python37Parser(Python37BaseParser):
 
             elif opname == "BEFORE_ASYNC_WITH":
                 rules_str = """
-                  stmt ::= async_with_stmt
+                  stmt               ::= async_with_stmt SETUP_ASYNC_WITH
+                  async_with_pre     ::= BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM SETUP_ASYNC_WITH
+                  async_with_post    ::= COME_FROM_ASYNC_WITH
+                                         WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                         WITH_CLEANUP_FINISH END_FINALLY
+
+                  stmt               ::= async_with_as_stmt
                   async_with_as_stmt ::= expr
-                               BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                         async_with_pre
-                               SETUP_ASYNC_WITH store
+                                         store
-                               suite_stmts_opt
+                                         suite_stmts_opt
-                               POP_BLOCK LOAD_CONST
+                                         POP_BLOCK LOAD_CONST
-                               COME_FROM_ASYNC_WITH
+                                         async_with_post
-                               WITH_CLEANUP_START
+
-                               GET_AWAITABLE LOAD_CONST YIELD_FROM
+                 async_with_stmt     ::= expr
-                               WITH_CLEANUP_FINISH END_FINALLY
+                                         async_with_pre
-                 stmt ::= async_with_as_stmt
+                                         POP_TOP
-                 async_with_stmt ::= expr
+                                         suite_stmts_opt
-                               BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                         POP_BLOCK LOAD_CONST
-                               SETUP_ASYNC_WITH POP_TOP suite_stmts_opt
+                                         async_with_post
-                               POP_BLOCK LOAD_CONST
+                 async_with_stmt     ::= expr
-                               COME_FROM_ASYNC_WITH
+                                         async_with_pre
-                               WITH_CLEANUP_START
+                                         POP_TOP
-                               GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                         suite_stmts_opt
-                               WITH_CLEANUP_FINISH END_FINALLY
+                                         async_with_post
                 """
                 self.addRule(rules_str, nop_func)
 

uncompyle6/parsers/parse37base.py

@@ -159,12 +159,11 @@ class Python37BaseParser(PythonParser):
               stmt ::= assign2_pypy
               assign3_pypy       ::= expr expr expr store store store
               assign2_pypy       ::= expr expr store store
-              stmt               ::= if_expr_lambda
+              stmt               ::= if_exp_lambda
-              stmt               ::= conditional_not_lambda
+              stmt               ::= if_exp_not_lambda
-              if_expr_lambda     ::= expr jmp_false expr return_if_lambda
+              if_exp_lambda      ::= expr jmp_false expr return_if_lambda
                                      return_lambda LAMBDA_MARKER
-              conditional_not_lambda
+              if_exp_not_lambda  ::= expr jmp_true expr return_if_lambda
-                                 ::= expr jmp_true expr return_if_lambda
                                      return_lambda LAMBDA_MARKER
               """,
                 nop_func,
@@ -211,37 +210,57 @@ class Python37BaseParser(PythonParser):
 
                 if self.version < 3.8:
                     rules_str += """
-                       async_with_stmt    ::= expr
+                      stmt                ::= async_with_stmt SETUP_ASYNC_WITH
-                                              BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                      async_with_stmt     ::= expr
-                                              SETUP_ASYNC_WITH POP_TOP suite_stmts_opt
+                                              async_with_pre
-                                              POP_BLOCK LOAD_CONST COME_FROM_ASYNC_WITH
+                                              POP_TOP
-                                              WITH_CLEANUP_START
+                                              suite_stmts_opt
-                                              GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                              POP_BLOCK LOAD_CONST
-                                              WITH_CLEANUP_FINISH END_FINALLY
+                                              async_with_post
-                       async_with_as_stmt ::= expr
+                      async_with_stmt     ::= expr
-                                              BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                              async_with_pre
-                                              SETUP_ASYNC_WITH store suite_stmts_opt
+                                              POP_TOP
-                                              POP_BLOCK LOAD_CONST COME_FROM_ASYNC_WITH
+                                              suite_stmts_opt
-                                              WITH_CLEANUP_START
+                                              async_with_post
-                                              GET_AWAITABLE LOAD_CONST YIELD_FROM
+                      async_with_as_stmt  ::= expr
-                                              WITH_CLEANUP_FINISH END_FINALLY
+                                              async_with_pre
+                                              store
+                                              suite_stmts_opt
+                                              POP_BLOCK LOAD_CONST
+                                              async_with_post
                     """
                 else:
                     rules_str += """
-                       async_with_stmt    ::= expr
+                      async_with_pre      ::= BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM SETUP_ASYNC_WITH
-                                              BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                      async_with_post     ::= BEGIN_FINALLY COME_FROM_ASYNC_WITH
-                                              SETUP_ASYNC_WITH POP_TOP suite_stmts
+                                              WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
-                                              POP_TOP POP_BLOCK BEGIN_FINALLY COME_FROM_ASYNC_WITH
-                                              WITH_CLEANUP_START
-                                              GET_AWAITABLE LOAD_CONST YIELD_FROM
                                               WITH_CLEANUP_FINISH END_FINALLY
-                       async_with_as_stmt ::= expr
+                      async_with_stmt     ::= expr
-                                              BEFORE_ASYNC_WITH GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                              async_with_pre
-                                              SETUP_ASYNC_WITH store suite_stmts
+                                              POP_TOP
-                                              POP_TOP POP_BLOCK BEGIN_FINALLY COME_FROM_ASYNC_WITH
+                                              suite_stmts
-                                              WITH_CLEANUP_START
+                                              POP_TOP POP_BLOCK
-                                              GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                              async_with_post
+                      async_with_stmt     ::= expr
+                                              async_with_pre
+                                              POP_TOP
+                                              suite_stmts
+                                              POP_BLOCK
+                                              BEGIN_FINALLY
+                                              WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
+                                              WITH_CLEANUP_FINISH POP_FINALLY LOAD_CONST RETURN_VALUE
+                                              COME_FROM_ASYNC_WITH
+                                              WITH_CLEANUP_START GET_AWAITABLE LOAD_CONST YIELD_FROM
                                               WITH_CLEANUP_FINISH END_FINALLY
+                      async_with_as_stmt  ::= expr
+                                              async_with_pre
+                                              store suite_stmts
+                                              POP_TOP POP_BLOCK
+                                              async_with_post
+                      async_with_as_stmt  ::= expr
+                                              async_with_pre
+                                              store suite_stmts
+                                              POP_BLOCK async_with_post
                     """
                 self.addRule(rules_str, nop_func)
 
@@ -993,6 +1012,7 @@ class Python37BaseParser(PythonParser):
             "_ifstmts_jump": ifstmts_jump,
             "and": and_check,
             "ifelsestmt": ifelsestmt,
+            "ifelsestmtl": ifelsestmt,
             "iflaststmt": iflaststmt,
             "iflaststmtl": iflaststmt,
             "ifstmt": ifstmt,
@@ -1013,6 +1033,7 @@ class Python37BaseParser(PythonParser):
         self.check_reduce["while1elsestmt"] = "noAST"
         self.check_reduce["_ifstmts_jump"] = "AST"
         self.check_reduce["ifelsestmt"] = "AST"
+        self.check_reduce["ifelsestmtl"] = "AST"
         self.check_reduce["iflaststmt"] = "AST"
         self.check_reduce["iflaststmtl"] = "AST"
         self.check_reduce["ifstmt"] = "AST"

uncompyle6/parsers/reducecheck/and_check.py

@@ -14,6 +14,9 @@ def and_check(self, lhs, n, rule, ast, tokens, first, last):
         if rule == ("and", ("expr", "jmp_false", "expr", "jmp_false")):
             jmp2_target = ast[3][0].attr
             return jmp_target != jmp2_target
+        elif rule == ("and", ("expr", "jmp_false", "expr", "POP_JUMP_IF_TRUE")):
+            jmp2_target = ast[3].attr
+            return jmp_target == jmp2_target
         elif rule == ("and", ("expr", "jmp_false", "expr")):
             if tokens[last] == "POP_JUMP_IF_FALSE":
                 # Ok if jump_target doesn't jump to last instruction

uncompyle6/parsers/reducecheck/and_not_check.py

@@ -0,0 +1,19 @@
+#  Copyright (c) 2020 Rocky Bernstein
+
+
+def and_not_check(
+    self, lhs, n, rule, ast, tokens, first, last
+) -> bool:
+    jmp = ast[1]
+    if jmp.kind.startswith("jmp_"):
+        if last == n:
+            return True
+        jmp_target = jmp[0].attr
+
+        if tokens[first].off2int() <= jmp_target < tokens[last].off2int():
+            return True
+        if rule == ("and_not", ("expr", "jmp_false", "expr", "POP_JUMP_IF_TRUE")):
+            jmp2_target = ast[3].attr
+            return jmp_target != jmp2_target
+        return jmp_target != tokens[last].off2int()
+    return False

uncompyle6/parsers/reducecheck/ifelsestmt.py

@@ -2,14 +2,9 @@
 
 from uncompyle6.scanners.tok import Token
 
-
+IFELSE_STMT_RULES = frozenset(
-def ifelsestmt(self, lhs, n, rule, ast, tokens, first, last):
+    [
-    if (last + 1) < n and tokens[last + 1] == "COME_FROM_LOOP":
+                (
-        # ifelsestmt jumped outside of loop. No good.
-        return True
-
-    if rule not in (
-        (
             "ifelsestmt",
             (
                 "testexpr",
@@ -29,6 +24,24 @@ def ifelsestmt(self, lhs, n, rule, ast, tokens, first, last):
                 "\\e__come_froms",
             ),
         ),
+        (
+            "ifelsestmtl",
+            (
+                "testexpr",
+                "c_stmts_opt",
+                "jump_forward_else",
+                "else_suitec",
+            ),
+        ),
+        (
+            "ifelsestmtc",
+            (
+                "testexpr",
+                "c_stmts_opt",
+                "jump_absolute_else",
+                "else_suitec"
+            ),
+        ),
         (
             "ifelsestmt",
             (
@@ -39,6 +52,16 @@ def ifelsestmt(self, lhs, n, rule, ast, tokens, first, last):
                 "\\e_opt_come_from_except",
             ),
         ),
+        (
+            "ifelsestmt",
+            (
+                "testexpr",
+                "c_stmts_opt",
+                "JUMP_FORWARD",
+                "else_suite",
+                "come_froms",
+            ),
+        ),
         (
             "ifelsestmt",
             ("testexpr", "c_stmts", "come_froms", "else_suite", "come_froms",),
@@ -62,81 +85,119 @@ def ifelsestmt(self, lhs, n, rule, ast, tokens, first, last):
                 "else_suite",
             ),
         ),
-    ):
+    ])
+
+def ifelsestmt(self, lhs, n, rule, ast, tokens, first, last):
+
+    if (last + 1) < n and tokens[last + 1] == "COME_FROM_LOOP" and lhs != "ifelsestmtc":
+        # ifelsestmt jumped outside of loop. No good.
+        return True
+
+    if rule not in IFELSE_STMT_RULES:
         return False
+
+    # Avoid if/else where the "then" is a "raise_stmt1" for an
+    # assert statemetn. Parse this as an "assert" instead.
+    stmts = ast[1]
+    if stmts in ("c_stmts",) and len(stmts) == 1:
+        raise_stmt1 = stmts[0]
+        if (
+                raise_stmt1 == "raise_stmt1" and
+                raise_stmt1[0] in ("LOAD_ASSERT",)
+        ):
+            return True
+
     # Make sure all of the "come froms" offset at the
     # end of the "if" come from somewhere inside the "if".
     # Since the come_froms are ordered so that lowest
     # offset COME_FROM is last, it is sufficient to test
     # just the last one.
-    come_froms = ast[-1]
+    if len(ast) == 5:
-    if come_froms.kind != "else_suite":
+        end_come_froms = ast[-1]
-        if come_froms == "opt_come_from_except" and len(come_froms) > 0:
+        if end_come_froms.kind != "else_suite" and self.version >= 3.0:
-            come_froms = come_froms[0]
+            if end_come_froms == "opt_come_from_except" and len(end_come_froms) > 0:
-        if not isinstance(come_froms, Token):
+                end_come_froms = end_come_froms[0]
-            if len(come_froms):
+            if not isinstance(end_come_froms, Token):
-                return tokens[first].offset > come_froms[-1].attr
+                if len(end_come_froms):
-        elif tokens[first].offset > come_froms.attr:
+                    return tokens[first].offset > end_come_froms[-1].attr
-            return True
+            elif tokens[first].offset > end_come_froms.attr:
+                return True
 
-    # For mysterious reasons a COME_FROM in tokens[last+1] might be part of the grammar rule
+        # FIXME: There is weirdness in the grammar we need to work around.
-    # even though it is not found in come_froms.
+        # we need to clean up the grammar.
-    # Work around this.
+        if self.version < 3.0:
-    if (
+            last_token = ast[-1]
-        last < n
+        else:
-        and tokens[last] == "COME_FROM"
+            last_token = tokens[last]
-        and tokens[first].offset > tokens[last].attr
+        if last_token == "COME_FROM" and tokens[first].offset > last_token.attr:
-    ):
+            if self.version < 3.0 and self.insts[self.offset2inst_index[last_token.attr]].opname != "SETUP_LOOP":
-        return True
+                return True
 
     testexpr = ast[0]
 
     # Check that the condition portion of the "if"
     # jumps to the "else" part.
     if testexpr[0] in ("testtrue", "testfalse"):
-        test = testexpr[0]
+        if_condition = testexpr[0]
 
         else_suite = ast[3]
-        assert else_suite == "else_suite"
+        assert else_suite.kind.startswith("else_suite")
 
-        if len(test) > 1 and test[1].kind.startswith("jmp_"):
+        if len(if_condition) > 1 and if_condition[1].kind.startswith("jmp_"):
             if last == n:
                 last -= 1
-            jmp = test[1]
+            jmp = if_condition[1]
             if self.version > 2.6:
                 jmp_target = jmp[0].attr
             else:
                 jmp_target = int(jmp[0].pattr)
 
 
-            # Make sure we don't jump at the end of the "then" inside the "else"
+            # Below we check that jmp_target is jumping to a feasible
-            # (jf_cf_pop may be a 2.6ish specific thing.)
+            # location. It should be to the transition after the "then"
-            jf_cf_pop = ast[2]
+            # block and to the beginning of the "else" block.
+            # However the "if/else" is inside a loop the false test can be
+            # back to the loop.
 
-            if jf_cf_pop == "jf_cf_pop" and jf_cf_pop[0] == "JUMP_FORWARD":
+            # FIXME: the below logic for jf_cfs could probably be
-                jump_forward = jf_cf_pop[0]
+            # simplified.
-                endif_target = int(jump_forward.pattr)
+            jump_else_end = ast[2]
+            if jump_else_end == "jf_cf_pop":
+                jump_else_end = jump_else_end[0]
+
+            jump_to_jump = False
+            if jump_else_end == "JUMP_FORWARD":
+                jump_to_jump = True
+                endif_target = int(jump_else_end.pattr)
                 last_offset = tokens[last].off2int()
                 if endif_target != last_offset:
                     return True
-
+            last_offset = tokens[last].off2int(prefer_last=False)
-            # The jump inside "else" check below should be added.
+            if jmp_target == last_offset:
-            # add this until we can find out what's wrong with
+                # jmp_target should be jumping to the end of the if/then/else
-            # not being able to parse:
+                # but is it jumping to the beginning of the "else"
-            #     if a and b or c:
+                return True
-            #         x = 1
+            if (
-            #     else:
+                jump_else_end in ("jf_cfs", "jump_forward_else")
-            #         x = 2
+                and jump_else_end[0] == "JUMP_FORWARD"
-
+            ):
-            # FIXME: add this
+                # If the "else" jump jumps before the end of the the "if .. else end", then this
-            # if jmp_target < else_suite.first_child().off2int():
+                # is not this kind of "ifelsestmt".
-            #     return True
+                jump_else_forward = jump_else_end[0]
+                jump_else_forward_target = jump_else_forward.attr
+                if jump_else_forward_target < last_offset:
+                    return True
+                pass
+            if (
+                jump_else_end in ("jb_elsec", "jb_elsel", "jf_cfs", "jb_cfs")
+                and jump_else_end[-1] == "COME_FROM"
+            ):
+                if jump_else_end[-1].off2int() != jmp_target:
+                    return True
 
             if tokens[first].off2int() > jmp_target:
                 return True
 
-            return (jmp_target > tokens[last].off2int()) and tokens[
+            return (jmp_target > last_offset) and tokens[last] != "JUMP_FORWARD"
-                last
-            ] != "JUMP_FORWARD"
 
     return False

uncompyle6/parsers/reducecheck/ifstmt.py

@@ -7,7 +7,9 @@ def ifstmt(self, lhs, n, rule, ast, tokens, first, last):
             last -= 1
             pass
         if tokens[last].attr and isinstance(tokens[last].attr, int):
-            return tokens[first].offset < tokens[last].attr
+            if tokens[first].offset >= tokens[last].attr:
+                return True
+            pass
         pass
 
     # Make sure jumps don't extend beyond the end of the if statement.
@@ -23,6 +25,9 @@ def ifstmt(self, lhs, n, rule, ast, tokens, first, last):
         # instead of POP_JUMP_IF, should we use op attributes?
         if t.kind in ("POP_JUMP_IF_FALSE", "POP_JUMP_IF_TRUE"):
             pjif_target = t.attr
+            target_instr = self.insts[self.offset2inst_index[pjif_target]]
+            if lhs == "iflaststmtl" and target_instr.opname == "JUMP_ABSOLUTE":
+                pjif_target = target_instr.arg
             if pjif_target > last_offset:
                 # In come cases, where we have long bytecode, a
                 # "POP_JUMP_IF_TRUE/FALSE" offset might be too

uncompyle6/parsers/reducecheck/ifstmts_jump.py

@@ -9,12 +9,6 @@ def ifstmts_jump(self, lhs, n, rule, ast, tokens, first, last):
         return False
 
     come_froms = ast[-1]
-    # Make sure all of the "come froms" offset at the
-    # end of the "if" come from somewhere inside the "if".
-    # Since the come_froms are ordered so that lowest
-    # offset COME_FROM is last, it is sufficient to test
-    # just the last one.
-
     # This is complicated, but note that the JUMP_IF instruction comes immediately
     # *before* _ifstmts_jump so that's what we have to test
     # the COME_FROM against. This can be complicated by intervening
@@ -28,31 +22,30 @@ def ifstmts_jump(self, lhs, n, rule, ast, tokens, first, last):
         "COME_FROM",
     ):
         pop_jump_index -= 1
-    come_froms = ast[-1]
 
     # FIXME: something is fishy when and EXTENDED ARG is needed before the
     # pop_jump_index instruction to get the argment. In this case, the
     # _ifsmtst_jump can jump to a spot beyond the come_froms.
     # That is going on in the non-EXTENDED_ARG case is that the POP_JUMP_IF
     # jumps to a JUMP_(FORWARD) which is changed into an EXTENDED_ARG POP_JUMP_IF
-    # to the jumped forwareded address
+    # to the jumped forwarded address
     if tokens[pop_jump_index].attr > 256:
         return False
 
+    pop_jump_offset = tokens[pop_jump_index].off2int(prefer_last=False)
     if isinstance(come_froms, Token):
         if (
-            tokens[pop_jump_index].attr < tokens[pop_jump_index].offset
+            tokens[pop_jump_index].attr < pop_jump_offset and ast[0] != "pass"
-            and ast[0] != "pass"
         ):
             # This is a jump backwards to a loop. All bets are off here when there the
             # unless statement is "pass" which has no instructions associated with it.
             return False
         return (
             come_froms.attr is not None
-            and tokens[pop_jump_index].offset > come_froms.attr
+            and pop_jump_offset > come_froms.attr
         )
 
     elif len(come_froms) == 0:
         return False
     else:
-        return tokens[pop_jump_index].offset > come_froms[-1].attr
+        return pop_jump_offset > come_froms[-1].attr

uncompyle6/parsers/reducecheck/or_check.py

@@ -1,9 +1,9 @@
 #  Copyright (c) 2020 Rocky Bernstein
 
-
+ASSERT_OPS = frozenset(["LOAD_ASSERT", "RAISE_VARARGS_1"])
 def or_check(self, lhs, n, rule, ast, tokens, first, last):
     if rule == ("or", ("expr", "jmp_true", "expr")):
-        if tokens[last] in ("LOAD_ASSERT", "RAISE_VARARGS_1",):
+        if tokens[last] in  ASSERT_OPS or tokens[last-1] in ASSERT_OPS:
             return True
 
         # The following test is be the most accurate. It prevents "or" from being

uncompyle6/parsers/reducecheck/tryelsestmt.py

@@ -7,6 +7,8 @@ def tryelsestmt(self, lhs, n, rule, ast, tokens, first, last):
     # inside the except handler to the end. If that happens
     # then this is a "try" with no "else".
     except_handler = ast[3]
+    if except_handler == "except_handler_else":
+        except_handler = except_handler[0]
     if except_handler == "except_handler":
 
         come_from = except_handler[-1]

uncompyle6/semantics/consts.py

@@ -56,12 +56,12 @@ PRECEDENCE = {
 
     '_mklambda':              30,
 
-    'conditional':            28, # Conditional expression
+    'if_exp':                 28, # If_Exp expression
-    'conditional_lamdba':     28, # Lambda expression
+    'if_exp_lamdba':          28, # Lambda expression
-    'conditional_not_lamdba': 28, # Lambda expression
+    'if_exp_not_lamdba':      28, # Lambda expression
-    'conditionalnot':         28,
+    'if_exp_not':             28,
-    'if_expr_true':           28,
+    'if_exp_true':            28,
-    'ret_cond':               28,
+    'if_exp_ret':             28,
 
     'or':                     26, # Boolean OR
     'ret_or':                 26,
@@ -92,6 +92,8 @@ PRECEDENCE = {
 
     'BINARY_POWER':           4,  # Exponentiation, *
 
+    'await_expr':             3,  # await x, *
+
     'attribute':              2,  # x.attribute
     'buildslice2':            2,  # x[index]
     'buildslice3':            2,  # x[index:index]
@@ -136,25 +138,24 @@ PASS = SyntaxTree('stmts',
                        [ SyntaxTree('pass', [])])])])
 
 ASSIGN_DOC_STRING = lambda doc_string, doc_load: \
-  SyntaxTree('stmt',
+  SyntaxTree("stmt",
-      [ SyntaxTree('assign',
+      [ SyntaxTree("assign",
-            [ SyntaxTree('expr', [ Token(doc_load, pattr=doc_string, attr=doc_string) ]),
+            [ SyntaxTree("expr", [ Token(doc_load, pattr=doc_string, attr=doc_string) ]),
-              SyntaxTree('store', [ Token('STORE_NAME', pattr='__doc__')])
+              SyntaxTree("store", [ Token("STORE_NAME", pattr="__doc__")])
             ])])
 
-NAME_MODULE = SyntaxTree('stmt',
+NAME_MODULE = SyntaxTree('assign',
-                [ SyntaxTree('assign',
                     [ SyntaxTree('expr',
                           [Token('LOAD_NAME', pattr='__name__', offset=0, has_arg=True)]),
                       SyntaxTree('store',
                           [ Token('STORE_NAME', pattr='__module__', offset=3, has_arg=True)])
-                      ])])
+                      ])
 
 # God intended \t, but Python has decided to use 4 spaces.
 # If you want real tabs, use Go.
-# TAB = '\t'
+# TAB = "\t"
-TAB = ' ' * 4
+TAB = " " * 4
-INDENT_PER_LEVEL = ' ' # additional intent per pretty-print level
+INDENT_PER_LEVEL = " " # additional intent per pretty-print level
 
 TABLE_R = {
     'STORE_ATTR':	( '%c.%[1]{pattr}', 0),
@@ -302,24 +303,24 @@ TABLE_DIRECT = {
     # which we don't use here.
     'aug_assign1':      ( '%|%c %c %c\n', 0, 2, 1),
 
-    'aug_assign2':	    ( '%|%c.%[2]{pattr} %c %c\n', 0, -3, -4 ),
+    'aug_assign2':      ( '%|%c.%[2]{pattr} %c %c\n', 0, -3, -4 ),
-    'designList':	    ( '%c = %c', 0, -1 ),
+    'designList':       ( '%c = %c', 0, -1 ),
     'and':          	( '%c and %c', 0, 2 ),
     'ret_and':        	( '%c and %c', 0, 2 ),
     'and2':          	( '%c', 3 ),
     'or':           	( '%c or %c', 0, 2 ),
     'ret_or':           ( '%c or %c', 0, 2 ),
-    'conditional':      ( '%p if %c else %c',
+    'if_exp':           ( '%p if %c else %c',
                           (2, 'expr', 27), 0, 4 ),
-    'if_expr_lambda':   ( '%p if %c else %c',
+    'if_exp_lambda':    ( '%p if %c else %c',
                           (2, 'expr', 27), (0, 'expr'), 4 ),
-    'if_expr_true':     ( '%p if 1 else %c', (0, 'expr', 27), 2 ),
+    'if_exp_true':      ( '%p if 1 else %c', (0, 'expr', 27), 2 ),
-    'ret_cond':         ( '%p if %p else %p', (2, 27), (0, 27), (-1, 27) ),
+    'if_exp_ret':       ( '%p if %p else %p', (2, 27), (0, 27), (-1, 27) ),
-    'conditional_not':  ( '%p if not %p else %p',
+    'if_exp_not':       ( '%p if not %p else %p',
                           (2, 27),
                           (0, "expr", PRECEDENCE['unary_not']),
                           (4, 27) ),
-    'conditional_not_lambda':
+    'if_exp_not_lambda':
                         ( '%p if not %c else %c',
                           (2, 'expr', 27), 0, 4 ),
 

uncompyle6/semantics/customize3.py

@@ -36,7 +36,7 @@ def customize_for_version3(self, version):
     TABLE_DIRECT.update(
         {
             "comp_for": (" for %c in %c", (2, "store"), (0, "expr")),
-            "conditionalnot": (
+            "if_exp_not": (
                 "%c if not %c else %c",
                 (2, "expr"),
                 (0, "expr"),

uncompyle6/semantics/customize35.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019 by Rocky Bernstein
+#  Copyright (c) 2019-2020 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -17,7 +17,12 @@
 
 from xdis.code import iscode
 from xdis.util import COMPILER_FLAG_BIT
-from uncompyle6.semantics.consts import INDENT_PER_LEVEL, TABLE_DIRECT
+from uncompyle6.semantics.consts import (
+    INDENT_PER_LEVEL,
+    PRECEDENCE,
+    TABLE_DIRECT,
+)
+
 from uncompyle6.semantics.helper import flatten_list, gen_function_parens_adjust
 
 #######################
@@ -26,7 +31,11 @@ from uncompyle6.semantics.helper import flatten_list, gen_function_parens_adjust
 def customize_for_version35(self, version):
     TABLE_DIRECT.update(
         {
-            "await_expr": ("await %c", 0),
+            # nested await expressions like:
+            #   return await (await bar())
+            # need parenthesis.
+            "await_expr": ("await %p", (0, PRECEDENCE["await_expr"]-1)),
+
             "await_stmt": ("%|%c\n", 0),
             "async_for_stmt": ("%|async for %c in %c:\n%+%|%c%-\n\n", 9, 1, 25),
             "async_forelse_stmt": (
@@ -36,12 +45,12 @@ def customize_for_version35(self, version):
                 25,
                 (27, "else_suite"),
             ),
-            "async_with_stmt": ("%|async with %c:\n%+%c%-", (0, "expr"), 7),
+            "async_with_stmt": ("%|async with %c:\n%+%c%-", (0, "expr"), 3),
             "async_with_as_stmt": (
                 "%|async with %c as %c:\n%+%c%-",
                 (0, "expr"),
-                (6, "store"),
+                (2, "store"),
-                7,
+                3,
             ),
             "unmap_dict": ("{**%C}", (0, -1, ", **")),
             # "unmapexpr":	       ( "{**%c}", 0), # done by n_unmapexpr

uncompyle6/semantics/customize36.py

@@ -49,29 +49,30 @@ def customize_for_version36(self, version):
 
     TABLE_DIRECT.update(
         {
-            "tryfinally36": ("%|try:\n%+%c%-%|finally:\n%+%c%-\n\n", (1, "returns"), 3),
-            "func_args36": ("%c(**", 0),
-            "try_except36": ("%|try:\n%+%c%-%c\n\n", 1, -2),
-            "except_return": ("%|except:\n%+%c%-", 3),
-            "unpack_list": ("*%c", (0, "list")),
-            "tryfinally_return_stmt": ("%|try:\n%+%c%-%|finally:\n%+%|return%-\n\n", 1),
-            "async_for_stmt36": (
-                "%|async for %c in %c:\n%+%c%-%-\n\n",
-                (9, "store"),
-                (1, "expr"),
-                (18, "for_block"),
-            ),
-            "call_ex": ("%c(%p)", (0, "expr"), (1, 100)),
-            # This comes from 3.7. Eventually we will rebase from 3.7
-            # and then this can go away
-            "conditional37": ("%p if %c else %c", (1, "expr", 27), 0, 3),
-            "store_annotation": ("%[1]{pattr}: %c", 0),
             "ann_assign_init_value": (
                 "%|%c = %p\n",
                 (-1, "store_annotation"),
                 (0, "expr", 200),
             ),
             "ann_assign_no_init": ("%|%c\n", (0, "store_annotation")),
+            "async_for_stmt36": (
+                "%|async for %c in %c:\n%+%c%-\n\n",
+                (9, "store"),
+                (1, "expr"),
+                (18, "for_block"),
+            ),
+            "call_ex": ("%c(%p)", (0, "expr"), (1, 100)),
+            "except_return": ("%|except:\n%+%c%-", 3),
+            "func_args36": ("%c(**", 0),
+            # This comes from 3.7. Eventually we will rebase from 3.7
+            # and then this can go away
+            "if_exp37": ("%p if %c else %c", (1, "expr", 27), 0, 3),
+            "ifstmtl": ("%|if %c:\n%+%c%-", (0, "testexpr"), (1, "_ifstmts_jumpl")),
+            "try_except36": ("%|try:\n%+%c%-%c\n\n", 1, -2),
+            "tryfinally36": ("%|try:\n%+%c%-%|finally:\n%+%c%-\n\n", (1, "returns"), 3),
+            "tryfinally_return_stmt": ("%|try:\n%+%c%-%|finally:\n%+%|return%-\n\n", 1),
+            "unpack_list": ("*%c", (0, "list")),
+            "store_annotation": ("%[1]{pattr}: %c", 0),
         }
     )
 

uncompyle6/semantics/customize37.py

@@ -55,24 +55,18 @@ def customize_for_version37(self, version):
                 (1, "expr"),
                 (17, "for_block"),
             ),
-            "async_for_stmt36": (
-                "%|async for %c in %c:\n%+%c%-%-\n\n",
-                (9, "store"),
-                (1, "expr"),
-                (18, "for_block"),
-            ),
             "async_for_stmt37": (
-                "%|async for %c in %c:\n%+%c%-%-\n\n",
+                "%|async for %c in %c:\n%+%c%-\n\n",
                 (7, "store"),
                 (1, "expr"),
                 (16, "for_block"),
             ),
-            "async_with_stmt": ("%|async with %c:\n%+%c%-", (0, "expr"), 7),
+            "async_with_stmt": ("%|async with %c:\n%+%c%-", (0, "expr"), 3),
             "async_with_as_stmt": (
                 "%|async with %c as %c:\n%+%c%-",
                 (0, "expr"),
-                (6, "store"),
+                (2, "store"),
-                7,
+                3,
             ),
             "async_forelse_stmt": (
                 "%|async for %c in %c:\n%+%c%-%|else:\n%+%c%-\n\n",
@@ -85,7 +79,12 @@ def customize_for_version37(self, version):
             "attributes37": ("%[0]{pattr} import %c",
                             (0, "IMPORT_NAME_ATTR"),
                             (1, "IMPORT_FROM")),
-            "await_expr": ("await %c", 0),
+
+            # nested await expressions like:
+            #   return await (await bar())
+            # need parenthesis.
+            "await_expr": ("await %p", (0, PRECEDENCE["await_expr"]-1)),
+
             "await_stmt": ("%|%c\n", 0),
             "call_ex": ("%c(%p)", (0, "expr"), (1, 100)),
             "compare_chained1a_37": (
@@ -121,8 +120,8 @@ def customize_for_version37(self, version):
                 (0, 19),
                 (6, 19),
             ),
-            'conditional37': ( '%p if %c else %c',
+            'if_exp37': ( '%p if %c else %c',
-                               (1, 'expr', 27), 0, 3 ),
+                          (1, 'expr', 27), 0, 3 ),
 
             "except_return": ("%|except:\n%+%c%-", 3),
             "if_exp_37a": (

uncompyle6/semantics/helper.py

@@ -51,7 +51,7 @@ def find_all_globals(node, globs):
 #         # print("XXX", n.kind, global_ops)
 #         if isinstance(n, SyntaxTree):
 #             # FIXME: do I need a caser for n.kind="mkfunc"?
-#             if n.kind in ("if_expr_lambda", "return_lambda"):
+#             if n.kind in ("if_exp_lambda", "return_lambda"):
 #                 globs = find_globals(n, globs, mklambda_globals)
 #             else:
 #                 globs = find_globals(n, globs, global_ops)

uncompyle6/semantics/make_function3.py

@@ -677,7 +677,7 @@ def make_function3(self, node, is_lambda, nested=1, code_node=None):
     # add in "yield" just so that the compiler will mark
     # the GENERATOR bit of the function. See for example
     # Python 3.x's test_generator.py test program.
-    if code.co_flags & CO_GENERATOR:
+    if not is_lambda and code.co_flags & CO_GENERATOR:
         need_bogus_yield = True
         for token in scanner_code._tokens:
             if token in ("YIELD_VALUE", "YIELD_FROM"):

uncompyle6/semantics/make_function36.py

@@ -356,7 +356,7 @@ def make_function36(self, node, is_lambda, nested=1, code_node=None):
     # add in "yield" just so that the compiler will mark
     # the GENERATOR bit of the function. See for example
     # Python 3.x's test_connection.py and test_contexlib_async test programs.
-    if code.co_flags & (CO_GENERATOR | CO_ASYNC_GENERATOR):
+    if not is_lambda and code.co_flags & (CO_GENERATOR | CO_ASYNC_GENERATOR):
         need_bogus_yield = True
         for token in scanner_code._tokens:
             if token == "YIELD_VALUE":

uncompyle6/semantics/pysource.py

@@ -260,7 +260,8 @@ class SourceWalker(GenericASTTraversal, object):
         self.ast_errors = []
         # FIXME: have p.insts update in a better way
         # modularity is broken here
-        self.p.insts = scanner.insts
+        self.insts = scanner.insts
+        self.offset2inst_index = scanner.offset2inst_index
 
         # This is in Python 2.6 on. It changes the way
         # strings get interpreted. See n_LOAD_CONST
@@ -883,7 +884,11 @@ class SourceWalker(GenericASTTraversal, object):
     def n_docstring(self, node):
 
         indent = self.indent
-        docstring = node[0].pattr
+        doc_node = node[0]
+        if doc_node.attr:
+            docstring = doc_node.attr
+        else:
+            docstring = node[0].pattr
 
         quote = '"""'
         if docstring.find(quote) >= 0:
@@ -2115,7 +2120,6 @@ class SourceWalker(GenericASTTraversal, object):
                 try:
                     self.write(eval(expr, d, d))
                 except:
-                    from trepan.api import debug; debug()
                     raise
             m = escape.search(fmt, i)
         self.write(fmt[i:])
@@ -2232,8 +2236,10 @@ class SourceWalker(GenericASTTraversal, object):
         assert ast == "stmts"
         for i in range(len(ast)):
             # search for an assign-statement
-            assert ast[i] == "sstmt"
+            if ast[i] == "sstmt":
-            node = ast[i][0]
+                node = ast[i][0]
+            else:
+                node = ast[i]
             if node == "assign" and node[0] == ASSIGN_TUPLE_PARAM(name):
                 # okay, this assigns '.n' to something
                 del ast[i]
@@ -2254,11 +2260,7 @@ class SourceWalker(GenericASTTraversal, object):
     def build_class(self, code):
         """Dump class definition, doc string and class body."""
 
-        try:
+        assert iscode(code)
-            assert iscode(code)
-        except:
-            from trepan.api import debug; debug()
-
         self.classes.append(self.currentclass)
         code = Code(code, self.scanner, self.currentclass)
 
@@ -2268,17 +2270,20 @@ class SourceWalker(GenericASTTraversal, object):
         code._tokens = None  # save memory
         assert ast == "stmts"
 
+        if ast[0] == "sstmt":
+            ast[0] = ast[0][0]
+        first_stmt = ast[0]
+
         if ast[0] == "docstring":
             self.println(self.traverse(ast[0]))
             del ast[0]
 
-        first_stmt = ast[0][0]
         if 3.0 <= self.version <= 3.3:
             try:
-                if first_stmt[0] == "store_locals":
+                if first_stmt == "store_locals":
                     if self.hide_internal:
                         del ast[0]
-                        first_stmt = ast[0][0]
+                        first_stmt = ast[0]
             except:
                 pass
 
@@ -2286,39 +2291,40 @@ class SourceWalker(GenericASTTraversal, object):
             if first_stmt == NAME_MODULE:
                 if self.hide_internal:
                     del ast[0]
-                    first_stmt = ast[0][0]
+                    first_stmt = ast[0]
             pass
         except:
             pass
 
         have_qualname = False
+        if len(ast):
+            if ast[0] == "sstmt":
+                ast[0] = ast[0][0]
+            first_stmt = ast[0]
+
         if self.version < 3.0:
             # Should we ditch this in favor of the "else" case?
             qualname = ".".join(self.classes)
             QUAL_NAME = SyntaxTree(
-                "stmt",
+                "assign",
                 [
+                    SyntaxTree("expr", [Token("LOAD_CONST", pattr=qualname)]),
                     SyntaxTree(
-                        "assign",
+                        "store", [Token("STORE_NAME", pattr="__qualname__")]
-                        [
+                    ),
-                            SyntaxTree("expr", [Token("LOAD_CONST", pattr=qualname)]),
-                            SyntaxTree(
-                                "store", [Token("STORE_NAME", pattr="__qualname__")]
-                            ),
-                        ],
-                    )
                 ],
             )
-            have_qualname = ast[0][0] == QUAL_NAME
+            # FIXME: is this right now that we've redone the grammar?
+            have_qualname = ast[0] == QUAL_NAME
         else:
             # Python 3.4+ has constants like 'cmp_to_key.<locals>.K'
             # which are not simple classes like the < 3 case.
             try:
                 if (
-                    first_stmt[0] == "assign"
+                    first_stmt == "assign"
-                    and first_stmt[0][0][0] == "LOAD_STR"
+                    and first_stmt[0][0] == "LOAD_STR"
-                    and first_stmt[0][1] == "store"
+                    and first_stmt[1] == "store"
-                    and first_stmt[0][1][0] == Token("STORE_NAME", pattr="__qualname__")
+                    and first_stmt[1][0] == Token("STORE_NAME", pattr="__qualname__")
                 ):
                     have_qualname = True
             except:
@@ -2340,6 +2346,7 @@ class SourceWalker(GenericASTTraversal, object):
                 do_doc = True
             if do_doc and self.hide_internal:
                 try:
+                    # FIXME: Is there an extra [0]?
                     docstring = ast[i][0][0][0][0].pattr
                 except:
                     docstring = code.co_consts[0]
@@ -2347,18 +2354,31 @@ class SourceWalker(GenericASTTraversal, object):
                     self.println()
                     del ast[i]
 
-        # the function defining a class normally returns locals(); we
+        # The function defining a class returns locals() in Python somewhere less than
-        # don't want this to show up in the source, thus remove the node
+        # 3.7.
-        if len(ast) > 0 and ast[-1][0] == RETURN_LOCALS:
+        #
-            if self.hide_internal:
+        # We don't want this to show up in the source, so remove the node.
-                del ast[-1]  # remove last node
+        if len(ast):
-        # else:
+            if ast == "stmts" and ast[-1] == "sstmt":
-        #    print ast[-1][-1]
+                return_locals_parent = ast[-1]
+                parent_index = 0
+            else:
+                return_locals_parent = ast
+                parent_index = -1
+            return_locals = return_locals_parent[parent_index]
+            if return_locals == RETURN_LOCALS:
+                if self.hide_internal:
+                    del return_locals_parent[parent_index]
+                    pass
+                pass
+            # else:
+            #    print stmt[-1]
 
+
+        # Add "global" declaration statements at the top
         globals, nonlocals = find_globals_and_nonlocals(
             ast, set(), set(), code, self.version
         )
-        # Add "global" declaration statements at the top
         # of the function
         for g in sorted(globals):
             self.println(indent, "global ", g)
@@ -2369,8 +2389,11 @@ class SourceWalker(GenericASTTraversal, object):
         old_name = self.name
         self.gen_source(ast, code.co_name, code._customize)
         self.name = old_name
+
+        # save memory by deleting no-longer-used structures
         code._tokens = None
-        code._customize = None  # save memory
+        code._customize = None
+
         self.classes.pop(-1)
 
     def gen_source(self, ast, name, customize, is_lambda=False, returnNone=False):
@@ -2413,9 +2436,11 @@ class SourceWalker(GenericASTTraversal, object):
                 # modularity is broken here
                 p_insts = self.p.insts
                 self.p.insts = self.scanner.insts
+                self.p.offset2inst_index = self.scanner.offset2inst_index
                 ast = python_parser.parse(self.p, tokens, customize)
                 self.customize(customize)
                 self.p.insts = p_insts
+
             except (python_parser.ParserError, AssertionError) as e:
                 raise ParserError(e, tokens)
             transform_ast = self.treeTransform.transform(ast)
@@ -2448,6 +2473,7 @@ class SourceWalker(GenericASTTraversal, object):
             # modularity is broken here
             p_insts = self.p.insts
             self.p.insts = self.scanner.insts
+            self.p.offset2inst_index = self.scanner.offset2inst_index
             self.p.opc = self.scanner.opc
             ast = python_parser.parse(self.p, tokens, customize)
             self.p.insts = p_insts

uncompyle6/semantics/transform.py

@@ -26,7 +26,7 @@ from uncompyle6.semantics.consts import RETURN_NONE
 
 def is_docstring(node):
     try:
-        return node[0].kind == "assign" and node[0][1][0].pattr == "__doc__"
+        return node.kind == "assign" and node[1][0].pattr == "__doc__"
     except:
         return False
 
@@ -113,7 +113,7 @@ class TreeTransform(GenericASTTraversal, object):
 
         testexpr = node[0]
 
-        if testexpr != "testexpr":
+        if testexpr not in ("testexpr", "testexprl"):
             return node
 
         if node.kind in ("ifstmt", "ifstmtl"):
@@ -121,31 +121,42 @@ class TreeTransform(GenericASTTraversal, object):
 
             if ifstmts_jump == "_ifstmts_jumpl" and ifstmts_jump[0] == "_ifstmts_jump":
                 ifstmts_jump = ifstmts_jump[0]
-            elif ifstmts_jump not in ("_ifstmts_jump", "ifstmts_jumpl"):
+            elif ifstmts_jump not in ("_ifstmts_jump", "_ifstmts_jumpl", "ifstmts_jumpl"):
                 return node
             stmts = ifstmts_jump[0]
         else:
             # iflaststmtl works this way
             stmts = node[1]
 
-        if stmts in ("c_stmts",) and len(stmts) == 1:
+        if stmts in ("c_stmts", "stmts", "stmts_opt") and len(stmts) == 1:
-            stmt = stmts[0]
+            raise_stmt = stmts[0]
-            raise_stmt = stmt[0]
+            if raise_stmt != "raise_stmt1":
+                raise_stmt = raise_stmt[0]
+
             testtrue_or_false = testexpr[0]
             if (
-                raise_stmt == "raise_stmt1"
+                raise_stmt.kind == "raise_stmt1"
                 and 1 <= len(testtrue_or_false) <= 2
                 and raise_stmt.first_child().pattr == "AssertionError"
             ):
-                if testtrue_or_false == "testtrue":
+                if testtrue_or_false in ("testtrue", "testtruel"):
                     # Skip over the testtrue because because it would
                     # produce a "not" and we don't want that here.
                     assert_expr = testtrue_or_false[0]
                     jump_cond = NoneToken
                 else:
+                    try:
+                        assert testtrue_or_false in ("testfalse", "testfalsel")
+                    except:
+                        from trepan.api import debug; debug()
                     assert_expr = testtrue_or_false[0]
+                    if assert_expr == "testfalse_not_and":
+                        # FIXME: come pack to stuff like this
+                        return node
+
                     jump_cond = testtrue_or_false[1]
                     assert_expr.kind = "assert_expr"
+
                 expr = raise_stmt[0]
                 RAISE_VARARGS_1 = raise_stmt[1]
                 call = expr[0]
@@ -172,7 +183,7 @@ class TreeTransform(GenericASTTraversal, object):
                         kind = "assert2not"
 
                     LOAD_ASSERT = call[0].first_child()
-                    if LOAD_ASSERT != "LOAD_ASSERT":
+                    if LOAD_ASSERT not in ( "LOAD_ASSERT", "LOAD_GLOBAL"):
                         return node
                     if isinstance(call[1], SyntaxTree):
                         expr = call[1][0]
@@ -397,15 +408,20 @@ class TreeTransform(GenericASTTraversal, object):
             # Disambiguate a string (expression) which appears as a "call_stmt" at
             # the beginning of a function versus a docstring. Seems pretty academic,
             # but this is Python.
-            call_stmt = ast[0][0][0]
+            call_stmt = ast[0][0]
             if is_not_docstring(call_stmt):
                 call_stmt.kind = "string_at_beginning"
                 call_stmt.transformed_by = "transform"
                 pass
         except:
             pass
+
         try:
             for i in range(len(self.ast)):
+                sstmt = ast[i]
+                if len(sstmt) == 1 and sstmt == "sstmt":
+                    ast[i] = ast[i][0]
+
                 if is_docstring(self.ast[i]):
                     docstring_ast = SyntaxTree(
                         "docstring",
@@ -414,7 +430,8 @@ class TreeTransform(GenericASTTraversal, object):
                                 "LOAD_STR",
                                 has_arg=True,
                                 offset=0,
-                                pattr=self.ast[i][0][0][0].pattr,
+                                attr=self.ast[i][0][0].attr,
+                                pattr=self.ast[i][0][0].pattr,
                             )
                         ],
                     )

uncompyle6/version.py

@@ -12,4 +12,4 @@
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
 # This file is suitable for sourcing inside bash as
 # well as importing into Python
-VERSION="3.6.3"  # noqa
+VERSION="3.6.4"  # noqa