Merge branch 'master' into python-2.4

fixa

NEWS.md

@@ -1,3 +1,30 @@
+3.6.3: 2020-1-26 Martin and Susanne
+===================================
+
+Of late, every release fixes major gaps and embarrassments of the last release....
+
+And in some cases, like this one, exposes lacuna and rot.
+
+I now have [control] flow under control, even if it isn't the most optimal way.
+
+I now have greatly expanded automated testing.
+
+On the most recent Python versions I regularly decompile thousands of Python programs that are distributed with Python.  when it is possible, I then decompile Python's standard test suite distributed with Python and run the decompiled source code which basically checks itself. This amounts to about 250 test programs per version. This is in addition to the 3 CI testing services which do different things.
+
+Does this mean the decompiler works perfectly? No. There are still a dozen or so failing programs, although the actual number of bugs is probably smaller though.
+
+However, in perparation of a more major refactoring of the parser grammar, this release was born.
+
+In many cases, decompilation is better. But there are some cases where decompilation has gotten worse. For lack of time (and interest) 3.0 bytecode suffered a hit. Possibly some code in the 3.x range did too. In time and with cleaner refactored code, this will come back.
+
+Commit c90ff51 was a local maxiumum before, I started reworking the grammar to separate productions that were specific to loops versus those that are not in loops.
+In the middle of that I added another grammar simplication to remove singleton productions of the form `sstmts-> stmts`. These were always was a bit ugly, and complicated output.
+
+At any rate if decompilation fails, you can try c90ff51. Or another decompiler. `unpyc37` is pretty good for 3.7. wibiti `uncompyle2` is great for 2.7. `pycdc` is mediocre for Python before 3.5 or so, and not that good for the most recent Python. Geerally these programs will give some sort of answer even if it isn't correct.
+
+decompyle3 isn't that good for 3.7 and worse for 3.8, but right now it does things no other Python decompiler like `unpyc37` or `pycdc` does. For example, `decompyle3` handles variable annotations. As always, the issue trackers for the various programs will give you a sense for what needs to be done. For now, I've given up on reporting issues in the other decompilers because there are already enough issues reported, and they are just not getting fixed anyway.
+
+
 3.6.2: 2020-1-5 Samish
 ======================
 

README.rst

@@ -231,7 +231,7 @@ See Also
 * https://github.com/rocky/python-xdis : Cross Python version disassembler
 * https://github.com/rocky/python-xasm : Cross Python version assembler
 * https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
-* https://github.com/zrax/pycdc : The README for this C++ code syas it aims to support all versions of Python. It is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
+* https://github.com/zrax/pycdc : The README for this C++ code says it aims to support all versions of Python. It is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
 
 
 .. _trepan: https://pypi.python.org/pypi/trepan2g

__pkginfo__.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2018 Rocky Bernstein <rocky@gnu.org>
+# Copyright (C) 2018, 2020 Rocky Bernstein <rocky@gnu.org>
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -23,7 +23,7 @@
 
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015-2019 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015-2020 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
 classifiers =  ["Development Status :: 5 - Production/Stable",

admin-tools/README.md

@@ -8,4 +8,15 @@ They are customized to my environment:
 - I have git repos for xdis, and spark parser at the same level as uncompyle6
 
 There may be other rocky-specific things that need customization.
-how-to-make-a-release.txt has overall how I make a release
+how-to-make-a-release.md has overall how I make a release
+
+Since this project uses python over a wide variety of release, some versions
+of projects that should be used for specific Python versions
+
+for 3.2.6:
+   pytest==2.9.2
+
+for 3.1.5
+   pytset==2.1.0
+   py=1.8.0 and comment out line 10 of _builtin.py # callable = callable
+   six==1.10.0

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.9 3.6.9 2.6.9 3.3.7 2.7.17 3.2.6 3.1.5 3.4.8 3.7.6 3.8.1'
+export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.17 3.2.6 3.1.5 3.4.10 3.7.6 3.8.1'

admin-tools/setup-master.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-PYTHON_VERSION=3.7.5
+PYTHON_VERSION=3.7.6
 
 # FIXME put some of the below in a common routine
 function finish {

admin-tools/setup-python-2.4.sh

@@ -15,3 +15,4 @@ cd $fulldir/..
     git checkout python-2.4 &&  pyenv local $PYTHON_VERSION && git pull
 cd $owd
 rm -v */.python-version || true
+pyenv local $PYTHON_VERSION

pytest/test_grammar.py

@@ -27,6 +27,9 @@ def test_grammar():
             expect_lhs.add("attribute")
 
         expect_lhs.add("get_iter")
+
+        if PYTHON_VERSION > 3.7 or PYTHON_VERSION < 3.0:
+            expect_lhs.add("stmts_opt")
     else:
         expect_lhs.add("async_with_as_stmt")
         expect_lhs.add("async_with_stmt")
@@ -42,6 +45,13 @@ def test_grammar():
         expect_lhs.add("kvlist")
         expect_lhs.add("kv3")
         unused_rhs.add("dict")
+    else:
+        # NOTE: this may disappear
+        expect_lhs.add("except_handler_else")
+
+        if PYTHON_VERSION < 3.7 and PYTHON_VERSION != 2.7:
+            # NOTE: this may disappear
+            expect_lhs.add("except_handler_else")
 
     if PYTHON3:
         expect_lhs.add("load_genexpr")

pytest/test_token.py

@@ -1,24 +1,25 @@
 from uncompyle6 import PYTHON_VERSION
 from uncompyle6.scanners.tok import Token
 
+
 def test_token():
     # Test token formatting of: LOAD_CONST None
-    t = Token('LOAD_CONST', offset=0, attr=None, pattr=None, has_arg=True)
-    expect = '           0  LOAD_CONST               None'
+    t = Token("LOAD_CONST", offset=0, attr=None, pattr=None, has_arg=True)
+    expect = "             0  LOAD_CONST               None"
     # print(t.format())
     assert t
     assert t.format() == expect
 
     # Make sure equality testing of tokens ignores offset
-    t2 = Token('LOAD_CONST', offset=2, attr=None, pattr=None, has_arg=True)
+    t2 = Token("LOAD_CONST", offset=2, attr=None, pattr=None, has_arg=True)
     assert t2 == t
 
-
     # Make sure formatting of: LOAD_CONST False. We assume False is the 0th index
     # of co_consts.
-    t = Token('LOAD_CONST', offset=1, attr=False, pattr=False, has_arg=True)
-    expect = '           1  LOAD_CONST               False'
+    t = Token("LOAD_CONST", offset=1, attr=False, pattr=False, has_arg=True)
+    expect = "             1  LOAD_CONST               False"
     assert t.format() == expect
 
-if __name__ == '__main__':
+
+if __name__ == "__main__":
     test_token()

pytest/testdata/if-2.7.right

@@ -1,12 +1,12 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_if.py
 
-   6       0  LOAD_NAME             0  'True'
-           3  POP_JUMP_IF_FALSE    15  'to 15'
+   6         0  LOAD_NAME             0  'True'
+             3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   7       6  LOAD_NAME             1  'False'
-           9  STORE_NAME            2  'b'
-          12  JUMP_FORWARD          0  'to 15'
-        15_0  COME_FROM            12  '12'
-          15  LOAD_CONST               None
-          18  RETURN_VALUE
+   7         6  LOAD_NAME             1  'False'
+             9  STORE_NAME            2  'b'
+            12  JUMP_FORWARD          0  'to 15'
+          15_0  COME_FROM            12  '12'
+            15  LOAD_CONST               None
+            18  RETURN_VALUE

pytest/testdata/ifelse-2.7.right

@@ -1,15 +1,15 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_ifelse.py
 
-   3       0  LOAD_NAME             0  'True'
-           3  POP_JUMP_IF_FALSE    15  'to 15'
+   3         0  LOAD_NAME             0  'True'
+             3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   4       6  LOAD_CONST               1
-           9  STORE_NAME            1  'b'
-          12  JUMP_FORWARD          6  'to 21'
+   4         6  LOAD_CONST               1
+             9  STORE_NAME            1  'b'
+            12  JUMP_FORWARD          6  'to 21'
 
-   6      15  LOAD_CONST               2
-          18  STORE_NAME            2  'd'
-        21_0  COME_FROM            12  '12'
-          21  LOAD_CONST               None
-          24  RETURN_VALUE
+   6        15  LOAD_CONST               2
+            18  STORE_NAME            2  'd'
+          21_0  COME_FROM            12  '12'
+            21  LOAD_CONST               None
+            24  RETURN_VALUE

test-unit/test_grammar.py

@@ -17,7 +17,7 @@ class TestGrammar(unittest.TestCase):
         p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
         (lhs, rhs, tokens,
          right_recursive, dup_rhs) = p.check_sets()
-        expect_lhs = set(['expr1024', 'pos_arg'])
+        expect_lhs = set(['pos_arg', 'get_iter', 'attribute'])
         unused_rhs = set(['list', 'call', 'mkfunc',
                           'mklambda',
                           'unpack',])

test/run-and-email.sh

@@ -14,27 +14,47 @@ function displaytime {
     printf '%d seconds\n' $S
 }
 
-PYVERSION=${PYVERSION:-"3.5.5 2.7.14 3.2.6 3.3.7 3.4.8 2.6.9 3.6.4"}
-# PYVERSION=${PYVERSION:-"3.5.5"}
+. ../admin-tools/pyenv-newer-versions
 
 USER=${USER:-rocky}
 EMAIL=${EMAIL:-rb@dustyfeet.com}
 MAX_TESTS=${MAX_TESTS:-800}
 typeset -i RUN_STARTTIME=$(date +%s)
 
-for VERSION in $PYVERSION ; do
+# PYVERSIONS="3.5.6"
+actual_versions=""
+for VERSION in $PYVERSIONS ; do
     typeset -i rc=0
     LOGFILE=/tmp/pyenvlib-$VERSION-$$.log
 
-    if [[ $VERSION == '3.5.5' ]] ; then
-	MAX_TESTS=224
-    elif [[ $VERSION == '3.2.6' ]] ; then
-	MAX_TESTS=700
-    elif [[ $VERSION == '3.6.4' ]] ; then
-	MAX_TESTS=400
-    else
-	MAX_TESTS=800
-    fi
+    case "$VERSION" in
+	3.7.6 | 3.8.1 | 3.1.5 | 3.0.1 )
+	    continue
+	    ;;
+	3.5.9 )
+	    MAX_TESTS=900
+	    ;;
+	3.2.6 )
+	    MAX_TESTS=900
+	    ;;
+	3.3.7 )
+	    MAX_TESTS=1300 # About 1256 exist
+	    ;;
+	3.4.10 )
+	    MAX_TESTS=800
+	    ;;
+	3.6.10 )
+	    MAX_TESTS=1300  # about 2139 exist
+	    ;;
+	2.6.9 )
+	    MAX_TESTS=1300
+	    ;;
+	* )
+	     MAX_TESTS=800
+	     ;;
+    esac
+
+    actual_versions="$actual_versions $VERSION"
 
     if ! pyenv local $VERSION ; then
 	rc=1
@@ -42,7 +62,9 @@ for VERSION in $PYVERSION ; do
       echo Python Version $(pyenv local) > $LOGFILE
       echo "" >> $LOGFILE
       typeset -i ALL_FILES_STARTTIME=$(date +%s)
-      python ./test_pyenvlib.py --max ${MAX_TESTS} --syntax-verify --$VERSION  >>$LOGFILE 2>&1
+      cmd="python ./test_pyenvlib.py --max ${MAX_TESTS} --syntax-verify --$VERSION"
+      echo "$cmd" >>$LOGFILE 2>&1
+      $cmd >>$LOGFILE 2>&1
       rc=$?
 
       echo Python Version $(pyenv local) >> $LOGFILE
@@ -66,4 +88,4 @@ done
 typeset -i RUN_ENDTIME=$(date +%s)
 (( time_diff =  RUN_ENDTIME - RUN_STARTTIME))
 elapsed_time=$(displaytime $time_diff)
-echo "Run complete $elapsed_time for versions $PYVERSION" | mail -s "pyenv weak verify in $elapsed_time" ${EMAIL}
+echo "Run complete $elapsed_time for versions $actual_versions" | mail -s "pyenv weak verify in $elapsed_time" ${EMAIL}

test/simple_source/bug26/04_if_and_bug.py

@@ -0,0 +1,52 @@
+def foo():
+    if julian == -1 and week_of_year != -1 and weekday != -1:
+        first_weekday = datetime_date(year, 1, 1).weekday()
+        preceeding_days = 7 - first_weekday
+        if preceeding_days == 7:
+            preceeding_days = 0
+        if weekday == 6 and week_of_year_start == 6:
+            week_of_year -= 1
+        if weekday == 0 and first_weekday == 0 and week_of_year_start == 6:
+            week_of_year += 1
+        if week_of_year == 0:
+            julian = 1 + weekday - first_weekday
+        else:
+            days_to_week = preceeding_days + 7 * (week_of_year - 1)
+            julian = 1 + days_to_week + weekday
+
+
+# 2.6 pstats.py
+# Bug is handling "for" with "elif" and "and"s.
+def eval_print_amount(a, b, c, d, list, msg=0):
+    if a:
+        for i in list:
+            msg = 1
+    elif b and c:
+        msg = 2
+    elif c and d:
+        msg = 3
+    return msg
+
+assert eval_print_amount(True, False, False, False, [1]) == 1
+assert eval_print_amount(True, False, False, False, []) == 0
+assert eval_print_amount(False, True, True, False, []) == 2
+assert eval_print_amount(False, False, True, True, []) == 3
+assert eval_print_amount(False, False, False, True, []) == 0
+
+
+# Bug in 2.6 was in including the part at x = value
+# at the end asa part of the "else"
+def eval_directive(a):
+    if a:
+        value = 2
+    else:
+        try:
+            value = 3
+        except:
+            pass
+
+    x = value
+    return x
+
+assert eval_directive(True)  == 2
+assert eval_directive(False) == 3

test/simple_source/bug30/06_listcomp.py

@@ -23,6 +23,17 @@ def columnize(l):
             if not isinstance(l[i], str)]
 assert [0, 2] == columnize([1, 'a', 2])
 
+# From 3.7.6 _collections_abc.py
+# Bug was handling "or" in listcomp
+def count(values, x):
+    return sum(1 for v in values if v or x)
+
+assert count([2, 2], False) == 2
+assert count([], False) == 0
+assert count([], True) == 0
+assert count([2], True) == 1
+assert count([0], False) == 0
+
 # From 3.7 test_generators
 # Bug was in handling the way list_if is optimized in 3.7+;
 # We need list_if37 and compare_chained37.

test/simple_source/bug31/04_def_annotate.py

@@ -150,3 +150,12 @@ ann2(1)
 assert test12(1, 2, 3, name='hi') == (1, (2, 3)), "a, *args, name"
 assert test13(1, 2, 3, name='hi') == ((1, 2, 3), 'hi'), "*args, name"
 assert test16('localhost', loop=2, limit=3, a='b') == ('localhost', None, 2, 3, {'a': 'b'})
+
+# From test 3.5 test_pydoc.py.
+# Bug was in 3.5 and earlier handling of the return type, typing.Tuple[...]
+try:
+    import typing
+    def foo() -> typing.Iterator[typing.Tuple[int, typing.Any]]:
+        ...
+except:
+    pass

test/simple_source/bug33/01_triple_compare.py

@@ -14,3 +14,23 @@ def _is_valid_netmask(netmask):
 
 # See in 2.6.9 quopri.py ishex():
 assert not '0' <= __file__ <= '9' or 'a' <= __file__ <= 'f' or 'A' <= __file__ <= 'F'
+
+# From 3.7 bug-grammar.py
+
+# Bug in 3.7 was handling the last line where compare_chained -> compare_chained37 and
+# therefore compare_chained has one child, not two as it normally does.
+
+def test_comparison():
+    ### comparison: expr (comp_op expr)*
+    ### comp_op: '<'|'>'|'=='|'>='|'<='|'!='|'in'|'not' 'in'|'is'|'is' 'not'
+    if 1: pass
+    x = (1 == 1)
+    if 1 == 1: pass
+    if 1 != 1: pass
+    if 1 < 1: pass
+    if 1 > 1: pass
+    if 1 <= 1: pass
+    if 1 >= 1: pass
+    if 1 in (): pass
+    if 1 not in (): pass
+    if 1 < 1 > 1 == 1 >= 1 <= 1 != 1 in 1 not in 1 is 1 is not 1: pass

test/simple_source/bug33/08_if_else.py

@@ -8,3 +8,33 @@ def init(modules=None):
 
 assert init() == set()
 assert init([1, 2, 3]) == set([1, 2, 3])
+
+# From 3.6 sre_parse
+# Bug was in handling multple COME_FROMS from nested if's
+def _escape(a, b, c, d, e):
+    if a:
+        if b:
+            if c:
+                if d:
+                    raise
+                return
+        if e:
+            if d:
+                raise
+            return
+        raise
+
+assert _escape(False, True, True,  True,  True) is None
+assert _escape(True,  True, True,  False, True) is None
+assert _escape(True,  True, False, False, True) is None
+
+for args in (
+        (True, True,  True, False, True),
+        (True, False, True, True,  True),
+        (True, False, True, True,  False),
+        ):
+    try:
+        _escape(*args)
+        assert False, args
+    except:
+        pass

test/simple_source/bug35/04_call_function.py

@@ -79,3 +79,7 @@ class ResultMixin(object):
 
 class SplitResult(namedtuple('SplitResult', 'scheme netloc path query fragment'), ResultMixin):
     pass
+
+# From 3.3.7 test_long.py
+# Bug was that we need parens around first "0"
+assert (0).bit_length() == 0

test/simple_source/bug35/04_importlist.py

@@ -5,12 +5,15 @@
 
 # RUNNABLE!
 import os.path as osp
-from sys import path
+from sys import platform
 from os import sep, name
 import collections.abc
 
 assert osp.basename("a") == "a"
-assert path
+
+assert isinstance(platform, str)
 assert sep
 assert name
 assert collections.abc
+import os.path as path
+assert path

test/simple_source/bug36/01_conditional.py

@@ -0,0 +1,21 @@
+# From 3.7.6 test_buffer.py
+
+# RUNNABLE!
+def foo(n):
+    zero_stride = True if n >= 95 and n & 1 else False
+    return zero_stride
+
+assert foo(95)
+assert not foo(94)
+assert not foo(96)
+
+# from test_buffer.py
+# Bug was handling "or" inside a conditional
+def rslice(a, b):
+    minlen = 0 if a or b else 1
+    return minlen
+
+assert rslice(False, False) == 1
+assert rslice(False, True) == 0
+assert rslice(True, False) == 0
+assert rslice(True, True) == 0

test/simple_source/bug36/01_extended_arg.py

@@ -1,2 +1,82 @@
 if __file__:
     0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0
+
+
+# From 3.7 test_buffer.py
+# Bug is in dealing with EXTENDED_ARG instructions.
+# In reduction-rule tests where we are testing the offset,
+# getting *which* offset to test against, when there are two
+# possible offset, can mess us up.
+
+def five(a):
+    return 5
+
+def test_ndarray_slice_multidim(a, f, listerr):
+    for slices in a:
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        nderr = None
+        if nderr or listerr:
+            return f(5)
+        else:
+            return 2
+
+assert test_ndarray_slice_multidim([1], five, False) == 2
+assert test_ndarray_slice_multidim([1], five, True) == 5
+
+# From 3.7 test_builtin.py
+def test_pow(self, m, a, b, c, f):
+
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+
+    shape_t = 0; shape_t = 1; shape_t = 2;
+
+    for z in m:
+        if a or \
+           b or \
+           c:
+            f(TypeError)
+        else:
+            x = 2
+
+    x = 3
+
+# From 3.7 test_exceptions.py
+# Bug is handling extended arg
+def testAttributes(exceptionList):
+    try:
+        x = 0
+    except:
+        pass
+
+    for exc in exceptionList:
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0
+
+# From test_urllibnet2.py
+# Bug was in detecting end of "if" flowing into an extended-arg "for".
+def _test_urls(retry, urls):
+    if retry:
+        urlopen = 1
+
+    for url in urls:
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0

test/simple_source/bug36/02_call_ex_kw.py

@@ -53,3 +53,20 @@ from collections import namedtuple
 Point = namedtuple('Point', 'x y')
 p = Point(11, 22)
 assert p == Point(**dict(x=11, y=22))
+
+# From 3.7 test/test_keywordonlyarg.py
+# Bug was in handling {"4":4} as a dictionary needing **
+def posonly_sum(pos_arg1, *arg, **kwarg):
+    return pos_arg1 + sum(arg) + sum(kwarg.values())
+assert 1+2+3+4 == posonly_sum(1,*(2,3),**{"4":4})
+
+# From 3.7 test_grammar.py
+# Bug was in handling keyword-only parameters when there are annotations.
+# The stack order from least- to most-recent is:
+# default, keyword, annotation, closure
+# This changes in between Python 3.5 and 3.6.
+def f(a, b: 1, c: 2, d, e: 3=4, f=5, *g: 6, h: 7, i=8, j: 9=10,
+      **k: 11) -> 12: pass
+
+assert f.__annotations__ == {'b': 1, 'c': 2, 'e': 3, 'g': 6, 'h': 7, 'j': 9,
+                             'k': 11, 'return': 12}

test/simple_source/bug36/10_fstring.py

@@ -67,3 +67,52 @@ def _repr_fn(fields):
 
 fields = ['a', 'b', 'c']
 assert _repr_fn(fields) == ['return xx + f"(a={self.a!r}, b={self.b!r}, c={self.c!r})"']
+
+
+#################################
+# From Python 3.7 test_fstring.py
+
+x = 5
+assert f'{(lambda y:x*y)("8")!r}' == "'88888'"
+assert f'{(lambda y:x*y)("8")!r:10}' == "'88888'   "
+assert f'{(lambda y:x*y)("8"):10}' == "88888     "
+
+try:
+    eval("f'{lambda x:x}'")
+except SyntaxError:
+    pass
+else:
+    assert False, "f'{lambda x:x}' should be a syntax error"
+
+(x, y, width) = ("foo", 2, 10)
+assert f'x={x*y:{width}}' == 'x=foofoo    '
+
+# Why the fact that the distinction of docstring versus stmt is a
+# string expression is important academic, but we will decompile an
+# equivalent thing. For compatiblity with older Python we'll use "%"
+# instead of a format string
+def f():
+    f'''Not a docstring'''
+def g():
+    '''Not a docstring''' \
+    f''
+
+assert f.__doc__ is None
+assert g.__doc__ is None
+
+import decimal
+width, precision, value = (10, 4, decimal.Decimal('12.34567'))
+
+# Make sure we don't have additional f'..' inside the format strings below.
+assert f'result: {value:{width}.{precision}}' == 'result:      12.35'
+assert f'result: {value:{width:0}.{precision:1}}' == 'result:      12.35'
+assert f'{2}\t' ==  '2\t'
+
+# But below we *do* need the additional f".."
+assert f'{f"{0}"*3}' == "000"
+
+# We need to make sure we have { {x:... not {{x: ...
+#                               ^
+# The former, {{ confuses the format strings so dictionary/set comprehensions
+# don't work.
+assert f'expr={ {x: y for x, y in [(1, 2), ]}}' == 'expr={1: 2}'

test/simple_source/bug37/01_assert2.py

@@ -6,3 +6,13 @@ def test_assert2(c):
         raise SyntaxError('Oops')
 
 test_assert2(5)
+
+# Bug is handling "assert" and confusing it with "or".
+# It is important that the assert be at the end of the loop.
+for x in (2, 4, 6):
+    assert x == x
+
+# Bug in 3.7 was not having a rule for 2-arg assert.
+# 2-arg assert code doesn't match "if not ... raise "
+for x in (1, 3, 5):
+    assert x == x, "foo"

test/simple_source/bug38/04_for_no_jump_back.py

@@ -1,6 +1,8 @@
 # from mult_by_const/instruction.py
 # Bug in 3.8 was handling no JUMP_BACK in "for" loop. It is
 # in the "if" instead
+
+# RUNNABLE!
 def instruction_sequence_value(instrs, a, b):
     for instr in instrs:
         if a:

test/simple_source/comprehension/05_dict_comp.py

@@ -3,12 +3,24 @@
 #
 # This code is RUNNABLE!
 def x(s):
-    return {k: v
-            for (k, v) in s
-            if not k.startswith('_')
-    }
+    return {k: v for (k, v) in s if not k.startswith("_")}
+
 
 # Yes, the print() is funny. This is
 # to test though a 2-arg assert where
 # the 2nd argument is not a string.
-assert x((('_foo', None),)) == {}, print("See issue #162")
+assert x((("_foo", None),)) == {}, print("See issue #162")
+
+# From 3.7 test_dictcomps.py
+assert {k: v for k in range(10) for v in range(10) if k == v} == {
+    0: 0,
+    1: 1,
+    2: 2,
+    3: 3,
+    4: 4,
+    5: 5,
+    6: 6,
+    7: 7,
+    8: 8,
+    9: 9,
+}

test/simple_source/def/01_class.py

@@ -8,8 +8,19 @@
 #   classdef ::= LOAD_CONST expr mkfunc CALL_FUNCTION_0 BUILD_CLASS store
 #   mkfunc ::= LOAD_CONST MAKE_FUNCTION_0
 
+# RUNNABLE!
 class A:
     pass
 
 class B(Exception):
     pass
+
+# From 3.x test_descr.py
+class MyInt(int):
+    class MyInt(int):
+        __slots__ = ()
+    try:
+        (1).__class__ = MyInt
+        assert False, "builtin types don't support __class__ assignment."
+    except TypeError:
+        pass

test/simple_source/looping/10_for_if_loopback.py

@@ -11,6 +11,7 @@
 #       101_0	COME_FROM         '50' statement: if name == ...to fictional "end if"
 #         101	JUMP_BACK         '16' jump as before to loop head
 
+# RUNNABLE!
 def _slotnames(cls):
     names = []
     for c in cls.__mro__:
@@ -21,3 +22,37 @@ def _slotnames(cls):
                     continue
                 else:
                     names.append(name) # 3.2 bug here jumping to outer for
+
+# From 3.2.6 pdb.py
+# There is no "come_from" after the "if" since the
+# if jumps back to the loop. For this we use
+# grammar rule "ifstmtl"
+def lasti2lineno(linestarts, a):
+    for i in linestarts:
+        if a:
+            return a
+    return -1
+
+assert lasti2lineno([], True) == -1
+assert lasti2lineno([], False) == -1
+assert lasti2lineno([1], False) == -1
+assert lasti2lineno([1], True) == 1
+
+# From 3.7 test_builtin.py
+# Bug was allowing if condition jump back to the
+# "for" loop as an acceptable "ifstmtl" rule.
+
+# RUNNABLE!
+def test_pow(m, b, c):
+    for a in m:
+        if a or \
+           b or \
+           c:
+            c = 1
+
+    return c
+
+assert test_pow([], 2, 3) == 3
+assert test_pow([1], 0, 5) == 1
+assert test_pow([1], 4, 2) == 1
+assert test_pow([0], 0, 0) == 0

test/stdlib/2.5-exclude.sh

@@ -0,0 +1,41 @@
+SKIP_TESTS=(
+    [test_coercion.py]=1
+    [test_decimal.py]=1
+    [test_dis.py]=1        # We change line numbers - duh!
+    [test_file.py]=1       # test assertion failures
+    [test_generators.py]=1 # Investigate
+    [test_grammar.py]=1    # Too many stmts. Handle large stmts
+    [test_grp.py]=1        # Long test - might work Control flow?
+    [test_macfs.py]=1      # it fails on its own
+    [test_macostools.py]=1 # it fails on its own
+    [test_mailbox.py]=1
+    [test_nis.py]=1        # it fails on its own
+    [test_normalization.py]=1 # it fails on its own
+    [test_optparse.py]=1   # it fails on its own
+    [test_ossaudiodev.py]=1 # it fails on its own
+    [test_pdb.py]=1        # Line-number specific
+    [test_pep277.py]=1    # it fails on its own
+    [test_pep352.py]=1     # Investigate
+    [test_plistlib.py]=1 # it fails on its own
+    [test_pwd.py]=1 # Long test - might work? Control flow?
+    [test_pyclbr.py]=1 # Investigate
+    [test_rgbimg.py]=1 #  it fails on its own
+    [test_queue.py]=1 # Control flow?
+    [test_scriptpackages.py]=1 # it fails on its own
+    [test_socketserver.py]=1 # Too long to run - 42 seconds
+    [test_sqlite.py]=1 # it fails on its own
+    [test_startfile.py]=1 # it fails on its own
+    [test_struct.py]=1 # "if and" confused for if .. assert and
+    [test_sunaudiodev.py]=1 # it fails on its own
+    [test_support.py]=1 # it fails on its own
+    [test_tcl.py=1] # it fails on its own
+    [test_threading.py]=1 # test takes too long to run: 11 seconds
+    [test_thread.py]=1 # test takes too long to run: 36 seconds
+    [test_trace.py]=1  # Line numbers are expected to be different
+    [test_urllib2net.py]=1 # is interactive?
+    [test_urllibnet.py]=1 # it fails on its own
+    [test_winreg.py]=1 # it fails on its own
+    [test_winsound.py[=1 # it fails on its own
+    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
+    [test_zlib]=1  # fails on its own
+)

test/stdlib/2.6-exclude.sh

@@ -0,0 +1,101 @@
+SKIP_TESTS=(
+
+    [test___all__.py]=1  # it fails on its own
+    [test___all__.py]=1 # it fails on its own
+    [test_aepack.py]=1 # Fails on its own
+    [test_aepack.py]=1 # it fails on its own
+    [test_al.py]=1 # it fails on its own
+    [test_anydbm.py]=1 # it fails on its own
+    [test_applesingle.py]=1 # it fails on its own
+
+    [test_bsddb185.py]=1 # it fails on its own
+    [test_bsddb3.py]=1 # it fails on its own
+    [test_bsddb.py]=1 # it fails on its own
+
+    [test_cd.py]=1 # it fails on its own
+    [test_cl.py]=1 # it fails on its own
+    [test_codecmaps_cn.py]=1 # it fails on its own
+    [test_codecmaps_jp.py]=1 # it fails on its own
+    [test_codecmaps_kr.py]=1 # it fails on its own
+    [test_codecmaps_tw.py]=1 # it fails on its own
+    [test_commands.py]=1 # it fails on its own
+    [test_curses.py]=1 # it fails on its own
+
+    [test_dbm.py]=1 # it fails on its own
+    [test_descr.py]=1
+    [test_doctest.py]=1    #
+    [test_dis.py]=1        # We change line numbers - duh!
+    [test_dl.py]=1 # it fails on its own
+
+    [test_file.py]=1 # it fails on its own
+    [test_future5.py]=1 # it fails on its own
+
+    [test_generators.py]=1 # Investigate
+    [test_gl.py]=1 # it fails on its own
+    [test_grp.py]=1      # Long test - might work Control flow?
+
+    [test_imageop.py]=1 # it fails on its own
+    [test_imaplib.py]=1 # it fails on its own
+    [test_imgfile.py]=1 # it fails on its own
+    [test_ioctl.py]=1 # it fails on its own
+
+    [test_kqueue.py]=1 # it fails on its own
+
+    [test_linuxaudiodev.py]=1 # it fails on its own
+
+    [test_macos.py]=1 # it fails on its own
+    [test_macostools.py]=1 # it fails on its own
+    [test_mailbox.py]=1 # FIXME: release 3.6.2 may have worked
+
+    [test_normalization.py]=1 # it fails on its own
+
+    [test_ossaudiodev.py]=1 # it fails on its own
+
+    [test_pep277.py]=1 # it fails on its own
+    [test_pep352.py]=1     # Investigate
+    [test_pyclbr.py]=1 # Investigate
+    [test_pwd.py]=1 # Long test - might work? Control flow?
+    [test_py3kwarn.py]=1 # it fails on its own
+
+    [test_scriptpackages.py]=1 # it fails on its own
+    [test_select.py]=1 # test takes too long to run: 11 seconds
+    [test_socket.py]=1 # test takes too long to run: 12 seconds
+    [test_startfile.py]=1 # it fails on its own
+    [test_structmembers.py]=1 # it fails on its own
+    [test_sunaudiodev.py]=1 # it fails on its own
+    [test_support.py]=1 # # it fails on it s own
+
+    [test_threading.py]=1  # fails on its own?
+    [test_trace.py]=1  # Line numbers are expected to be different
+
+    [test_urllib2_localnet.py]=1 # test takes too long to run: 12 seconds
+    [test_urllib2net.py]=1 # test takes too long to run: 11 seconds
+    [test_urllibnet.py]=1 # it fails on its own
+
+
+    [test_whichdb.py]=1 # it fails on its own
+    [test_winreg.py]=1 # it fails on its own
+    [test_winsound.py]=1 # it fails on its own
+
+    [test_zipimport_support.py]=1
+    [test_zipfile64.py]=1  # Skip Long test
+    [test_zlib.py]=1  #
+    # .pyenv/versions/2.6.9/lib/python2.6/lib2to3/refactor.pyc
+    # .pyenv/versions/2.6.9/lib/python2.6/pyclbr.pyc
+)
+# About 305 unit-test files in about 12 minutes
+
+if (( batch )) ; then
+    # Fails in crontab environment?
+    # Figure out what's up here
+    SKIP_TESTS[test_aifc.py]=1
+    SKIP_TESTS[test_array.py]=1
+
+    # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
+    SKIP_TESTS[test_base64.py]=1
+    SKIP_TESTS[test_decimal.py]=1 # Might be a POWER math thing
+
+    # output indicates expected == output, but this fails anyway.
+    # Maybe the underlying encoding is subtlely different so it
+    # looks the same?
+fi

test/stdlib/2.7-exclude.sh

@@ -0,0 +1,49 @@
+SKIP_TESTS=(
+    [test_bsddb3.py]=1 # test takes too long to run: 110 seconds
+    [test_compile.py]=1  # Code introspects on co_consts in a non-decompilable way
+    [test_curses.py]=1  # Possibly fails on its own but not detected
+    [test_cmd_line.py]=1 # Takes too long, maybe hangs, or looking for interactive input?
+    [test_dis.py]=1   # We change line numbers - duh!
+    [test_doctest.py]=1 # Fails on its own
+    [test_exceptions.py]=1
+    [test_format.py]=1 # Control flow "and" vs nested "if"
+    [test_grammar.py]=1     # Too many stmts. Handle large stmts
+    [test_grp.py]=1     # test takes to long, works interactively though
+    [test_io.py]=1 # Test takes too long to run
+    [test_ioctl.py]=1 # Test takes too long to run
+    [test_lib2to3.py]=1 # test takes too long to run: 28 seconds
+    [test_math.py]=1
+    [test_memoryio.py]=1 # FIX
+    [test_modulefinder.py]=1 # FIX
+    [test_multiprocessing.py]=1 # On uncompyle2, takes 24 secs
+    [test_poll.py]=1  # test takes too long to run: 11 seconds
+    [test_pwd.py]=1     # Takes too long
+    [test_regrtest.py]=1 #
+    [test_runpy.py]=1   # Long and fails on its own
+    [test_select.py]=1  # Runs okay but takes 11 seconds
+    [test_socket.py]=1  # Runs ok but takes 22 seconds
+    [test_ssl.py]=1  #
+    [test_subprocess.py]=1 # Runs ok but takes 22 seconds
+    [test_sys_settrace.py]=1 # Line numbers are expected to be different
+    [test_tokenize.py]=1 # test takes too long to run: 19 seconds
+    [test_traceback.py]=1 # Line numbers change - duh.
+    [test_unicode.py]=1  # Too long to run 11 seconds
+    [test_xpickle.py]=1 # Runs ok but takes 72 seconds
+    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
+    [test_zipimport.py]=1  #
+)
+# 334 unit-test files in about 15 minutes
+
+if (( batch )) ; then
+    # Fails in crontab environment?
+    # Figure out what's up here
+    SKIP_TESTS[test_array.py]=1
+    SKIP_TESTS[test_ast.py]=1
+    SKIP_TESTS[test_audioop.py]=1
+    SKIP_TESTS[test_doctest2.py]=1 # a POWER thing?
+    SKIP_TESTS[test_httplib.py]=1  # Ok, but POWER has problems with this
+    SKIP_TESTS[test_pdb.py]=1 # Ok, but POWER has problems with this
+
+    # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
+    SKIP_TESTS[test_base64.py]=1
+fi

test/stdlib/3.2-exclude.sh

@@ -0,0 +1,48 @@
+SKIP_TESTS=(
+    [test_decorators.py]=1 # FIXME: Works on c90ff51
+    [test_optparse.py]=1 # FIXME: Works on c90ff51
+    [test_os.py]=1 # FIXME: Works on c90ff51
+    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
+    [test_strftime.py]=1 # FIXME: Works on c90ff51
+
+    [test_binop.py]=1 # FIXME: Works on c90ff51?
+    [test_cgi.py]=1 # FIXME: Works on c90ff51?
+    [test_descr.py]=1 # FIXME: Works on c90ff51?
+    [test_doctest2.py]=1 # FIXME: Works on c90ff51?
+
+    [test_cmd_line.py]=1
+    [test_collections.py]=1
+    [test_concurrent_futures.py]=1 # too long to run over 46 seconds by itself
+    [test_datetimetester.py]=1
+    [test_decimal.py]=1
+    [test_dictcomps.py]=1 # FIXME: semantic error: actual = {k:v for k in }
+    [test_doctest.py]=1 # test failures
+    [test_dis.py]=1   # We change line numbers - duh!
+    [test_exceptions.py]=1 # parse error
+
+    [test_modulefinder.py]=1 # test failures
+    [test_multiprocessing.py]=1 # test takes too long to run: 35 seconds
+
+    [test_peepholer.py]=1
+    [test_pep352.py]=1
+
+    [test_quopri.py]=1 # TypeError: Can't convert 'bytes' object to str implicitly
+
+    [test_runpy.py]=1
+
+    [test_ssl.py]=1 # too installation specific
+    [test_startfile.py]=1 # fails on its own
+    [test_subprocess.py]=1  # fails on its own
+    [test_tcl.py]=1 # it fails on its own
+    [test_tk.py]=1 # it fails on its own
+    [test_traceback.py]=1 # it fails on its own
+    [test_zipfile64.py]=1 # Too long to run
+
+)
+
+if (( batch )) ; then
+    # Fails in crontab environment?
+    # Figure out what's up here
+    SKIP_TESTS[test_exception_variations.py]=1
+    SKIP_TESTS[test_quopri.py]=1
+fi

test/stdlib/3.3-exclude.sh

@@ -0,0 +1,80 @@
+SKIP_TESTS=(
+    [test_binop.py]=1 # FIXME: Works on c90ff51
+    [test_cgi.py]=1 # FIXME: Works on c90ff51
+    [test_decorators.py]=1 # FIXME: Works on c90ff51
+    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
+    [test_optparse.py]=1 # FIXME: Works on c90ff51
+    [test_os.py]=1 # FIXME: Works on c90ff51
+    [test_pep352.py]=1 # FIXME: Works on c90ff51
+    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
+    [test_shutil.py]=1 # FIXME: Works on c90ff51
+    [test_strftime.py]=1 # FIXME: Works on c90ff51
+    [test_symtable.py]=1 # FIXME: Works on c90ff51
+
+    [test_atexit.py]=1  # The atexit test starting at 3.3 looks for specific comments in error lines
+
+    [test_buffer.py]=1  # parse error
+
+    [test_cmd_line.py]=1 # too long?
+    [test_concurrent_futures.py]=1  # too long?
+
+    [test_decimal.py]=1 # test takes too long to run: 18 seconds
+    [test_descr.py]=1  # test assertion errors
+    [test_doctest.py]=1  # test assertion errors
+    [test_doctest2.py]=1  # test assertion errors
+    [test_dis.py]=1   # We change line numbers - duh!
+
+    [test_exceptions.py]=1   #
+    [test_faulthandler.py]=1
+    [test_fork1.py]=1 # test takes too long to run: 12 seconds
+
+    [test_grammar.py]=1
+
+    [test_io.py]=1  # test takes too long to run: 34 seconds
+
+    [test_lib2to3.py]=1
+    [test_logging.py]=1 # test takes too long to run: 13 seconds
+    [test_long.py]=1 # test assert failure AttributeError: 'Rat' object has no attribute 'd'
+
+    [test_math.py]=1
+    [test_modulefinder.py]=1
+    [test_multiprocessing.py]=1
+
+    [test_nntplib.py]=1
+
+    [test_peepholer.py]=1
+    [test_poll.py]=1  # test takes too long to run: 11 seconds
+    [test_pty.py]=1  # FIXME: Needs grammar loop isolation separation
+
+    [test_queue.py]=1
+
+    [test_re.py]=1
+    [test_resource.py]=1
+    [test_runpy.py]=1
+
+    [test_scope.py]=1
+    [test_select.py]=1
+    [test_signal.py]=1
+    [test_socket.py]=1
+    [test_ssl.py]=1 # too installation specific
+    [test_subprocess.py]=1  # test takes too long to run: 28 seconds
+    [test_sys_setprofile.py]=1 # test assertion errors
+    [test_sys_settrace.py]=1 # test assertion errors
+
+    [test_tcl.py]=1  # installation specific; it fails on its own
+    [test_timeout.py]=1 # Too long to run: 19 seconds
+    [test_traceback.py]=1 # Probably introspects code
+
+    [test_zipfile64.py]=1 # Too long to run
+)
+# About 300 unit-test files in about 20 minutes
+
+if (( batch )) ; then
+    SKIP_TESTS[test_ftplib.py]=1  # Runs too long on POWER; over 15 seconds
+    SKIP_TESTS[test_idle.py]=1  # No tk
+    SKIP_TESTS[test_pep352.py]=1  # UnicodeDecodeError may be funny on weird environments
+    SKIP_TESTS[test_pep380.py]=1  # test_delegating_generators_claim_to_be_running ?
+    # Fails in crontab environment?
+    # Figure out what's up here
+    # SKIP_TESTS[test_exception_variations.py]=1
+fi

test/stdlib/3.4-exclude.sh

@@ -0,0 +1,79 @@
+SKIP_TESTS=(
+    [test_buffer.py]=1 # FIXME: Works on c90ff51
+    [test_decorators.py]=1 # FIXME: Works on c90ff51
+    [test_faulthandler.py]=1 # FIXME: too long to run 20 seconds. Works on c90ff51
+    [test_decimal.py]=1 # FIXME: Works on c90ff51
+    [test_optparse.py]=1 # FIXME: Works on c90ff51
+    [test_os.py]=1 # FIXME: Works on c90ff51
+    [test_shutil.py]=1 # FIXME: Works on c90ff51
+    [test_strftime.py]=1 # FIXME: Works on c90ff51
+
+    [test___all__.py]=1  # it fails on its own
+    [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
+    [test_cmd_line.py]=1 # takes too long to run
+    [test_concurrent_futures.py]=1  # too long?
+    [test_configparser.py]=1  # Doesn't terminate
+    [test_ctypes.py]=1 # it fails on its own
+    [test_curses.py]=1 # Investigate
+    [test_dbm_gnu.py]=1   # fails on its own
+    [test_devpoll.py]=1 # it fails on its own
+    [test_descr.py]=1   # test assertion errors
+    [test_dis.py]=1   # We change line numbers - duh!
+    [test_distutils.py]=1 # it fails on its own
+    [test_doctest2.py]=1
+    [test_doctest.py]=1  # test assert failures
+    [test_docxmlrpc.py]=1
+    [test_enum.py]=1 # compile syntax?
+    [test_exceptions.py]=1
+    [test_faulthandler.py]=1
+    [test_file_eintr.py]=1   # parse error
+    [test_fork1.py]=1 # too long
+    [test_gdb.py]=1 # it fails on its own
+    [test_grammar.py]=1 # parse error
+    [test_httplib.py]=1 # it fails on its own
+    [test_import.py]=1 # it fails on its own
+    [test_io.py]=1
+    [test_ioctl.py]=1 # it fails on its own
+    [test_inspect.py]=1 # Syntax error Investigate
+    [test_logging.py]=1 # Too long to run
+    [test_long.py]=1 # FIXME: Works on c90ff51
+    [test_modulefinder.py]=1  # test assertion error
+    [test_multiprocessing_fork.py]=1 # doesn't terminate
+    [test_multiprocessing_forkserver.py]=1 # doesn't terminate
+    [test_multiprocessing_main_handling.py]=1  # doesn't terminate
+    [test_multiprocessing_spawn.py]=1 # doesn't terminate
+    [test_nntplib.py]=1 # too long to run
+    [test_peepholer.py]=1 # control flow?
+    [test_pep352.py]=1 # test assert failures
+    [test_pickle.py]=1 # test assert failures
+    [test_pkgimport.py]=1 # long
+    [test_poll.py]=1 # Too long to run: 11 seconds
+    [test_pydoc.py]=1 # test assertion failures
+    [test_runpy.py]=1 # Too long:
+    [test_select.py]=1 # Too long: 11 seconds
+    [test_selectors.py]=1  # Too long: 11 seconds
+    [test_signal.py]=1 # Too long: 22 seconds
+    [test_sndhdr.py]=1
+    [test_socket.py]=1 # long 25 seconds
+    [test_socketserver.py]=1 # long 25 seconds
+    [test_subprocess.py]=1 # Too long
+    [test_symtable.py]=1 # Investigate bad output
+    [test_sys_settrace.py]=1 # test assert failures
+    [test_tcl.py]=1 # May be implementation specific. On POWER though it fails
+    [test_threading.py]=1 # Too long
+    [test_threadsignals.py]=1 # Too long to run: 12 seconds
+    [test_timeout.py]=1 # Too long to run: 19 seconds
+    [test_traceback.py]=1 # introspects on code
+    [test_urllib2net.py]=1 # Doesn't terminate
+    [test_zipfile64.py]=1
+    [test_zlib.py]=1
+)
+# 304 unit-test file in about 20 minutes
+
+if (( batch )) ; then
+    # Fails in crontab environment?
+    # Figure out what's up here
+    SKIP_TESTS[test_exception_variations.py]=1
+    SKIP_TESTS[test_mailbox.py]=1 # Takes to long on POWER; over 15 secs
+    SKIP_TESTS[test_quopri.py]=1
+fi

test/stdlib/3.5-exclude.sh

@@ -0,0 +1,154 @@
+SKIP_TESTS=(
+    [test_buffer.py]=1 # FIXME: Works on c90ff51
+    [test_decorators.py]=1 # FIXME: Works on c90ff51
+    [test_faulthandler.py]=1 # FIXME: too long to run 20 seconds. Works on c90ff51
+    [test_ftplib.py]=1 # Works on c90ff51
+    [test_marshal.py]=1 # FIXME: Works on c90ff51
+    [test_optparse.py]=1 # FIXME: Works on c90ff51
+    [test_os.py]=1 # FIXME: Works on c90ff51
+    [test_platform.py]=1 # FIXME: Works on c90ff51
+    [test_poplib.py]=1 # FIXME: Works on c90ff51
+    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
+    [test_smtplib.py]=1 # FIXME: Works on c90ff51
+    [test_strftime.py]=1 # FIXME: Works on c90ff51
+    [test_shutil.py]=1 # FIXME: Works on c90ff51
+    [test_sysconfig.py]=1 # FIXME: Works on c90ff51
+    [test_tempfile.py]=1 # FIXME: Works on c90ff51
+    [test_uu.py]=1 # FIXME: Works on c90ff51
+
+    [test___all__.py]=1 # it fails on its own
+    [test_aifc.py]=1  #
+    [test_asdl_parser.py]=1 # it fails on its own
+    [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
+
+    [test_bisect.py]=1  # it fails on its own
+    [test_cmath]=1
+    [test_cmd_line.py]=1 #
+    [test_codecmaps_cn.py]=1 # it fails on its own
+    [test_codecmaps_hk.py]=1 # it fails on its own
+    [test_codecmaps_jp.py]=1 # it fails on its own
+    [test_codecmaps_kr.py]=1 # it fails on its own
+    [test_codecmaps_tw.py]=1 # it fails on its own
+    [test_codecs.py]=1 # test assertion failure
+    [test_collections.py]=1
+    [test_compile.py]=1  # Code introspects on co_consts in a non-decompilable way
+    [test_concurrent_futures.py]=1 # Takes long to run
+    [test_coroutines.py]=1 # Syntax error Investigate
+    [test_curses.py]=1 #
+
+    [test_devpoll.py]=1 # it fails on its own
+    [test_dict.py]=1   #
+    [test_dis.py]=1   # We change line numbers - duh!
+    [test_distutils.py]=1 # it fails on its own
+    [test_dbm_ndbm.py]=1 # it fails on its own
+    [test_descr.py]=1   # test assertion errors
+    [test_doctest2.py]=1  #
+    [test_doctest.py]=1  #
+
+    [file_eintr.py]=1
+    [test_enum.py]=1  #
+    [test_exceptions.py]=1 # parse error
+
+    [test_format.py]=1
+
+    [test_gdb.py]=1 # it fails on its own
+    [test_glob.py]=1 #
+    [test_grammar.py]=1 # parse error
+    [test_grp.py]=1  # Long test
+
+    [test_heapq.py]=1 # test assertion failures
+
+    [test_imaplib.py]=1
+    [test_inspect.py]=1 # Syntax error Investigate
+    [test_io.py]=1 # Long run time.
+
+    [test_logging.py]=1 #
+    [test_long.py]=1 # too long run time: 20 seconds
+
+    [test_math.py]=1 # test assertion errors TypeError: a float is required
+    [test_modulefinder.py]=1  # test assertion error
+    [test_msilib.py]=1 # it fails on its own
+    [test_multiprocessing_fork.py]=1  # long
+    [test_multiprocessing_forkserver.py]=1  # long
+    [test_multiprocessing_spawn.py]=1 # long
+
+    [test_nntplib.py]=1 # too long 25 seconds
+    [test_normalization.py]=1 # it fails on its own
+
+    [test_pdb.py]=1 # probably relies on comments in code
+    [test_peepholer.py]=1 # parse error; Investigate
+    [test_pep352.py]=1 # test assertion error
+    [test_pep380.py]=1 # Investigate test assertion differ
+    [test_pickle.py]=1 # test assert errors
+    [test_pkgimport.py]=1 # long
+    [test_pkgutil.py]=1 # it fails on its own
+    [test_print.py]=1 # test assert errors
+    [test_pty.py]=1  # FIXME: Needs grammar loop isolation separation
+    [test_pydoc.py]=1 # test assertion: help text difference
+
+    [test_queue.py]=1 # Possibly parameter differences - investigate
+
+    [test_raise.py]=1 # Test assert error
+    [test_regrtest.py]=1 # test assertion errors
+    [test_robotparser.py]=1 # fails on its own
+    [test_runpy.py]=1 # decompile takes too long?
+
+    [test_sched.py]=1
+    [test_scope.py]=1
+    [test_select.py]=1 # Takes too long 11 seconds
+    [test_selectors.py]=1 # Takes too long 17 seconds
+    [test_set.py]=1 # # test assert failure and doesn't terminate
+    [test_signal.py]=1 # too long?
+    [test_smtpd.py]=1 # test failures
+    [test_socket.py]=1 # long
+    [test_socketserver.py]=1
+    [test_strtod.py]=1 # Test assert failure
+    [test_subprocess.py]=1
+    [test_sys_setprofile.py]=1 # test assert fail
+    [test_sys_settrace.py]=1 # test assert fail
+
+    [test_tcl.py]=1  # it fails on its own
+    [test_thread.py]=1
+    [test_threading.py]=1
+    [test_timeout.py]=1
+    [test_trace.py]=1 # it fails on its own
+    [test_traceback.py]=1
+    [test_ttk_guionly.py]=1 # it fails on its own
+    [test_ttk_textonly.py]=1 # it fails on its own
+    [test_typing.py]=1 # investigate syntax error
+
+    [test_urllib.py]=1 # it fails on its own
+    [test_urllib2.py]=1 # it fails on its own
+    [test_urllib2_localnet.py]=1  # doesn't terminate
+    [test_urllib2net.py]=1 # it fails on its own
+    [test_urllibnet.py]=1 # it fails on its own
+    [test_urlparse.py]=1 # test assert error
+
+    [test_winreg.py]=1 # it fails on its own
+    [test_winsound.py]=1 # it fails on its own
+
+    [test_xmlrpc.py]=1
+
+    [test_zipfile64.py]=1
+    [test_zipimport.py]=1
+    [test_zipimport_support.py]=1
+    [test_zipfile.py]=1 # it fails on its own
+    [test_zlib.py]=1
+)
+# About 260 unit-test in about 16 minutes
+
+if (( batch )) ; then
+    # Fails in crontab environment?
+    # Figure out what's up here
+    SKIP_TESTS[test_bisect.py]=1
+    SKIP_TESTS[test_buffer.py]=1  # too long
+    SKIP_TESTS[test_compileall.py]=1  # Something weird on POWER
+    SKIP_TESTS[test_codeccallbacks.py]=1 # Something differenet in locale?
+    SKIP_TESTS[test_distutils.py]=1
+
+    SKIP_TESTS[test_exception_variations.py]=1
+    SKIP_TESTS[test_quopri.py]=1
+    SKIP_TESTS[test_ioctl.py]=1 # it fails on its own
+    SKIP_TESTS[test_tarfile.py]=1 # too long to run on POWER 15 secs
+    SKIP_TESTS[test_venv.py]=1 # takes too long 11 seconds
+fi

test/stdlib/3.6-exclude.sh

@@ -0,0 +1,202 @@
+SKIP_TESTS=(
+    [test_binop.py]=1 # FIXME: Works on c90ff51
+    [test_cmath.py]=1 # FIXME: Works on c90ff51
+    [test_decorators.py]=1 # FIXME: Works on c90ff51
+    [test_format.py]=1  # FIXME: works on c90ff51
+    [test_locale.py]=1 # FIXME: Works on c90ff51
+    [test_optparse.py]=1 # FIXME: Works on c90ff51
+    [test_os.py]=1 # FIXME: Works on c90ff51
+    [test_pyclbr.py]=1 # FIXME: Works on c90ff51
+    [test_sort.py]=1 # FIXME: Works on c90ff51
+    [test_strftime.py]=1 # FIXME: Works on c90ff51
+
+    [test___all__.py]=1  # it fails on its own
+    [test_aifc.py]=1  #
+    [test_argparse.py]=1 # it fails on its own
+    [test_asdl_parser.py]=1 # it fails on its own
+    [test_asyncgen.py]=1  # parse error
+    [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
+
+    [test_baseexception.py]=1 # test assert error
+    [test_bdb.py]=1  #
+    [test_bisect.py]=1  # it fails on its own
+    [test_buffer.py]=1  # parse error
+    [test_builtin.py]=1  # Fails on its own
+    [test test_capi.py]=1 # it fails on its own
+    [test_cmd_line.py]=1 # Interactive?
+
+    [test_codecencodings_cn.py]=1 # it fails on its own
+    [test_codecencodings_hk.py]=1 # it fails on its own
+    [test_codecencodings_iso2022.py]=1 # it fails on its own
+    [test_codecencodings_jp.py]=1 # it fails on its own
+    [test_codecencodings_kr.py]=1 # it fails on its own
+    [test_codecencodings_tw.py]=1 # it fails on its own
+    [test_codecmaps_cn.py]=1 # it fails on its own
+    [test_codecmaps_hk.py]=1 # it fails on its own
+    [test_codecmaps_jp.py]=1 # it fails on its own
+    [test_codecmaps_kr.py]=1 # it fails on its own
+    [test_codecmaps_tw.py]=1 # it fails on its own
+    [test_codecs.py]=1
+    [test_collections.py]= # it fails on its own
+    [test_compile.py]=1  # Code introspects on co_consts in a non-decompilable way
+    [test_concurrent_futures.py]=1 # Takes long
+    [test_contextlib.py]=1 # test assertion failure
+    [test_contextlib_async.py]=1 # Investigate
+    [test_coroutines.py]=1 # Parse error
+    [test_curses.py]=1 # Parse error
+    [test_ctypes.py]=1 # it fails on its own
+
+    [test_datetime.py]=1 # it fails on its own
+    [test_dbm_ndbm.py]=1 # it fails on its own
+    [test_decimal.py]=1
+    [test_descr.py]=1   # syntax error: Investigate
+    [test_devpoll.py]=1 # it fails on its own
+    [test_dict.py]=1 # it fails on its own
+    [test_dis.py]=1   # We change line numbers - duh!
+    [test_doctest2.py]=1  #
+    [test_doctest.py]=1  #
+    [test_docxmlrpc.py]=1 # it fails on its own
+    [test_dtrace.py]=1 # it fails on its own
+    [test_dummy_thread.py]=1 # it fails on its own
+
+    [test_enum.py]=1  #
+    [test_exceptions.py]=1   # parse error
+
+    [test_filecmp.py]=1   # parse error
+    [test_file_eintr.py]=1   # parse error
+    [test_fileinput.py]=1 # doesn't terminate
+    [test_finalization.py]=1
+    [test_float.py]=1 # it fails on its own
+    [test_functools.py]=1 # it fails on its own
+    [test_fstring.py]=1 # need to disambiguate leading fstrings from docstrings
+    [test___future__.py]=1 # syntax error: Investigate
+
+    [test_gdb.py]=1 # it fails on its own
+    [test_generators.py]=1 # FIXME: Invalid syntax: f2 = lambda : (yield from g())        if False:
+    [test_genexps.py]=1 #
+    [test_glob.py]=1 #
+    [test_grammar.py]=1 # parse error
+
+    [test_hashlib.py]=1 # it fails on its own
+    [test_heapq.py]=1 # it fails on its own
+
+    [test_io.py]=1 # it fails on its own
+    [test_imaplib.py]=1
+    [test_inspect.py]=1 # Syntax error Investigate
+    [test_itertools.py]=1 # test assertion failures
+
+    [test_kqueue.py]=1 # it fails on its own
+
+    [test_lib2to3.py]=1 # it fails on its own
+    [test_logging.py]=1 # it fails on its own
+    [test_long.py]=1 #
+    [test_lzma.py]=1 # fails on its own
+
+    [test_mailbox.py]=1 # it fails on its own
+    [test_marshal.py]=1 #
+    [test_math.py]=1 # test assert errors call param Investigate
+    [test_metaclass.py]=1
+    [test_modulefinder.py]=1  # test assertion error
+    [test_msilib.py]=1 # it fails on its own
+    [test_multiprocessing_fork.py]=1 # it fails on its own
+    [test_multiprocessing_forkserver.py]=1 # it fails on its own
+    [test_multiprocessing_main_handling.py]=1 # takes too long -  11 seconds
+    [test_multiprocessing_spawn.py]=1 # it fails on its own
+
+    [test_nntplib.py]=1 # test takes too long to run: 31 seconds
+    [test_normalization.py]=1 # it fails on its own
+
+    [test_ordered_dict.py]= # it fails on its own
+    [test_ossaudiodev.py]=1 # it fails on its own
+
+    [test_pdb.py]=1 # Probably introspection
+    [test_peepholer.py]=1
+    [test_pickle.py]=1
+    [test_pkgimport.py]=1 # it fails on its own
+    [test_platform.py]=1
+    [test_plistlib.py]=1
+    [test_poll.py]=1 # Takes too long 11 seconds
+    [test_poplib.py]=1
+    [test_pprint.py]=1 # it fails on its own
+    [test_pulldom.py]=1
+    [test_pyclbr.py]=1 # it fails on its own
+    [test_pydoc.py]=1 # it fails on its own
+
+    [test_quopri.py]=1      # AssertionError: b'123=four' != '123=four'
+
+    [test_random.py]=1 # it fails on its own
+    [test_range.py]=1
+    [test_regrtest.py]=1 # test takes too long to run: 12 seconds
+    [test_robotparser.py]=1
+    [test_runpy.py]=1 # decompile takes too long?
+
+    [test_sax.py]=1 # it fails on its own
+    [test_sched.py]=1
+    [test_scope.py]=1
+    [test_secrets.py]=1 # it fails on its own
+    [test_select.py]=1 # test takes too long to run: 11 seconds
+    [test_selectors.py]=1 # it fails on its own
+    [test_shutil.py]=1 # it fails on its own
+    [test_signal.py]=1 # it fails on its own
+    [test_site.py]=1 # it fails on its own
+    [test_smtpd.py]=1
+    [test_smtplib.py]=1 # it fails on its own
+    [test_socket.py]=1 # long
+    [test_socketserver.py]=1
+    [test_ssl.py]=1 # it fails on its own
+    [test_startfile.py]=1 # it fails on its own
+    [test_statistics.py]=1 # it fails on its own
+    [test_string_literals.py]=1
+    [test_strtod.py]=1 # it fails on its own
+    [test_struct.py]=1  # test assertion errors
+    [test_subprocess.py]=1
+    [test_sys.py]=1 # Investigate confusing "and" with nested "if" when there is an "else
+    [test_sys_settrace.py]=1 # parse error
+
+    [test_tarfile.py]=1 # it fails on its own
+    [test_tcl.py]=1 # Test assert failures
+    [test_telnetlib.py]=1 # takes more than 15 seconds to run
+    [test_thread.py]=1 # it fails on its own
+    [test_threading.py]=1
+    [test_threadsignals.py]=1
+    [test_timeout.py]=1
+    [test_tix.py]=1 # it fails on its own
+    [test_tk.py]=1 # it fails on its own
+    [test_tokenize.py]=1 # test takes too long to run: 80 seconds
+    [test_trace.py]= # it fails on its own
+    [test_traceback.py]=1
+    [test_tracemalloc.py]=1
+    [test_ttk_guionly.py]= # it fails on its own
+    [test_ttk_textonly.py]=1 # it fails on its own
+    [test_turtle.py]=1 # it fails on its own
+    [test_typing.py]=1 # investigate syntax error
+
+    [test_ucn.py]=1 # it fails on its own
+    [test_urllib2_localnet.py]=1 # long
+    [test_urllib2net.py]=1 # it fails on its own
+    [test_urllib2.py]=1 # it fails on its own
+    [test_urllibnet.py]=1 # it fails on its own
+    [test_urllib.py]=1 # it fails on its own
+
+    [test_venv.py]=1 # test takes too long to run: 13 seconds
+
+    [test_weakref.py]=1
+    [test_winconsoleio.py]=1 # it fails on its own
+    [test_winreg.py]=1 # it fails on its own
+    [test_winsound.py]=1 # it fails on its own
+
+    [test_xmlrpc.py]=1 # it fails on its own
+
+    [test_zipfile.py]=1 # Too long - 11 seconds
+    [test_zipfile64.py]=1
+    [test_zipimport.py]=1
+    [test_zipimport_support.py]=1
+    [test_zlib.py]=1
+)
+# 236 unit-test files in about 13 minutes
+
+if (( batch )) ; then
+    SKIP_TESTS[test_codeccallbacks.py]=1
+    # locale on test machine is probably customized
+    SKIP_TESTS[test__locale.py]=1
+fi

test/stdlib/3.7-exclude.sh

@@ -0,0 +1,136 @@
+SKIP_TESTS=(
+    [test___all__.py]=1 # it fails on its own
+    [test_argparse.py]=1 #- it fails on its own
+    [test_asdl_parser.py]=1 # it fails on its own
+    [test_ast.py]=1  # Depends on comments in code
+    [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
+    [test_baseexception.py]=1  # UnboundLocalError: local variable 'exc' referenced before assignment
+    [test_bdb.py]=1  #
+    [test_buffer.py]=1  # test assertion errors
+    [test_builtin.py]=1  # parse error, but decompyle3 doesn't have this. (It has test assert failures though)
+    [test_clinic.py]=1 # it fails on its own
+    [test_cmath.py]=1 # test assertion failure
+    [test_cmd_line.py]=1  # Interactive?
+    [test_cmd_line_script.py]=1
+    [test_compare.py]=1 # Weird test assert faiure AssertionError: [1] == [1]
+    [test_compileall.py]=1 # fails on its own
+    [test_compile.py]=1  # Code introspects on co_consts in a non-decompilable way
+    [test_concurrent_futures.py]=1 # too long
+    [test_context.py]=1
+    [test_coroutines.py]=1 # Investigate: Parse error - async/yield  stuff?
+    [test_codecs.py]=1 # test assert failures; encoding/decoding stuff
+    [test_ctypes.py]=1 # it fails on its own
+    [test_curses.py]=1 # probably byte string not handled properly
+    [test_dataclasses.py]=1   # FIXME: control flow probably: AssertionError: unknown result 'exception'
+    [test_datetime.py]=1   # Takes too long
+    [test_dbm_gnu.py]=1   # Takes too long
+    [test_dbm_ndbm.py]=1 # it fails on its own
+    [test_decimal.py]=1   # test assertion failures
+    [test_descr.py]=1   # test assertion failures
+    [test_devpoll.py]=1 # it fails on its own
+    [test_dis.py]=1   # We change line numbers - duh!
+    [test_doctest2.py]=1 # FIXME: assert failure - works on decompyle3
+    [test_docxmlrpc.py]=1
+    [test_enum.py]=1   # probably bad control flow
+    [test_faulthandler.py]=1   # takes too long
+    [test_fcntl.py]=1
+    [test_fileinput.py]=1 # Test assertion failures
+    [test_format.py]=1 # Probably not handling bytestrings properly
+    [test_frame.py]=1 # test assertion errors
+    [test_ftplib.py]=1 # parse error
+    [test_functools.py]=1 # parse error
+    [test_fstring.py]=1 # need to disambiguate leading fstrings from docstrings
+    [test_gdb.py]=1 # it fails on its own
+    [test_generators.py]=1  # Investigate improper lamdba with bogus "False" added
+    [test_glob.py]=1  # TypeError: join() argument must be str or bytes, not 'tuple'
+    [test_grammar.py]=1 # investigate: index out of range in decompiler (template_engine?)
+    [test_grp.py]=1 # Doesn't terminate (killed)
+    [test_hashlib.py]=1 # test assert failures
+    [test_imaplib-3.7.py]=1  # test assert failures
+    [test_idle.py]=1 # Probably installation specific
+    [test_io.py]=1 # test takes too long to run: 37 seconds
+    [test_imaplib.py]=1 # test assert failures
+    [test_inspect.py]=1 # Investigate test failures involving lambda
+    [test_kqueue.py]=1 # it fails on its own
+    [test_lib2to3.py]=1 # it fails on its own
+    [test_long.py]=1 # FIX: if boundaries wrong in Rat __init__
+    [test_logging.py]=1 # test takes too long to run: 20 seconds
+    [test_mailbox.py]=1
+    [test_marshal.py]=1
+    [test_math.py]=1  # test assert failures
+    [test_modulefinder.py]=1
+    [test_msilib.py]=1
+    [test_multiprocessing_fork.py]=1 # test takes too long to run: 62 seconds
+    [test_multiprocessing_forkserver.py]=1
+    [test_multiprocessing_spawn.py]=1
+    [test_normalization.py]=1 # probably control flow (uninitialized variable)
+    [test_nntplib.py]=1
+    [test_optparse.py]=1 # doesn't terminate at test_consume_separator_stop_at_option
+    [test_os.py]=1 # probably control flow (uninitialized variable)
+    [test_ossaudiodev.py]=1 # it fails on its own
+    [test_pdb.py]=1 # Probably relies on comments
+    [test_peepholer.py]=1 # test assert error
+    [test_pickle.py]=1 # Probably relies on comments
+    [test_poll.py]=1
+    [test_poplib.py]=1
+    [test_pydoc.py]=1 # it fails on its own
+    [test_runpy.py]=1  #
+    [test_pkg.py]=1 # Investigate: lists differ
+    [test_pkgutil.py]=1 # Investigate:
+    [test_platform.py]=1 # probably control flow: uninitialized variable
+    [test_pow.py]=1 # probably control flow: test assertion failure
+    [test_pwd.py]=1 # killing - doesn't terminate
+    [test_regrtest.py]=1 # lists differ
+    [test_re.py]=1 # test assertion error
+    [test_richcmp.py]=1 # parse error
+    [test_select.py]=1 # test takes too long to run: 11 seconds
+    [test_selectors.py]=1
+    [test_shutil.py]=1 # fails on its own
+    [test_signal.py]=1 #
+    [test_slice.py]=1 # test assert error in data; Investigate
+    [test_smtplib.py]=1 #
+    [test_socket.py]=1
+    [test_socketserver.py]=1
+    [test_sort.py]=1 # Probably control flow; unintialized varaible
+    [test_ssl.py]=1 # Probably control flow; unintialized varaible
+    [test_startfile.py]=1 # it fails on its own
+    [test_statistics.py]=1 # Probably control flow; unintialized varaible
+    [test_string_literals.py]=1 # Investigate boolean parsing
+    [test_strptime.py]=1 # test assertions failed
+    [test_strtod.py]=1 # test assertions failed
+    [test_structmembers.py]=1 # test assertions failed
+    [test_struct.py]=1 # test assertions failed
+    [test_subprocess.py]=1
+    [test_sys_setprofile.py]=1 # test assertions failed
+    [test_sys_settrace.py]=1 # parse error
+    [test_tarfile.py]=1 # test assertions failed
+    [test_threading.py]=1 #
+    [test_timeit.py]=1 # probably control flow uninitialized variable
+    [test_tk.py]=1  # test takes too long to run: 13 seconds
+    [test_tokenize.py]=1
+    [test_trace.py]=1  # it fails on its own
+    [test_traceback.py]=1 # Probably uses comment for testing
+    [test_tracemalloc.py]=1 #
+    [test_ttk_guionly.py]=1  # implementation specfic and test takes too long to run: 19 seconds
+    [test_typing.py]=1 # parse error
+    [test_types.py]=1 # parse error
+    [test_unicode.py]=1 # unicode thing
+    [test_urllib2_localnet.py]=1 #
+    [test_urllibnet.py]=1 # probably control flow - uninitialized variable
+    [test_weakref.py]=1 # probably control flow - uninitialized variable
+    [test_with.py]=1 # probably control flow - uninitialized variable
+    [test_xml_dom_minicompat.py]=1 # FIXME: parse error; imports again
+    [test_winconsoleio.py]=1 # it fails on its own
+    [test_winreg.py]=1 # it fails on its own
+    [test_winsound.py]=1 # it fails on its own
+    [test_zipfile.py]=1 # it fails on its own
+    [test_zipfile64.py]=1 # Too long to run
+)
+# 278 unit-test files in about 15 minutes
+
+if (( batch )) ; then
+    SKIP_TESTS[test_distutils.py]=1
+    SKIP_TESTS[test_fileio.py]=1
+    SKIP_TESTS[test_gc.py]=1
+    SKIP_TESTS[test_zipimport_support.py]=1
+fi

test/stdlib/3.8-exclude.sh

@@ -0,0 +1,152 @@
+SKIP_TESTS=(
+    [test___all__.py]=1 # it fails on its own
+    [test_argparse.py]=1 #- it fails on its own
+    [test_asdl_parser.py]=1 # it fails on its own
+    [test_ast.py]=1  # Depends on comments in code
+    [test_atexit.py]=1  # The atexit test looks for specific comments in error lines
+
+    [test_baseexception.py]=1  #
+    [test_bdb.py]=1  #
+    [test_buffer.py]=1  # parse error
+    [test_builtin.py]=1  # parse error
+
+    [test_clinic.py]=1 # it fails on its own
+    [test_cmath.py]=1 # test assertion failure
+    [test_cmd_line.py]=1  # Interactive?
+    [test_cmd_line_script.py]=1
+    [test_collections.py]=1
+    [test_compare.py]=1
+    [test_compileall.py]=1 # fails on its own
+    [test_compile.py]=1
+    [test_concurrent_futures.py]=1 # too long
+    [test_configparser.py]=1
+    [test_context.py]=1
+    [test_coroutines.py]=1 # Parse error
+    [test_codecs.py]=1
+    [test_compile.py]=1  # Parse error, but after that, the code introspects on co_consts in a non-decompilable way
+    [test_complex.py]=1 # Investigate
+    [test_crypt.py]=1 # Parse error
+    [test_ctypes.py]=1 # it fails on its own
+    [test_curses.py]=1 # Parse error
+
+    [test_dataclasses.py]=1  # test assertion errors
+    [test_datetime.py]=1   # Takes too long
+    [test_dbm_gnu.py]=1   # Takes too long
+    [test_dbm_ndbm.py]=1 # it fails on its own
+    [test_decimal.py]=1   # Parse error
+    [test_descr.py]=1   # Parse error
+    [test_devpoll.py]=1 # it fails on its own
+    [test_dictcomps.py]=1 # Bad semantics - Investigate
+    [test_dis.py]=1   # We change line numbers - duh!
+    [test_docxmlrpc.py]=1
+
+    [test_exceptions.py]=1   # parse error
+    [test_enumerate.py]=1   #
+    [test_enum.py]=1   #
+
+    [test_faulthandler.py]=1   # takes too long
+    [test_fcntl.py]=1
+    [test_fileinput.py]=1
+    [test_float.py]=1
+    [test_format.py]=1
+    [test_frame.py]=1
+    [test_fstring.py]=1 # Investigate
+    [test_ftplib.py]=1
+    [test_functools.py]=1
+
+    [test_gdb.py]=1 # it fails on its own
+    [test_generators.py]=1  # improper decompile of assert i < n and (n-i) % 3 == 0
+    [test_glob.py]=1  #
+    [test_grammar.py]=1
+    [test_grp.py]=1 # Doesn't terminate (killed)
+    [test_gzip.py]=1 # parse error
+
+    [test_hashlib.py]=1 # test assert failures
+    [test_httplib.py]=1 # parse error
+    [test_http_cookiejar.py]=1
+    [test_imaplib-3.7.py]=1
+    [test_idle.py]=1 # Probably installation specific
+    [test_io.py]=1 # test takes too long to run: 37 seconds
+    [test_imaplib.py]=1
+    [test_index.py]=1
+    [test_inspect.py]=1
+    [test_itertools.py]=1 # parse error
+    [test_keywordonlyarg.py]=1 # parse error
+    [test_kqueue.py]=1 # it fails on its own
+    [test_lib2to3.py]=1 # it fails on its own
+    [test_long.py]=1 # investigate
+    [test_logging.py]=1 # test takes too long to run: 20 seconds
+    [test_mailbox.py]=1
+    [test_marshal.py]=1
+    [test_math.py]=1
+    [test_modulefinder.py]=1
+    [test_msilib.py]=1
+    [test_multiprocessing_fork.py]=1 # test takes too long to run: 62 seconds
+    [test_multiprocessing_forkserver.py]=1
+    [test_multiprocessing_spawn.py]=1
+    [test_normalization.py]=1 # probably control flow (uninitialized variable)
+    [test_nntplib.py]=1
+    [test_optparse.py]=1 # doesn't terminate (killed)
+    [test_os.py]=1 # probably control flow (uninitialized variable)
+    [test_ossaudiodev.py]=1 # it fails on its own
+    [test_pathlib.py]=1 # parse error
+    [test_pdb.py]=1 # Probably relies on comments
+    [test_peepholer.py]=1 # decompile takes a long time; then test assert error
+    [test_pickle.py]=1 # Probably relies on comments
+    [test_poll.py]=1
+    [test_poplib.py]=1
+    [test_pydoc.py]=1 # it fails on its own
+    [test_runpy.py]=1  #
+    [test_pkg.py]=1 # parse error; Investigate: lists differ
+    [test_pkgutil.py]=1 # parse error
+    [test_platform.py]=1 # parse error
+    [test_pwd.py]=1 # killing - doesn't terminate
+    [test_regrtest.py]=1 # parse error; test assertion error: lists differ
+    [test_re.py]=1 # parse error; test assertion error
+    [test_richcmp.py]=1 # Investigate: data[i] index error in semantic handling
+    [test_select.py]=1 # test takes too long to run: 11 seconds
+    [test_selectors.py]=1
+    [test_shutil.py]=1 # fails on its own
+    [test_signal.py]=1 #
+    [test_slice.py]=1 # Investigate: test assertion error
+    [test_smtplib.py]=1 #
+    [test_socket.py]=1
+    [test_socketserver.py]=1
+    [test_sort.py]=1 # parse error;
+    [test_ssl.py]=1 # parse error
+    [test_startfile.py]=1 # it fails on its own
+    [test_statistics.py]=1 # Takes more than 15 secs to run. Assert failures
+    [test_stat.py]=1 # parse error; test assertions failed
+    [test_string_literals.py]=1 # parse error; Investigate boolean parsing
+    [test_strptime.py]=1 # test assertions failed
+    [test_strtod.py]=1 # test assertions failed
+    [test_structmembers.py]=1 # test assertions failed
+    [test_struct.py]=1 # test assertions failed
+    [test_subprocess.py]=1
+    [test_sys_setprofile.py]=1 # test assertions failed
+    [test_sys_settrace.py]=1 # parse error
+    [test_tarfile.py]=1 # parse error
+    [test_threading.py]=1 #
+    [test_timeit.py]=1 # probably control flow uninitialized variable
+    [test_tk.py]=1  # test takes too long to run: 13 seconds
+    [test_tokenize.py]=1
+    [test_trace.py]=1  # it fails on its own
+    [test_traceback.py]=1 # Probably uses comment for testing
+    [test_tracemalloc.py]=1 #
+    [test_ttk_guionly.py]=1  # implementation specfic and test takes too long to run: 19 seconds
+    [test_typing.py]=1 # parse error
+    [test_types.py]=1 # parse error
+    [test_unicode.py]=1 # unicode thing
+    [test_urllib2.py]=1 #
+    [test_urllib2_localnet.py]=1 #
+    [test_urllibnet.py]=1 # probably control flow - uninitialized variable
+    [test_weakref.py]=1 # probably control flow - uninitialized variable
+    [test_with.py]=1 # probably control flow - uninitialized variable
+    [test_xml_dom_minicompat.py]=1 # parse error
+    [test_winconsoleio.py]=1 # it fails on its own
+    [test_winreg.py]=1 # it fails on its own
+    [test_winsound.py]=1 # it fails on its own
+    [test_zipfile.py]=1 # it fails on its own
+    [test_zipfile64.py]=1 #
+)
+# 268 About unit-test files, in about 11 minutes

test/stdlib/compare-un33.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+STOP_ONERROR=0  ./runun33.sh 'test_[a-b]*.py'
+STOP_ONERROR=0  ./runun33.sh 'test_[a-b]*.py'

test/stdlib/run-and-email.sh

@@ -1,16 +1,48 @@
 #!/bin/bash
+
+function displaytime {
+    printf "ran in "
+    local T=$1
+    local D=$((T/60/60/24))
+    local H=$((T/60/60%24))
+    local M=$((T/60%60))
+    local S=$((T%60))
+    (( $D > 0 )) && printf '%d days ' $D
+    (( $H > 0 )) && printf '%d hours ' $H
+    (( $M > 0 )) && printf '%d minutes ' $M
+    (( $D > 0 || $H > 0 || $M > 0 )) && printf 'and '
+    printf '%d seconds\n' $S
+}
+
+. ../../admin-tools/pyenv-newer-versions
+
 USER=${USER:-rocky}
 EMAIL=${EMAIL:-rb@dustyfeet.com}
 SUBJECT_PREFIX="stdlib unit testing for"
-for VERSION in 2.6.9 2.7.14 3.4.8 3.5.5 3.6.4 ; do
+
+typeset -i RUN_STARTTIME=$(date +%s)
+
+actual_versions=""
+DEBUG=""  # -x
+
+for VERSION in $PYVERSIONS ; do
     typeset -i rc=0
     LOGFILE=/tmp/runtests-$VERSION-$$.log
+
+    case "$VERSION" in
+	3.0.1 | 3.1.5 | 3.2.6 | 3.8.1 )
+	    continue
+	    ;;
+    esac
+    actual_versions="$actual_versions $VERSION"
+
     if ! pyenv local $VERSION ; then
 	rc=1
     else
-      /bin/bash ./runtests.sh  >$LOGFILE 2>&1
+      STOP_ONERROR=1 /bin/bash $DEBUG ./runtests.sh  >$LOGFILE 2>&1
       rc=$?
     fi
+    SUBJECT_PREFIX="runtests verify for"
     if ((rc == 0)); then
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION ok" ${USER}@localhost
     else
@@ -18,3 +50,8 @@ for VERSION in 2.6.9 2.7.14 3.4.8 3.5.5 3.6.4 ; do
 	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" $EMAIL
     fi
 done
+
+typeset -i RUN_ENDTIME=$(date +%s)
+(( time_diff =  RUN_ENDTIME - RUN_STARTTIME))
+elapsed_time=$(displaytime $time_diff)
+echo "Run complete $elapsed_time for versions $actual_versions" | mail -s "runtests in $elapsed_time" ${EMAIL}

test/stdlib/runtests.sh

@@ -28,15 +28,37 @@ MINOR=${FULLVERSION##?.?.}
 
 STOP_ONERROR=${STOP_ONERROR:-1}
 
+typeset -i timeout=15
+
+function timeout_cmd {
+
+  (
+    $@ &
+    child=$!
+    trap -- "" SIGTERM
+    (
+	sleep "$timeout"
+	if ps -p $child >/dev/null ; then
+	    echo ""
+	    echo >&1 "**Killing ${2}; takes more than $timeout seconds to run"
+	    kill -TERM ${child}
+	fi
+    ) &
+    wait "$child"
+  )
+}
+
 typeset -A SKIP_TESTS
 case $PYVERSION in
     2.4)
 	SKIP_TESTS=(
+	    [test_decimal.py]=1  #
 	    [test_dis.py]=1   # We change line numbers - duh!
+	    [test_generators.py]=1  # Investigate
+	    [test_grammar.py]=1    # Too many stmts. Handle large stmts
 	    [test_grp.py]=1      # Long test - might work Control flow?
 	    [test_pep247.py]=1 # Long test - might work? Control flow?
 	    [test_pwd.py]=1 # Long test - might work? Control flow?
-	    [test_pyclbr.py]=1 # Investigate
 	    [test_socketserver.py]=1 # -- test takes too long to run: 40 seconds
 	    [test_threading.py]=1 # test takes too long to run: 11 seconds
 	    [test_thread.py]=1 # test takes too long to run: 36 seconds
@@ -45,107 +67,18 @@ case $PYVERSION in
 	)
 	;;
     2.5)
-	SKIP_TESTS=(
-	    [test_dis.py]=1        # We change line numbers - duh!
-	    [test_grammar.py]=1    # Too many stmts. Handle large stmts
-	    [test_grp.py]=1        # Long test - might work Control flow?
-	    [test_pdb.py]=1        # Line-number specific
-	    [test_pep247.py]=1     # "assert xxx or .." not detected properly in check_hash_module()
-	    [test_pep352.py]=1     # Investigate
-	    [test_pwd.py]=1 # Long test - might work? Control flow?
-	    [test_pyclbr.py]=1 # Investigate
-	    [test_struct.py]=1 # "if and" confused for if .. assert and
-	    [test_threading.py]=1 # test takes too long to run: 11 seconds
-	    [test_thread.py]=1 # test takes too long to run: 36 seconds
-	    [test_trace.py]=1  # Line numbers are expected to be different
-	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
-	)
+	. ./2.5-exclude.sh
 	;;
     2.6)
-	SKIP_TESTS=(
-	    [test_aepack.py]=1 # Fails on its own
-	    [test_codeccallbacks.py]=1  # Fails on its own
-	    [test_compile.py]=1  # Intermittent - sometimes works and sometimes doesn't
-	    [test_exceptions.py]=1
-	    [test_grp.py]=1      # Long test - might work Control flow?
-	    [test_pep352.py]=1     # Investigate
-	    [test_pprint.py]=1
-	    [test_pyclbr.py]=1 # Investigate
-	    [test_pwd.py]=1 # Long test - might work? Control flow?
-	    [test_trace.py]=1  # Line numbers are expected to be different
-	    [test_urllib2net.py]=1 # Fails on its own. May need interactive input
-	    [test_zipfile64.py]=1  # Skip Long test
-	    [test_zlib.py]=1  # Takes too long to run (more than 3 minutes 39 seconds)
-	    # .pyenv/versions/2.6.9/lib/python2.6/lib2to3/refactor.pyc
-	    # .pyenv/versions/2.6.9/lib/python2.6/pyclbr.pyc
-	)
-	if (( batch )) ; then
-	    # Fails in crontab environment?
-	    # Figure out what's up here
-	    SKIP_TESTS[test_aifc.py]=1
-	    SKIP_TESTS[test_array.py]=1
-
-	    # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
-	    SKIP_TESTS[test_base64.py]=1
-
-	    # output indicates expected == output, but this fails anyway.
-	    # Maybe the underlying encoding is subtlely different so it
-	    # looks the same?
-	    SKIP_TESTS[test_pprint.py]=1
-	fi
+	. ./2.6-exclude.sh
 	;;
     2.7)
-	SKIP_TESTS=(
-	    # These are ok, but our test machine POWER has problems
-	    # so we skip..
-	    [test_httplib.py]=1  # Ok, but POWER has problems with this
-	    [test_pdb.py]=1 # Ok, but POWER has problems with this
-
-	    [test_capi.py]=1
-	    [test_curses.py]=1  # Possibly fails on its own but not detected
-	    [test_cmd_line.py]=1 # Takes too long, maybe hangs, or looking for interactive input?
-	    [test_compilex.py]=1 # Probably complex literals again. Investigate
-	    [test_dis.py]=1   # We change line numbers - duh!
-	    [test_doctest.py]=1 # Fails on its own
-	    [test_exceptions.py]=1
-	    [test_format.py]=1  # control flow. uncompyle2 does not have problems here
-	    [test_grammar.py]=1     # Too many stmts. Handle large stmts
-	    [test_grp.py]=1     # test takes to long, works interactively though
-	    [test_io.py]=1 # Test takes too long to run
-	    [test_ioctl.py]=1 # Test takes too long to run
-	    [test_long.py]=1
-	    [test_long_future.py]=1
-	    [test_math.py]=1
-	    [test_memoryio.py]=1 # FIX
-	    [test_modulefinder.py]=1 # FIX
-	    [test_multiprocessing.py]=1 # On uncompyle2, takes 24 secs
-	    [test_pwd.py]=1     # Takes too long
-	    [test_pty.py]=1
-	    [test_runpy.py]=1   # Long and fails on its own
-	    [test_select.py]=1  # Runs okay but takes 11 seconds
-	    [test_socket.py]=1  # Runs ok but takes 22 seconds
-	    [test_subprocess.py]=1 # Runs ok but takes 22 seconds
-	    [test_sys_setprofile.py]=1
-	    [test_sys_settrace.py]=1 # Line numbers are expected to be different
-	    [test_traceback.py]=1 # Line numbers change - duh.
-	    [test_unicode.py]=1  # Too long to run 11 seconds
-	    [test_xpickle.py]=1 # Runs ok but takes 72 seconds
-	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
-        )
-	if (( batch )) ; then
-	    # Fails in crontab environment?
-	    # Figure out what's up here
-	    SKIP_TESTS[test_array.py]=1
-	    SKIP_TESTS[test_ast.py]=1
-	    SKIP_TESTS[test_audioop.py]=1
-
-	    # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
-	    SKIP_TESTS[test_base64.py]=1
-	fi
+	. ./2.7-exclude.sh
 	;;
     3.0)
 	SKIP_TESTS=(
 	    [test_array.py]=1  # Handling of bytestring
+	    [test_binascii.py]=1 # handling of bytearray?
 	    [test_concurrent_futures.py]=1 # too long to run over 46 seconds by itself
 	    [test_datetimetester.py]=1
 	    [test_decimal.py]=1
@@ -161,10 +94,7 @@ case $PYVERSION in
 	;;
     3.1)
 	SKIP_TESTS=(
-	    [test_collections.py]=1
 	    [test_concurrent_futures.py]=1 # too long to run over 46 seconds by itself
-	    [test_datetimetester.py]=1
-	    [test_decimal.py]=1
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_fileio.py]=1
 	)
@@ -176,128 +106,25 @@ case $PYVERSION in
 	fi
 	;;
     3.2)
-	SKIP_TESTS=(
-	    [test_ast.py]=1  # Look at: AssertionError: b'hi' != 'hi'
-	    [test_cmd_line.py]=1
-	    [test_collections.py]=1
-	    [test_concurrent_futures.py]=1 # too long to run over 46 seconds by itself
-	    [test_datetimetester.py]=1
-	    [test_decimal.py]=1
-	    [test_dis.py]=1   # We change line numbers - duh!
-	    [test_quopri.py]=1 # TypeError: Can't convert 'bytes' object to str implicitly
-	)
-	if (( batch )) ; then
-	    # Fails in crontab environment?
-	    # Figure out what's up here
-	    SKIP_TESTS[test_exception_variations.py]=1
-	    SKIP_TESTS[test_quopri.py]=1
-	fi
+	. ./3.2-exclude.sh
 	;;
-
     3.3)
-	SKIP_TESTS=(
-	    [test_atexit.py]=1  #
-	)
-	if (( batch )) ; then
-	    # Fails in crontab environment?
-	    # Figure out what's up here
-	    SKIP_TESTS[test_exception_variations.py]=1
-	    SKIP_TESTS[test_quopri.py]=1
-	fi
+	. ./3.3-exclude.sh
 	;;
-
     3.4)
-	SKIP_TESTS=(
-	    [test_asynchat.py]=1  #
-	    [test_asyncore.py]=1  #
-	    [test_atexit.py]=1  #
-	    [test_bdb.py]=1  #
-	    [test_binascii]=1
-	    [test_dis.py]=1   # We change line numbers - duh!
-	)
-	if (( batch )) ; then
-	    # Fails in crontab environment?
-	    # Figure out what's up here
-	    SKIP_TESTS[test_exception_variations.py]=1
-	    SKIP_TESTS[test_quopri.py]=1
-	fi
+	. ./3.4-exclude.sh
 	;;
     3.5)
-	SKIP_TESTS=(
-	    [test_ast.py]=1  # line 379, in test_literal_eval  self.assertEqual(ast.literal_eval('b"hi"'), 'hi')
-	    [test_atexit.py]=1  #
-	    [test_builtin.py]=1  #
-	    [test_compare.py]=1
-	    [test_dis.py]=1   # We change line numbers - duh!
-	)
-	if (( batch )) ; then
-	    # Fails in crontab environment?
-	    # Figure out what's up here
-	    SKIP_TESTS[test_exception_variations.py]=1
-	    SKIP_TESTS[test_quopri.py]=1
-	fi
+	. ./3.5-exclude.sh
 	;;
-
     3.6)
-	SKIP_TESTS=(
-	    [test_ast.py]=1  #
-	    [test_atexit.py]=1  #
-	    [test_bdb.py]=1  #
-	    [test_builtin.py]=1  #
-	    [test_compare.py]=1
-	    [test_compile.py]=1
-	    [test_contains.py]=1    # Code "while False: yield None" is optimized away in compilation
-	    [test_contextlib_async.py]=1 # Investigate
-	    [test_coroutines.py]=1 # Parse error
-	    [test_curses.py]=1 # Parse error
-	    [test_dis.py]=1   # We change line numbers - duh!
-	    [test_quopri.py]=1      # AssertionError: b'123=four' != '123=four'
-	)
+	. ./3.6-exclude.sh
 	;;
     3.7)
-	SKIP_TESTS=(
-	    [test_ast.py]=1  #
-	    [test_atexit.py]=1  #
-	    [test_baseexception.py]=1  #
-	    [test_bdb.py]=1  #
-	    [test_buffer.py]=1  # parse error
-	    [test_builtin.py]=1  # parser error
-	    [test_cmdline.py]=1  # Interactive?
-	    [test_collections.py]=1  # Fixed I think in decompyle3 - pull from there
-	    [test_compare.py]=1
-	    [test_compile.py]=1
-	    [test_configparser.py]=1
-	    [test_contains.py]=1    # Code "while False: yield None" is optimized away in compilation
-	    [test_contextlib_async.py]=1 # Investigate
-	    [test_context.py]=1
-	    [test_coroutines.py]=1 # Parse error
-	    [test_crypt.py]=1 # Parse error
-	    [test_curses.py]=1 # Parse error
-	    [test_dataclasses.py]=1   # parse error
-	    [test_datetime.py]=1   # Takes too long
-	    [test_dbm_gnu.py]=1   # Takes too long
-	    [test_decimal.py]=1   # Parse error
-	    [test_descr.py]=1   # Parse error
-	    [test_dictcomps.py]=1 # Bad semantics - Investigate
-	    [test_dis.py]=1   # We change line numbers - duh!
-	    [test_enumerate.py]=1   #
-	    [test_enum.py]=1   #
-	    [test_faulthandler.py]=1   # takes too long
-	    [test_generators.py]=1  # improper decompile of assert i < n and (n-i) % 3 == 0
-	    # ...
-	)
+	. ./3.7-exclude.sh
 	;;
     3.8)
-	SKIP_TESTS=(
-	    [test_contains.py]=1    # Code "while False: yield None" is optimized away in compilation
-	    [test_collections.py]=1  # Investigate
-	    [test_decorators.py]=1  # Control flow wrt "if elif"
-	    [test_exceptions.py]=1   # parse error
-	    [test_dis.py]=1   # We change line numbers - duh!
-	    [test_pow.py]=1         # Control flow wrt "continue"
-	    [test_quopri.py]=1      # Only fails on POWER
-	    # ...
-	)
+	. ./3.8-exclude.sh
 	;;
     *)
 	SKIP_TESTS=( [test_aepack.py]=1
@@ -322,6 +149,13 @@ fi
 
 PYENV_ROOT=${PYENV_ROOT:-$HOME/.pyenv}
 pyenv_local=$(pyenv local)
+
+# pyenv version update
+for dir in ../ ../../ ; do
+    cp -v .python-version $dir
+done
+
+
 mkdir $TESTDIR || exit $?
 cp -r ${PYENV_ROOT}/versions/${PYVERSION}.${MINOR}/lib/python${PYVERSION}/test $TESTDIR
 cd $TESTDIR/test
@@ -354,7 +188,10 @@ for file in $files; do
 
     # If the fails *before* decompiling, skip it!
     typeset -i STARTTIME=$(date +%s)
-    if ! python $file >/dev/null 2>&1 ; then
+    if [ ! -r $file ]; then
+	echo "Skipping test $file -- not readable. Does it exist?"
+	continue
+    elif ! python $file >/dev/null 2>&1 ; then
 	echo "Skipping test $file -- it fails on its own"
 	continue
     fi
@@ -377,7 +214,7 @@ for file in $files; do
     rc=$?
     if (( rc == 0 )) ; then
 	echo ========== $(date +%X) Running $file ===========
-	python $file
+	timeout_cmd python $file
 	rc=$?
     else
 	echo ======= Skipping $file due to compile/decompile errors ========
@@ -392,7 +229,8 @@ typeset -i ALL_FILES_ENDTIME=$(date +%s)
 
 (( time_diff =  ALL_FILES_ENDTIME - ALL_FILES_STARTTIME))
 
-printf "Ran $i unit-test files in "
+printf "Ran $i unit-test files, $allerrs errors; Elapsed time: "
 displaytime $time_diff
 echo "Skipped $skipped test for known failures."
+cd $fulldir/../.. && pyenv local $FULLVERSION
 exit $allerrs

uncompyle6/main.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2018-2019 Rocky Bernstein <rocky@gnu.org>
+# Copyright (C) 2018-2020 Rocky Bernstein <rocky@gnu.org>
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -79,35 +79,42 @@ def decompile(
     assert iscode(co)
 
     if is_pypy:
-        co_pypy_str = 'PyPy '
+        co_pypy_str = "PyPy "
     else:
-        co_pypy_str = ''
+        co_pypy_str = ""
 
     if IS_PYPY:
-        run_pypy_str = 'PyPy '
+        run_pypy_str = "PyPy "
     else:
-        run_pypy_str = ''
+        run_pypy_str = ""
 
     if magic_int:
         m = str(magic_int)
     else:
         m = ""
 
-    sys_version_lines = sys.version.split('\n')
+    sys_version_lines = sys.version.split("\n")
     if source_encoding:
-        write('# -*- coding: %s -*-' % source_encoding)
-    write('# uncompyle6 version %s\n'
-          '# %sPython bytecode %s%s\n# Decompiled from: %sPython %s' %
-          (VERSION, co_pypy_str, bytecode_version,
-               " (%s)" % m, run_pypy_str,
-           '\n# '.join(sys_version_lines)))
+        write("# -*- coding: %s -*-" % source_encoding)
+    write(
+        "# uncompyle6 version %s\n"
+        "# %sPython bytecode %s%s\n# Decompiled from: %sPython %s" %
+          (VERSION,
+           co_pypy_str,
+           bytecode_version,
+           " (%s)" % m, run_pypy_str,
+           "\n# ".join(sys_version_lines),
+          )
+    )
+    if bytecode_version >= 3.0:
+        write("# Warning: this version has problems handling the Python 3 byte type in contants properly.\n")
     if co.co_filename:
         write("# Embedded file name: %s" % co.co_filename,)
     if timestamp:
-        write('# Compiled at: %s' %
+        write("# Compiled at: %s" %
               datetime.datetime.fromtimestamp(timestamp))
     if source_size:
-        real_out.write('# Size of source mod 2**32: %d bytes\n' %
+        real_out.write("# Size of source mod 2**32: %d bytes\n" %
                        source_size)
 
     debug_opts = {"asm": showasm, "ast": showast, "grammar": showgrammar}

uncompyle6/parser.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2019 Rocky Bernstein
+#  Copyright (c) 2015-2020 Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
@@ -26,21 +26,24 @@ from xdis.magics import py_str2float
 from spark_parser import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.show import maybe_show_asm
 
+
 class ParserError(Exception):
     def __init__(self, token, offset):
         self.token = token
         self.offset = offset
 
     def __str__(self):
-        return "Parse error at or near `%r' instruction at offset %s\n" % \
-               (self.token, self.offset)
+        return "Parse error at or near `%r' instruction at offset %s\n" % (
+            self.token,
+            self.offset,
+        )
 
 
 def nop_func(self, args):
     return None
 
-class PythonParser(GenericASTBuilder):
 
+class PythonParser(GenericASTBuilder):
     def __init__(self, SyntaxTree, start, debug):
         super(PythonParser, self).__init__(SyntaxTree, start, debug)
         # FIXME: customize per python parser version
@@ -50,24 +53,32 @@ class PythonParser(GenericASTBuilder):
         #   stmts -> stmts stmt -> stmts stmt stmt ...
         # collect as stmts -> stmt stmt ...
         nt_list = [
-            'stmts', 'except_stmts', '_stmts', 'attributes',
-            'exprlist', 'kvlist', 'kwargs', 'come_froms', '_come_froms',
-            'importlist',
+            "_come_froms",
+            "_stmts",
+            "attributes",
+            "come_froms",
+            "except_stmts",
+            "exprlist",
+            "importlist",
+            "kvlist",
+            "kwargs",
+            "l_stmts",
+            "stmts",
             # Python < 3
-            'print_items',
+            "print_items",
             # PyPy:
-            'imports_cont',
-            'kvlist_n',
+            "imports_cont",
+            "kvlist_n",
             # Python 3.6+
-            'come_from_loops',
+            "come_from_loops",
             # Python 3.7+
-            'importlist37',
-            ]
+            "importlist37",
+        ]
         self.collect = frozenset(nt_list)
 
         # For these items we need to keep the 1st epslion reduction since
         # the nonterminal name is used in a semantic action.
-        self.keep_epsilon = frozenset(('kvlist_n', 'kvlist'))
+        self.keep_epsilon = frozenset(("kvlist_n", "kvlist"))
 
         # ??? Do we need a debug option to skip eliding singleton reductions?
         # Time will tell if it if useful in debugging
@@ -75,20 +86,22 @@ class PythonParser(GenericASTBuilder):
         # FIXME: optional_nt is a misnomer. It's really about there being a
         # singleton reduction that we can simplify. It also happens to be optional
         # in its other derivation
-        self.optional_nt |= frozenset(('come_froms', 'suite_stmts', 'l_stmts_opt',
-                                       'c_stmts_opt'))
+        self.optional_nt |= frozenset(
+            ("come_froms", "suite_stmts", "l_stmts_opt", "c_stmts_opt", "stmts_opt", "stmt")
+        )
 
         # Reduce singleton reductions in these nonterminals:
         # FIXME: would love to do expr, sstmts, stmts and
         # so on but that would require major changes to the
         # semantic actions
-        self.singleton = frozenset(('str', 'store', '_stmts', 'suite_stmts_opt',
-                                    'inplace_op'))
+        self.singleton = frozenset(
+            ("str", "store", "_stmts", "suite_stmts_opt", "inplace_op")
+        )
         # Instructions filled in from scanner
         self.insts = []
 
     def ast_first_offset(self, ast):
-        if hasattr(ast, 'offset'):
+        if hasattr(ast, "offset"):
             return ast.offset
         else:
             return self.ast_first_offset(ast[0])
@@ -101,6 +114,7 @@ class PythonParser(GenericASTBuilder):
            many arguments it has. Often it is not used.
         """
         if rule not in self.new_rules:
+            # print("XXX ", rule) # debug
             self.new_rules.add(rule)
             self.addRule(rule, nop_func)
             customize[opname] = arg_count
@@ -115,7 +129,7 @@ class PythonParser(GenericASTBuilder):
         for rule in rules:
             if len(rule) == 0:
                 continue
-            opname = rule.split('::=')[0].strip()
+            opname = rule.split("::=")[0].strip()
             self.add_unique_rule(rule, opname, 0, customize)
         return
 
@@ -143,49 +157,52 @@ class PythonParser(GenericASTBuilder):
         """Customized format and print for our kind of tokens
         which gets called in debugging grammar reduce rules
         """
+
         def fix(c):
             s = str(c)
-            last_token_pos = s.find('_')
+            last_token_pos = s.find("_")
             if last_token_pos == -1:
                 return s
             else:
                 return s[:last_token_pos]
 
-        prefix = ''
+        prefix = ""
         if parent and tokens:
             p_token = tokens[parent]
-            if hasattr(p_token, 'linestart') and p_token.linestart:
-                prefix = 'L.%3d: ' % p_token.linestart
+            if hasattr(p_token, "linestart") and p_token.linestart:
+                prefix = "L.%3d: " % p_token.linestart
             else:
-                prefix = '       '
-            if hasattr(p_token, 'offset'):
+                prefix = "       "
+            if hasattr(p_token, "offset"):
                 prefix += "%3s" % fix(p_token.offset)
                 if len(rule[1]) > 1:
-                    prefix += '-%-3s ' % fix(tokens[last_token_pos-1].offset)
+                    prefix += "-%-3s " % fix(tokens[last_token_pos - 1].offset)
                 else:
-                    prefix += '     '
+                    prefix += "     "
         else:
-            prefix = '               '
+            prefix = "               "
 
-        print("%s%s ::= %s (%d)" % (prefix, rule[0], ' '.join(rule[1]), last_token_pos))
+        print("%s%s ::= %s (%d)" % (prefix, rule[0], " ".join(rule[1]), last_token_pos))
 
     def error(self, instructions, index):
         # Find the last line boundary
         start, finish = -1, -1
         for start in range(index, -1, -1):
-            if instructions[start].linestart:  break
+            if instructions[start].linestart:
+                break
             pass
-        for finish in range(index+1, len(instructions)):
-            if instructions[finish].linestart:  break
+        for finish in range(index + 1, len(instructions)):
+            if instructions[finish].linestart:
+                break
             pass
         if start > 0:
             err_token = instructions[index]
             print("Instruction context:")
             for i in range(start, finish):
                 if i != index:
-                    indent = '   '
+                    indent = "   "
                 else:
-                    indent = '-> '
+                    indent = "-> "
                 print "%s%s" % (indent, instructions[i])
             raise ParserError(err_token, err_token.offset)
         else:
@@ -196,8 +213,8 @@ class PythonParser(GenericASTBuilder):
         represented by the attr field of token"""
         # Low byte indicates number of positional paramters,
         # high byte number of keyword parameters
-        args_pos = token.attr & 0xff
-        args_kw = (token.attr >> 8) & 0xff
+        args_pos = token.attr & 0xFF
+        args_kw = (token.attr >> 8) & 0xFF
         return args_pos, args_kw
 
     def nonterminal(self, nt, args):
@@ -213,7 +230,7 @@ class PythonParser(GenericASTBuilder):
             #  stmts -> stmts stmt -> stmts stmt -> ...
             #  stmms -> stmt stmt ...
             #
-            if not hasattr(args[0], 'append'):
+            if not hasattr(args[0], "append"):
                 # Was in self.optional_nt as a single item, but we find we have
                 # more than one now...
                 rv = GenericASTBuilder.nonterminal(self, nt, [args[0]])
@@ -228,7 +245,7 @@ class PythonParser(GenericASTBuilder):
                 rv.append(args[1])
         elif n == 1 and args[0] in self.singleton:
             rv = GenericASTBuilder.nonterminal(self, nt, args[0])
-            del args[0] # save memory
+            del args[0]  # save memory
         elif n == 1 and nt in self.optional_nt:
             rv = args[0]
         else:
@@ -241,10 +258,10 @@ class PythonParser(GenericASTBuilder):
         return GenericASTBuilder.ambiguity(self, children)
 
     def resolve(self, list):
-        if len(list) == 2 and 'function_def' in list and 'assign' in list:
-            return 'function_def'
-        if 'grammar' in list and 'expr' in list:
-            return 'expr'
+        if len(list) == 2 and "function_def" in list and "assign" in list:
+            return "function_def"
+        if "grammar" in list and "expr" in list:
+            return "expr"
         # print >> sys.stderr, 'resolve', str(list)
         return GenericASTBuilder.resolve(self, list)
 
@@ -252,17 +269,17 @@ class PythonParser(GenericASTBuilder):
     #  Common Python 2 and Python 3 grammar rules #
     ###############################################
     def p_start(self, args):
-        '''
+        """
         # The start or goal symbol
         stmts ::= sstmt+
-        '''
+        """
 
     def p_call_stmt(self, args):
-        '''
+        """
         # eval-mode compilation.  Single-mode interactive compilation
         # adds another rule.
         call_stmt ::= expr POP_TOP
-        '''
+        """
 
     def p_stmt(self, args):
         """
@@ -283,6 +300,9 @@ class PythonParser(GenericASTBuilder):
         c_stmts_opt ::= c_stmts
         c_stmts_opt ::= pass
 
+        stmts_opt ::= _stmts
+        stmts_opt ::= pass
+
         # statements inside a loop
         l_stmts ::= _stmts
         l_stmts ::= returns
@@ -352,7 +372,7 @@ class PythonParser(GenericASTBuilder):
         pass
 
     def p_function_def(self, args):
-        '''
+        """
         stmt               ::= function_def
         function_def       ::= mkfunc store
         stmt               ::= function_def_deco
@@ -362,15 +382,15 @@ class PythonParser(GenericASTBuilder):
         mkfuncdeco0        ::= mkfunc
         load_closure       ::= load_closure LOAD_CLOSURE
         load_closure       ::= LOAD_CLOSURE
-        '''
+        """
 
     def p_generator_exp(self, args):
-        '''
+        """
         expr ::= generator_exp
         stmt ::= genexpr_func
 
         genexpr_func ::= LOAD_FAST FOR_ITER store comp_iter JUMP_BACK
-        '''
+        """
 
     def p_jump(self, args):
         """
@@ -390,7 +410,7 @@ class PythonParser(GenericASTBuilder):
         """
 
     def p_augmented_assign(self, args):
-        '''
+        """
         stmt ::= aug_assign1
         stmt ::= aug_assign2
 
@@ -416,10 +436,10 @@ class PythonParser(GenericASTBuilder):
         inplace_op ::= INPLACE_AND
         inplace_op ::= INPLACE_XOR
         inplace_op ::= INPLACE_OR
-        '''
+        """
 
     def p_assign(self, args):
-        '''
+        """
         stmt ::= assign
         assign ::= expr DUP_TOP designList
         assign ::= expr store
@@ -428,7 +448,7 @@ class PythonParser(GenericASTBuilder):
         stmt ::= assign3
         assign2 ::= expr expr ROT_TWO store store
         assign3 ::= expr expr expr ROT_THREE ROT_TWO store store store
-        '''
+        """
 
     def p_forstmt(self, args):
         """
@@ -609,8 +629,8 @@ def parse(p, tokens, customize):
 
 
 def get_python_parser(
-        version, debug_parser=PARSER_DEFAULT_DEBUG, compile_mode='exec',
-        is_pypy = False):
+    version, debug_parser=PARSER_DEFAULT_DEBUG, compile_mode="exec", is_pypy=False
+):
     """Returns parser object for Python version 2 or 3, 3.2, 3.5on,
     etc., depending on the parameters passed.  *compile_mode* is either
     'exec', 'eval', or 'single'. See
@@ -631,91 +651,106 @@ def get_python_parser(
         if version < 2.2:
             if version == 1.0:
                 import uncompyle6.parsers.parse10 as parse10
-                if compile_mode == 'exec':
+
+                if compile_mode == "exec":
                     p = parse10.Python10Parser(debug_parser)
                 else:
                     p = parse10.Python01ParserSingle(debug_parser)
             elif version == 1.1:
                 import uncompyle6.parsers.parse11 as parse11
-                if compile_mode == 'exec':
+
+                if compile_mode == "exec":
                     p = parse11.Python11Parser(debug_parser)
                 else:
                     p = parse11.Python11ParserSingle(debug_parser)
             if version == 1.2:
                 import uncompyle6.parsers.parse12 as parse12
-                if compile_mode == 'exec':
+
+                if compile_mode == "exec":
                     p = parse12.Python12Parser(debug_parser)
                 else:
                     p = parse12.Python12ParserSingle(debug_parser)
             if version == 1.3:
                 import uncompyle6.parsers.parse13 as parse13
-                if compile_mode == 'exec':
+
+                if compile_mode == "exec":
                     p = parse13.Python13Parser(debug_parser)
                 else:
                     p = parse13.Python13ParserSingle(debug_parser)
             elif version == 1.4:
                 import uncompyle6.parsers.parse14 as parse14
-                if compile_mode == 'exec':
+
+                if compile_mode == "exec":
                     p = parse14.Python14Parser(debug_parser)
                 else:
                     p = parse14.Python14ParserSingle(debug_parser)
             elif version == 1.5:
                 import uncompyle6.parsers.parse15 as parse15
-                if compile_mode == 'exec':
+
+                if compile_mode == "exec":
                     p = parse15.Python15Parser(debug_parser)
                 else:
                     p = parse15.Python15ParserSingle(debug_parser)
             elif version == 1.6:
                 import uncompyle6.parsers.parse16 as parse16
-                if compile_mode == 'exec':
+
+                if compile_mode == "exec":
                     p = parse16.Python16Parser(debug_parser)
                 else:
                     p = parse16.Python16ParserSingle(debug_parser)
             elif version == 2.1:
                 import uncompyle6.parsers.parse21 as parse21
-                if compile_mode == 'exec':
+
+                if compile_mode == "exec":
                     p = parse21.Python21Parser(debug_parser)
                 else:
                     p = parse21.Python21ParserSingle(debug_parser)
         elif version == 2.2:
             import uncompyle6.parsers.parse22 as parse22
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse22.Python22Parser(debug_parser)
             else:
                 p = parse22.Python22ParserSingle(debug_parser)
         elif version == 2.3:
             import uncompyle6.parsers.parse23 as parse23
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse23.Python23Parser(debug_parser)
             else:
                 p = parse23.Python23ParserSingle(debug_parser)
         elif version == 2.4:
             import uncompyle6.parsers.parse24 as parse24
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse24.Python24Parser(debug_parser)
             else:
                 p = parse24.Python24ParserSingle(debug_parser)
         elif version == 2.5:
             import uncompyle6.parsers.parse25 as parse25
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse25.Python25Parser(debug_parser)
             else:
                 p = parse25.Python25ParserSingle(debug_parser)
         elif version == 2.6:
             import uncompyle6.parsers.parse26 as parse26
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse26.Python26Parser(debug_parser)
             else:
                 p = parse26.Python26ParserSingle(debug_parser)
         elif version == 2.7:
             import uncompyle6.parsers.parse27 as parse27
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse27.Python27Parser(debug_parser)
             else:
                 p = parse27.Python27ParserSingle(debug_parser)
         else:
             import uncompyle6.parsers.parse2 as parse2
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse2.Python2Parser(debug_parser)
             else:
                 p = parse2.Python2ParserSingle(debug_parser)
@@ -724,62 +759,72 @@ def get_python_parser(
         pass
     else:
         import uncompyle6.parsers.parse3 as parse3
+
         if version == 3.0:
             import uncompyle6.parsers.parse30 as parse30
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse30.Python30Parser(debug_parser)
             else:
                 p = parse30.Python30ParserSingle(debug_parser)
         elif version == 3.1:
             import uncompyle6.parsers.parse31 as parse31
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse31.Python31Parser(debug_parser)
             else:
                 p = parse31.Python31ParserSingle(debug_parser)
         elif version == 3.2:
             import uncompyle6.parsers.parse32 as parse32
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse32.Python32Parser(debug_parser)
             else:
                 p = parse32.Python32ParserSingle(debug_parser)
         elif version == 3.3:
             import uncompyle6.parsers.parse33 as parse33
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse33.Python33Parser(debug_parser)
             else:
                 p = parse33.Python33ParserSingle(debug_parser)
         elif version == 3.4:
             import uncompyle6.parsers.parse34 as parse34
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse34.Python34Parser(debug_parser)
             else:
                 p = parse34.Python34ParserSingle(debug_parser)
         elif version == 3.5:
             import uncompyle6.parsers.parse35 as parse35
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse35.Python35Parser(debug_parser)
             else:
                 p = parse35.Python35ParserSingle(debug_parser)
         elif version == 3.6:
             import uncompyle6.parsers.parse36 as parse36
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse36.Python36Parser(debug_parser)
             else:
                 p = parse36.Python36ParserSingle(debug_parser)
         elif version == 3.7:
             import uncompyle6.parsers.parse37 as parse37
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse37.Python37Parser(debug_parser)
             else:
                 p = parse37.Python37ParserSingle(debug_parser)
         elif version == 3.8:
             import uncompyle6.parsers.parse38 as parse38
-            if compile_mode == 'exec':
+
+            if compile_mode == "exec":
                 p = parse38.Python38Parser(debug_parser)
             else:
                 p = parse38.Python38ParserSingle(debug_parser)
         else:
-            if compile_mode == 'exec':
+            if compile_mode == "exec":
                 p = parse3.Python3Parser(debug_parser)
             else:
                 p = parse3.Python3ParserSingle(debug_parser)
@@ -787,18 +832,25 @@ def get_python_parser(
     # p.dump_grammar() # debug
     return p
 
+
 class PythonParserSingle(PythonParser):
     def p_call_stmt_single(self, args):
-        '''
+        """
         # single-mode compilation. Eval-mode interactive compilation
         # drops the last rule.
 
         call_stmt ::= expr PRINT_EXPR
-        '''
+        """
 
 
-def python_parser(version, co, out=sys.stdout, showasm=False,
-                  parser_debug=PARSER_DEFAULT_DEBUG, is_pypy=False):
+def python_parser(
+    version,
+    co,
+    out=sys.stdout,
+    showasm=False,
+    parser_debug=PARSER_DEFAULT_DEBUG,
+    is_pypy=False,
+):
     """
     Parse a code object to an abstract syntax tree representation.
 
@@ -815,6 +867,7 @@ def python_parser(version, co, out=sys.stdout, showasm=False,
 
     assert iscode(co)
     from uncompyle6.scanner import get_scanner
+
     scanner = get_scanner(version, is_pypy)
     tokens, customize = scanner.ingest(co)
     maybe_show_asm(showasm, tokens)
@@ -826,10 +879,12 @@ def python_parser(version, co, out=sys.stdout, showasm=False,
     return parse(p, tokens, customize)
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
+
     def parse_test(co):
         from uncompyle6 import PYTHON_VERSION, IS_PYPY
+
         ast = python_parser(PYTHON_VERSION, co, showasm=True, is_pypy=IS_PYPY)
         print(ast)
         return
-    # parse_test(parse_test.__code__)
+    parse_test(parse_test.func_code)

uncompyle6/parsers/__init__.py

@@ -0,0 +1,11 @@
+"""Here we have parser grammars for the different Python versions.
+Instead of full grammars, we have full grammars for certain Python versions
+and the others indicate differences between a neighboring version.
+
+For example Python 2.6, 2.7, 3.2, and 3.7 are largely "base" versions
+which work off off parse2.py, parse3.py, and parse37base.py.
+
+Some examples:
+Python 3.3 diffs off of 3.2; 3.1 and 3.0 diff off of 3.2; Python 1.0..Python 2.5 diff off of
+Python 2.6 and Python 3.8 diff off of 3.7
+"""

uncompyle6/parsers/parse2.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2019 Rocky Bernstein
+#  Copyright (c) 2015-2020 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #
 #  Copyright (c) 1999 John Aycock
@@ -25,6 +25,7 @@ If we succeed in creating a parse tree, then we have a Python program
 that a later phase can turn into a sequence of ASCII text.
 """
 
+from uncompyle6.parsers.reducecheck import (except_handler_else, ifelsestmt, tryelsestmt)
 from uncompyle6.parser import PythonParser, PythonParserSingle, nop_func
 from uncompyle6.parsers.treenode import SyntaxTree
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
@@ -144,10 +145,10 @@ class Python2Parser(PythonParser):
 
         # Move to 2.7? 2.6 may use come_froms
         tryelsestmtc    ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
-                            except_handler else_suitec COME_FROM
+                            except_handler_else else_suitec COME_FROM
 
         tryelsestmtl    ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
-                            except_handler else_suitel COME_FROM
+                            except_handler_else else_suitel COME_FROM
 
         try_except      ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
                             except_handler COME_FROM
@@ -159,6 +160,8 @@ class Python2Parser(PythonParser):
         except_handler  ::= jmp_abs COME_FROM except_stmts
                              END_FINALLY
 
+        except_handler_else  ::= except_handler
+
         except_stmts ::= except_stmt+
 
         except_stmt ::= except_cond1 except_suite
@@ -635,12 +638,23 @@ class Python2Parser(PythonParser):
             self.addRule(rule, nop_func)
             pass
 
+        self.reduce_check_table = {
+            # "and": and_check,
+            "except_handler_else": except_handler_else,
+            "ifelsestmt": ifelsestmt,
+            # "or": or_check,
+            "tryelsestmt": tryelsestmt,
+            "tryelsestmtl": tryelsestmt,
+        }
+        self.check_reduce["and"] = "AST"
+        self.check_reduce["except_handler_else"] = "tokens"
         self.check_reduce["raise_stmt1"] = "tokens"
         self.check_reduce["assert_expr_and"] = "AST"
         self.check_reduce["tryelsestmt"] = "AST"
         self.check_reduce["tryelsestmtl"] = "AST"
         self.check_reduce["aug_assign2"] = "AST"
         self.check_reduce["or"] = "AST"
+        self.check_reduce["ifstmt"] = "tokens"
         # self.check_reduce['_stmts'] = 'AST'
 
         # Dead code testing...
@@ -651,11 +665,40 @@ class Python2Parser(PythonParser):
         if tokens is None:
             return False
         lhs = rule[0]
+        n = len(tokens)
 
+        fn = self.reduce_check_table.get(lhs, None)
+        if fn:
+            return fn(self, lhs, n, rule, ast, tokens, first, last)
+
+        if rule == ("and", ("expr", "jmp_false", "expr", "\\e_come_from_opt")):
+            # If the instruction after the instructions forming the "and"  is an "YIELD_VALUE"
+            # then this is probably an "if" inside a comprehension.
+            if tokens[last] == "YIELD_VALUE":
+                # Note: We might also consider testing last+1 being "POP_TOP"
+                return True
+
+            # Test that jump_false jump somewhere beyond the end of the "and"
+            # it might not be exactly the end of the "and" because this and can
+            # be a part of a larger condition. Oddly in 2.7 there doesn't seem to be
+            # an optimization where the "and" jump_false is back to a loop.
+            jmp_false = ast[1]
+            if jmp_false[0] == "POP_JUMP_IF_FALSE":
+                while (first < last and isinstance(tokens[last].offset, str)):
+                    last -= 1
+                if jmp_false[0].attr < tokens[last].offset:
+                    return True
+
+            # Test that jmp_false jumps to the end of "and"
+            # or that it jumps to the same place as the end of "and"
+            jmp_false = ast[1][0]
+            jmp_target = jmp_false.offset + jmp_false.attr + 3
+            return not (jmp_target == tokens[last].offset or
+                        tokens[last].pattr == jmp_false.pattr)
         # Dead code testing...
         # if lhs == 'while1elsestmt':
         #     from trepan.api import debug; debug()
-        if (
+        elif (
             lhs in ("aug_assign1", "aug_assign2")
             and ast[0]
             and ast[0][0] in ("and", "or")
@@ -674,33 +717,6 @@ class Python2Parser(PythonParser):
         elif lhs in ("delete_subscript", "del_expr"):
             op = ast[0][0]
             return op.kind in ("and", "or")
-        elif lhs in ("tryelsestmt", "tryelsestmtl"):
-            # Check the end of the except handler that there isn't a jump from
-            # inside the except handler to the end. If that happens
-            # then this is a "try" with no "else".
-            except_handler = ast[3]
-            if except_handler == "except_handler":
-
-                come_from = except_handler[-1]
-                # We only care about the *first* come_from because that is the
-                # the innermost one. So if the "tryelse" is invalid (should be a "try")
-                # ti will be invalid here.
-                if come_from == "COME_FROM":
-                    first_come_from = except_handler[-1]
-                else:
-                    assert come_from == "come_froms"
-                    first_come_from = come_from[0]
-                leading_jump = except_handler[0]
-
-                # We really don't care that this is a jump per-se. But
-                # we could also check that this jumps to the end of the except if
-                # desired.
-                if isinstance(leading_jump, SyntaxTree):
-                    except_handler_first_offset = leading_jump.first_child().off2int()
-                else:
-                    except_handler_first_offset = leading_jump.off2int()
-                return first_come_from.attr > except_handler_first_offset
-
 
         return False
 

uncompyle6/parsers/parse21.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2017 Rocky Bernstein
+#  Copyright (c) 2016-2017, 2020 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
@@ -19,9 +19,6 @@ class Python21Parser(Python22Parser):
         for         ::= SETUP_LOOP expr for_iter store
                         l_stmts_opt _jump_back
                         POP_BLOCK COME_FROM
-
-        expr        ::= conditional
-        conditional ::= expr jmp_false expr JUMP_ABSOLUTE expr
         """
 
     def p_import21(self, args):

uncompyle6/parsers/parse24.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2018 Rocky Bernstein
+#  Copyright (c) 2016-2018, 2020 Rocky Bernstein
 """
 spark grammar differences over Python2.5 for Python 2.4.
 """
@@ -56,6 +56,12 @@ class Python24Parser(Python25Parser):
         kv2    ::= DUP_TOP expr expr ROT_THREE STORE_SUBSCR
         '''
 
+    def remove_rules_24(self):
+        self.remove_rules("""
+        expr ::= conditional
+        """)
+
+
     def customize_grammar_rules(self, tokens, customize):
         self.remove_rules("""
         gen_comp_body ::= expr YIELD_VALUE POP_TOP
@@ -72,6 +78,7 @@ class Python24Parser(Python25Parser):
         stmt ::= withasstmt
         """)
         super(Python24Parser, self).customize_grammar_rules(tokens, customize)
+        self.remove_rules_24()
         if self.version == 2.4:
             self.check_reduce['nop_stmt'] = 'tokens'