Readme link typo

until we can more fully untangle if stmts in loops.
Current tests break urllib2.pyc and cgi.pyc

.circleci/config.yml

@@ -12,9 +12,8 @@ jobs:
       COMPILE: --compile
     # To see the list of pre-built images that CircleCI provides for most common languages see
     # https://circleci.com/docs/2.0/circleci-images/
-    machine:
-      python:
-        version: 2.7.14
+    docker:
+      - image: circleci/python:3.6.9
     steps:
     # Machine Setup
     #   If you break your build into multiple jobs with workflows, you will probably want to do the parts of this that are relevant in each
@@ -27,7 +26,7 @@ jobs:
     # This is based on your 1.0 configuration file or project settings
     - run:
         working_directory: ~/rocky/python-uncompyle6
-        command: pip install virtualenv && pip install nose && pip install pep8 && pyenv rehash
+        command: pip install --user virtualenv && pip install --user nose && pip install --user pep8
     # Dependencies
     #   This would typically go in either a build or a build-and-test job when using workflows
     # Restore the dependency cache
@@ -38,9 +37,9 @@ jobs:
           - v2-dependencies-
 
     # This is based on your 1.0 configuration file or project settings
-    - run: pip install --upgrade setuptools
-    - run: pip install -e .
-    - run: pip install -r requirements-dev.txt
+    - run: pip install --user --upgrade setuptools
+    - run: pip install --user -e .
+    - run: pip install --user -r requirements-dev.txt
 
     # Save dependency cache
     - save_cache:
@@ -58,7 +57,7 @@ jobs:
     # Test
     #   This would typically be a build job when using workflows, possibly combined with build
     # This is based on your 1.0 configuration file or project settings
-    - run: python ./setup.py develop && make check-2.7
+    - run: sudo python ./setup.py develop && make check-3.6
     - run: cd ./test/stdlib && bash ./runtests.sh 'test_[p-z]*.py'
     # Teardown
     #   If you break your build into multiple jobs with workflows, you will probably want to do the parts of this that are relevant in each

NEWS.md

@@ -1,3 +1,80 @@
+3.6.4: 2020-2-9 Plateau
+=======================
+
+The main focus in this release was fix some of the more glaring problems creapt in from the last release due to that refactor.
+
+`uncompyle6` code is at a plateau where what is most needed is a code refactoring. In doing this, until everything refactored and replaced, decomplation may get worse.
+Therefore, this release largely serves as a checkpoint before more major upheaval.
+
+The upheaval, in  started last release, I believe the pinnicle was around c90ff51 which wasn't a release. I suppose I should tag that.
+
+After c90ff5, I started down the road of redoing control flow in a more comprehensible, debuggable, and scalable way. See [The Control Flow Mess](https://github.com/rocky/python-uncompyle6/wiki/The-Control-Flow-Mess)
+
+The bulk of the refactoring going on in the [decompyle3](https://github.com/rocky/python-decompil3) project, but I try to trickle down the changes.
+
+It is tricky because the changes are large and I have to figure decompose things so that little testable pieces can be done. And there is also the problem that what is in decompyle3 is incomplete as well.
+
+Other than control flow, another change that will probably happen in the next release is to redo the grammar for lambda expressions. Right now, we treat them as Python statements, you know, things with compound statements in them. But lambda aren't that. And so there is hackery to paper over difference making a statement out of an expression the wrong thing to do. For example, a return of an "and" expression can be expressed as nested "if" statements with return inside them, but the "if" variant of the bytecode is not valid in a lambda.
+
+In the decompyle3 code, I've gone down the road making the grammar goal symbol be an expression. This also offers the opportunity to split the grammar making parsing inside lambda not only more reliable because the wrong choices don't exist, but also simpler and faster because all those rules just need don't need to exist in parsing.
+
+I cringe in thinking about how the code has lived for so long without noticing such a simple stupidity, and lapse of sufficient thought.
+
+Some stats from testing. The below give numbers of decompiled tests from Python's test suite which succesfully ran
+
+```
+   Version  test-suites passing
+   -------  -------------------
+   2.4.6     243
+   2.5.6     265
+   2.6.9     305
+   3.3.7     300
+   3.4.10    304
+   3.5.9     260
+   3.6.10    236
+   3.7.6     306
+   3.8.1     114
+```
+
+Decompiled bytecode files distributed with Python (syntax check only):
+
+```
+2.7.17  647 files:   0 failed
+3.2.6   900 files:   0 failed
+3.3.7  1256 files:   0 failed
+3.4.10  800 files:   0 failed
+3.5.9   900 files:   0 failed
+3.6.10 1300 files:  28 failed
+```
+
+
+3.6.3: 2020-1-26 Martin and Susanne
+===================================
+
+Of late, every release fixes major gaps and embarrassments of the last release....
+
+And in some cases, like this one, exposes lacuna and rot.
+
+I now have [control] flow under control, even if it isn't the most optimal way.
+
+I now have greatly expanded automated testing.
+
+On the most recent Python versions I regularly decompile thousands of Python programs that are distributed with Python.  when it is possible, I then decompile Python's standard test suite distributed with Python and run the decompiled source code which basically checks itself. This amounts to about 250 test programs per version. This is in addition to the 3 CI testing services which do different things.
+
+Does this mean the decompiler works perfectly? No. There are still a dozen or so failing programs, although the actual number of bugs is probably smaller though.
+
+However, in perparation of a more major refactoring of the parser grammar, this release was born.
+
+In many cases, decompilation is better. But there are some cases where decompilation has gotten worse. For lack of time (and interest) 3.0 bytecode suffered a hit. Possibly some code in the 3.x range did too. In time and with cleaner refactored code, this will come back.
+
+Commit c90ff51 was a local maxiumum before, I started reworking the grammar to separate productions that were specific to loops versus those that are not in loops.
+In the middle of that I added another grammar simplication to remove singleton productions of the form `sstmts-> stmts`. These were always was a bit ugly, and complicated output.
+
+At any rate if decompilation fails, you can try c90ff51. Or another decompiler. `unpyc37` is pretty good for 3.7. wibiti `uncompyle2` is great for 2.7. `pycdc` is mediocre for Python before 3.5 or so, and not that good for the most recent Python. Geerally these programs will give some sort of answer even if it isn't correct.
+
+decompyle3 isn't that good for 3.7 and worse for 3.8, but right now it does things no other Python decompiler like `unpyc37` or `pycdc` does. For example, `decompyle3` handles variable annotations. As always, the issue trackers for the various programs will give you a sense for what needs to be done. For now, I've given up on reporting issues in the other decompilers because there are already enough issues reported, and they are just not getting fixed anyway.
+
+
 3.6.2: 2020-1-5 Samish
 ======================
 
@@ -226,8 +303,7 @@ Lots of decomplation bugs, especially in the 3.x series fixed. Don't worry thoug
 3.3.0 2019-04-14 Holy Week
 ==========================
 
-* First cut at Python 3.8 (many bugs remain)
-* Reinstate -c | --compile (compile before disassembly) option
+* First cut at Python 3.8 (many bug remain)
 * The usual smattering of bug and doc fixes
 
 3.2.6 2019-03-23 Mueller Report

README.rst

@@ -54,7 +54,7 @@ only; another patched that and handled only 3.3.  You get the
 idea. This code pulls all of these forks together and *moves
 forward*. There is some serious refactoring and cleanup in this code
 base over those old forks. Even more experimental refactoring is going
-on in decompile3_.
+on in decompyle3_.
 
 This demonstrably does the best in decompiling Python across all
 Python versions. And even when there is another project that only
@@ -140,9 +140,16 @@ Python syntax changes, you should use this option if the bytecode is
 the right bytecode for the Python interpreter that will be checking
 the syntax.
 
-You can also cross compare the results with another python decompiler
-like pycdc_ . Since they work differently, bugs here often aren't in
-that, and vice versa.
+You can also cross compare the results with either another version of
+`uncompyle6` since there are are sometimes regressions in decompiling
+specific bytecode as the overall quality improves.
+
+For Python 3.7 and above, the code in decompyle3_ is generally
+better.
+
+Or try specific another python decompiler like uncompyle2_, unpyc37_,
+or pycdc_.  Since the later two work differently, bugs here often
+aren't in that, and vice versa.
 
 There is an interesting class of these programs that is readily
 available give stronger verification: those programs that when run
@@ -232,7 +239,7 @@ See Also
 * https://github.com/rocky/python-xdis : Cross Python version disassembler
 * https://github.com/rocky/python-xasm : Cross Python version assembler
 * https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
-* https://github.com/zrax/pycdc : The README for this C++ code syas it aims to support all versions of Python. It is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
+* https://github.com/zrax/pycdc : The README for this C++ code says it aims to support all versions of Python. It is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
 
 
 .. _trepan: https://pypi.python.org/pypi/trepan2g
@@ -241,7 +248,9 @@ See Also
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake
 .. _pycdc: https://github.com/zrax/pycdc
-.. _decompile3: https://github.com/rocky/python-decompile3
+.. _decompyle3: https://github.com/rocky/python-decompile3
+.. _uncompyle2: https://github.com/wibiti/uncompyle2
+.. _unpyc37: https://github.com/andrew-tavera/unpyc37
 .. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
 .. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
 		 :target: https://travis-ci.org/rocky/python-uncompyle6

admin-tools/README.md

@@ -8,4 +8,15 @@ They are customized to my environment:
 - I have git repos for xdis, and spark parser at the same level as uncompyle6
 
 There may be other rocky-specific things that need customization.
-how-to-make-a-release.txt has overall how I make a release
+how-to-make-a-release.md has overall how I make a release
+
+Since this project uses python over a wide variety of release, some versions
+of projects that should be used for specific Python versions
+
+for 3.2.6:
+   pytest==2.9.2
+
+for 3.1.5
+   pytset==2.1.0
+   py=1.8.0 and comment out line 10 of _builtin.py # callable = callable
+   six==1.10.0

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.9 3.6.9 2.6.9 3.3.7 2.7.17 3.2.6 3.1.5 3.4.8 3.7.6 3.8.1'
+export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.17 3.2.6 3.1.5 3.4.10 3.7.6 3.8.1'

admin-tools/setup-master.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-PYTHON_VERSION=3.7.5
+PYTHON_VERSION=3.7.6
 
 # FIXME put some of the below in a common routine
 function finish {

pytest/test_grammar.py

@@ -27,6 +27,9 @@ def test_grammar():
             expect_lhs.add("attribute")
 
         expect_lhs.add("get_iter")
+
+        if PYTHON_VERSION > 3.7 or PYTHON_VERSION < 3.0:
+            expect_lhs.add("stmts_opt")
     else:
         expect_lhs.add("async_with_as_stmt")
         expect_lhs.add("async_with_stmt")
@@ -43,6 +46,10 @@ def test_grammar():
         expect_lhs.add("kv3")
         unused_rhs.add("dict")
 
+        if PYTHON_VERSION < 3.7 and PYTHON_VERSION != 2.7:
+            # NOTE: this may disappear
+            expect_lhs.add("except_handler_else")
+
     if PYTHON3:
         expect_lhs.add("load_genexpr")
 

pytest/test_token.py

@@ -1,24 +1,25 @@
 from uncompyle6 import PYTHON_VERSION
 from uncompyle6.scanners.tok import Token
 
+
 def test_token():
     # Test token formatting of: LOAD_CONST None
-    t = Token('LOAD_CONST', offset=0, attr=None, pattr=None, has_arg=True)
-    expect = '           0  LOAD_CONST               None'
+    t = Token("LOAD_CONST", offset=0, attr=None, pattr=None, has_arg=True)
+    expect = "             0  LOAD_CONST               None"
     # print(t.format())
     assert t
     assert t.format() == expect
 
     # Make sure equality testing of tokens ignores offset
-    t2 = Token('LOAD_CONST', offset=2, attr=None, pattr=None, has_arg=True)
+    t2 = Token("LOAD_CONST", offset=2, attr=None, pattr=None, has_arg=True)
     assert t2 == t
 
-
     # Make sure formatting of: LOAD_CONST False. We assume False is the 0th index
     # of co_consts.
-    t = Token('LOAD_CONST', offset=1, attr=False, pattr=False, has_arg=True)
-    expect = '           1  LOAD_CONST               False'
+    t = Token("LOAD_CONST", offset=1, attr=False, pattr=False, has_arg=True)
+    expect = "             1  LOAD_CONST               False"
     assert t.format() == expect
 
-if __name__ == '__main__':
+
+if __name__ == "__main__":
     test_token()

pytest/testdata/if-2.7.right

@@ -1,12 +1,12 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_if.py
 
-   6       0  LOAD_NAME             0  'True'
-           3  POP_JUMP_IF_FALSE    15  'to 15'
+   6         0  LOAD_NAME             0  'True'
+             3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   7       6  LOAD_NAME             1  'False'
-           9  STORE_NAME            2  'b'
-          12  JUMP_FORWARD          0  'to 15'
-        15_0  COME_FROM            12  '12'
-          15  LOAD_CONST               None
-          18  RETURN_VALUE     
+   7         6  LOAD_NAME             1  'False'
+             9  STORE_NAME            2  'b'
+            12  JUMP_FORWARD          0  'to 15'
+          15_0  COME_FROM            12  '12'
+            15  LOAD_CONST               None
+            18  RETURN_VALUE     

pytest/testdata/ifelse-2.7.right

@@ -1,15 +1,15 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_ifelse.py
 
-   3       0  LOAD_NAME             0  'True'
-           3  POP_JUMP_IF_FALSE    15  'to 15'
+   3         0  LOAD_NAME             0  'True'
+             3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   4       6  LOAD_CONST               1
-           9  STORE_NAME            1  'b'
-          12  JUMP_FORWARD          6  'to 21'
+   4         6  LOAD_CONST               1
+             9  STORE_NAME            1  'b'
+            12  JUMP_FORWARD          6  'to 21'
 
-   6      15  LOAD_CONST               2
-          18  STORE_NAME            2  'd'
-        21_0  COME_FROM            12  '12'
-          21  LOAD_CONST               None
-          24  RETURN_VALUE     
+   6        15  LOAD_CONST               2
+            18  STORE_NAME            2  'd'
+          21_0  COME_FROM            12  '12'
+            21  LOAD_CONST               None
+            24  RETURN_VALUE     

test/.gitignore

@@ -1,2 +1,3 @@
 /.coverage
+/.python-version
 /nohup.out

test/decompyle/test_loops2.py

@@ -8,7 +8,7 @@ See http://www.goebel-consult.de/decompyle/ for download and
 for further information
 """
 
-# This is a seperate test pattern, since 'continue' within 'try'
+# This is a separate test pattern, since 'continue' within 'try'
 # was not allowed till Python 2.1
 
 for term in args:

test/run-and-email.sh

@@ -14,27 +14,47 @@ function displaytime {
     printf '%d seconds\n' $S
 }
 
-PYVERSION=${PYVERSION:-"3.5.5 2.7.14 3.2.6 3.3.7 3.4.8 2.6.9 3.6.4"}
-# PYVERSION=${PYVERSION:-"3.5.5"}
+. ../admin-tools/pyenv-newer-versions
 
 USER=${USER:-rocky}
 EMAIL=${EMAIL:-rb@dustyfeet.com}
 MAX_TESTS=${MAX_TESTS:-800}
 typeset -i RUN_STARTTIME=$(date +%s)
 
-for VERSION in $PYVERSION ; do
+# PYVERSIONS="3.5.6"
+actual_versions=""
+for VERSION in $PYVERSIONS ; do
     typeset -i rc=0
     LOGFILE=/tmp/pyenvlib-$VERSION-$$.log
 
-    if [[ $VERSION == '3.5.5' ]] ; then
-	MAX_TESTS=224
-    elif [[ $VERSION == '3.2.6' ]] ; then
-	MAX_TESTS=700
-    elif [[ $VERSION == '3.6.4' ]] ; then
-	MAX_TESTS=400
-    else
-	MAX_TESTS=800
-    fi
+    case "$VERSION" in
+	3.7.6 | 3.8.1 | 3.1.5 | 3.0.1 )
+	    continue
+	    ;;
+	3.5.9 )
+	    MAX_TESTS=900
+	    ;;
+	3.2.6 )
+	    MAX_TESTS=900
+	    ;;
+	3.3.7 )
+	    MAX_TESTS=1300 # About 1256 exist
+	    ;;
+	3.4.10 )
+	    MAX_TESTS=800
+	    ;;
+	3.6.10 )
+	    MAX_TESTS=1300  # about 2139 exist
+	    ;;
+	2.6.9 )
+	    MAX_TESTS=1300
+	    ;;
+	* )
+	     MAX_TESTS=800
+	     ;;
+    esac
+
+    actual_versions="$actual_versions $VERSION"
 
     if ! pyenv local $VERSION ; then
 	rc=1
@@ -42,7 +62,9 @@ for VERSION in $PYVERSION ; do
       echo Python Version $(pyenv local) > $LOGFILE
       echo "" >> $LOGFILE
       typeset -i ALL_FILES_STARTTIME=$(date +%s)
-      python ./test_pyenvlib.py --max ${MAX_TESTS} --syntax-verify --$VERSION  >>$LOGFILE 2>&1
+      cmd="python ./test_pyenvlib.py --max ${MAX_TESTS} --syntax-verify --$VERSION"
+      echo "$cmd" >>$LOGFILE 2>&1
+      $cmd >>$LOGFILE 2>&1
       rc=$?
 
       echo Python Version $(pyenv local) >> $LOGFILE
@@ -66,4 +88,4 @@ done
 typeset -i RUN_ENDTIME=$(date +%s)
 (( time_diff =  RUN_ENDTIME - RUN_STARTTIME))
 elapsed_time=$(displaytime $time_diff)
-echo "Run complete $elapsed_time for versions $PYVERSION" | mail -s "pyenv weak verify in $elapsed_time" ${EMAIL}
+echo "Run complete $elapsed_time for versions $actual_versions" | mail -s "pyenv weak verify in $elapsed_time" ${EMAIL}

test/simple_source/branching/02_ifelse_lambda.py

@@ -22,7 +22,7 @@ assert i[0]('a') == True
 assert i[0]('A') == False
 
 # Issue #170. Bug is needing an "conditional_not_lambda" grammar rule
-# in addition the the "if_expr_lambda" rule
+# in addition the the "if_exp_lambda" rule
 j = lambda a: False if not a else True
 assert j(True) == True
 assert j(False) == False

test/simple_source/branching/10_if_else_ternary.py

@@ -1,8 +1,8 @@
 # Tests:
 
 # ret_expr_or_cond ::= ret_expr
-# ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
-# ret_expr_or_cond ::= ret_cond
+# if_exp_ret ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
+# ret_expr_or_cond ::= if_exp_ret
 # ret_or ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
 
 # See https://github.com/rocky/python-uncompyle6/issues/5

test/simple_source/bug26/00_future_divide.py

@@ -0,0 +1,4 @@
+# From 2.7.17 fractions
+"""Rational, infinite-precision, real numbers."""
+
+from __future__ import division

test/simple_source/bug26/03_weird26.py

@@ -8,7 +8,7 @@ list(x for x in range(10) if x % 2 if x % 3)
 
 # expresion which evaluates True unconditionally,
 # but leave dead code or junk around that we have to match on.
-# Tests "if_expr_true" rule
+# Tests "if_exp_true" rule
 5 if 1 else 2
 
 0 or max(5, 3) if 0 else 3

test/simple_source/bug26/04_if_and_bug.py

@@ -0,0 +1,52 @@
+def foo():
+    if julian == -1 and week_of_year != -1 and weekday != -1:
+        first_weekday = datetime_date(year, 1, 1).weekday()
+        preceeding_days = 7 - first_weekday
+        if preceeding_days == 7:
+            preceeding_days = 0
+        if weekday == 6 and week_of_year_start == 6:
+            week_of_year -= 1
+        if weekday == 0 and first_weekday == 0 and week_of_year_start == 6:
+            week_of_year += 1
+        if week_of_year == 0:
+            julian = 1 + weekday - first_weekday
+        else:
+            days_to_week = preceeding_days + 7 * (week_of_year - 1)
+            julian = 1 + days_to_week + weekday
+
+
+# 2.6 pstats.py
+# Bug is handling "for" with "elif" and "and"s.
+def eval_print_amount(a, b, c, d, list, msg=0):
+    if a:
+        for i in list:
+            msg = 1
+    elif b and c:
+        msg = 2
+    elif c and d:
+        msg = 3
+    return msg
+
+assert eval_print_amount(True, False, False, False, [1]) == 1
+assert eval_print_amount(True, False, False, False, []) == 0
+assert eval_print_amount(False, True, True, False, []) == 2
+assert eval_print_amount(False, False, True, True, []) == 3
+assert eval_print_amount(False, False, False, True, []) == 0
+
+
+# Bug in 2.6 was in including the part at x = value
+# at the end asa part of the "else"
+def eval_directive(a):
+    if a:
+        value = 2
+    else:
+        try:
+            value = 3
+        except:
+            pass
+
+    x = value
+    return x
+
+assert eval_directive(True)  == 2
+assert eval_directive(False) == 3

test/simple_source/bug27+/03_not_dead_code.py

@@ -1,6 +1,6 @@
 # Bug found in 2.7 test_itertools.py
-# Bug was erroneously using reduction to if_expr_true
-# A proper fix would be to use if_expr_true only when we
+# Bug was erroneously using reduction to if_exp_true
+# A proper fix would be to use if_exp_true only when we
 # can determine there is or was dead code.
 from itertools import izip_longest
 for args in [['abc', range(6)]]:

test/simple_source/bug30/05_block_fallback.py

@@ -0,0 +1,32 @@
+# Adapted from 3.7.6 test_contains
+# The bug was in reconstructing something equivalent to
+# "while False: yelid None" which is *needed* in __iter__().
+# Sheeh!
+
+# RUNNABLE!
+def test_block_fallback():
+    # blocking fallback with __contains__ = None
+    class ByContains(object):
+        def __contains__(self, other):
+            return False
+    c = ByContains()
+    class BlockContains(ByContains):
+        """Is not a container
+
+        This class is a perfectly good iterable (as tested by
+        list(bc)), as well as inheriting from a perfectly good
+        container, but __contains__ = None prevents the usual
+        fallback to iteration in the container protocol. That
+        is, normally, 0 in bc would fall back to the equivalent
+        of any(x==0 for x in bc), but here it's blocked from
+        doing so.
+        """
+        def __iter__(self):
+            while False:
+                yield None
+        __contains__ = None
+    bc = BlockContains()
+    assert not (0 in c)
+    assert not (0 in list(bc))
+
+test_block_fallback()

test/simple_source/bug30/06_listcomp.py

@@ -23,6 +23,17 @@ def columnize(l):
             if not isinstance(l[i], str)]
 assert [0, 2] == columnize([1, 'a', 2])
 
+# From 3.7.6 _collections_abc.py
+# Bug was handling "or" in listcomp
+def count(values, x):
+    return sum(1 for v in values if v or x)
+
+assert count([2, 2], False) == 2
+assert count([], False) == 0
+assert count([], True) == 0
+assert count([2], True) == 1
+assert count([0], False) == 0
+
 # From 3.7 test_generators
 # Bug was in handling the way list_if is optimized in 3.7+;
 # We need list_if37 and compare_chained37.

test/simple_source/bug31/04_def_annotate.py

@@ -150,3 +150,12 @@ ann2(1)
 assert test12(1, 2, 3, name='hi') == (1, (2, 3)), "a, *args, name"
 assert test13(1, 2, 3, name='hi') == ((1, 2, 3), 'hi'), "*args, name"
 assert test16('localhost', loop=2, limit=3, a='b') == ('localhost', None, 2, 3, {'a': 'b'})
+
+# From test 3.5 test_pydoc.py.
+# Bug was in 3.5 and earlier handling of the return type, typing.Tuple[...]
+try:
+    import typing
+    def foo() -> typing.Iterator[typing.Tuple[int, typing.Any]]:
+        ...
+except:
+    pass

test/simple_source/bug33/01_triple_compare.py

@@ -14,3 +14,23 @@ def _is_valid_netmask(netmask):
 
 # See in 2.6.9 quopri.py ishex():
 assert not '0' <= __file__ <= '9' or 'a' <= __file__ <= 'f' or 'A' <= __file__ <= 'F'
+
+# From 3.7 bug-grammar.py
+
+# Bug in 3.7 was handling the last line where compare_chained -> compare_chained37 and
+# therefore compare_chained has one child, not two as it normally does.
+
+def test_comparison():
+    ### comparison: expr (comp_op expr)*
+    ### comp_op: '<'|'>'|'=='|'>='|'<='|'!='|'in'|'not' 'in'|'is'|'is' 'not'
+    if 1: pass
+    x = (1 == 1)
+    if 1 == 1: pass
+    if 1 != 1: pass
+    if 1 < 1: pass
+    if 1 > 1: pass
+    if 1 <= 1: pass
+    if 1 >= 1: pass
+    if 1 in (): pass
+    if 1 not in (): pass
+    if 1 < 1 > 1 == 1 >= 1 <= 1 != 1 in 1 not in 1 is 1 is not 1: pass

test/simple_source/bug33/08_if_else.py

@@ -8,3 +8,33 @@ def init(modules=None):
 
 assert init() == set()
 assert init([1, 2, 3]) == set([1, 2, 3])
+
+# From 3.6 sre_parse
+# Bug was in handling multple COME_FROMS from nested if's
+def _escape(a, b, c, d, e):
+    if a:
+        if b:
+            if c:
+                if d:
+                    raise
+                return
+        if e:
+            if d:
+                raise
+            return
+        raise
+
+assert _escape(False, True, True,  True,  True) is None
+assert _escape(True,  True, True,  False, True) is None
+assert _escape(True,  True, False, False, True) is None
+
+for args in (
+        (True, True,  True, False, True),
+        (True, False, True, True,  True),
+        (True, False, True, True,  False),
+        ):
+    try:
+        _escape(*args)
+        assert False, args
+    except:
+        pass

test/simple_source/bug35/02_async_for.py

@@ -1,3 +1,12 @@
 async def a(b, c):
     async for b in c:
         pass
+
+# From 3.7 test_generators.py
+# Bug was getting indentation correct for multiple async's
+async def foo(X):
+    async for i in X:
+        pass
+    async for i in X:
+        pass
+    raise Done

test/simple_source/bug35/04_call_function.py

@@ -79,3 +79,7 @@ class ResultMixin(object):
 
 class SplitResult(namedtuple('SplitResult', 'scheme netloc path query fragment'), ResultMixin):
     pass
+
+# From 3.3.7 test_long.py
+# Bug was that we need parens around first "0"
+assert (0).bit_length() == 0

test/simple_source/bug35/04_importlist.py

@@ -5,12 +5,15 @@
 
 # RUNNABLE!
 import os.path as osp
-from sys import path
+from sys import platform
 from os import sep, name
 import collections.abc
 
 assert osp.basename("a") == "a"
-assert path
+
+assert isinstance(platform, str)
 assert sep
 assert name
 assert collections.abc
+import os.path as path
+assert path

test/simple_source/bug36/01_conditional.py

@@ -0,0 +1,21 @@
+# From 3.7.6 test_buffer.py
+
+# RUNNABLE!
+def foo(n):
+    zero_stride = True if n >= 95 and n & 1 else False
+    return zero_stride
+
+assert foo(95)
+assert not foo(94)
+assert not foo(96)
+
+# from test_buffer.py
+# Bug was handling "or" inside a conditional
+def rslice(a, b):
+    minlen = 0 if a or b else 1
+    return minlen
+
+assert rslice(False, False) == 1
+assert rslice(False, True) == 0
+assert rslice(True, False) == 0
+assert rslice(True, True) == 0

test/simple_source/bug36/01_extended_arg.py

@@ -1,2 +1,82 @@
 if __file__:
     0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0
+
+
+# From 3.7 test_buffer.py
+# Bug is in dealing with EXTENDED_ARG instructions.
+# In reduction-rule tests where we are testing the offset,
+# getting *which* offset to test against, when there are two
+# possible offset, can mess us up.
+
+def five(a):
+    return 5
+
+def test_ndarray_slice_multidim(a, f, listerr):
+    for slices in a:
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        nderr = None
+        if nderr or listerr:
+            return f(5)
+        else:
+            return 2
+
+assert test_ndarray_slice_multidim([1], five, False) == 2
+assert test_ndarray_slice_multidim([1], five, True) == 5
+
+# From 3.7 test_builtin.py
+def test_pow(self, m, a, b, c, f):
+
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+    shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+
+    shape_t = 0; shape_t = 1; shape_t = 2;
+
+    for z in m:
+        if a or \
+           b or \
+           c:
+            f(TypeError)
+        else:
+            x = 2
+
+    x = 3
+
+# From 3.7 test_exceptions.py
+# Bug is handling extended arg
+def testAttributes(exceptionList):
+    try:
+        x = 0
+    except:
+        pass
+
+    for exc in exceptionList:
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0; x = 1; x = 2; x = 3; x = 4; x = 5; x = 6; x = 7; x = 8; x = 9;
+        x = 0
+
+# From test_urllibnet2.py
+# Bug was in detecting end of "if" flowing into an extended-arg "for".
+def _test_urls(retry, urls):
+    if retry:
+        urlopen = 1
+
+    for url in urls:
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0; shape_t = 1; shape_t = 2; shape_t = 3; shape_t = 4; shape_t = 5; shape_t = 6; shape_t = 7; shape_t = 8; shape_t = 9
+        shape_t = 0

test/simple_source/bug36/02_call_ex_kw.py

@@ -53,3 +53,20 @@ from collections import namedtuple
 Point = namedtuple('Point', 'x y')
 p = Point(11, 22)
 assert p == Point(**dict(x=11, y=22))
+
+# From 3.7 test/test_keywordonlyarg.py
+# Bug was in handling {"4":4} as a dictionary needing **
+def posonly_sum(pos_arg1, *arg, **kwarg):
+    return pos_arg1 + sum(arg) + sum(kwarg.values())
+assert 1+2+3+4 == posonly_sum(1,*(2,3),**{"4":4})
+
+# From 3.7 test_grammar.py
+# Bug was in handling keyword-only parameters when there are annotations.
+# The stack order from least- to most-recent is:
+# default, keyword, annotation, closure
+# This changes in between Python 3.5 and 3.6.
+def f(a, b: 1, c: 2, d, e: 3=4, f=5, *g: 6, h: 7, i=8, j: 9=10,
+      **k: 11) -> 12: pass
+
+assert f.__annotations__ == {'b': 1, 'c': 2, 'e': 3, 'g': 6, 'h': 7, 'j': 9,
+                             'k': 11, 'return': 12}

test/simple_source/bug36/10_fstring.py

@@ -67,3 +67,52 @@ def _repr_fn(fields):
 
 fields = ['a', 'b', 'c']
 assert _repr_fn(fields) == ['return xx + f"(a={self.a!r}, b={self.b!r}, c={self.c!r})"']
+
+
+#################################
+# From Python 3.7 test_fstring.py
+
+x = 5
+assert f'{(lambda y:x*y)("8")!r}' == "'88888'"
+assert f'{(lambda y:x*y)("8")!r:10}' == "'88888'   "
+assert f'{(lambda y:x*y)("8"):10}' == "88888     "
+
+try:
+    eval("f'{lambda x:x}'")
+except SyntaxError:
+    pass
+else:
+    assert False, "f'{lambda x:x}' should be a syntax error"
+
+(x, y, width) = ("foo", 2, 10)
+assert f'x={x*y:{width}}' == 'x=foofoo    '
+
+# Why the fact that the distinction of docstring versus stmt is a
+# string expression is important academic, but we will decompile an
+# equivalent thing. For compatiblity with older Python we'll use "%"
+# instead of a format string
+def f():
+    f'''Not a docstring'''
+def g():
+    '''Not a docstring''' \
+    f''
+
+assert f.__doc__ is None
+assert g.__doc__ is None
+
+import decimal
+width, precision, value = (10, 4, decimal.Decimal('12.34567'))
+
+# Make sure we don't have additional f'..' inside the format strings below.
+assert f'result: {value:{width}.{precision}}' == 'result:      12.35'
+assert f'result: {value:{width:0}.{precision:1}}' == 'result:      12.35'
+assert f'{2}\t' ==  '2\t'
+
+# But below we *do* need the additional f".."
+assert f'{f"{0}"*3}' == "000"
+
+# We need to make sure we have { {x:... not {{x: ...
+#                               ^
+# The former, {{ confuses the format strings so dictionary/set comprehensions
+# don't work.
+assert f'expr={ {x: y for x, y in [(1, 2), ]}}' == 'expr={1: 2}'

test/simple_source/bug37/01_assert2.py

@@ -6,3 +6,13 @@ def test_assert2(c):
         raise SyntaxError('Oops')
 
 test_assert2(5)
+
+# Bug is handling "assert" and confusing it with "or".
+# It is important that the assert be at the end of the loop.
+for x in (2, 4, 6):
+    assert x == x
+
+# Bug in 3.7 was not having a rule for 2-arg assert.
+# 2-arg assert code doesn't match "if not ... raise "
+for x in (1, 3, 5):
+    assert x == x, "foo"

test/simple_source/bug37/04_async.py

@@ -10,3 +10,14 @@ async def test_enter(self):
             x = 1
             y = 2
         assert manager is context
+
+# From 3.7.6 test_coroutines.py
+# Bug was different form of code for "async with" below
+class CoroutineTest():
+    def test_with_8(self):
+        CNT = 0
+        async def foo():
+            nonlocal CNT
+            async with CM():
+                CNT += 1
+                return

test/simple_source/bug38/04_for_no_jump_back.py

@@ -1,6 +1,8 @@
 # from mult_by_const/instruction.py
 # Bug in 3.8 was handling no JUMP_BACK in "for" loop. It is
 # in the "if" instead
+
+# RUNNABLE!
 def instruction_sequence_value(instrs, a, b):
     for instr in instrs:
         if a:

test/simple_source/comprehension/05_dict_comp.py

@@ -3,12 +3,24 @@
 #
 # This code is RUNNABLE!
 def x(s):
-    return {k: v
-            for (k, v) in s
-            if not k.startswith('_')
-    }
+    return {k: v for (k, v) in s if not k.startswith("_")}
+
 
 # Yes, the print() is funny. This is
 # to test though a 2-arg assert where
 # the 2nd argument is not a string.
-assert x((('_foo', None),)) == {}, print("See issue #162")
+assert x((("_foo", None),)) == {}, print("See issue #162")
+
+# From 3.7 test_dictcomps.py
+assert {k: v for k in range(10) for v in range(10) if k == v} == {
+    0: 0,
+    1: 1,
+    2: 2,
+    3: 3,
+    4: 4,
+    5: 5,
+    6: 6,
+    7: 7,
+    8: 8,
+    9: 9,
+}