Get ready for release 3.3.4

Add 3.6 STORE_ANNOTATION

.travis.yml

@@ -3,7 +3,6 @@ language: python
 python:
   - '3.5'
   - '2.7'
-  - '2.6'
   - '3.4'
   - '3.6'
 

HISTORY.md

@@ -115,7 +115,7 @@ mechanisms and addressed problems and extensions by some other means.
 Specifically, in `uncompyle`, decompilation of python bytecode 2.5 &
 2.6 is done by transforming the byte code into a pseudo-2.7 Python
 bytecode and is based on code from Eloi Vanderbeken. A bit of this
-could have bene easily added by modifying grammar rules.
+could have been easily added by modifying grammar rules.
 
 This project, `uncompyle6`, abandons that approach for various
 reasons. Having a grammar per Python version is much cleaner and it

NEWS.md

@@ -1,7 +1,78 @@
-3.3.0 2019-03-23 Holy Week
+3.3.4 2019-05-19 Fleetwood at 65
+================================
+
+Most of the work in this is release is thanks to x0ret.
+
+- Major work was done by x0ret to correct function signatures and include annotation types
+- Handle Python 3.6 STORE_ANNOTATION [#58](https://github.com/rocky/python-uncompyle6/issues/58)
+- Friendlier assembly output
+- `LOAD_CONST` replaced by `LOAD_STR` where appropriate to simplify parsing and improve clarity
+- remove unneeded parenthesis in a generator expression when it is the single argument to the function [#247](https://github.com/rocky/python-uncompyle6/issues/246)
+- Bug in noting an async function [#246](https://github.com/rocky/python-uncompyle6/issues/246)
+- Handle unicode docstrings and fix docstring bugs [#241](https://github.com/rocky/python-uncompyle6/issues/241)
+- Add short option -T as an alternate for --tree+
+- Some grammar cleanup
+
+3.3.3 2019-05-19 Henry and Lewis
+================================
+
+As before, decomplation bugs fixed. The focus has primarily been on
+Python 3.7. But with this release, releases will be put on hold,as a
+better control-flow detection is worked on . This has been needed for a
+while, and is long overdue. It will probably also take a while to get
+done as good as what we have now.
+
+However this work will be done in a new project
+[decompyle3](https://github.com/rocky/python-decompile3).  In contrast
+to _uncompyle6_ the code will be written assuming a modern Python 3,
+e.g. 3.7. It is originally intended to decompile Python version 3.7
+and greater.
+
+* A number of Python 3.7+ chained comparisons were fixed
+* Revise Python 3.6ish format string handling
+* Go over operator precedence, e.g. for AST `IfExp`
+
+Reported Bug Fixes
+------------------
+
+* [#239: 3.7 handling of 4-level attribute import](https://github.com/rocky/python-uncompyle6/issues/239),
+* [#229: Inconsistent if block in python3.6](https://github.com/rocky/python-uncompyle6/issues/229),
+* [#227: Args not appearing in decompiled src when kwargs is specified explicitly (call_ex_kw)](https://github.com/rocky/python-uncompyle6/issues/227)
+2.7 confusion around "and" versus comprehension "if"
+* [#225: 2.7 confusion around "and" vs comprehension "if"](https://github.com/rocky/python-uncompyle6/issues/225)
+
+3.3.2 2019-05-03 Better Friday
+==============================
+
+As before, lots of decomplation bugs fixed. The focus has primarily
+been on Python 3.6. We can now parse the entire 3.6.8 Python library
+and verify that without an error. The same is true for 3.5.8. A number
+of the bugs fixed though are not contained to these versions. In fact
+some span back as far as 2.x
+
+But as before, many more remain in the 3.7 and 3.8 range which will
+get addressed in future releases
+
+Pypy 3.6 support was started. Pypy 3.x detection fixed (via xdis)
+
+3.3.1 2019-04-19 Good Friday
 ==========================
 
-* First cut at Python 3.8 (many bug remain)
+Lots of decomplation bugs, especially in the 3.x series fixed. Don't worry though, many more remain.
+
+* Add annotation return values in 3.6+
+* Fix 3.6+ lambda parameter handling decompilation
+* Fix 3.7+ chained comparison decompilation
+* split out semantic-action customization into more separate files
+* Add 3.8 try/else
+* Fix 2.7 generator decompilation
+* Fix some parser failures fixes in 3.4+ using test_pyenvlib
+* Add more run tests
+
+3.3.0 2019-04-14 Holy Week
+==========================
+
+* First cut at Python 3.8 (many bugs remain)
 * Reinstate -c | --compile (compile before disassembly) option
 * The usual smattering of bug and doc fixes
 
@@ -13,23 +84,25 @@ Mostly more of the same: bug fixes and pull requests.
 Bug Fixes
 -----------
 
-* [#155: Python 3.x bytecode confusing "try/else" with "try" in a loop](https://github.com/rocky/python-uncompyle6/issues/155),
-* [#200: Python 3 bug in not detecting end bounds of an "if" ... "elif"](https://github.com/rocky/python-uncompyle6/issues/200),
-* [#208: Comma placement in 3.6 and 3.7 **kwargs](https://github.com/rocky/python-uncompyle6/issues/208),
-* [#209: Fix "if" return boundary in 3.6+](https://github.com/rocky/python-uncompyle6/issues/209),
+* [#221: Wrong grammar for nested ifelsestmt (in Python 3.7 at least)](https://github.com/rocky/python-uncompyle6/issues/221)
 * [#215: 2.7 can have two JUMP_BACKs at the end of a while loop](https://github.com/rocky/python-uncompyle6/issues/215)
+* [#209: Fix "if" return boundary in 3.6+](https://github.com/rocky/python-uncompyle6/issues/209),
+* [#208: Comma placement in 3.6 and 3.7 **kwargs](https://github.com/rocky/python-uncompyle6/issues/208),
+* [#200: Python 3 bug in not detecting end bounds of an "if" ... "elif"](https://github.com/rocky/python-uncompyle6/issues/200),
+* [#155: Python 3.x bytecode confusing "try/else" with "try" in a loop](https://github.com/rocky/python-uncompyle6/issues/155),
+
 
 Pull Requests
 ----------------
 
-* [#202: Better "assert" statement detemination in Python 2.7](https://github.com/rocky/python-uncompyle6/pull/211)
+* [#202: Better "assert" statement determination in Python 2.7](https://github.com/rocky/python-uncompyle6/pull/211)
 * [#204: Python 3.7 testing](https://github.com/rocky/python-uncompyle6/pull/204)
 * [#205: Run more f-string tests on Python 3.7](https://github.com/rocky/python-uncompyle6/pull/205)
 * [#211: support utf-8 chars in Python 3 sourcecode](https://github.com/rocky/python-uncompyle6/pull/202)
 
 
 
-3.2.5 2018-12-30 Clearout sale
+3.2.5 2018-12-30 Clear-out sale
 ======================================
 
 - 3.7.2 Remove deprecation warning on regexp string that isn't raw
@@ -94,14 +167,14 @@ Jesus on Friday's New York Times puzzle: "I'm stuck on 2A"
 - reduce 3.5, 3.6 control-flow bugs
 - reduce ambiguity in rules that lead to long (exponential?) parses
 - limit/isolate some 2.6/2.7,3.x grammar rules
-- more runtime testing of decompiled code
-- more removal of parenthesis around calls via setting precidence
+- more run-time testing of decompiled code
+- more removal of parenthesis around calls via setting precedence
 
 3.1.0 2018-03-21 Equinox
 ==============================
 
 - Add code_deparse_with_offset() fragment function.
-- Correct paramenter call fragment deparse_code()
+- Correct parameter call fragment deparse_code()
 - Lots of 3.6, 3.x, and 2.7 bug fixes
   About 5% of 3.6 fail parsing now. But
   semantics still needs much to be desired.

README.rst

@@ -12,7 +12,7 @@ Introduction
 
 *uncompyle6* translates Python bytecode back into equivalent Python
 source code. It accepts bytecodes from Python version 1.3 to version
-3.7, spanning over 22 years of Python releases. We include Dropbox's
+3.8, spanning over 24 years of Python releases. We include Dropbox's
 Python 2.5 bytecode and some PyPy bytecode.
 
 Why this?
@@ -76,7 +76,7 @@ Requirements
 The code here can be run on Python versions 2.6 or later, PyPy 3-2.4,
 or PyPy-5.0.1.  Python versions 2.4-2.7 are supported in the
 python-2.4 branch.  The bytecode files it can read have been tested on
-Python bytecodes from versions 1.4, 2.1-2.7, and 3.0-3.6 and the
+Python bytecodes from versions 1.4, 2.1-2.7, and 3.0-3.8 and the
 above-mentioned PyPy versions.
 
 Installation
@@ -93,8 +93,8 @@ This uses setup.py, so it follows the standard Python routine:
 A GNU makefile is also provided so :code:`make install` (possibly as root or
 sudo) will do the steps above.
 
-Testing
--------
+Running Tests
+-------------
 
 ::
 
@@ -133,18 +133,8 @@ You can also cross compare the results with pycdc_ . Since they work
 differently, bugs here often aren't in that, and vice versa.
 
 
-Known Bugs/Restrictions
------------------------
-
-The biggest known and possibly fixable (but hard) problem has to do
-with handling control flow. (Python has probably the most diverse and
-screwy set of compound statements I've ever seen; there
-are "else" clauses on loops and try blocks that I suspect many
-programmers don't know about.)
-
-All of the Python decompilers that I have looked at have problems
-decompiling Python's control flow. In some cases we can detect an
-erroneous decompilation and report that.
+Verification
+------------
 
 In older versions of Python it was possible to verify bytecode by
 decompiling bytecode, and then compiling using the Python interpreter
@@ -152,7 +142,7 @@ for that bytecode version. Having done this the bytecode produced
 could be compared with the original bytecode. However as Python's code
 generation got better, this is no longer feasible.
 
-There verification that we use that doesn't check bytecode for
+The verification that we use that doesn't check bytecode for
 equivalence but does check to see if the resulting decompiled source
 is a valid Python program by running the Python interpreter. Because
 the Python language has changed so much, for best results you should
@@ -167,6 +157,19 @@ And already Python has a set of programs like this: the test suite
 for the standard library that comes with Python. We have some
 code in `test/stdlib` to facilitate this kind of checking.
 
+Known Bugs/Restrictions
+-----------------------
+
+The biggest known and possibly fixable (but hard) problem has to do
+with handling control flow. (Python has probably the most diverse and
+screwy set of compound statements I've ever seen; there
+are "else" clauses on loops and try blocks that I suspect many
+programmers don't know about.)
+
+All of the Python decompilers that I have looked at have problems
+decompiling Python's control flow. In some cases we can detect an
+erroneous decompilation and report that.
+
 Python support is strongest in Python 2 for 2.7 and drops off as you
 get further away from that. Support is also probably pretty good for
 python 2.3-2.4 since a lot of the goodness of early the version of the
@@ -194,8 +197,12 @@ Between Python 3.5, 3.6 and 3.7 there have been major changes to the
 
 Currently not all Python magic numbers are supported. Specifically in
 some versions of Python, notably Python 3.6, the magic number has
-changes several times within a version. We support only the released
-magic. There are also customized Python interpreters, notably Dropbox,
+changes several times within a version.
+
+**We support only released versions, not candidate versions.** Note however
+that the magic of a released version is usually the same as the *last* candidate version prior to release.
+
+There are also customized Python interpreters, notably Dropbox,
 which use their own magic and encrypt bytcode. With the exception of
 the Dropbox's old Python 2.5 interpreter this kind of thing is not
 handled.
@@ -218,7 +225,7 @@ See Also
 * https://github.com/zrax/pycdc : purports to support all versions of Python. It is written in C++ and is most accurate for Python versions around 2.7 and 3.3 when the code was more actively developed. Accuracy for more recent versions of Python 3 and early versions of Python are especially lacking. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
-* https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situtations where `uncompyle6` results are incorrect while `uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because `uncompyle6` adheres to accuracy over idiomatic Python, `uncompyle2` can produce more natural-looking code when it is correct. Currently `uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details
+* https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where `uncompyle6` results are incorrect while `uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because `uncompyle6` adheres to accuracy over idiomatic Python, `uncompyle2` can produce more natural-looking code when it is correct. Currently `uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details
 * `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_
 * The HISTORY_ file.
 * https://github.com/rocky/python-xdis : Cross Python version disassembler
@@ -226,7 +233,7 @@ See Also
 * https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
 
 
-.. _trepan: https://pypi.python.org/pypi/trepan2
+.. _trepan: https://pypi.python.org/pypi/trepan2g
 .. _compiler: https://pypi.python.org/pypi/spark_parser
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
 .. _debuggers: https://pypi.python.org/pypi/trepan3k

__pkginfo__.py

@@ -43,6 +43,7 @@ classifiers =  ['Development Status :: 5 - Production/Stable',
                 'Programming Language :: Python :: 3.5',
                 'Programming Language :: Python :: 3.6',
                 'Programming Language :: Python :: 3.7',
+                'Programming Language :: Python :: 3.8',
                 'Topic :: Software Development :: Debuggers',
                 'Topic :: Software Development :: Libraries :: Python Modules',
                 ]
@@ -57,7 +58,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ['spark-parser >= 1.8.7, < 1.9.0',
-                      'xdis >= 4.0.0, < 4.1.0']
+                      'xdis >= 4.0.2, < 4.1.0']
 
 license            = 'GPL3'
 mailing_list       = 'python-debugger@googlegroups.com'

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.2.6 3.6.8 3.7.2 2.6.9 3.3.7 2.7.15 3.2.6 3.1.5 3.4.8'
+export PYVERSIONS='3.6.8 3.7.3 2.6.9 3.3.7 2.7.16 3.2.6 3.1.5 3.4.8'

admin-tools/pyenv-older-versions

@@ -6,4 +6,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='2.4.6 2.5.6'
+export PYVERSIONS='2.4.6 2.5.6 2.6.9'

admin-tools/setup-master.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-PYTHON_VERSION=3.6.5
+PYTHON_VERSION=3.6.8
 
 # FIXME put some of the below in a common routine
 function finish {

pytest/test_docstring.py

@@ -1,78 +0,0 @@
-import sys
-from uncompyle6 import PYTHON3
-if PYTHON3:
-    from io import StringIO
-    minint = -sys.maxsize-1
-    maxint = sys.maxsize
-else:
-    from StringIO import StringIO
-    minint = -sys.maxint-1
-    maxint = sys.maxint
-from uncompyle6.semantics.helper import print_docstring
-
-class PrintFake():
-    def __init__(self):
-        self.pending_newlines = 0
-        self.f = StringIO()
-
-    def write(self, *data):
-        if (len(data) == 0) or (len(data) == 1 and data[0] == ''):
-            return
-        out = ''.join((str(j) for j in data))
-        n = 0
-        for i in out:
-            if i == '\n':
-                n += 1
-                if n == len(out):
-                    self.pending_newlines = max(self.pending_newlines, n)
-                    return
-            elif n:
-                self.pending_newlines = max(self.pending_newlines, n)
-                out = out[n:]
-                break
-            else:
-                break
-
-        if self.pending_newlines > 0:
-            self.f.write('\n'*self.pending_newlines)
-            self.pending_newlines = 0
-
-        for i in out[::-1]:
-            if i == '\n':
-                self.pending_newlines += 1
-            else:
-                break
-
-        if self.pending_newlines:
-            out = out[:-self.pending_newlines]
-        self.f.write(out)
-    def println(self, *data):
-        if data and not(len(data) == 1 and data[0] == ''):
-            self.write(*data)
-        self.pending_newlines = max(self.pending_newlines, 1)
-        return
-    pass
-
-def test_docstring():
-
-    for doc, expect in (
-            ("Now is the time",
-             '  """Now is the time"""'),
-           ("""
-Now is the time
-""",
-            '''  """
-  Now is the time
-  """''')
-
-            # (r'''func placeholder - ' and with ("""\nstring\n  """)''',
-            #  """  r'''func placeholder - ' and with (\"\"\"\nstring\n\"\"\")'''"""),
-            # (r"""func placeholder - ' and with ('''\nstring\n''') and \"\"\"\nstring\n\"\"\" """,
-            # """  r\"\"\"func placeholder - ' and with ('''\nstring\n''') and \"\"\"\nstring\n\"\"\" \"\"\"""")
-            ):
-
-        o = PrintFake()
-        # print(doc)
-        # print(expect)
-        print_docstring(o, '  ', doc)
-        assert expect == o.f.getvalue()

pytest/test_fstring.py

@@ -134,6 +134,10 @@ if PYTHON_VERSION > 2.6:
         deparsed = deparse_code(PYTHON_VERSION, code, compile_mode='single')
         recompiled = compile(deparsed.text, '<string>', 'single')
         if recompiled != code:
+            print(recompiled)
+            print('================')
+            print(code)
+            print('----------------')
             assert 'dis(' + deparsed.text.strip('\n') + ')' == 'dis(' + expr.strip('\n') + ')'
 
 
@@ -144,7 +148,7 @@ if PYTHON_VERSION > 2.6:
         run_test(fstring)
 
 
-    @pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need Python 3.6+')
+    @pytest.mark.skipif(PYTHON_VERSION != 3.6, reason='need Python 3.6+')
     @pytest.mark.parametrize('fstring', [
         "f'{abc}{abc!s}'",
         "f'{abc}0'",

pytest/test_grammar.py

@@ -88,7 +88,7 @@ def test_grammar():
             COME_FROM_EXCEPT_CLAUSE
             COME_FROM_LOOP COME_FROM_WITH
             COME_FROM_FINALLY ELSE
-            LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP
+            LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_STR
             LAMBDA_MARKER
             RETURN_END_IF RETURN_END_IF_LAMBDA RETURN_VALUE_LAMBDA RETURN_LAST
             """.split())

pytest/test_pysource.py

@@ -127,11 +127,17 @@ def test_tables():
                             "Full entry: %s" %
                             (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
                             )
-                        assert len(tup) == 2
+                        assert 2 <= len(tup) <= 3
                         for j, x in enumerate(tup):
-                            assert isinstance(x, int), (
-                                "%s[%s][%d][%d] type '%s' is '%s should be an int but is %s. Full entry: %s" %
-                                (name, k, arg, j, typ, x, type(x), entry)
+                            if len(tup) == 3 and j == 1:
+                                assert isinstance(x, str), (
+                                    "%s[%s][%d][%d] type '%s' is '%s should be an string but is %s. Full entry: %s" %
+                                    (name, k, arg, j, typ, x, type(x), entry)
+                                    )
+                            else:
+                                assert isinstance(x, int), (
+                                    "%s[%s][%d][%d] type '%s' is '%s should be an int but is %s. Full entry: %s" %
+                                    (name, k, arg, j, typ, x, type(x), entry)
                                 )
                         pass
                     arg += 1

pytest/test_token.py

@@ -1,3 +1,4 @@
+from uncompyle6 import PYTHON_VERSION
 from uncompyle6.scanners.tok import Token
 
 def test_token():
@@ -16,7 +17,7 @@ def test_token():
     # Make sure formatting of: LOAD_CONST False. We assume False is the 0th index
     # of co_consts.
     t = Token('LOAD_CONST', offset=1, attr=False, pattr=False, has_arg=True)
-    expect = '           1  LOAD_CONST            0  False'
+    expect = '           1  LOAD_CONST               False'
     assert t.format() == expect
 
 if __name__ == '__main__':

pytest/testdata/if-2.7.right

@@ -8,5 +8,5 @@
            9  STORE_NAME            2  'b'
           12  JUMP_FORWARD          0  'to 15'
         15_0  COME_FROM            12  '12'
-          15  LOAD_CONST            0  None
+          15  LOAD_CONST               None
           18  RETURN_VALUE     

pytest/testdata/ifelse-2.7.right

@@ -4,12 +4,12 @@
    3       0  LOAD_NAME             0  'True'
            3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   4       6  LOAD_CONST            0  1
+   4       6  LOAD_CONST               1
            9  STORE_NAME            1  'b'
           12  JUMP_FORWARD          6  'to 21'
 
-   6      15  LOAD_CONST            1  2
+   6      15  LOAD_CONST               2
           18  STORE_NAME            2  'd'
         21_0  COME_FROM            12  '12'
-          21  LOAD_CONST            2  None
+          21  LOAD_CONST               None
           24  RETURN_VALUE     

test/Makefile

@@ -176,7 +176,7 @@ grammar-coverage-2.6:
 grammar-coverage-2.7:
 	-rm $(COVER_DIR)/spark-grammar-2.7.cover || true
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.7.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.7.cover $(PYTHON) test_pyenvlib.py --2.7.14 --max=600
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.7.cover $(PYTHON) test_pyenvlib.py --2.7.16 --max=600
 
 #: Get grammar coverage for Python 3.0
 grammar-coverage-3.0:
@@ -219,7 +219,12 @@ grammar-coverage-3.5:
 grammar-coverage-3.6:
 	rm $(COVER_DIR)/spark-grammar-3.6.cover || /bin/true
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.6.cover $(PYTHON) test_pythonlib.py --bytecode-3.6
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.6.cover $(PYTHON) test_pyenvlib.py --3.6.4 --max=280
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.6.cover $(PYTHON) test_pyenvlib.py --3.6.8 --max=280
+
+#: Get grammar coverage for Python 3.7
+grammar-coverage-3.7:
+	rm $(COVER_DIR)/spark-grammar-3.7.cover || /bin/true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.7.cover $(PYTHON) test_pyenvlib.py --3.7.3 --max=500
 
 #: Check deparsing Python 2.6
 check-bytecode-2.6:
@@ -263,15 +268,17 @@ check-bytecode-3.5:
 
 #: Check deparsing Python 3.6
 check-bytecode-3.6:
-	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-3.6-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify
 
 #: Check deparsing Python 3.7
 check-bytecode-3.7:
+	$(PYTHON) test_pythonlib.py --bytecode-3.7-run --verify-run
 	$(PYTHON) test_pythonlib.py --bytecode-3.7 --weak-verify
 
 #: Check deparsing Python 3.8
 check-bytecode-3.8:
+	$(PYTHON) test_pythonlib.py --bytecode-3.8-run --verify-run
 	$(PYTHON) test_pythonlib.py --bytecode-3.8 --weak-verify
 
 #: short tests for bytecodes only for this version of Python
@@ -307,6 +314,12 @@ pypy-2.7 5.0 5.3 6.0:
 pypy-3.2 2.4:
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.2 --verify
 
+#: PyPy 5.0.x with Python 3.6 ...
+7.1:
+	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
+
+
+
 clean: clean-py-dis clean-dis clean-unverified
 
 clean-dis:

test/grammar-cover/grammar.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 # Remake Python grammar statistics
 
-typeset -A ALL_VERS=([2.4]=2.4.6 [2.5]=2.5.6 [2.6]=2.6.9 [2.7]=2.7.14 [3.2]=3.2.6 [3.3]=3.3.6 [3.4]=3.4.8 [3.5]=3.5.5 [3.6]=3.6.4)
+typeset -A ALL_VERS=([2.4]=2.4.6 [2.5]=2.5.6 [2.6]=2.6.9 [2.7]=2.7.16 [3.2]=3.2.6 [3.3]=3.3.6 [3.4]=3.4.8 [3.5]=3.5.6 [3.6]=3.6.8, [3.7]=3.7.3)
 
 if (( $# == 0 )); then
     echo 1>&2 "usage: $0 two-digit-version"

test/simple_source/branching/02_ifelse_lambda.py

@@ -22,7 +22,7 @@ assert i[0]('a') == True
 assert i[0]('A') == False
 
 # Issue #170. Bug is needing an "conditional_not_lambda" grammar rule
-# in addition the the "conditional_lambda" rule
+# in addition the the "if_expr_lambda" rule
 j = lambda a: False if not a else True
 assert j(True) == True
 assert j(False) == False

test/simple_source/bug26/01_ifelse_listcomp.py

@@ -2,3 +2,10 @@
 # This is RUNNABLE!
 assert [False, True, True, True, True] == [False if not a else True for a in range(5)]
 assert [True, False, False, False, False] == [False if a else True for a in range(5)]
+
+# From bug #225
+m = ['hi', 'he', 'ih', 'who', 'ho']
+ms = {}
+for f in (f for f in m if f.startswith('h')):
+    ms[f] = 5
+assert ms == {'hi': 5, 'he': 5, 'ho': 5}

test/simple_source/bug26/03_weird26.py

@@ -8,7 +8,7 @@ list(x for x in range(10) if x % 2 if x % 3)
 
 # expresion which evaluates True unconditionally,
 # but leave dead code or junk around that we have to match on.
-# Tests "conditional_true" rule
+# Tests "if_expr_true" rule
 5 if 1 else 2
 
 0 or max(5, 3) if 0 else 3

test/simple_source/bug26/04_ifelse_parens.py

@@ -0,0 +1,9 @@
+# From 3.7.3 dataclasses.py
+# Bug was handling precedence. Need parenthesis before IfExp.
+#
+# RUNNABLE!
+def _hash_add(fields):
+    flds = [f for f in fields if (4 if f is None else f)]
+    return flds
+
+assert _hash_add([None, True, False, 3]) == [None, True, 3]

test/simple_source/bug27+/03_not_dead_code.py

@@ -1,6 +1,6 @@
 # Bug found in 2.7 test_itertools.py
-# Bug was erroneously using reduction to unconditional_true
-# A proper fix would be to use unconditional_true only when we
+# Bug was erroneously using reduction to if_expr_true
+# A proper fix would be to use if_expr_true only when we
 # can determine there is or was dead code.
 from itertools import izip_longest
 for args in [['abc', range(6)]]:

test/simple_source/bug30/04_and_del.py

@@ -0,0 +1,20 @@
+# From 2.5.6 osxemxpath.py
+# Bug is in getting "and" and "del" correct
+def normpath(comps):
+    i = 0
+    while i < len(comps):
+        if comps[i] == '.':
+            del comps[i]
+        elif comps[i] == '..' and i > 0 and comps[i-1] not in ('', '..'):
+            del comps[i-1:i+1]
+            i = i - 1
+        elif comps[i] == '' and i > 0 and comps[i-1] != '':
+            del comps[i]
+        else:
+            i = i + 1
+    return comps
+
+assert normpath(['.']) == []
+assert normpath(['a', 'b', '..']) == ['a']
+assert normpath(['a', 'b', '', 'c']) == ['a', 'b', 'c']
+assert normpath(['a', 'b', '.', '', 'c', '..']) == ['a', 'b']

test/simple_source/bug31/02_ifelse_comprehension.py

@@ -1,12 +1,18 @@
-# Python 2.7 sqlalchemy-1.013/sql/crud.py
-def _extend_values_for_multiparams(compiler, stmt, c):
-    c(
-        [
-            (
-                (compiler() if compiler()
-                    else compiler())
-                if c in stmt else compiler(),
-            )
-        ]
-        for i in enumerate(stmt)
+# Adapted from Python 2.7 sqlalchemy-1.013/sql/crud.py
+# Bug was in handling generator comprehension
+# In 2.6, 2.7 JUMP_ABSOLUTEs rather than JUMP_BACKs are generated
+
+# RUNNABLE!
+def extend(stmt, a, c, c1, c2, c3):
+    return c(
+        ([ (5 if c1 else c2)
+          if a else c3
+        ] for i in enumerate(stmt))
     )
+
+def foo(gen):
+    return list(gen)
+
+assert extend([0], 0, foo, True, 'c2', 'c3') == [['c3']]
+assert extend([0, 1], 1, foo, False, 'c2', 'c3') == [['c2'], ['c2']]
+assert extend([0, 1], False, foo, False, 'c2', 'c3') == [['c3'], ['c3']]

test/simple_source/bug31/04_def_annotate.py

@@ -1,13 +1,42 @@
-# Python 3 annotations
+# Python 3 positional, kwonly, varargs, and annotations. Ick.
 
-def foo(a, b: 'annotating b', c: int) -> float:
-    print(a + b + c)
+# RUNNABLE!
+def test1(args_1, c: int, w=4, *varargs: int, **kwargs: 'annotating kwargs') -> tuple:
+    return (args_1, c, w, kwargs)
+
+def test2(args_1, args_2, c: int, w=4, *varargs: int, **kwargs: 'annotating kwargs'):
+    return (args_1, args_2, c, w, varargs, kwargs)
+
+def test3(c: int, w=4, *varargs: int, **kwargs: 'annotating kwargs') -> float:
+    return 5.4
+
+def test4(a: float, c: int, *varargs: int, **kwargs: 'annotating kwargs') -> float:
+    return 5.4
+
+def test5(a: float, c: int = 5, *varargs: int, **kwargs: 'annotating kwargs') -> float:
+    return 5.4
+
+def test6(a: float, c: int, test=None):
+    return (a, c, test)
+
+def test7(*varargs: int, **kwargs):
+    return (varargs, kwargs)
+
+def test8(x=55, *varargs: int, **kwargs) -> list:
+    return (x, varargs, kwargs)
+
+def test9(arg_1=55, *varargs: int, y=5, **kwargs):
+    return x, varargs, int, y, kwargs
+
+def test10(args_1, b: 'annotating b', c: int) -> float:
+    return 5.4
+
+class IOBase:
+    pass
 
 # Python 3.1 _pyio.py uses the  -> "IOBase" annotation
-def open(file, mode = "r", buffering = None,
-         encoding = None, errors = None,
-         newline = None, closefd = True) -> "IOBase":
-    return text
+def o(f, mode = "r", buffering = None) -> "IOBase":
+    return (f, mode, buffering)
 
 def foo1(x: 'an argument that defaults to 5' = 5):
     print(x)
@@ -18,13 +47,77 @@ def div(a: dict(type=float, help='the dividend'),
     """Divide a by b"""
     return a / b
 
-class TestSignatureObject(unittest.TestCase):
+class TestSignatureObject1():
     def test_signature_on_wkwonly(self):
-        def test(*, a:float, b:str) -> int:
+        def test(*, a:float, b:str, c:str = 'test', **kwargs: int) -> int:
             pass
 
-class SupportsInt(_Protocol):
+class TestSignatureObject2():
+    def test_signature_on_wkwonly(self):
+        def test(*, c='test', a:float, b:str="S", **kwargs: int) -> int:
+            pass
+
+class TestSignatureObject3():
+    def test_signature_on_wkwonly(self):
+        def test(*, c='test', a:float, kwargs:str="S", **b: int) -> int:
+            pass
+
+class TestSignatureObject4():
+    def test_signature_on_wkwonly(self):
+        def test(x=55, *args, c:str='test', a:float, kwargs:str="S", **b: int) -> int:
+            pass
+
+class TestSignatureObject5():
+    def test_signature_on_wkwonly(self):
+        def test(x=55, *args: int, c='test', a:float, kwargs:str="S", **b: int) -> int:
+            pass
+
+class TestSignatureObject5():
+    def test_signature_on_wkwonly(self):
+        def test(x:int=55, *args: (int, str), c='test', a:float, kwargs:str="S", **b: int) -> int:
+            pass
+
+class TestSignatureObject7():
+    def test_signature_on_wkwonly(self):
+        def test(c='test', kwargs:str="S", **b: int) -> int:
+            pass
+
+class TestSignatureObject8():
+    def test_signature_on_wkwonly(self):
+        def test(**b: int) -> int:
+            pass
+
+class TestSignatureObject9():
+    def test_signature_on_wkwonly(self):
+        def test(a, **b: int) -> int:
+            pass
+
+class SupportsInt():
 
-    @abstractmethod
     def __int__(self) -> int:
         pass
+
+def ann1(args_1, b: 'annotating b', c: int, *varargs: str) -> float:
+    assert ann1.__annotations__['b'] == 'annotating b'
+    assert ann1.__annotations__['c'] == int
+    assert ann1.__annotations__['varargs'] == str
+    assert ann1.__annotations__['return'] == float
+
+def ann2(args_1, b: int = 5, **kwargs: float) -> float:
+    assert ann2.__annotations__['b'] == int
+    assert ann2.__annotations__['kwargs'] == float
+    assert ann2.__annotations__['return'] == float
+    assert b == 5
+
+
+assert test1(1, 5) == (1, 5, 4, {})
+assert test1(1, 5, 6, foo='bar') == (1, 5, 6, {'foo': 'bar'})
+assert test2(2, 3, 4) == (2, 3, 4, 4, (), {})
+assert test3(10, foo='bar') == 5.4
+assert test4(9.5, 7, 6, 4, bar='baz') == 5.4
+### FIXME: fill in...
+assert test6(1.2, 3) == (1.2, 3, None)
+assert test6(2.3, 4, 5) == (2.3, 4, 5)
+
+ann1(1, 'test', 5)
+ann2(1)

test/simple_source/bug33/01_triple_compare.py

@@ -1,8 +1,11 @@
 # In Python 3.3+ this uses grammar rule
 # compare_chained2 ::= expr COMPARE_OP RETURN_VALUE
+# In Python 3.6 uses this uses grammar rule
+# compare_chained2 ::= expr COMPARE_OP come_froms JUMP_FORWARD
 
 # Seen in Python 3.3 ipaddress.py
 
+# RUNNABLE!
 def _is_valid_netmask(netmask):
     return 0 <= netmask <= 10
 
@@ -10,4 +13,4 @@ def _is_valid_netmask(netmask):
 # detections
 
 # See in 2.6.9 quopri.py ishex():
-'0' <= __file__ <= '9' or 'a' <= __file__ <= 'f' or 'A' <= __file__ <= 'F'
+assert not '0' <= __file__ <= '9' or 'a' <= __file__ <= 'f' or 'A' <= __file__ <= 'F'

test/simple_source/bug33/02_pos_args.py

@@ -1,6 +1,12 @@
 # From Python 3.3.6 hmac.py
-# Problem was getting wrong placement of positional args
+# Problem was getting wrong placement of positional args.
+# In 3.6+ paramter handling changes
+
+# RUNNABLE!
+
 digest_cons = lambda d=b'': 5
 
 # Handle single kwarg
-lambda *, d=0: None
+x = lambda *, d=0: d
+assert x(d=1) == 1
+assert x() == 0

test/simple_source/bug33/08_if_else.py

@@ -0,0 +1,10 @@
+# From python 3.3.7 trace
+# Bug was not having not having semantic rule for conditional not
+
+# RUNNABLE!
+def init(modules=None):
+    mods = set() if not modules else set(modules)
+    return mods
+
+assert init() == set()
+assert init([1, 2, 3]) == set([1, 2, 3])

test/simple_source/bug35/02_async.py

@@ -0,0 +1,17 @@
+# From 3.7.3 asyncio/base_events.py
+# We had (still have) screwy logic. Python 3.5 code node detection was off too.
+
+async def create_connection(self):
+    infos = await self._ensure_resolved()
+
+    laddr_infos = await self._ensure_resolved()
+    for family in infos:
+        for laddr in laddr_infos:
+            family = 1
+        else:
+            continue
+        await self.sock_connect()
+    else:
+        raise OSError('Multiple exceptions: {}' for exc in family)
+
+    return

test/simple_source/bug35/06_while_return.py

@@ -1,28 +1,32 @@
 # From Python 3.4 asynchat.py
 # Tests presence or absense of
 # SETUP_LOOP testexpr return_stmts POP_BLOCK COME_FROM_LOOP
+# Note: that there is no JUMP_BACK because of the return_stmts.
 
-def initiate_send(self, num_sent, first):
-    while self.producer_fifo and self.connected:
+def initiate_send(a, b, c, num_sent):
+    while a and b:
         try:
-            5
-        except OSError:
-            return
+            1 / (b - 1)
+        except ZeroDivisionError:
+            return 1
 
         if num_sent:
-            if first:
-                self.producer_fifo = '6'
-            else:
-                del self.producer_fifo[0]
-        return
+            c = 2
+        return c
 
 
-# FIXME: this causes a parse error:
-# def initiate_send(self):
-#     while self.producer_fifo and self.connected:
-#         try:
-#             6
-#         except OSError:
-#             return
+def initiate_send2(a, b):
+    while a and b:
+        try:
+            1 / (b - 1)
+        except ZeroDivisionError:
+            return 1
 
-#         return
+        return 2
+
+assert initiate_send(1, 1, 2, False) == 1
+assert initiate_send(1, 2, 3, False) == 3
+assert initiate_send(1, 2, 3, True) == 2
+
+assert initiate_send2(1, 1) == 1
+assert initiate_send2(1, 2) == 2

test/simple_source/bug36/01_fstring.py

@@ -4,8 +4,8 @@
 var1 = 'x'
 var2 = 'y'
 abc  = 'def'
-assert (f'interpolate {var1} strings {var2!r} {var2!s} py36' ==
-        "interpolate x strings 'y' y py36")
+assert (f"interpolate {var1} strings {var2!r} {var2!s} 'py36" ==
+        "interpolate x strings 'y' y 'py36")
 assert 'def0' == f'{abc}0'
 assert 'defdef' == f'{abc}{abc!s}'
 
@@ -17,3 +17,52 @@ x = f"{k}={v!r}"
 y = f"functools.{x}({', '.join(v)})"
 assert x == "1=['2']"
 assert y == "functools.1=['2'](2)"
+
+# From 3.6 http/client.py
+# Bug is in handling  X
+chunk = ['a', 'b', 'c']
+chunk2 = 'd'
+chunk = f'{len(chunk):X}' + chunk2
+assert chunk == '3d'
+
+chunk = b'abc'
+chunk2 = 'd'
+chunk = f'{len(chunk):X}\r\n'.encode('ascii') + chunk \
+        + b'\r\n'
+assert chunk == b'3\r\nabc\r\n'
+
+# From 3.6.8 idlelib/pyshell.py
+# Bug was handling '''
+import os
+filename = '.'
+source = 'foo'
+source = (f"__file__ = r'''{os.path.abspath(filename)}'''\n"
+          + source + "\ndel __file__")
+
+# Note how { and } are *not* escaped here
+f = 'one'
+name = 'two'
+assert(f"{f}{'{{name}}'} {f}{'{name}'}") == 'one{{name}} one{name}'
+
+# From 3.7.3 dataclasses.py
+log_rounds  = 5
+assert "05$" == f'{log_rounds:02d}$'
+
+
+def testit(a, b, l):
+    # print(l)
+    return l
+
+# The call below shows the need for BUILD_STRING to count expr arguments.
+# Also note that we use {{ }} to escape braces in contrast to the example
+# above.
+def _repr_fn(fields):
+    return testit('__repr__',
+                  ('self',),
+                  ['return xx + f"(' +
+                   ', '.join([f"{f}={{self.{f}!r}}"
+                              for f in fields]) +
+                   ')"'])
+
+fields = ['a', 'b', 'c']
+assert _repr_fn(fields) == ['return xx + f"(a={self.a!r}, b={self.b!r}, c={self.c!r})"']

test/simple_source/bug36/02_call_ex_kw.py

@@ -0,0 +1,47 @@
+# From #227
+# Bug was not handling call_ex_kw correctly
+# This appears in
+#   showparams(c, test="A", **extra_args)
+# below
+
+def showparams(c, test, **extra_args):
+    return {'c': c, **extra_args, 'test': test}
+
+def f(c, **extra_args):
+    return showparams(c, test="A", **extra_args)
+
+def f1(c, d, **extra_args):
+    return showparams(c, test="B", **extra_args)
+
+def f2(**extra_args):
+    return showparams(1, test="C", **extra_args)
+
+def f3(c, *args, **extra_args):
+    return showparams(c, *args, **extra_args)
+
+assert f(1, a=2, b=3) == {'c': 1, 'a': 2, 'b': 3, 'test': 'A'}
+
+a = {'param1': 2}
+assert f1('2', '{\'test\': "4"}', test2='a', **a) \
+    == {'c': '2', 'test2': 'a', 'param1': 2, 'test': 'B'}
+assert f1(2, '"3"', test2='a', **a) \
+    == {'c': 2, 'test2': 'a', 'param1': 2, 'test': 'B'}
+assert f1(False, '"3"', test2='a', **a) \
+    == {'c': False, 'test2': 'a', 'param1': 2, 'test': 'B'}
+assert f(2, test2='A', **a) \
+    == {'c': 2, 'test2': 'A', 'param1': 2, 'test': 'A'}
+assert f(str(2) + str(1), test2='a', **a) \
+    == {'c': '21', 'test2': 'a', 'param1': 2, 'test': 'A'}
+assert f1((a.get('a'), a.get('b')), a, test3='A', **a) \
+    == {'c': (None, None), 'test3': 'A', 'param1': 2, 'test': 'B'}
+
+b = {'b1': 1, 'b2': 2}
+assert f2(**a, **b) == \
+    {'c': 1, 'param1': 2, 'b1': 1, 'b2': 2, 'test': 'C'}
+
+c = (2,)
+d = (2, 3)
+assert f(2, **a) == {'c': 2, 'param1': 2, 'test': 'A'}
+assert f3(2, *c, **a) == {'c': 2, 'param1': 2, 'test': 2}
+assert f3(*d, **a) == {'c': 2, 'param1': 2, 'test': 3}
+

test/simple_source/bug36/03_build_tuple_call.py

@@ -0,0 +1,4 @@
+# From Python 3.6.5 email/message.py
+# Bug is in handling 'related' parameter
+def add_related(self, *args, **kw):
+    self._add_multipart('related', *args, _disp='inline', **kw)

test/simple_source/bug36/05_ann_mopdule2.py

@@ -0,0 +1,37 @@
+# This is from Python 3.6's test directory.
+"""
+Some correct syntax for variable annotation here.
+More examples are in test_grammar and test_parser.
+"""
+
+from typing import no_type_check, ClassVar
+
+i: int = 1
+j: int
+x: float = i/10
+
+def f():
+    class C: ...
+    return C()
+
+f().new_attr: object = object()
+
+class C:
+    def __init__(self, x: int) -> None:
+        self.x = x
+
+c = C(5)
+c.new_attr: int = 10
+
+__annotations__ = {}
+
+
+@no_type_check
+class NTC:
+    def meth(self, param: complex) -> None:
+        ...
+
+class CV:
+   var: ClassVar['CV']
+
+CV.var = CV()

test/simple_source/bug36/05_try_whiletrue.py

@@ -0,0 +1,19 @@
+# From 3.6 _collections.abc.py
+# Bug was try/execpt parsing detection since 3.6 removes
+# a JUMP_FORWARD from earlier 3.xs.
+# This could also get confused with try/else.
+
+# RUNNABLE!
+def iter(self):
+    i = 0
+    try:
+        while True:
+            v = self[i]
+            yield v
+            i += 1
+    except IndexError:
+        return
+
+
+A = [10, 20, 30]
+assert list(iter(A)) == A

test/simple_source/bug36/08_comp_gen_for.py

@@ -0,0 +1,19 @@
+# Bug in 3.3 weakset
+# Bug was not having a rule for 3.x "comp_for"
+
+# RUNNABLE!
+class WeakSet:
+    def __init__(self, data=None):
+        self.data = set(data)
+
+    def __iter__(self):
+        for item in self.data:
+            if item is not None:
+                yield item
+
+    def union(self, other):
+        return self.__class__(e for s in (self, other) for e in s)
+
+a = WeakSet([1, 2, 3])
+b = WeakSet([1, 3, 5])
+assert list(a.union(b)) == [1, 2, 3, 5]

test/simple_source/bug37/01_and_not_else.py

@@ -0,0 +1,16 @@
+# From 3.7.3 base64.py
+# Bug was handling "and not" in an
+# if/else in the presence of better Python bytecode generatation
+
+# RUNNABLE!
+def foo(foldnuls, word):
+    x = 5 if foldnuls and not word else 6
+    return x
+
+for expect, foldnuls, word in (
+        (6, True, True),
+        (5, True, False),
+        (6, False, True),
+        (6, False, False)
+        ):
+   assert foo(foldnuls, word) == expect