Get ready for release 3.7.4

build_ast() added a code argument
n_dict(): was out of sync with corresponding pysource routine
pysource.py: small doc typo

.github/ISSUE_TEMPLATE/bug-report.md

@@ -4,28 +4,40 @@ about: Tell us about uncompyle6 bugs
 
 ---
 
-<!-- __Note:__ Have you read https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md ?
+<!-- __Note:__ Bugs are not for asking questions about a problem you
+are trying to solve that involve the use of uncompyle6 along the way,
+although I may be more tolerent of this if you sponsor the project.
+
+Also, the unless you are a sponsor of the project, it may take a
+while, maybe a week or so, before the bug report is noticed, let alone
+acted upon.
+
+To set expectations, some legitimate bugs can take years
+to fix, but they eventually do get fixed. Funding the project was
+added to address the problem that there are lots of people seeking
+help and reporting bugs, but few people who are willing or capable of
+providing help or fixing bugs.
+
+Finally, have you read https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md
+?
+
 
 Please remove any of the optional sections if they are not applicable.
 
-Prerequisites
+Prerequisites/Caveats
 
 * Make sure the bytecode you have can be disassembled with a
   disassembler and produces valid results.
 * Don't put bytecode and corresponding source code on any service that
   requires registration to download.
-* When you open a bug report there is no privacy. If the legitimacy of
-  the activity is deemed suspicous, I may flag it as suspicious,
+* When you open a bug report there is no privacy. If you need privacy, then
+  contact me by email and explain who you are and the need for privacy.
+  But be mindful that you may be asked to sponsor the project for the
+  personal and private help that you are requesting.
+* If the legitimacy of the activity is deemed suspicous, I may flag it as suspicious,
   making the issue even more easy to detect.
 
-Bug reports that violate a prerequisite may be discarded.
-
-Note that there are way more bug-fix requestors than there are bug
-fixers. If you want you need more immediate, confidential or urgent
-assistance
-
-http://www.crazy-compilers.com/decompyle/ offers a byte-code
-decompiler service for versions of Python up to 2.6.
+Bug reports that violate the above may be discarded.
 
 -->
 

HOW-TO-REPORT-A-BUG.md

@@ -30,31 +30,9 @@ files that I know of that will cause problems. See, for example, the
 list in
 [`test/stdlib/runtests.sh`](https://github.com/rocky/python-uncompyle6/blob/master/test/stdlib/runtests.sh).
 
-But I understand: you would the bugs _you_ encounter addressed before
-all the other known bugs.
+There are far more bug reporters than there are bug fixers.
 
-From my standpoint, the good thing about the bugs listed in
-`runtests.sh` is that each test case is small and isolated to a single
-kind of problem. And I'll tend to fix easier, more isolated cases than
-generic "something's wrong" kinds of bugs where I'd have to do a bit
-of work to figure out what's up, if not use some sort of mind reading,
-make some guesses, and perform some experiments to see if the guesses
-are correct. I can't read minds, nor am I into guessing games; I'd
-rather devote the effort spent instead towards fixing bugs that are
-precisely defined.
-
-And it often turns out that by just fixing the well-defined and
-prescribed cases, the ill-defined amorphous cases as well will get
-handled as well.
-
-In sum, you may need to do some work to have the bug you have found
-handled before the hundreds of other bugs, and other things I could be
-doing.
-
-No one is getting paid to work to work on this project, let alone the
-bugs you may have an interest in. If you require decompiling bytecode
-immediately, consider using a decompilation service, listed further
-down in this document.
+Unless you are a sponsor of this project, it may take a while, maybe a week or so, before the bug report is noticed, let alone acted upon. Things eventually get fixed, but it may take years.
 
 # Is it really a bug?
 
@@ -176,11 +154,7 @@ provide the input command and the output from that, please give:
 
 ## But I don't *have* the source code!
 
-Sure, I get it. No problem. There is Python assembly code on parse
-errors, so simply by hand decompile that. To get a full disassembly,
-use `pydisasm` from the [xdis](https://pypi.python.org/pypi/xdis)
-package. Opcodes are described in the documentation for
-the [dis](https://docs.python.org/3.6/library/dis.html) module.
+There is Python assembly code on parse errors, so simply by hand decompile that. To get a full disassembly, use `pydisasm` from the [xdis](https://pypi.python.org/pypi/xdis) package. Opcodes are described in the documentation for the[dis](https://docs.python.org/3.6/library/dis.html) module.
 
 ### But I don't *have* the source code and am incapable of figuring how to do a hand disassembly!
 
@@ -189,7 +163,7 @@ disassemble Python bytecode. And as Richard Feynman once said, "What
 one fool can learn, so can another."
 
 If this is too difficult, or too time consuming, or not of interest to
-you, then perhaps what require is a decompilation service. [Crazy
+you, then you might consider sponsoring the project. [Crazy
 Compilers](http://www.crazy-compilers.com/decompyle/) offers a
 byte-code decompiler service for versions of Python up to 2.6. (If
 there are others around let me know and I'll list them here.)
@@ -217,10 +191,7 @@ likely the problem will be fixed and fixed sooner.
 # Karma
 
 I realize that following the instructions given herein puts a bit of
-burden on the bug reporter. In my opinion, this is justified as
-attempts to balance somewhat the burden and effort needed to fix the
-bug and the attempts to balance number of would-be bug reporters with
-the number of bug fixers. Better bug reporters are more likely to move
+burden on the bug reporter. This is justified as attempts to balance somewhat the burden and effort needed to fix the bug and the attempts to balance number of would-be bug reporters with the number of bug fixers. Better bug reporters are more likely to move
 in the category of bug fixers.
 
 The barrier to reporting a big is pretty small: all you really need is
@@ -228,22 +199,14 @@ a github account, and the ability to type something after clicking
 some buttons. So the reality is that many people just don't bother to
 read these instructions, let alone follow it to any simulacrum.
 
-And the reality is also that bugs sometimes get fixed even though
-these instructions are not followed.
+That said, bugs sometimes get fixed even though these instructions are not followed.
 
-So one factors I may take into consideration is the bug reporter's karma.
+I may take into consideration is the bug reporter's karma.
+
+* Have you demonstrably contributed to open source? I may look at your  github profile to see what contributions you have made, how popular those contributions are, or how popular you are.
+* How appreciative are you? Have you starred this project that you are   seeking help from? Have you starred _any_ github project? And the above  two kind of feed into ...
+* Attitude. Some people feel that they are doing me and the world a great favor by just pointing out that there is a problem whose solution would greatly benefit them. (This might account partially those that have this attitude often don't read or follow instructions such as those given here.)
 
-* Have you demonstrably contributed to open source? I may look at your
-  github profile to see what contributions you have made, how popular
-  those contributions are, or how popular you are.
-* How appreciative are you? Have you starred this project that you are
-  seeking help from? Have you starred _any_ github project? And the above
-  two kind of feed into ...
-* Attitude. Some people feel that they are doing me and the world a
-  great favor by just pointing out that there is a problem whose solution
-  would greatly benefit them. Perhaps this is why they feel that
-  instructions are not to be followed by them, nor any need for
-  showing evidence gratitude when help is offered them.
 
 # Confidentiality of Bug Reports
 
@@ -255,6 +218,8 @@ remains would not be an issue.
 However feel free to remove any comments, and modify variable names
 or constants in the source code.
 
+If there is some legitimate reason to keep confidentiality, you can contact me by email to explain the extenuating circumstances. However I tend to discard without reading anonymous email.
+
 # Ethics
 
 I do not condone using this program for unethical or illegal purposes.

NEWS.md

@@ -1,3 +1,26 @@
+3.7.4: 2020-8-05
+================
+
+* Fragment parsing was borked. This means deparsing in trepan2/trepan3k was broken
+* 3.7+: narrow precedence for call tatement
+* del_stmt -> delete to better match Python AST
+* 3.8+ Add another `forelsestmt` (found only in a loop)
+* 3.8+ Add precedence on walrus operator
+* More files blackened
+* bump min xdis version
+
+3.7.3: 2020-7-25
+================
+
+Mostly small miscellaneous bug fixes
+
+* `__doc__ = DocDescr()` from `test_descr.py` was getting confused as a docstring.
+* detect 2.7 exchandler range better
+* Add for .. else reduction checks on 2.6 and before
+* Add reduce check for 2.7 augmented assign
+* Add VERSION in a pydoc-friendly way
+
+
 3.7.2: 2020-6-27
 ================
 

README.rst

@@ -68,10 +68,9 @@ are syntactically correct by running the Python interpreter for that
 bytecode version.  Finally, in cases where the program has a test for
 itself, we can run the check on the decompiled code.
 
-We are serious about testing, and use automated processes to find
-bugs. In the issue trackers for other decompilers, you will find a
-number of bugs we've found along the way. Very few to none of them are
-fixed in the other decompilers.
+We use an automated processes to find bugs. In the issue trackers for
+other decompilers, you will find a number of bugs we've found along
+the way. Very few to none of them are fixed in the other decompilers.
 
 Requirements
 ------------
@@ -172,15 +171,11 @@ All of the Python decompilers that I have looked at have problems
 decompiling Python's control flow. In some cases we can detect an
 erroneous decompilation and report that.
 
-Python support is strongest in Python 2 for 2.7 and drops off as you
-get further away from that. Support is also probably pretty good for
-python 2.3-2.4 since a lot of the goodness of early the version of the
-decompiler from that era has been preserved (and Python compilation in
-that era was minimal)
+Python support is pretty good for Python 2
 
-There is some work to do on the lower end Python versions which is
-more difficult for us to handle since we don't have a Python
-interpreter for versions 1.6, and 2.0.
+On the lower end of Python versions, decompilation seems pretty good although
+we don't have any automated testing in place for Python's distributed tests.
+Also, we don't have a Python interpreter for versions 1.6, and 2.0.
 
 In the Python 3 series, Python support is is strongest around 3.4 or
 3.3 and drops off as you move further away from those versions. Python
@@ -195,7 +190,7 @@ added. So in sum handling control flow by ad hoc means as is currently
 done is worse.
 
 Between Python 3.5, 3.6, 3.7 there have been major changes to the
-:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.  Python
+:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.
 
 Python 3.8 removes :code:`SETUP_LOOP`, :code:`SETUP_EXCEPT`,
 :code:`BREAK_LOOP`, and :code:`CONTINUE_LOOP`, instructions which may
@@ -215,17 +210,43 @@ which use their own magic and encrypt bytecode. With the exception of
 the Dropbox's old Python 2.5 interpreter this kind of thing is not
 handled.
 
-We also don't handle PJOrion_ obfuscated code. For that try: PJOrion
-Deobfuscator_ to unscramble the bytecode to get valid bytecode before
-trying this tool. This program can't decompile Microsoft Windows EXE
-files created by Py2EXE_, although we can probably decompile the code
-after you extract the bytecode properly. For situations like this, you
-might want to consider a decompilation service like `Crazy Compilers
-<http://www.crazy-compilers.com/decompyle/>`_.  Handling
-pathologically long lists of expressions or statements is slow.
+We also don't handle PJOrion_ or otherwise obfuscated code. For
+PJOrion try: PJOrion Deobfuscator_ to unscramble the bytecode to get
+valid bytecode before trying this tool. This program can't decompile
+Microsoft Windows EXE files created by Py2EXE_, although we can
+probably decompile the code after you extract the bytecode
+properly. Handling pathologically long lists of expressions or
+statements is slow. We don't handle Cython_ or MicroPython which don't
+use bytecode.
 
+There are numerous bugs in decompilation. And that's true for every
+other CPython decompiler I have encountered, even the ones that
+claimed to be "perfect" on some particular version like 2.4.
 
-There is lots to do, so please dig in and help.
+As Python progresses decompilation also gets harder because the
+compilation is more sophisticated and the language itself is more
+sophisticated. I suspect that attempts there will be fewer ad-hoc
+attempts like unpyc37_ (which is based on a 3.3 decompiler) simply
+because it is harder to do so. The good news, at least from my
+standpoint, is that I think I understand what's needed to address the
+problems in a more robust way. But right now until such time as
+project is better funded, I do not intend to make any serious effort
+to support Python versions 3.8 or 3.9, including bugs that might come
+in. I imagine at some point I may be interested in it.
+
+You can easily find bugs by running the tests against the standard
+test suite that Python uses to check itself. At any given time, there are
+dozens of known problems that are pretty well isolated and that could
+be solved if one were to put in the time to do so. The problem is that
+there aren't that many people who have been working on bug fixing.
+
+Some of the bugs in 3.7 and 3.8 are simply a matter of back-porting
+the fixes in decmopyle3.
+
+You may run across a bug, that you want to report. Please do so. But
+be aware that it might not get my attention for a while. If you
+sponsor or support the project in some way, I'll prioritize your
+issues above the queue of other things I might be doing instead.
 
 See Also
 --------
@@ -242,6 +263,7 @@ See Also
 * https://github.com/zrax/pycdc : The README for this C++ code says it aims to support all versions of Python. It is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
 
 
+.. _Cython: https://en.wikipedia.org/wiki/Cython
 .. _trepan: https://pypi.python.org/pypi/trepan2g
 .. _compiler: https://pypi.python.org/pypi/spark_parser
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md

__pkginfo__.py

@@ -33,65 +33,71 @@
 # 3.4            | pip     | 19.1.1       |
 
 # Things that change more often go here.
-copyright   = """
+copyright = """
 Copyright (C) 2015-2020 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
-classifiers =  ["Development Status :: 5 - Production/Stable",
-                "Intended Audience :: Developers",
-                "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
-                "Operating System :: OS Independent",
-                "Programming Language :: Python",
-                "Programming Language :: Python :: 2.4",
-                "Programming Language :: Python :: 2.5",
-                "Programming Language :: Python :: 2.6",
-                "Programming Language :: Python :: 2.7",
-                "Programming Language :: Python :: 3.0",
-                "Programming Language :: Python :: 3.1",
-                "Programming Language :: Python :: 3.2",
-                "Programming Language :: Python :: 3.3",
-                "Programming Language :: Python :: 3.4",
-                "Programming Language :: Python :: 3.5",
-                "Programming Language :: Python :: 3.6",
-                "Programming Language :: Python :: 3.7",
-                "Programming Language :: Python :: 3.8",
-                "Topic :: Software Development :: Debuggers",
-                "Topic :: Software Development :: Libraries :: Python Modules",
-                ]
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 2.4",
+    "Programming Language :: Python :: 2.5",
+    "Programming Language :: Python :: 2.6",
+    "Programming Language :: Python :: 2.7",
+    "Programming Language :: Python :: 3.0",
+    "Programming Language :: Python :: 3.1",
+    "Programming Language :: Python :: 3.2",
+    "Programming Language :: Python :: 3.3",
+    "Programming Language :: Python :: 3.4",
+    "Programming Language :: Python :: 3.5",
+    "Programming Language :: Python :: 3.6",
+    "Programming Language :: Python :: 3.7",
+    "Programming Language :: Python :: 3.8",
+    "Topic :: Software Development :: Debuggers",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
 
 # The rest in alphabetic order
-author             = "Rocky Bernstein, Hartmut Goebel, John Aycock, and others"
-author_email       = "rb@dustyfeet.com"
-entry_points       = {
+author = "Rocky Bernstein, Hartmut Goebel, John Aycock, and others"
+author_email = "rb@dustyfeet.com"
+entry_points = {
     "console_scripts": [
         "uncompyle6=uncompyle6.bin.uncompile:main_bin",
         "pydisassemble=uncompyle6.bin.pydisassemble:main",
-    ]}
-ftp_url            = None
-install_requires   = ["spark-parser >= 1.8.9, < 1.9.0",
-                      "xdis >= 4.7.0, <5.1.0"]
+    ]
+}
+ftp_url = None
+install_requires = ["spark-parser >= 1.8.9, < 1.9.0", "xdis >= 5.0.4, <5.1.0"]
 
-license            = "GPL3"
-mailing_list       = "python-debugger@googlegroups.com"
-modname            = "uncompyle6"
-py_modules         = None
-short_desc         = "Python cross-version byte-code decompiler"
-web                = "https://github.com/rocky/python-uncompyle6/"
+license = "GPL3"
+mailing_list = "python-debugger@googlegroups.com"
+modname = "uncompyle6"
+py_modules = None
+short_desc = "Python cross-version byte-code decompiler"
+web = "https://github.com/rocky/python-uncompyle6/"
 
 # tracebacks in zip files are funky and not debuggable
 zip_safe = True
 
 
 import os.path
+
+
 def get_srcdir():
     filename = os.path.normcase(os.path.dirname(os.path.abspath(__file__)))
     return os.path.realpath(filename)
 
+
 srcdir = get_srcdir()
 
+
 def read(*rnames):
     return open(os.path.join(srcdir, *rnames)).read()
 
+
 # Get info from files; set: long_description and VERSION
-long_description   = ( read("README.rst") + "\n" )
+long_description = read("README.rst") + "\n"
 exec(read("uncompyle6/version.py"))

admin-tools/how-to-make-a-release.md

@@ -39,7 +39,7 @@
 
 # Make sure pyenv is running and check newer versions
 
-    $ pyenv local && source admin-tools/check-newer-versions.sh
+    $ admin-tools/check-newer-versions.sh
 
 # Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.
 
@@ -50,20 +50,21 @@
 
 # Check against older versions
 
-    $ source admin-tools/check-older-versions.sh
+    $ admin-tools/check-older-versions.sh
 
 # Make packages and tag
 
     $ . ./admin-tools/make-dist-older.sh
-	$ pyenv local 3.8.3
+	$ pyenv local 3.8.5
 	$ twine check dist/uncompyle6-$VERSION*
-    $ . ./admin-tools/make-dist-newer.sh
+    $ git tag release-python-2.4-$VERSION
+    $ ./admin-tools/make-dist-newer.sh
 	$ twine check dist/uncompyle6-$VERSION*
 
 # Check package on github
 
-	$ mkdir /tmp/gittest; pushd /tmp/gittest
-	$ pyenv local 3.7.5
+	$ [[ ! -d /tmp/gittest ]] && mkdir /tmp/gittest; pushd /tmp/gittest
+	$ pyenv local 3.8.3
 	$ pip install -e git://github.com/rocky/python-uncompyle6.git#egg=uncompyle6
 	$ uncompyle6 --help
 	$ pip uninstall uncompyle6
@@ -93,3 +94,7 @@ Todo: turn this into a script in `admin-tools`
 
     $ git push --tags
     $ git pull --tags
+
+# Move dist files to uploaded
+
+	$ mv -v dist/uncompyle6-${VERSION}* dist/uploaded

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.18 3.2.6 3.1.5 3.4.10 3.7.7 3.8.3'
+export PYVERSIONS='3.5.9 3.6.12 2.6.9 3.3.7 2.7.18 3.2.6 3.1.5 3.4.10 3.7.9 3.8.5'

test/add-test.py

@@ -3,14 +3,22 @@
 """
 import os, sys, py_compile
 
-assert len(sys.argv) >= 2
+assert (2 <= len(sys.argv) <= 4)
 version = sys.version[0:3]
+vers = sys.version_info[:2]
 if sys.argv[1] in ("--run", "-r"):
     suffix = "_run"
     py_source = sys.argv[2:]
+    i = 2
 else:
     suffix = ""
     py_source = sys.argv[1:]
+    i = 1
+try:
+    optimize = int(sys.argv[-1])
+    py_source = sys.argv[i:-1]
+except:
+    optimize = 2
 
 for path in py_source:
     short = os.path.basename(path)
@@ -19,7 +27,10 @@ for path in py_source:
     else:
         cfile = "bytecode_%s%s/%s" % (version, suffix, short) + "c"
     print("byte-compiling %s to %s" % (path, cfile))
-    optimize = 2
-    py_compile.compile(path, cfile, optimize=optimize)
-    if isinstance(version, str) or version >= (2, 6, 0):
+    optimize = optimize
+    if vers > (3, 1):
+        py_compile.compile(path, cfile, optimize=optimize)
+    else:
+        py_compile.compile(path, cfile)
+    if vers >= (2, 6):
         os.system("../bin/uncompyle6 -a -T %s" % cfile)

test/run-and-email.sh

@@ -51,7 +51,7 @@ for VERSION in $PYVERSIONS ; do
     LOGFILE=/tmp/${MAIN}-$VERSION-$$.log
 
     case "$VERSION" in
-	3.7.7 | 3.8.3 | 3.1.5 | 3.0.1 )
+	3.7.8 | 3.8.5 | 3.1.5 | 3.0.1 )
 	    continue
 	    ;;
 	3.5.9 )
@@ -66,7 +66,7 @@ for VERSION in $PYVERSIONS ; do
 	3.4.10 )
 	    MAX_TESTS=800
 	    ;;
-	3.6.10 )
+	3.6.11 )
 	    # MAX_TESTS=1300  # about 2139 exist
 	    # fails on _pyio.cpython-36.opt-1.pyc
 	    MAX_TESTS=34

test/simple_source/bug14/01_for_else_try_else.py

@@ -0,0 +1,17 @@
+# Adapted from 1.4 anydbm
+"""This program is self-checking!"""
+def scan(items):
+    for i in items:
+            try:
+                    5 / i
+            except:
+                    continue
+            else:
+                    break
+    else:
+            return 2
+    return i
+
+assert scan((0, 1)) == 1
+assert scan((0, 0)) == 2
+assert scan((3, 2, 1)) == 3

test/simple_source/bug27+/01_module_doc.py

@@ -0,0 +1,9 @@
+# From 2.7.17 test_bdb.py
+# The problem was detecting a docstring at the begining of the module
+# It must be detected and change'd or else the "from __future__" below
+# is invalid.
+# Note that this has to be compiled with optimation < 2 or else optimization
+# will remove the docstring
+"""Rational, infinite-precision, real numbers."""
+
+from __future__ import division

test/simple_source/bug27+/03_doc_assign.py

@@ -0,0 +1,27 @@
+# From 2.7 test_descr.py
+# Testing __doc__ descriptor...
+# The bug in decompilation was erroneously matching
+# __doc__ = as a docstring
+
+"""This program is self-checking!"""
+
+def test_doc_descriptor():
+    # Testing __doc__ descriptor...
+    # Python SF bug 542984
+    class DocDescr(object):
+        def __get__(self, object, otype):
+            if object:
+                object = object.__class__.__name__ + ' instance'
+            if otype:
+                otype = otype.__name__
+            return 'object=%s; type=%s' % (object, otype)
+    class OldClass:
+        __doc__ = DocDescr()
+    class NewClass(object):
+        __doc__ = DocDescr()
+    assert OldClass.__doc__ == 'object=None; type=OldClass'
+    assert OldClass().__doc__ == 'object=OldClass instance; type=OldClass'
+    assert NewClass.__doc__ == 'object=None; type=NewClass'
+    assert NewClass().__doc__ == 'object=NewClass instance; type=NewClass'
+
+test_doc_descriptor()

test/simple_source/stmts/10_del.py

@@ -8,7 +8,7 @@ b = [4, 5, 6]
 del b[1]
 del b[:]
 
-# del_stmt ::= expr expr DELETE_SLICE+1
+# delete ::= expr expr DELETE_SLICE+1
 l = [None] * 10
 del l[-2:]
 

test/stdlib/2.4-exclude.sh

@@ -45,7 +45,6 @@ SKIP_TESTS=(
     [test_grammar.py]=1    # Too many stmts. Handle large stmts
     [test_grp.py]=1      # Long test - might work Control flow?
     [test_pep247.py]=1 # Long test - might work? Control flow?
-    [test_pwd.py]=1 # Long test - might work? Control flow?
     [test_socketserver.py]=1 # -- test takes too long to run: 40 seconds
     [test_threading.py]=1 # test takes too long to run: 11 seconds
     [test_thread.py]=1 # test takes too long to run: 36 seconds

test/stdlib/2.5-exclude.sh

@@ -56,10 +56,8 @@ SKIP_TESTS=(
     [test_pdb.py]=1        # Line-number specific
     [test_pep277.py]=1    # it fails on its own
     [test_plistlib.py]=1 # it fails on its own
-    [test_pwd.py]=1 # Long test - might work? Control flow?
     [test_pyclbr.py]=1 # Investigate
     [test_rgbimg.py]=1 #  it fails on its own
-    [test_queue.py]=1 # Control flow?
     [test_scriptpackages.py]=1 # it fails on its own
     [test_socketserver.py]=1 # Too long to run - 42 seconds
     [test_sqlite.py]=1 # it fails on its own

test/stdlib/2.6-exclude.sh

@@ -7,7 +7,6 @@ SKIP_TESTS=(
     #   assert 0  # shouldn't reach here.
     [test_shutil.py]=1
 
-
     [test___all__.py]=1  # it fails on its own
     [test___all__.py]=1 # it fails on its own
     [test_aepack.py]=1 # Fails on its own
@@ -61,7 +60,6 @@ SKIP_TESTS=(
 
     [test_pep277.py]=1 # it fails on its own
     [test_pyclbr.py]=1 # Investigate
-    [test_pwd.py]=1 # Long test - might work? Control flow?
     [test_py3kwarn.py]=1 # it fails on its own
 
     [test_scriptpackages.py]=1 # it fails on its own

test/stdlib/2.7-exclude.sh

@@ -19,7 +19,6 @@ SKIP_TESTS=(
     [test_modulefinder.py]=1 # FIX
     [test_multiprocessing.py]=1 # On uncompyle2, takes 24 secs
     [test_poll.py]=1  # test takes too long to run: 11 seconds
-    [test_pwd.py]=1     # Takes too long
     [test_regrtest.py]=1 #
     [test_runpy.py]=1   # Long and fails on its own
     [test_select.py]=1  # Runs okay but takes 11 seconds

test/stdlib/3.8-exclude.sh

@@ -8,6 +8,37 @@ SKIP_TESTS=(
     [test_urllib2.py]=1 # FIXME: works on uncompyle6?
     [test_zipimport.py]=1 # FIXME: works on uncompyle6
 
+    # From decompyle3 excludes
+    # Very Simple example. Compare with 3.7 Need 3.8 parse rules for exception handling return
+    #    for proto in p:
+    #    try:
+    #        drop = 5
+    #    except StopIteration:
+    #        continue
+
+    [test_dict.py]=1 #
+
+    # Simple example. Compare with 3.7 Need 3.8 parse rules for exception handling return
+    #    try:
+    #    return 5
+    # except KeyError:
+    #    return res
+    # except TypeError:
+    #    return 10
+
+    # These and the above may be due to new code generation or tests
+    # between 3.8.3 and 3.8.5 ?
+    [test_decorators.py]=1 #
+
+    [test_dtrace.py]=1 #
+    [test_exceptions.py]=1 #
+    [test_ftplib.py]=1 #
+    [test_gc.py]=1 #
+    [test_gzip.py]=1 #
+    [test_hashlib.py]=1 #
+    [test_iter.py]=1 #
+    [test_itertools.py]=1 #
+
     [test___all__.py]=1 # it fails on its own
     [test_argparse.py]=1 #- it fails on its own
     [test_array.py]=1 #- parse error

test/stdlib/run-and-email.sh

@@ -52,7 +52,7 @@ for VERSION in $PYVERSIONS ; do
     LOGFILE=/tmp/runtests-$VERSION-$$.log
 
     case "$VERSION" in
-	3.0.1 | 3.1.5 | 3.2.6 | 3.8.1 )
+	3.0.1 | 3.1.5 | 3.2.6 | 3.8.5 )
 	    continue
 	    ;;
     esac

tox.ini

@@ -3,7 +3,7 @@
 [flake8]
 exclude = .tox,./build,./trepan/processor/command/tmp
 filename = *.py
-ignore = C901,E113,E121,E122,E123,E124,E125,E126,E127,E128,E129,E201,E202,E203,E221,E222,E225,E226,E241,E242,E251,E261,E271,E272,E302,E401,E402,E501,F401,E701,E702
+ignore = C901,E113,E121,E122,E123,E124,E125,E126,E127,E128,E129,E201,E202,E203,E221,E222,E225,E226,E241,E242,E251,E261,E271,E272,E302,E401,E402,E501,F401,E701,E702,W503
 
 [tox]
 envlist = py27, py34, pypy

uncompyle6/__init__.py

@@ -28,20 +28,25 @@
 
 import sys
 
-__docformat__ = 'restructuredtext'
+__docformat__ = "restructuredtext"
 
-PYTHON3 = (sys.version_info >= (3, 0))
+from uncompyle6.version import VERSION
+
+# This ensures VERSION will appear in pydoc
+__version__  = VERSION
+
+PYTHON3 = sys.version_info >= (3, 0)
 
 # We do this crazy way to support Python 2.6 which
 # doesn't support version_major, and has a bug in
 # floating point so we can't divide 26 by 10 and get
 # 2.6
 PYTHON_VERSION = sys.version_info[0] + (sys.version_info[1] / 10.0)
-PYTHON_VERSION_STR  = "%s.%s" % (sys.version_info[0], sys.version_info[1])
+PYTHON_VERSION_STR = "%s.%s" % (sys.version_info[0], sys.version_info[1])
 
-IS_PYPY = '__pypy__' in sys.builtin_module_names
+IS_PYPY = "__pypy__" in sys.builtin_module_names
 
-if hasattr(sys, 'setrecursionlimit'):
+if hasattr(sys, "setrecursionlimit"):
     # pyston doesn't have setrecursionlimit
     sys.setrecursionlimit(5000)
 

uncompyle6/main.py

@@ -101,7 +101,7 @@ def decompile(
     )
     if PYTHON_VERSION < 3.0 and bytecode_version >= 3.0:
         write(
-            '# Warning: this version has problems handling the Python 3 "byte" type in constants properly.\n'
+            '# Warning: this version of Python has problems handling the Python 3 "byte" type in constants properly.\n'
         )
 
     if co.co_filename:

uncompyle6/parser.py

@@ -354,10 +354,10 @@ class PythonParser(GenericASTBuilder):
         stmt ::= with
         stmt ::= withasstmt
 
-        stmt ::= del_stmt
-        del_stmt ::= DELETE_FAST
-        del_stmt ::= DELETE_NAME
-        del_stmt ::= DELETE_GLOBAL
+        stmt   ::= delete
+        delete ::= DELETE_FAST
+        delete ::= DELETE_NAME
+        delete ::= DELETE_GLOBAL
 
 
         stmt   ::= return
@@ -623,7 +623,7 @@ class PythonParser(GenericASTBuilder):
         """
 
 
-def parse(p, tokens, customize):
+def parse(p, tokens, customize, code):
     p.customize_grammar_rules(tokens, customize)
     ast = p.parse(tokens)
     #  p.cleanup()
@@ -878,7 +878,7 @@ def python_parser(
     # parser_debug = {'rules': True, 'transition': True, 'reduce' : True,
     #                 'showstack': 'full'}
     p = get_python_parser(version, parser_debug)
-    return parse(p, tokens, customize)
+    return parse(p, tokens, customize, co)
 
 
 if __name__ == "__main__":

uncompyle6/parsers/parse2.py

@@ -99,9 +99,9 @@ class Python2Parser(PythonParser):
         for         ::= SETUP_LOOP expr for_iter store
                         for_block POP_BLOCK _come_froms
 
-        del_stmt         ::= delete_subscript
+        delete           ::= delete_subscript
         delete_subscript ::= expr expr DELETE_SUBSCR
-        del_stmt         ::= expr DELETE_ATTR
+        delete           ::= expr DELETE_ATTR
 
         _mklambda ::= load_closure mklambda
         kwarg     ::= LOAD_CONST expr
@@ -423,17 +423,17 @@ class Python2Parser(PythonParser):
                 custom_seen_ops.add(opname)
                 continue
             elif opname == "DELETE_ATTR":
-                self.addRule("del_stmt ::= expr DELETE_ATTR", nop_func)
+                self.addRule("delete ::= expr DELETE_ATTR", nop_func)
                 custom_seen_ops.add(opname)
                 continue
             elif opname.startswith("DELETE_SLICE"):
                 self.addRule(
                     """
                 del_expr ::= expr
-                del_stmt ::= del_expr DELETE_SLICE+0
-                del_stmt ::= del_expr del_expr DELETE_SLICE+1
-                del_stmt ::= del_expr del_expr DELETE_SLICE+2
-                del_stmt ::= del_expr del_expr del_expr DELETE_SLICE+3
+                delete   ::= del_expr DELETE_SLICE+0
+                delete   ::= del_expr del_expr DELETE_SLICE+1
+                delete   ::= del_expr del_expr DELETE_SLICE+2
+                delete   ::= del_expr del_expr del_expr DELETE_SLICE+3
                 """,
                     nop_func,
                 )
@@ -453,7 +453,7 @@ class Python2Parser(PythonParser):
             elif opname == "DELETE_SUBSCR":
                 self.addRule(
                     """
-                    del_stmt ::= delete_subscript
+                    delete ::= delete_subscript
                     delete_subscript ::= expr expr DELETE_SUBSCR
                    """,
                     nop_func,

uncompyle6/parsers/parse23.py

@@ -48,7 +48,7 @@ class Python23Parser(Python24Parser):
         while1stmt ::= _while1test l_stmts JUMP_BACK
                        POP_TOP POP_BLOCK
 
-        list_comp  ::= BUILD_LIST_0 DUP_TOP LOAD_ATTR store list_iter del_stmt
+        list_comp  ::= BUILD_LIST_0 DUP_TOP LOAD_ATTR store list_iter delete
         list_for   ::= expr for_iter store list_iter JUMP_BACK come_froms POP_TOP JUMP_BACK
 
         lc_body ::= LOAD_NAME expr CALL_FUNCTION_1 POP_TOP

uncompyle6/parsers/parse26.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2017-2019 Rocky Bernstein
+#  Copyright (c) 2017-2020 Rocky Bernstein
 """
 spark grammar differences over Python2 for Python 2.6.
 """
@@ -6,6 +6,7 @@ spark grammar differences over Python2 for Python 2.6.
 from uncompyle6.parser import PythonParserSingle
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.parsers.parse2 import Python2Parser
+from uncompyle6.parsers.reducecheck import (except_handler, tryelsestmt)
 
 class Python26Parser(Python2Parser):
 
@@ -24,7 +25,11 @@ class Python26Parser(Python2Parser):
         except_handler ::= JUMP_FORWARD COME_FROM except_stmts
                            come_froms_pop END_FINALLY come_froms
 
-        except_handler ::= JUMP_FORWARD COME_FROM except_stmts END_FINALLY
+        except_handler ::= JUMP_FORWARD COME_FROM except_stmts
+                           END_FINALLY
+
+        except_handler ::= JUMP_FORWARD COME_FROM except_stmts
+                           POP_TOP END_FINALLY
                            come_froms
 
         except_handler ::= jmp_abs COME_FROM except_stmts
@@ -33,6 +38,7 @@ class Python26Parser(Python2Parser):
         except_handler ::= jmp_abs COME_FROM except_stmts
                            END_FINALLY JUMP_FORWARD
 
+
         # Sometimes we don't put in COME_FROM to the next statement
         # like we do in 2.7. Perhaps we should?
         try_except     ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
@@ -87,8 +93,8 @@ class Python26Parser(Python2Parser):
 
         cf_jb_cf_pop ::= _come_froms JUMP_BACK come_froms POP_TOP
 
-        bp_come_from    ::= POP_BLOCK COME_FROM
-        jb_pb_come_from ::= JUMP_BACK bp_come_from
+        pb_come_from    ::= POP_BLOCK COME_FROM
+        jb_pb_come_from ::= JUMP_BACK pb_come_from
 
         _ifstmts_jump ::= c_stmts_opt JUMP_FORWARD COME_FROM POP_TOP
         _ifstmts_jump ::= c_stmts_opt JUMP_FORWARD come_froms POP_TOP COME_FROM
@@ -142,7 +148,7 @@ class Python26Parser(Python2Parser):
         while1stmt     ::= SETUP_LOOP l_stmts_opt CONTINUE _come_froms
 
         whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt jb_pop POP_BLOCK _come_froms
-        whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt jb_cf_pop bp_come_from
+        whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt jb_cf_pop pb_come_from
         whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt jb_cf_pop POP_BLOCK
         whilestmt      ::= SETUP_LOOP testexpr returns POP_BLOCK COME_FROM
 
@@ -224,9 +230,9 @@ class Python26Parser(Python2Parser):
         list_iter  ::= list_if JUMP_BACK
         list_iter  ::= list_if JUMP_BACK COME_FROM POP_TOP
         list_comp  ::= BUILD_LIST_0 DUP_TOP
-                       store list_iter del_stmt
+                       store list_iter delete
         list_comp  ::= BUILD_LIST_0 DUP_TOP
-	               store list_iter JUMP_BACK del_stmt
+	               store list_iter JUMP_BACK delete
         lc_body    ::= LOAD_NAME expr LIST_APPEND
 	lc_body    ::= LOAD_FAST expr LIST_APPEND
 
@@ -345,18 +351,30 @@ class Python26Parser(Python2Parser):
                 WITH_CLEANUP END_FINALLY
         """)
         super(Python26Parser, self).customize_grammar_rules(tokens, customize)
+        self.reduce_check_table = {
+            "except_handler": except_handler,
+            "tryelsestmt": tryelsestmt,
+            "tryelsestmtl": tryelsestmt,
+        }
+
+
         self.check_reduce['and'] = 'AST'
         self.check_reduce['assert_expr_and'] = 'AST'
+        self.check_reduce["except_handler"] = "tokens"
         self.check_reduce["ifstmt"] = "tokens"
         self.check_reduce["ifelsestmt"] = "AST"
+        self.check_reduce["forelselaststmtl"] = "tokens"
+        self.check_reduce["forelsestmt"] = "tokens"
         self.check_reduce['list_for'] = 'AST'
         self.check_reduce['try_except'] = 'tokens'
         self.check_reduce['tryelsestmt'] = 'AST'
+        self.check_reduce['tryelsestmtl'] = 'AST'
 
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
         invalid = super(Python26Parser,
                         self).reduce_is_invalid(rule, ast,
                                                 tokens, first, last)
+        lhs = rule[0]
         if invalid or tokens is None:
             return invalid
         if rule in (
@@ -389,6 +407,27 @@ class Python26Parser(Python2Parser):
             return not (jmp_target == tokens[test_index].offset or
                         tokens[last].pattr == jmp_false.pattr)
 
+        elif lhs in ("forelselaststmtl", "forelsestmt"):
+            # print("XXX", first, last)
+            # for t in range(first, last):
+            #     print(tokens[t])
+            # print("=" * 30)
+
+            # If the SETUP_LOOP jumps to the tokens[last] then
+            # this is a "for" not a "for else".
+
+            # However, in Python 2.2 and before there is a SET_LINENO
+            # instruction which might have gotten removed. So we need
+            # to account for that.  bytecode-1.4/anydbm.pyc exhibits
+            # this behavior.
+
+            # Also we need to use the setup_loop instruction (not opcode)
+            # since the operand can be a relative offset rather than
+            # an absolute offset.
+            setup_inst = self.insts[self.offset2inst_index[tokens[first].offset]]
+            if self.version <= 2.2 and tokens[last] == "COME_FROM":
+                last += 1
+            return tokens[last-1].off2int() > setup_inst.argval
         elif rule == ("ifstmt", ("testexpr", "_ifstmts_jump")):
             for i in range(last-1, last-4, -1):
                 t = tokens[i]
@@ -405,7 +444,7 @@ class Python26Parser(Python2Parser):
             # The JUMP_ABSOLUTE has to be to the last POP_TOP or this is invalid
             ja_attr = ast[4].attr
             return tokens[last].offset != ja_attr
-        elif rule[0] == 'try_except':
+        elif lhs == 'try_except':
             # We need to distingush try_except from tryelsestmt and we do that
             # by checking the jump before the END_FINALLY
             # If we have:
@@ -422,12 +461,12 @@ class Python26Parser(Python2Parser):
                 last -= 1
             if (tokens[last] == 'COME_FROM'
                 and tokens[last-1] == 'END_FINALLY'
-                    and tokens[last-2] == 'POP_TOP'):
+                   and tokens[last-2] == 'POP_TOP'):
                 # A jump of 2 is a jump around POP_TOP, END_FINALLY which
                 # would indicate try/else rather than try
                 return (tokens[last-3].kind not in frozenset(('JUMP_FORWARD', 'RETURN_VALUE'))
                         or (tokens[last-3] == 'JUMP_FORWARD' and tokens[last-3].attr != 2))
-        elif rule[0] == 'tryelsestmt':
+        elif lhs == 'tryelsestmt':
 
             # We need to distingush try_except from tryelsestmt and we do that
             # by making sure that the jump before the except handler jumps to

uncompyle6/parsers/parse27.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2019 Rocky Bernstein
+#  Copyright (c) 2016-2020 Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 
@@ -7,8 +7,10 @@ from xdis import next_offset
 from uncompyle6.parser import PythonParserSingle, nop_func
 from uncompyle6.parsers.parse2 import Python2Parser
 from uncompyle6.parsers.reducecheck import (
+    aug_assign1_check,
     or_check,
     tryelsestmt,
+    except_handler,
 )
 
 class Python27Parser(Python2Parser):
@@ -91,7 +93,7 @@ class Python27Parser(Python2Parser):
         iflaststmtl     ::= testexpr c_stmts
 
         _ifstmts_jump   ::= c_stmts_opt JUMP_FORWARD come_froms
-        bp_come_from    ::= POP_BLOCK COME_FROM
+        pb_come_from    ::= POP_BLOCK COME_FROM
 
         # FIXME: Common with 3.0+
         jmp_false ::= POP_JUMP_IF_FALSE
@@ -163,7 +165,7 @@ class Python27Parser(Python2Parser):
         while1elsestmt    ::= SETUP_LOOP l_stmts JUMP_BACK
                               else_suitel COME_FROM
 
-        while1stmt        ::= SETUP_LOOP returns bp_come_from
+        while1stmt        ::= SETUP_LOOP returns pb_come_from
         while1stmt        ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_BLOCK COME_FROM
 
         whilestmt         ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK _come_froms
@@ -230,12 +232,15 @@ class Python27Parser(Python2Parser):
         # FIXME: Put more in this table
         self.reduce_check_table = {
             # "ifelsestmt": ifelsestmt,
+            "aug_assign1": aug_assign1_check,
+            "except_handler": except_handler,
             "or": or_check,
             "tryelsestmt": tryelsestmt,
             "tryelsestmtl": tryelsestmt,
         }
 
         self.check_reduce["and"] = "AST"
+        self.check_reduce["aug_assign1"] = "AST"
         self.check_reduce["if_exp"] = "AST"
 
         self.check_reduce["except_handler"] = "tokens"

uncompyle6/parsers/parse3.py

@@ -254,7 +254,7 @@ class Python3Parser(PythonParser):
                                   END_FINALLY _jump
 
         except_var_finalize ::= POP_BLOCK POP_EXCEPT LOAD_CONST COME_FROM_FINALLY
-                                LOAD_CONST store del_stmt
+                                LOAD_CONST store delete
 
         except_suite ::= returns
 
@@ -933,7 +933,7 @@ class Python3Parser(PythonParser):
                 self.addRule("continue ::= CONTINUE_LOOP", nop_func)
                 custom_ops_processed.add(opname)
             elif opname == "DELETE_ATTR":
-                self.addRule("del_stmt ::= expr DELETE_ATTR", nop_func)
+                self.addRule("delete ::= expr DELETE_ATTR", nop_func)
                 custom_ops_processed.add(opname)
             elif opname == "DELETE_DEREF":
                 self.addRule(
@@ -947,7 +947,7 @@ class Python3Parser(PythonParser):
             elif opname == "DELETE_SUBSCR":
                 self.addRule(
                     """
-                    del_stmt ::= delete_subscript
+                    delete ::= delete_subscript
                     delete_subscript ::= expr expr DELETE_SUBSCR
                    """,
                     nop_func,

uncompyle6/parsers/parse31.py

@@ -18,15 +18,15 @@ class Python31Parser(Python32Parser):
         with           ::= expr setupwith SETUP_FINALLY
                            suite_stmts_opt
                            POP_BLOCK LOAD_CONST COME_FROM_FINALLY
-                           load del_stmt WITH_CLEANUP END_FINALLY
+                           load delete WITH_CLEANUP END_FINALLY
 
         # Keeps Python 3.1 withas desigator in the same position as it is in other version
-        setupwithas31  ::= setupwithas SETUP_FINALLY load del_stmt
+        setupwithas31  ::= setupwithas SETUP_FINALLY load delete
 
         withasstmt     ::= expr setupwithas31 store
                            suite_stmts_opt
                            POP_BLOCK LOAD_CONST COME_FROM_FINALLY
-                           load del_stmt WITH_CLEANUP END_FINALLY
+                           load delete WITH_CLEANUP END_FINALLY
 
         store ::= STORE_NAME
         load  ::= LOAD_FAST

uncompyle6/parsers/parse37.py

@@ -113,10 +113,10 @@ class Python37Parser(Python37BaseParser):
         stmt ::= tryelsestmt
         stmt ::= tryfinallystmt
 
-        stmt ::= del_stmt
-        del_stmt ::= DELETE_FAST
-        del_stmt ::= DELETE_NAME
-        del_stmt ::= DELETE_GLOBAL
+        stmt   ::= delete
+        delete ::= DELETE_FAST
+        delete ::= DELETE_NAME
+        delete ::= DELETE_GLOBAL
 
         stmt   ::= return
         return ::= ret_expr RETURN_VALUE
@@ -889,7 +889,7 @@ class Python37Parser(Python37BaseParser):
                                   END_FINALLY _jump
 
         except_var_finalize ::= POP_BLOCK POP_EXCEPT LOAD_CONST COME_FROM_FINALLY
-                                LOAD_CONST store del_stmt
+                                LOAD_CONST store delete
 
         except_suite ::= returns
 

uncompyle6/parsers/parse37base.py

@@ -541,7 +541,7 @@ class Python37BaseParser(PythonParser):
                 self.addRule("continue ::= CONTINUE_LOOP", nop_func)
                 custom_ops_processed.add(opname)
             elif opname == "DELETE_ATTR":
-                self.addRule("del_stmt ::= expr DELETE_ATTR", nop_func)
+                self.addRule("delete ::= expr DELETE_ATTR", nop_func)
                 custom_ops_processed.add(opname)
             elif opname == "DELETE_DEREF":
                 self.addRule(
@@ -555,7 +555,7 @@ class Python37BaseParser(PythonParser):
             elif opname == "DELETE_SUBSCR":
                 self.addRule(
                     """
-                    del_stmt ::= delete_subscript
+                    delete ::= delete_subscript
                     delete_subscript ::= expr expr DELETE_SUBSCR
                    """,
                     nop_func,

uncompyle6/parsers/parse38.py

@@ -112,6 +112,8 @@ class Python38Parser(Python37Parser):
         for38              ::= expr get_for_iter store for_block
 
         forelsestmt38      ::= expr get_for_iter store for_block POP_BLOCK else_suite
+        forelsestmt38      ::= expr get_for_iter store for_block JUMP_BACK _come_froms else_suite
+
         forelselaststmt38  ::= expr get_for_iter store for_block POP_BLOCK else_suitec
         forelselaststmtl38 ::= expr get_for_iter store for_block POP_BLOCK else_suitel
 

uncompyle6/parsers/reducecheck/__init__.py

@@ -1,4 +1,6 @@
 from uncompyle6.parsers.reducecheck.and_check import *
+from uncompyle6.parsers.reducecheck.aug_assign import *
+from uncompyle6.parsers.reducecheck.except_handler import *
 from uncompyle6.parsers.reducecheck.except_handler_else import *
 from uncompyle6.parsers.reducecheck.ifelsestmt import *
 from uncompyle6.parsers.reducecheck.iflaststmt import *

uncompyle6/parsers/reducecheck/aug_assign.py

@@ -0,0 +1,10 @@
+#  Copyright (c) 2020 Rocky Bernstein
+
+def aug_assign1_check(self, lhs, n, rule, ast, tokens, first, last):
+    # print("XXX", first, last, rule)
+    # for t in range(first, last): print(tokens[t])
+    # print("="*40)
+
+    expr = ast[0]
+    return expr == "expr" and expr[0] == "or"
+    return False

uncompyle6/parsers/reducecheck/except_handler.py

@@ -0,0 +1,19 @@
+#  Copyright (c) 2020 Rocky Bernstein
+
+def except_handler(self, lhs, n, rule, ast, tokens, first, last):
+    end_token = tokens[last-1]
+
+    # print("XXX", first, last)
+    # for t in range(first, last):
+    #     print(tokens[t])
+    # print("=" * 30)
+
+    # FIXME: Figure out why this doesn't work on
+    # bytecode-1.4/anydbm.pyc
+    if self.version == 1.4:
+        return False
+
+    # Make sure come froms all come from within "except_handler".
+    if end_token != "COME_FROM":
+        return False
+    return end_token.attr < tokens[first].offset

uncompyle6/parsers/reducecheck/tryelsestmt.py

@@ -6,7 +6,13 @@ def tryelsestmt(self, lhs, n, rule, ast, tokens, first, last):
     # Check the end of the except handler that there isn't a jump from
     # inside the except handler to the end. If that happens
     # then this is a "try" with no "else".
+
+    # for t in range(first, last):
+    #     print(tokens[t])
+    # print("=" * 30)
+
     except_handler = ast[3]
+
     if except_handler == "except_handler_else":
         except_handler = except_handler[0]
     if except_handler == "except_handler":
@@ -32,5 +38,8 @@ def tryelsestmt(self, lhs, n, rule, ast, tokens, first, last):
             except_handler_first_offset = leading_jump.first_child().off2int()
         else:
             except_handler_first_offset = leading_jump.off2int()
+
+        if first_come_from.attr < tokens[first].offset:
+            return True
         return first_come_from.attr > except_handler_first_offset
     return False

uncompyle6/scanner.py

@@ -74,8 +74,8 @@ if PYTHON3:
     intern = sys.intern
     L65536 = 65536
 
-    def long(l):
-        return l
+    def long(num):
+        return num
 
 
 else:

uncompyle6/semantics/aligner.py

@@ -142,7 +142,7 @@ def code_deparse_align(co, out=sys.stderr, version=None, is_pypy=None,
                             is_pypy = is_pypy)
 
     isTopLevel = co.co_name == '<module>'
-    deparsed.ast = deparsed.build_ast(tokens, customize, isTopLevel=isTopLevel)
+    deparsed.ast = deparsed.build_ast(tokens, customize, co, isTopLevel=isTopLevel)
 
     assert deparsed.ast == 'stmts', 'Should have parsed grammar start'
 

uncompyle6/semantics/consts.py

@@ -142,17 +142,12 @@ PASS = SyntaxTree(
 )
 
 ASSIGN_DOC_STRING = lambda doc_string, doc_load: SyntaxTree(
-    "stmt",
+    "assign",
     [
         SyntaxTree(
-            "assign",
-            [
-                SyntaxTree(
-                    "expr", [Token(doc_load, pattr=doc_string, attr=doc_string)]
-                ),
-                SyntaxTree("store", [Token("STORE_NAME", pattr="__doc__")]),
-            ],
-        )
+            "expr", [Token(doc_load, pattr=doc_string, attr=doc_string)]
+        ),
+        SyntaxTree("store", [Token("STORE_NAME", pattr="__doc__")]),
     ],
 )
 
@@ -332,7 +327,10 @@ TABLE_DIRECT = {
     # For compatibility with older Python, we'll use "%" instead of
     # a format string.
     "string_at_beginning": ('%|"%%s" %% %c\n', 0),
-    "call_stmt": ("%|%p\n", (0, 200)),
+    "call_stmt":	   ( "%|%p\n",
+                             # When a call statement contains only a named_expr (:=)
+                             # the named_expr should have parenthesis around it.
+                             (0, PRECEDENCE["named_expr"]-1)),
     "break": ("%|break\n",),
     "continue": ("%|continue\n",),
     "raise_stmt0": ("%|raise\n",),
@@ -438,7 +436,7 @@ MAP_R = (TABLE_R, -1)
 MAP = {
     "stmt": MAP_R,
     "call": MAP_R,
-    "del_stmt": MAP_R,
+    "delete": MAP_R,
     "store": MAP_R,
     "exprlist": MAP_R0,
 }

uncompyle6/semantics/customize3.py

@@ -90,7 +90,7 @@ def customize_for_version3(self, version):
         code_obj = node[1].attr
         assert iscode(code_obj)
         code = Code(code_obj, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
 
         # skip over: sstmt, stmt, return, ret_expr

uncompyle6/semantics/customize37.py

@@ -149,6 +149,11 @@ def customize_for_version37(self, version):
                 (1, "store"), (0, "get_aiter"), (3, "list_iter"),
             ),
 
+            "list_afor": (
+                " async for %[1]{%c} in %c%[1]{%c}",
+                (1, "store"), (0, "get_aiter"), (3, "list_iter"),
+            ),
+
             "list_if37": (" if %p%c", (0, 27), 1),
             "list_if37_not": (" if not %p%c", (0, 27), 1),
             "testfalse_not_or": ("not %c or %c", (0, "expr"), (2, "expr")),

uncompyle6/semantics/customize38.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019 by Rocky Bernstein
+#  Copyright (c) 2019-2020 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -19,7 +19,7 @@
 # Python 3.8+ changes
 #######################
 
-from uncompyle6.semantics.consts import TABLE_DIRECT
+from uncompyle6.semantics.consts import PRECEDENCE, TABLE_DIRECT
 
 def customize_for_version38(self, version):
 
@@ -37,14 +37,15 @@ def customize_for_version38(self, version):
             '%|async for %c in %c:\n%+%c%-%|else:\n%+%c%-\n\n',
             (7, 'store'), (0, 'expr'), (8, 'for_block'), (-1, 'else_suite') ),
 
-        'async_with_stmt38': (
-            '%|async with %c:\n%+%|%c%-',
-            (0, 'expr'), 7),
+        "async_with_stmt38": (
+            "%|async with %c:\n%+%|%c%-",
+            (0, "expr"), 7),
 
-        'async_with_as_stmt38':  (
-            '%|async with %c as %c:\n%+%|%c%-',
-            (0, 'expr'), (6, 'store'),
-            (7, 'suite_stmts') ),
+        "async_with_as_stmt38":  (
+            "%|async with %c as %c:\n%+%|%c%-",
+            (0, "expr"), (6, "store"),
+            (7, "suite_stmts")
+        ),
 
         "call_stmt": (
             "%|%c\n", 0
@@ -80,11 +81,13 @@ def customize_for_version38(self, version):
             (2, 'store'),
             (0, 'expr'),
             (3, 'for_block') ),
-        'forelsestmt38':    (
-            '%|for %c in %c:\n%+%c%-%|else:\n%+%c%-\n\n',
-            (2, 'store'),
-            (0, 'expr'),
-            (3, 'for_block'), -2 ),
+
+        "forelsestmt38":    (
+            "%|for %c in %c:\n%+%c%-%|else:\n%+%c%-\n\n",
+            (2, "store"),
+            (0, "expr"),
+            (3, "for_block"), -1 ),
+
         'forelselaststmt38': (
             '%|for %c in %c:\n%+%c%-%|else:\n%+%c%-',
             (2, 'store'),
@@ -153,6 +156,6 @@ def customize_for_version38(self, version):
             (2, "suite_stmts_opt"),
             (8, "suite_stmts_opt") ),
         "named_expr": ( # AKA "walrus operator"
-            "%c := %c", (2, "store"), (0, "expr")
+            "%c := %p", (2, "store"), (0, "expr", PRECEDENCE["named_expr"]-1)
             )
     })

uncompyle6/semantics/fragments.py

@@ -258,7 +258,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
             raise GenericASTTraversalPruningException
 
     n_slice0 = n_slice1 = n_slice2 = n_slice3 = n_subscript = table_r_node
-    n_aug_assign_1 = n_print_item = exec_stmt = print_to_item = del_stmt = table_r_node
+    n_aug_assign_1 = n_print_item = exec_stmt = print_to_item = delete = table_r_node
     n_classdefco1 = n_classdefco2 = except_cond1 = except_cond2 = table_r_node
 
     def n_pass(self, node):
@@ -681,7 +681,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
         assert iscode(cn.attr)
 
         code = Code(cn.attr, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
         ast = ast[0][0][0]
 
@@ -728,7 +728,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
         code_name = code.co_name
         code = Code(code, self.scanner, self.currentclass)
 
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
 
         self.customize(code._customize)
         if ast[0] == "sstmt":
@@ -850,7 +850,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
         self.prec = 27
 
         code = Code(node[1].attr, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
         if node == "set_comp":
             ast = ast[0][0][0]
@@ -992,7 +992,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
         self.prec = 27
 
         code = Code(node[1].attr, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
         ast = ast[0][0][0]
         store = ast[3]
@@ -1142,9 +1142,8 @@ class FragmentsWalker(pysource.SourceWalker, object):
         self.name = old_name
         self.return_none = rn
 
-    def build_ast(
-        self, tokens, customize, is_lambda=False, noneInNames=False, isTopLevel=False
-    ):
+    def build_ast(self, tokens, customize, code, is_lambda=False,
+                  noneInNames=False, isTopLevel=False):
 
         # FIXME: DRY with pysource.py
 
@@ -1201,7 +1200,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
             p_insts = self.p.insts
             self.p.insts = self.scanner.insts
             self.p.offset2inst_index = self.scanner.offset2inst_index
-            ast = parser.parse(self.p, tokens, customize)
+            ast = parser.parse(self.p, tokens, customize, code)
             self.p.insts = p_insts
         except (parser.ParserError, AssertionError) as e:
             raise ParserError(e, tokens, {})
@@ -1545,7 +1544,8 @@ class FragmentsWalker(pysource.SourceWalker, object):
         line_seperator = ",\n" + self.indent
         sep = INDENT_PER_LEVEL[:-1]
         start = len(self.f.getvalue())
-        self.write("{")
+        if node[0] != "dict_entry":
+            self.write("{")
         self.set_pos_info(node[0], start, start + 1)
 
         if self.version >= 3.0 and not self.is_pypy:
@@ -1571,7 +1571,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
                     i += 2
                     pass
                 pass
-            elif node[1].kind.startswith("kvlist"):
+            elif len(node) > 1 and node[1].kind.startswith("kvlist"):
                 # Python 3.0..3.4 style key/value list in dict
                 kv_node = node[1]
                 l = list(kv_node)
@@ -1597,37 +1597,98 @@ class FragmentsWalker(pysource.SourceWalker, object):
                     i += 3
                     pass
                 pass
+            elif node[-1].kind.startswith("BUILD_CONST_KEY_MAP"):
+                # Python 3.6+ style const map
+                keys = node[-2].pattr
+                values = node[:-2]
+                # FIXME: Line numbers?
+                for key, value in zip(keys, values):
+                    self.write(sep)
+                    self.write(repr(key))
+                    line_number = self.line_number
+                    self.write(":")
+                    self.write(self.traverse(value[0]))
+                    sep = ", "
+                    if line_number != self.line_number:
+                        sep += "\n" + self.indent + INDENT_PER_LEVEL[:-1]
+                        line_number = self.line_number
+                    else:
+                        sep += " "
+                        pass
+                    pass
+                if sep.startswith(",\n"):
+                    self.write(sep[1:])
+                pass
+            elif node[0].kind.startswith("dict_entry"):
+                assert self.version >= 3.5
+                template = ("%C", (0, len(node[0]), ", **"))
+                self.template_engine(template, node[0])
+                sep = ""
+            elif node[-1].kind.startswith("BUILD_MAP_UNPACK") or node[
+                -1
+            ].kind.startswith("dict_entry"):
+                assert self.version >= 3.5
+                # FIXME: I think we can intermingle dict_comp's with other
+                # dictionary kinds of things. The most common though is
+                # a sequence of dict_comp's
+                kwargs = node[-1].attr
+                template = ("**%C", (0, kwargs, ", **"))
+                self.template_engine(template, node)
+                sep = ""
+
             pass
         else:
-            # Python 2 style kvlist
-            assert node[-1].kind.startswith("kvlist")
-            kv_node = node[-1]  # goto kvlist
+            # Python 2 style kvlist. Find beginning of kvlist.
+            indent = self.indent + "  "
+            line_number = self.line_number
+            if node[0].kind.startswith("BUILD_MAP"):
+                if len(node) > 1 and node[1].kind in ("kvlist", "kvlist_n"):
+                    kv_node = node[1]
+                else:
+                    kv_node = node[1:]
+                self.kv_map(kv_node, sep, line_number, indent)
 
-            for kv in kv_node:
-                assert kv in ("kv", "kv2", "kv3")
-                # kv ::= DUP_TOP expr ROT_TWO expr STORE_SUBSCR
-                # kv2 ::= DUP_TOP expr expr ROT_THREE STORE_SUBSCR
-                # kv3 ::= expr expr STORE_MAP
-                if kv == "kv":
-                    name = self.traverse(kv[-2], indent="")
-                    kv[1].parent = kv_node
-                    value = self.traverse(
-                        kv[1], indent=self.indent + (len(name) + 2) * " "
-                    )
-                elif kv == "kv2":
-                    name = self.traverse(kv[1], indent="")
-                    kv[-3].parent = kv_node
-                    value = self.traverse(
-                        kv[-3], indent=self.indent + (len(name) + 2) * " "
-                    )
-                elif kv == "kv3":
-                    name = self.traverse(kv[-2], indent="")
-                    kv[0].parent = kv_node
-                    value = self.traverse(
-                        kv[0], indent=self.indent + (len(name) + 2) * " "
-                    )
-                self.write(sep, name, ": ", value)
-                sep = line_seperator
+            else:
+                sep = ""
+                opname = node[-1].kind
+                if self.is_pypy and self.version >= 3.5:
+                    if opname.startswith("BUILD_CONST_KEY_MAP"):
+                        keys = node[-2].attr
+                        # FIXME: DRY this and the above
+                        for i in range(len(keys)):
+                            key = keys[i]
+                            value = self.traverse(node[i], indent="")
+                            self.write(sep, key, ": ", value)
+                            sep = ", "
+                            if line_number != self.line_number:
+                                sep += "\n" + self.indent + "  "
+                                line_number = self.line_number
+                                pass
+                            pass
+                        pass
+                    else:
+                        if opname.startswith("kvlist"):
+                            list_node = node[0]
+                        else:
+                            list_node = node
+
+                        assert list_node[-1].kind.startswith("BUILD_MAP")
+                        for i in range(0, len(list_node) - 1, 2):
+                            key = self.traverse(list_node[i], indent="")
+                            value = self.traverse(list_node[i + 1], indent="")
+                            self.write(sep, key, ": ", value)
+                            sep = ", "
+                            if line_number != self.line_number:
+                                sep += "\n" + self.indent + "  "
+                                line_number = self.line_number
+                                pass
+                            pass
+                        pass
+                elif opname.startswith("kvlist"):
+                    kv_node = node[-1]
+                    self.kv_map(node[-1], sep, line_number, indent)
+
+                pass
         self.write("}")
         finish = len(self.f.getvalue())
         self.set_pos_info(node, start, finish)
@@ -1932,7 +1993,7 @@ def code_deparse(
     co,
     out=StringIO(),
     version=None,
-    is_pypy=None,
+    is_pypy=IS_PYPY,
     debug_opts=DEFAULT_DEBUG_OPTS,
     code_objects={},
     compile_mode="exec",
@@ -1961,9 +2022,8 @@ def code_deparse(
 
     if version is None:
         version = sysinfo2float()
-    if is_pypy is None:
-        is_pypy = IS_PYPY
 
+    # store final output stream for case of error
     scanner = get_scanner(version, is_pypy=is_pypy)
 
     show_asm = debug_opts.get("asm", None)
@@ -1990,14 +2050,17 @@ def code_deparse(
         is_pypy=is_pypy,
     )
 
-    deparsed.ast = deparsed.build_ast(tokens, customize)
+    isTopLevel = co.co_name == "<module>"
+    deparsed.ast = deparsed.build_ast(tokens, customize, co, isTopLevel=isTopLevel)
 
     assert deparsed.ast == "stmts", "Should have parsed grammar start"
 
-    del tokens  # save memory
+    # save memory
+    del tokens
 
     # convert leading '__doc__ = "..." into doc string
     assert deparsed.ast == "stmts"
+
     (deparsed.mod_globs, nonlocals) = pysource.find_globals_and_nonlocals(
         deparsed.ast, set(), set(), co, version
     )

uncompyle6/semantics/make_function2.py

@@ -100,6 +100,7 @@ def make_function2(self, node, is_lambda, nested=1, code_node=None):
         ast = self.build_ast(
             code._tokens,
             code._customize,
+            code,
             is_lambda=is_lambda,
             noneInNames=("None" in code.co_names),
         )

uncompyle6/semantics/make_function3.py

@@ -132,6 +132,7 @@ def make_function3_annotate(
         ast = self.build_ast(
             code._tokens,
             code._customize,
+            code,
             is_lambda=is_lambda,
             noneInNames=("None" in code.co_names),
         )
@@ -491,6 +492,7 @@ def make_function3(self, node, is_lambda, nested=1, code_node=None):
         ast = self.build_ast(
             scanner_code._tokens,
             scanner_code._customize,
+            scanner_code,
             is_lambda=is_lambda,
             noneInNames=("None" in code.co_names),
         )

uncompyle6/semantics/make_function36.py

@@ -173,6 +173,7 @@ def make_function36(self, node, is_lambda, nested=1, code_node=None):
         ast = self.build_ast(
             scanner_code._tokens,
             scanner_code._customize,
+            scanner_code,
             is_lambda=is_lambda,
             noneInNames=("None" in code.co_names),
         )

uncompyle6/semantics/pysource.py

@@ -135,7 +135,7 @@ import sys
 IS_PYPY = "__pypy__" in sys.builtin_module_names
 PYTHON3 = sys.version_info >= (3, 0)
 
-from xdis import iscode, COMPILER_FLAG_BIT
+from xdis import iscode, COMPILER_FLAG_BIT, sysinfo2float
 
 from uncompyle6.parser import get_python_parser
 from uncompyle6.parsers.treenode import SyntaxTree
@@ -164,7 +164,6 @@ from uncompyle6.semantics.consts import (
     NONE,
     RETURN_NONE,
     PASS,
-    ASSIGN_DOC_STRING,
     NAME_MODULE,
     TAB,
     INDENT_PER_LEVEL,
@@ -588,8 +587,7 @@ class SourceWalker(GenericASTTraversal, object):
         if n == "LOAD_CONST" and repr(n.pattr)[0] == "-":
             self.prec = 6
 
-        # print(n.kind, p, "<", self.prec)
-        # print(self.f.getvalue())
+        # print("XXX", n.kind, p, "<", self.prec)
 
         if p < self.prec:
             self.write("(")
@@ -605,7 +603,6 @@ class SourceWalker(GenericASTTraversal, object):
             # If expr is yield we want parens.
             self.prec = PRECEDENCE["yield"] - 1
             self.n_expr(node[0])
-            p = self.prec
         else:
             self.n_expr(node)
 
@@ -983,7 +980,7 @@ class SourceWalker(GenericASTTraversal, object):
                 self.n_list_comp_pypy27(node)
                 return
             n = node[-1]
-        elif node[-1] == "del_stmt":
+        elif node[-1] == "delete":
             if node[-2] == "JUMP_BACK":
                 n = node[-3]
             else:
@@ -1120,7 +1117,7 @@ class SourceWalker(GenericASTTraversal, object):
         assert iscode(cn.attr)
 
         code = Code(cn.attr, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
 
         # Remove single reductions as in ("stmts", "sstmt"):
@@ -1204,7 +1201,7 @@ class SourceWalker(GenericASTTraversal, object):
         assert iscode(code), node[code_index]
         code = Code(code, self.scanner, self.currentclass)
 
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
 
         # skip over: sstmt, stmt, return, ret_expr
@@ -1399,7 +1396,7 @@ class SourceWalker(GenericASTTraversal, object):
         self.prec = 27
 
         code = Code(node[1].attr, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
 
         # Remove single reductions as in ("stmts", "sstmt"):
@@ -1663,7 +1660,7 @@ class SourceWalker(GenericASTTraversal, object):
         """
         prettyprint a dict
         'dict' is something like k = {'a': 1, 'b': 42}"
-        We will source-code use line breaks to guide us when to break.
+        We will use source-code line breaks to guide us when to break.
         """
         p = self.prec
         self.prec = 100
@@ -2313,7 +2310,7 @@ class SourceWalker(GenericASTTraversal, object):
 
         indent = self.indent
         # self.println(indent, '#flags:\t', int(code.co_flags))
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         code._tokens = None  # save memory
         assert ast == "stmts"
 
@@ -2324,6 +2321,7 @@ class SourceWalker(GenericASTTraversal, object):
         if ast[0] == "docstring":
             self.println(self.traverse(ast[0]))
             del ast[0]
+            first_stmt = ast[0]
 
         if 3.0 <= self.version <= 3.3:
             try:
@@ -2387,10 +2385,10 @@ class SourceWalker(GenericASTTraversal, object):
         # if docstring exists, dump it
         if code.co_consts and code.co_consts[0] is not None and len(ast) > 0:
             do_doc = False
-            if is_docstring(ast[0]):
+            if is_docstring(ast[0], self.version, code.co_consts):
                 i = 0
                 do_doc = True
-            elif len(ast) > 1 and is_docstring(ast[1]):
+            elif len(ast) > 1 and is_docstring(ast[1], self.version, code.co_consts):
                 i = 1
                 do_doc = True
             if do_doc and self.hide_internal:
@@ -2466,7 +2464,7 @@ class SourceWalker(GenericASTTraversal, object):
         self.return_none = rn
 
     def build_ast(
-        self, tokens, customize, is_lambda=False, noneInNames=False, isTopLevel=False
+        self, tokens, customize, code, is_lambda=False, noneInNames=False, isTopLevel=False
     ):
 
         # FIXME: DRY with fragments.py
@@ -2486,13 +2484,13 @@ class SourceWalker(GenericASTTraversal, object):
                 p_insts = self.p.insts
                 self.p.insts = self.scanner.insts
                 self.p.offset2inst_index = self.scanner.offset2inst_index
-                ast = python_parser.parse(self.p, tokens, customize)
+                ast = python_parser.parse(self.p, tokens, customize, code)
                 self.customize(customize)
                 self.p.insts = p_insts
 
             except (python_parser.ParserError, AssertionError) as e:
                 raise ParserError(e, tokens, self.p.debug['reduce'])
-            transform_ast = self.treeTransform.transform(ast)
+            transform_ast = self.treeTransform.transform(ast, code)
             self.maybe_show_tree(ast)
             del ast  # Save memory
             return transform_ast
@@ -2524,7 +2522,7 @@ class SourceWalker(GenericASTTraversal, object):
             self.p.insts = self.scanner.insts
             self.p.offset2inst_index = self.scanner.offset2inst_index
             self.p.opc = self.scanner.opc
-            ast = python_parser.parse(self.p, tokens, customize)
+            ast = python_parser.parse(self.p, tokens, customize, code)
             self.p.insts = p_insts
         except (python_parser.ParserError, AssertionError) as e:
             raise ParserError(e, tokens, self.p.debug['reduce'])
@@ -2532,7 +2530,7 @@ class SourceWalker(GenericASTTraversal, object):
         checker(ast, False, self.ast_errors)
 
         self.customize(customize)
-        transform_ast = self.treeTransform.transform(ast)
+        transform_ast = self.treeTransform.transform(ast, code)
 
         self.maybe_show_tree(ast)
 
@@ -2566,7 +2564,7 @@ def code_deparse(
     assert iscode(co)
 
     if version is None:
-        version = float(sys.version[0:3])
+        version = sysinfo2float()
 
     # store final output stream for case of error
     scanner = get_scanner(version, is_pypy=is_pypy)
@@ -2594,7 +2592,7 @@ def code_deparse(
     )
 
     isTopLevel = co.co_name == "<module>"
-    deparsed.ast = deparsed.build_ast(tokens, customize, isTopLevel=isTopLevel)
+    deparsed.ast = deparsed.build_ast(tokens, customize, co, isTopLevel=isTopLevel)
 
     #### XXX workaround for profiling
     if deparsed.ast is None:

uncompyle6/semantics/transform.py

@@ -13,7 +13,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-from xdis import iscode
 from uncompyle6.show import maybe_show_tree
 from copy import copy
 from spark_parser import GenericASTTraversal, GenericASTTraversalPruningException
@@ -21,16 +20,39 @@ from spark_parser import GenericASTTraversal, GenericASTTraversalPruningExceptio
 from uncompyle6.semantics.helper import find_code_node
 from uncompyle6.parsers.treenode import SyntaxTree
 from uncompyle6.scanners.tok import NoneToken, Token
-from uncompyle6.semantics.consts import RETURN_NONE
+from uncompyle6.semantics.consts import RETURN_NONE, ASSIGN_DOC_STRING
 
 
-def is_docstring(node):
+def is_docstring(node, version, co_consts):
     if node == "sstmt":
         node = node[0]
-    try:
-        return node.kind == "assign" and node[1][0].pattr == "__doc__"
-    except:
-        return False
+    # TODO: the test below on 2.7 succeeds for
+    #   class OldClass:
+    #     __doc__ = DocDescr()
+    # which produces:
+    #
+    #   assign (2)
+    #      0. expr
+    #         call (2)
+    #              0. expr
+    #                  L.  16         6  LOAD_DEREF            0  'DocDescr'
+    #              1.                 9  CALL_FUNCTION_0       0  None
+    #      1. store
+    #
+    # See Python 2.7 test_descr.py
+
+    # If ASSIGN_DOC_STRING doesn't work we need something like the below
+    # but more elaborate to address the above.
+
+    # try:
+    #     return node.kind == "assign" and node[1][0].pattr == "__doc__"
+    # except:
+    #     return False
+    if version <= 2.7:
+        doc_load = "LOAD_CONST"
+    else:
+        doc_load = "LOAD_STR"
+    return node == ASSIGN_DOC_STRING(co_consts[0], doc_load)
 
 
 def is_not_docstring(call_stmt_node):
@@ -417,7 +439,7 @@ class TreeTransform(GenericASTTraversal, object):
         node = self.preorder(node)
         return node
 
-    def transform(self, ast):
+    def transform(self, ast, code):
         self.maybe_show_tree(ast)
         self.ast = copy(ast)
         self.ast = self.traverse(self.ast, is_lambda=False)
@@ -438,9 +460,9 @@ class TreeTransform(GenericASTTraversal, object):
             for i in range(len(self.ast)):
                 sstmt = ast[i]
                 if len(sstmt) == 1 and sstmt == "sstmt":
-                    ast[i] = ast[i][0]
+                    self.ast[i] = self.ast[i][0]
 
-                if is_docstring(self.ast[i]):
+                if is_docstring(self.ast[i], self.version, code.co_consts):
                     load_const = self.ast[i].first_child()
                     docstring_ast = SyntaxTree(
                         "docstring",

uncompyle6/show.py

@@ -1,4 +1,4 @@
-# Copyright (C) 2018 Rocky Bernstein <rocky@gnu.org>
+# Copyright (C) 2018, 2020 Rocky Bernstein <rocky@gnu.org>
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -47,8 +47,11 @@ def maybe_show_tree(walker, ast):
             stream = walker.showast
         else:
             stream = sys.stdout
-        if (isinstance(walker.showast, dict) and walker.showast.get("Full", False)
-            and hasattr(walker, "str_with_template")):
+        if (
+            isinstance(walker.showast, dict)
+            and walker.showast.get("Full", False)
+            and hasattr(walker, "str_with_template")
+        ):
             walker.str_with_template(ast)
         else:
             stream.write(str(ast))

uncompyle6/util.py

@@ -5,10 +5,12 @@
 from math import copysign
 from uncompyle6 import PYTHON_VERSION
 
+
 def is_negative_zero(n):
     """Returns true if n is -0.0"""
     return n == 0.0 and copysign(1, n) == -1
 
+
 def better_repr(v, version):
     """Work around Python's unorthogonal and unhelpful repr() for primitive float
     and complex."""
@@ -41,7 +43,6 @@ def better_repr(v, version):
         else:
             return s
     elif isinstance(v, list):
-        l = better_repr(v)
         if len(v) == 1:
             return "[%s,]" % better_repr(v[0], version)
         return "[%s]" % ", ".join(better_repr(i) for i in v)

uncompyle6/version.py

@@ -12,4 +12,4 @@
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
 # This file is suitable for sourcing inside POSIX shell as
 # well as importing into Python
-VERSION="3.7.2"  # noqa
+VERSION="3.7.4"  # noqa