Get ready for release 3.7.3

Note: this is an upheaval because we need to pass "code" or at least
"code.co_consts" to the docstring detection routine

.circleci/config.yml

@@ -36,10 +36,11 @@ jobs:
           # fallback to using the latest cache if no exact match is found
           - v2-dependencies-
 
-    # This is based on your 1.0 configuration file or project settings
-    - run: pip install --user --upgrade setuptools
-    - run: pip install --user -e .
-    - run: pip install --user -r requirements-dev.txt
+    - run:
+        command: | # Use pip to install dependengcies
+          pip install --user --upgrade setuptools
+          pip install --user -e .
+          pip install --user -r requirements-dev.txt
 
     # Save dependency cache
     - save_cache:

.github/FUNDING.yml

@@ -0,0 +1,12 @@
+# These are supported funding model platforms
+
+github: [rocky]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

.github/ISSUE_TEMPLATE/bug-report.md

@@ -4,28 +4,40 @@ about: Tell us about uncompyle6 bugs
 
 ---
 
-<!-- __Note:__ Have you read https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md ?
+<!-- __Note:__ Bugs are not for asking questions about a problem you
+are trying to solve that involve the use of uncompyle6 along the way,
+although I may be more tolerent of this if you sponsor the project.
+
+Also, the unless you are a sponsor of the project, it may take a
+while, maybe a week or so, before the bug report is noticed, let alone
+acted upon.
+
+To set expectations, some legitimate bugs can take years
+to fix, but they eventually do get fixed. Funding the project was
+added to address the problem that there are lots of people seeking
+help and reporting bugs, but few people who are willing or capable of
+providing help or fixing bugs.
+
+Finally, have you read https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md
+?
+
 
 Please remove any of the optional sections if they are not applicable.
 
-Prerequisites
+Prerequisites/Caveats
 
 * Make sure the bytecode you have can be disassembled with a
   disassembler and produces valid results.
 * Don't put bytecode and corresponding source code on any service that
   requires registration to download.
-* When you open a bug report there is no privacy. If the legitimacy of
-  the activity is deemed suspicous, I may flag it as suspicious,
+* When you open a bug report there is no privacy. If you need privacy, then
+  contact me by email and explain who you are and the need for privacy.
+  But be mindful that you may be asked to sponsor the project for the
+  personal and private help that you are requesting.
+* If the legitimacy of the activity is deemed suspicous, I may flag it as suspicious,
   making the issue even more easy to detect.
 
-Bug reports that violate a prerequisite may be discarded.
-
-Note that there are way more bug-fix requestors than there are bug
-fixers. If you want you need more immediate, confidential or urgent
-assistance
-
-http://www.crazy-compilers.com/decompyle/ offers a byte-code
-decompiler service for versions of Python up to 2.6.
+Bug reports that violate the above may be discarded.
 
 -->
 

.gitignore

@@ -10,6 +10,7 @@
 /.pytest_cache
 /.python-version
 /.tox
+.mypy_cache
 /.venv*
 /README
 /__pkginfo__.pyc

HOW-TO-REPORT-A-BUG.md

@@ -30,31 +30,9 @@ files that I know of that will cause problems. See, for example, the
 list in
 [`test/stdlib/runtests.sh`](https://github.com/rocky/python-uncompyle6/blob/master/test/stdlib/runtests.sh).
 
-But I understand: you would the bugs _you_ encounter addressed before
-all the other known bugs.
+There are far more bug reporters than there are bug fixers.
 
-From my standpoint, the good thing about the bugs listed in
-`runtests.sh` is that each test case is small and isolated to a single
-kind of problem. And I'll tend to fix easier, more isolated cases than
-generic "something's wrong" kinds of bugs where I'd have to do a bit
-of work to figure out what's up, if not use some sort of mind reading,
-make some guesses, and perform some experiments to see if the guesses
-are correct. I can't read minds, nor am I into guessing games; I'd
-rather devote the effort spent instead towards fixing bugs that are
-precisely defined.
-
-And it often turns out that by just fixing the well-defined and
-prescribed cases, the ill-defined amorphous cases as well will get
-handled as well.
-
-In sum, you may need to do some work to have the bug you have found
-handled before the hundreds of other bugs, and other things I could be
-doing.
-
-No one is getting paid to work to work on this project, let alone the
-bugs you may have an interest in. If you require decompiling bytecode
-immediately, consider using a decompilation service, listed further
-down in this document.
+Unless you are a sponsor of this project, it may take a while, maybe a week or so, before the bug report is noticed, let alone acted upon. Things eventually get fixed, but it may take years.
 
 # Is it really a bug?
 
@@ -176,11 +154,7 @@ provide the input command and the output from that, please give:
 
 ## But I don't *have* the source code!
 
-Sure, I get it. No problem. There is Python assembly code on parse
-errors, so simply by hand decompile that. To get a full disassembly,
-use `pydisasm` from the [xdis](https://pypi.python.org/pypi/xdis)
-package. Opcodes are described in the documentation for
-the [dis](https://docs.python.org/3.6/library/dis.html) module.
+There is Python assembly code on parse errors, so simply by hand decompile that. To get a full disassembly, use `pydisasm` from the [xdis](https://pypi.python.org/pypi/xdis) package. Opcodes are described in the documentation for the[dis](https://docs.python.org/3.6/library/dis.html) module.
 
 ### But I don't *have* the source code and am incapable of figuring how to do a hand disassembly!
 
@@ -189,7 +163,7 @@ disassemble Python bytecode. And as Richard Feynman once said, "What
 one fool can learn, so can another."
 
 If this is too difficult, or too time consuming, or not of interest to
-you, then perhaps what require is a decompilation service. [Crazy
+you, then you might consider sponsoring the project. [Crazy
 Compilers](http://www.crazy-compilers.com/decompyle/) offers a
 byte-code decompiler service for versions of Python up to 2.6. (If
 there are others around let me know and I'll list them here.)
@@ -217,10 +191,7 @@ likely the problem will be fixed and fixed sooner.
 # Karma
 
 I realize that following the instructions given herein puts a bit of
-burden on the bug reporter. In my opinion, this is justified as
-attempts to balance somewhat the burden and effort needed to fix the
-bug and the attempts to balance number of would-be bug reporters with
-the number of bug fixers. Better bug reporters are more likely to move
+burden on the bug reporter. This is justified as attempts to balance somewhat the burden and effort needed to fix the bug and the attempts to balance number of would-be bug reporters with the number of bug fixers. Better bug reporters are more likely to move
 in the category of bug fixers.
 
 The barrier to reporting a big is pretty small: all you really need is
@@ -228,22 +199,14 @@ a github account, and the ability to type something after clicking
 some buttons. So the reality is that many people just don't bother to
 read these instructions, let alone follow it to any simulacrum.
 
-And the reality is also that bugs sometimes get fixed even though
-these instructions are not followed.
+That said, bugs sometimes get fixed even though these instructions are not followed.
 
-So one factors I may take into consideration is the bug reporter's karma.
+I may take into consideration is the bug reporter's karma.
+
+* Have you demonstrably contributed to open source? I may look at your  github profile to see what contributions you have made, how popular those contributions are, or how popular you are.
+* How appreciative are you? Have you starred this project that you are   seeking help from? Have you starred _any_ github project? And the above  two kind of feed into ...
+* Attitude. Some people feel that they are doing me and the world a great favor by just pointing out that there is a problem whose solution would greatly benefit them. (This might account partially those that have this attitude often don't read or follow instructions such as those given here.)
 
-* Have you demonstrably contributed to open source? I may look at your
-  github profile to see what contributions you have made, how popular
-  those contributions are, or how popular you are.
-* How appreciative are you? Have you starred this project that you are
-  seeking help from? Have you starred _any_ github project? And the above
-  two kind of feed into ...
-* Attitude. Some people feel that they are doing me and the world a
-  great favor by just pointing out that there is a problem whose solution
-  would greatly benefit them. Perhaps this is why they feel that
-  instructions are not to be followed by them, nor any need for
-  showing evidence gratitude when help is offered them.
 
 # Confidentiality of Bug Reports
 
@@ -255,6 +218,8 @@ remains would not be an issue.
 However feel free to remove any comments, and modify variable names
 or constants in the source code.
 
+If there is some legitimate reason to keep confidentiality, you can contact me by email to explain the extenuating circumstances. However I tend to discard without reading anonymous email.
+
 # Ethics
 
 I do not condone using this program for unethical or illegal purposes.

NEWS.md

@@ -1,3 +1,47 @@
+3.7.3: 2020-7-25
+================
+
+Mostly small miscellaneous bug fixes
+
+* `__doc__ = DocDescr()` from `test_descr.py` was getting confused as a docstring.
+* detect 2.7 exchandler range better
+* Add for .. else reduction checks on 2.6 and before
+* Add reduce check for 2.7 augmented assign
+* Add VERSION in a pydoc-friendly way
+
+
+3.7.2: 2020-6-27
+================
+
+* Use newer xdis
+* Docstrings (again) which were broken again on earlier Python
+* Fix 2.6 and 2.7 decompilation bug in handling "list if" comprehensions
+
+
+
+3.7.1: 2020-6-12 Fleetwood66
+====================================================
+
+Released to pick up new xdis version which has fixes to read bytestings better on 3.x
+
+* Handle 3.7+ "else" branch removal adAs seen in `_cmp()` of `python3.8/distutils/version.py` with optimization `-O2`
+* 3.6+ "with" and "with .. as" grammar improvements
+* ast-check for "for" loop was missing some grammar rules
+
+3.7.0: 2020-5-19 Primidi 1st Prairial - Alfalfa - HF
+====================================================
+
+The main impetus for this release is to pull in the recent changes from xdis.
+We simplify imports using xdis 4.6.0.
+
+There were some bugfixes to Python 3.4-3.8. See the ChangeLog for details
+
+
+3.6.7: 2020-4-27 xdis again
+===========================
+
+More upheaval in xdis which we need to track here.
+
 3.6.6: 2020-4-20 Love in the time of Cholera
 ============================================
 

README.rst

@@ -68,10 +68,9 @@ are syntactically correct by running the Python interpreter for that
 bytecode version.  Finally, in cases where the program has a test for
 itself, we can run the check on the decompiled code.
 
-We are serious about testing, and use automated processes to find
-bugs. In the issue trackers for other decompilers, you will find a
-number of bugs we've found along the way. Very few to none of them are
-fixed in the other decompilers.
+We use an automated processes to find bugs. In the issue trackers for
+other decompilers, you will find a number of bugs we've found along
+the way. Very few to none of them are fixed in the other decompilers.
 
 Requirements
 ------------
@@ -172,15 +171,11 @@ All of the Python decompilers that I have looked at have problems
 decompiling Python's control flow. In some cases we can detect an
 erroneous decompilation and report that.
 
-Python support is strongest in Python 2 for 2.7 and drops off as you
-get further away from that. Support is also probably pretty good for
-python 2.3-2.4 since a lot of the goodness of early the version of the
-decompiler from that era has been preserved (and Python compilation in
-that era was minimal)
+Python support is pretty good for Python 2
 
-There is some work to do on the lower end Python versions which is
-more difficult for us to handle since we don't have a Python
-interpreter for versions 1.6, and 2.0.
+On the lower end of Python versions, decompilation seems pretty good although
+we don't have any automated testing in place for Python's distributed tests.
+Also, we don't have a Python interpreter for versions 1.6, and 2.0.
 
 In the Python 3 series, Python support is is strongest around 3.4 or
 3.3 and drops off as you move further away from those versions. Python
@@ -195,7 +190,7 @@ added. So in sum handling control flow by ad hoc means as is currently
 done is worse.
 
 Between Python 3.5, 3.6, 3.7 there have been major changes to the
-:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.  Python
+:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.
 
 Python 3.8 removes :code:`SETUP_LOOP`, :code:`SETUP_EXCEPT`,
 :code:`BREAK_LOOP`, and :code:`CONTINUE_LOOP`, instructions which may
@@ -211,26 +206,52 @@ however that the magic of a released version is usually the same as
 the *last* candidate version prior to release.
 
 There are also customized Python interpreters, notably Dropbox,
-which use their own magic and encrypt bytcode. With the exception of
+which use their own magic and encrypt bytecode. With the exception of
 the Dropbox's old Python 2.5 interpreter this kind of thing is not
 handled.
 
-We also don't handle PJOrion_ obfuscated code. For that try: PJOrion
-Deobfuscator_ to unscramble the bytecode to get valid bytecode before
-trying this tool. This program can't decompile Microsoft Windows EXE
-files created by Py2EXE_, although we can probably decompile the code
-after you extract the bytecode properly. For situations like this, you
-might want to consider a decompilation service like `Crazy Compilers
-<http://www.crazy-compilers.com/decompyle/>`_.  Handling
-pathologically long lists of expressions or statements is slow.
+We also don't handle PJOrion_ or otherwise obfuscated code. For
+PJOrion try: PJOrion Deobfuscator_ to unscramble the bytecode to get
+valid bytecode before trying this tool. This program can't decompile
+Microsoft Windows EXE files created by Py2EXE_, although we can
+probably decompile the code after you extract the bytecode
+properly. Handling pathologically long lists of expressions or
+statements is slow. We don't handle Cython_ or MicroPython which don't
+use bytecode.
 
+There are numerous bugs in decompilation. And that's true for every
+other CPython decompiler I have encountered, even the ones that
+claimed to be "perfect" on some particular version like 2.4.
 
-There is lots to do, so please dig in and help.
+As Python progresses decompilation also gets harder because the
+compilation is more sophisticated and the language itself is more
+sophisticated. I suspect that attempts there will be fewer ad-hoc
+attempts like unpyc37_ (which is based on a 3.3 decompiler) simply
+because it is harder to do so. The good news, at least from my
+standpoint, is that I think I understand what's needed to address the
+problems in a more robust way. But right now until such time as
+project is better funded, I do not intend to make any serious effort
+to support Python versions 3.8 or 3.9, including bugs that might come
+in. I imagine at some point I may be interested in it.
+
+You can easily find bugs by running the tests against the standard
+test suite that Python uses to check itself. At any given time, there are
+dozens of known problems that are pretty well isolated and that could
+be solved if one were to put in the time to do so. The problem is that
+there aren't that many people who have been working on bug fixing.
+
+Some of the bugs in 3.7 and 3.8 are simply a matter of back-porting
+the fixes in decmopyle3.
+
+You may run across a bug, that you want to report. Please do so. But
+be aware that it might not get my attention for a while. If you
+sponsor or support the project in some way, I'll prioritize your
+issues above the queue of other things I might be doing instead.
 
 See Also
 --------
 
-* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back ehre.
+* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back here.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
 * https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details
@@ -242,6 +263,7 @@ See Also
 * https://github.com/zrax/pycdc : The README for this C++ code says it aims to support all versions of Python. It is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
 
 
+.. _Cython: https://en.wikipedia.org/wiki/Cython
 .. _trepan: https://pypi.python.org/pypi/trepan2g
 .. _compiler: https://pypi.python.org/pypi/spark_parser
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md

__pkginfo__.py

@@ -21,6 +21,17 @@
 # less elegant than having it here with reduced code, albeit there
 # still is some room for improvement.
 
+# Python-version | package | last-version |
+# -----------------------------------------
+# 2.5            | pip     |  1.1         |
+# 2.6            | pip     |  1.5.6       |
+# 2.7            | pip     | 19.2.3       |
+# 2.7            | pip     |  1.2.1       |
+# 3.1            | pip     |  1.5.6       |
+# 3.2            | pip     |  7.1.2       |
+# 3.3            | pip     | 10.0.1       |
+# 3.4            | pip     | 19.1.1       |
+
 # Things that change more often go here.
 copyright   = """
 Copyright (C) 2015-2020 Rocky Bernstein <rb@dustyfeet.com>.
@@ -58,7 +69,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ["spark-parser >= 1.8.9, < 1.9.0",
-                      "xdis >= 4.4.0, < 4.5.0"]
+                      "xdis >= 4.7.0, <5.1.0"]
 
 license            = "GPL3"
 mailing_list       = "python-debugger@googlegroups.com"

admin-tools/how-to-make-a-release.md

@@ -2,17 +2,17 @@
 **Table of Contents**
 
 - [Get latest sources:](#get-latest-sources)
-- [Change version in uncompyle6/version.py](#change-version-in-uncompyle6versionpy)
+- [Change version in uncompyle6/version.py:](#change-version-in-uncompyle6versionpy)
 - [Update ChangeLog:](#update-changelog)
-- [Update NEWS from ChangeLog:](#update-news-from-changelog)
+- [Update NEWS.md from ChangeLog:](#update-newsmd-from-changelog)
 - [Make sure pyenv is running and check newer versions](#make-sure-pyenv-is-running-and-check-newer-versions)
 - [Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.](#switch-to-python-24-sync-that-up-and-build-that-first-since-it-creates-a-tarball-which-we-dont-want)
-- [Update NEWS from master branch](#update-news-from-master-branch)
-- [Check against all versions](#check-against-all-versions)
+- [Check against older versions](#check-against-older-versions)
 - [Make packages and tag](#make-packages-and-tag)
-- [Upload single package and look at Rst Formating](#upload-single-package-and-look-at-rst-formating)
-- [Upload rest of versions](#upload-rest-of-versions)
-- [Push tags:](#push-tags)
+- [Check package on github](#check-package-on-github)
+- [Release on Github](#release-on-github)
+- [Get onto PyPI](#get-onto-pypi)
+- [Update tags:](#update-tags)
 
 <!-- markdown-toc end -->
 # Get latest sources:
@@ -39,7 +39,7 @@
 
 # Make sure pyenv is running and check newer versions
 
-    $ pyenv local && source admin-tools/check-newer-versions.sh
+    $ admin-tools/check-newer-versions.sh
 
 # Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.
 
@@ -50,39 +50,46 @@
 
 # Check against older versions
 
-    $ source admin-tools/check-older-versions.sh
+    $ admin-tools/check-older-versions.sh
 
 # Make packages and tag
 
     $ . ./admin-tools/make-dist-older.sh
-	$ pyenv local 3.8.2
+	$ pyenv local 3.8.4
 	$ twine check dist/uncompyle6-$VERSION*
-    $ git tag release-python-2.4-$VERSION
-	$ twine check dist/uncompyle6-$VERSION*
-    $ . ./admin-tools/make-dist-newer.sh
+    $ ./admin-tools/make-dist-newer.sh
 	$ twine check dist/uncompyle6-$VERSION*
 
-# Upload single package and look at Rst Formating
+# Check package on github
 
-	$ twine check dist/uncompyle6-${VERSION}*
-    $ twine upload dist/uncompyle6-${VERSION}-py3.3.egg
+	$ [[ ! -d /tmp/gittest ]] && mkdir /tmp/gittest; pushd /tmp/gittest
+	$ pyenv local 3.8.3
+	$ pip install -e git://github.com/rocky/python-uncompyle6.git#egg=uncompyle6
+	$ uncompyle6 --help
+	$ pip uninstall uncompyle6
+	$ popd
 
-# Upload rest of versions
-
-    $ twine upload dist/uncompyle6-${VERSION}*
+# Release on Github
 
 Goto https://github.com/rocky/python-uncompyle6/releases
 
-# Push tags:
+Now check the *tagged* release. (Checking the untagged release was previously done).
+
+Todo: turn this into a script in `admin-tools`
+
+	$ pushd /tmp/gittest
+	$ pip install -e git://github.com/rocky/python-uncompyle6.git@$VERSION#egg=uncompyle6
+	$ uncompyle6 --help
+	$ pip uninstall uncompyle6
+	$ popd
+
+
+# Get onto PyPI
+
+    $ twine upload dist/uncompyle6-${VERSION}*
+
+
+# Update tags:
 
     $ git push --tags
-
-# Check on a VM
-
-    $ cd /virtual/vagrant/virtual/vagrant/ubuntu-zesty
-	$ vagrant up
-	$ vagrant ssh
-	$ pyenv local 3.5.2
-	$ pip install --upgrade uncompyle6
-	$ exit
-	$ vagrant halt
+    $ git pull --tags

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.17 3.2.6 3.1.5 3.4.10 3.7.7 3.8.2'
+export PYVERSIONS='3.5.9 3.6.10 2.6.9 3.3.7 2.7.18 3.2.6 3.1.5 3.4.10 3.7.7 3.8.4'

appveyor.yml

@@ -53,6 +53,7 @@ install:
   # compiled extensions and are not provided as pre-built wheel packages,
   # pip will build them from source using the MSVC compiler matching the
   # target Python version and architecture
+  - "%CMD_IN_ENV% pip install git+git://github.com/rocky/python-uncompyle6.git#egg=uncompyle6-3.6.6"
   - "%CMD_IN_ENV% pip install -r requirements.txt"
 
 build_script:

pytest/validate.py

@@ -12,8 +12,7 @@ import functools
 from uncompyle6 import PYTHON_VERSION, PYTHON3, IS_PYPY, code_deparse
 
 # TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
-from xdis.bytecode import Bytecode
-from xdis.main import get_opcode
+from xdis import Bytecode, get_opcode
 
 opc = get_opcode(PYTHON_VERSION, IS_PYPY)
 Bytecode = functools.partial(Bytecode, opc=opc)

setup.py

@@ -4,40 +4,54 @@ import sys
 """Setup script for the 'uncompyle6' distribution."""
 
 SYS_VERSION = sys.version_info[0:2]
-if not ((2, 6) <= SYS_VERSION  <= (3, 9)):
+if not ((2, 6) <= SYS_VERSION <= (3, 9)):
     mess = "Python Release 2.6 .. 3.9 are supported in this code branch."
-    if ((2, 4) <= SYS_VERSION <= (2, 7)):
-        mess += ("\nFor your Python, version %s, use the python-2.4 code/branch." %
-                 sys.version[0:3])
+    if (2, 4) <= SYS_VERSION <= (2, 7):
+        mess += (
+            "\nFor your Python, version %s, use the python-2.4 code/branch."
+            % sys.version[0:3]
+        )
     elif SYS_VERSION < (2, 4):
-        mess += ("\nThis package is not supported for Python version %s."
-                 % sys.version[0:3])
+        mess += (
+            "\nThis package is not supported for Python version %s." % sys.version[0:3]
+        )
     print(mess)
     raise Exception(mess)
 
-from __pkginfo__ import \
-    author,           author_email,       install_requires,          \
-    license,          long_description,   classifiers,               \
-    entry_points,     modname,            py_modules,                \
-    short_desc,       VERSION,            web,                       \
-    zip_safe
+from __pkginfo__ import (
+    author,
+    author_email,
+    install_requires,
+    license,
+    long_description,
+    classifiers,
+    entry_points,
+    modname,
+    py_modules,
+    short_desc,
+    VERSION,
+    web,
+    zip_safe,
+)
 
 from setuptools import setup, find_packages
+
 setup(
-       author             = author,
-       author_email       = author_email,
-       classifiers        = classifiers,
-       description        = short_desc,
-       entry_points       = entry_points,
-       install_requires   = install_requires,
-       license            = license,
-       long_description   = long_description,
-       long_description_content_type = "text/x-rst",
-       name               = modname,
-       packages           = find_packages(),
-       py_modules         = py_modules,
-       test_suite         = 'nose.collector',
-       url                = web,
-       tests_require      = ['nose>=1.0'],
-       version            = VERSION,
-       zip_safe           = zip_safe)
+    author=author,
+    author_email=author_email,
+    classifiers=classifiers,
+    description=short_desc,
+    entry_points=entry_points,
+    install_requires=install_requires,
+    license=license,
+    long_description=long_description,
+    long_description_content_type="text/x-rst",
+    name=modname,
+    packages=find_packages(),
+    py_modules=py_modules,
+    test_suite="nose.collector",
+    url=web,
+    tests_require=["nose>=1.0"],
+    version=VERSION,
+    zip_safe=zip_safe,
+)

test/.gitignore

@@ -1,3 +1,8 @@
 /.coverage
 /.python-version
 /nohup.out
+/pycdc
+/test_pycdc_tests.sh
+/test_uncompyle2.py
+/test_unpy33.py
+/test_unpy37.py

test/add-test.py

@@ -3,14 +3,22 @@
 """
 import os, sys, py_compile
 
-assert len(sys.argv) >= 2
+assert (2 <= len(sys.argv) <= 4)
 version = sys.version[0:3]
+vers = sys.version_info[:2]
 if sys.argv[1] in ("--run", "-r"):
     suffix = "_run"
     py_source = sys.argv[2:]
+    i = 2
 else:
     suffix = ""
     py_source = sys.argv[1:]
+    i = 1
+try:
+    optimize = int(sys.argv[-1])
+    py_source = sys.argv[i:-1]
+except:
+    optimize = 2
 
 for path in py_source:
     short = os.path.basename(path)
@@ -19,6 +27,10 @@ for path in py_source:
     else:
         cfile = "bytecode_%s%s/%s" % (version, suffix, short) + "c"
     print("byte-compiling %s to %s" % (path, cfile))
-    py_compile.compile(path, cfile)
-    if isinstance(version, str) or version >= (2, 6, 0):
+    optimize = optimize
+    if vers > (3, 1):
+        py_compile.compile(path, cfile, optimize=optimize)
+    else:
+        py_compile.compile(path, cfile)
+    if vers >= (2, 6):
         os.system("../bin/uncompyle6 -a -T %s" % cfile)

test/run-and-email.sh

@@ -51,7 +51,7 @@ for VERSION in $PYVERSIONS ; do
     LOGFILE=/tmp/${MAIN}-$VERSION-$$.log
 
     case "$VERSION" in
-	3.7.7 | 3.8.2 | 3.1.5 | 3.0.1 )
+	3.7.7 | 3.8.3 | 3.1.5 | 3.0.1 )
 	    continue
 	    ;;
 	3.5.9 )

test/simple-decompile-code-test.py

@@ -3,7 +3,7 @@
 from __future__ import print_function
 
 from uncompyle6.main import decompile
-from xdis.magics import sysinfo2float
+from xdis import sysinfo2float
 import sys, inspect
 
 def uncompyle_test():

test/simple_source/bug14/01_for_else_try_else.py

@@ -0,0 +1,17 @@
+# Adapted from 1.4 anydbm
+"""This program is self-checking!"""
+def scan(items):
+    for i in items:
+            try:
+                    5 / i
+            except:
+                    continue
+            else:
+                    break
+    else:
+            return 2
+    return i
+
+assert scan((0, 1)) == 1
+assert scan((0, 0)) == 2
+assert scan((3, 2, 1)) == 3

test/simple_source/bug27+/01_module_doc.py

@@ -0,0 +1,9 @@
+# From 2.7.17 test_bdb.py
+# The problem was detecting a docstring at the begining of the module
+# It must be detected and change'd or else the "from __future__" below
+# is invalid.
+# Note that this has to be compiled with optimation < 2 or else optimization
+# will remove the docstring
+"""Rational, infinite-precision, real numbers."""
+
+from __future__ import division

test/simple_source/bug27+/03_doc_assign.py

@@ -0,0 +1,27 @@
+# From 2.7 test_descr.py
+# Testing __doc__ descriptor...
+# The bug in decompilation was erroneously matching
+# __doc__ = as a docstring
+
+"""This program is self-checking!"""
+
+def test_doc_descriptor():
+    # Testing __doc__ descriptor...
+    # Python SF bug 542984
+    class DocDescr(object):
+        def __get__(self, object, otype):
+            if object:
+                object = object.__class__.__name__ + ' instance'
+            if otype:
+                otype = otype.__name__
+            return 'object=%s; type=%s' % (object, otype)
+    class OldClass:
+        __doc__ = DocDescr()
+    class NewClass(object):
+        __doc__ = DocDescr()
+    assert OldClass.__doc__ == 'object=None; type=OldClass'
+    assert OldClass().__doc__ == 'object=OldClass instance; type=OldClass'
+    assert NewClass.__doc__ == 'object=None; type=NewClass'
+    assert NewClass().__doc__ == 'object=NewClass instance; type=NewClass'
+
+test_doc_descriptor()

test/simple_source/bug33/02_pos_args.py

@@ -1,6 +1,8 @@
+# Python 3.3+
+#
 # From Python 3.3.6 hmac.py
 # Problem was getting wrong placement of positional args.
-# In 3.6+ paramter handling changes
+# In 3.6+ parameter handling changes
 
 # RUNNABLE!
 

test/simple_source/bug35/04_call_function.py

@@ -2,6 +2,7 @@
 # Bug was code not knowing which Python versions
 # have kwargs coming before positional args in code.
 
+"""This program is self-checking!"""
 # RUNNABLE!
 
 def tometadata(self, metadata, schema, Table, args, name=None):

test/simple_source/bug36/02_var_annotate.py

@@ -1,6 +1,9 @@
 # 3.6+ type annotations on variables
 from typing import List
 
+# This test program is part of the uncompyle6 test suite
+# tests STORE_ANNOTATION and SETUP_ANOTATIONS
+
 # RUNNABLE!
 y = 2
 x: bool

test/simple_source/bug37/03_else_removal.py

@@ -0,0 +1,12 @@
+# From python3.8/distutils/version.py with optimization -O2
+# The bug was that the other "else" constant propagated removed.
+
+# NOTE: this program needs to be compile with optimization
+def _cmp (b, c):
+    if b:
+        if c:
+            return 0
+        else:
+            return 1
+    else:
+        assert False, "never get here"

test/simple_source/bug37/04_async.py

@@ -1,23 +0,0 @@
-# from 3.7 test_contextlib_async.py
-# Bugs were not adding "async" when a function is a decorator,
-# and a misaligment when using "async with as".
-@_async_test
-async def test_enter(self):
-    self.assertIs(await manager.__aenter__(), manager)
-
-    async with manager as context:
-        async with woohoo() as x:
-            x = 1
-            y = 2
-        assert manager is context
-
-# From 3.7.6 test_coroutines.py
-# Bug was different form of code for "async with" below
-class CoroutineTest():
-    def test_with_8(self):
-        CNT = 0
-        async def foo():
-            nonlocal CNT
-            async with CM():
-                CNT += 1
-                return

test/simple_source/bug37/10_async.py

@@ -0,0 +1,70 @@
+# from 3.7 test_contextlib_async.py
+# Bugs were not adding "async" when a function is a decorator,
+# and a misaligment when using "async with ... as".
+
+"""This program is self-checking!"""
+import asyncio
+from contextlib import asynccontextmanager, AbstractAsyncContextManager
+import functools
+
+
+def _async_test(func):
+    """Decorator to turn an async function into a test case."""
+
+    @functools.wraps(func)
+    def wrapper(*args, **kwargs):
+        coro = func(*args, **kwargs)
+        loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(loop)
+        try:
+            return loop.run_until_complete(coro)
+        finally:
+            loop.close()
+            asyncio.set_event_loop(None)
+
+    return wrapper
+
+
+state = []
+
+
+@asynccontextmanager
+async def woohoo():
+    state.append(1)
+    yield 42
+    state.append(999)
+
+
+@_async_test
+async def test_enter():
+    class DefaultEnter(AbstractAsyncContextManager):
+        async def __aexit__(*args):
+            return
+            # await super().__aexit__(*args)
+
+    manager = DefaultEnter()
+    got_manager = await manager.__aenter__()
+    # print(got_manager, manager)
+    assert got_manager is manager
+
+    async with manager as context:
+        async with woohoo() as x:
+            x = 1
+            y = 2
+        assert manager is context
+
+
+# From 3.7.6 test_coroutines.py
+# Bug was different form of code for "async with" below
+class CoroutineTest:
+    def test_with_8(self):
+        CNT = 0
+
+        async def foo():
+            nonlocal CNT
+            async with CM():
+                CNT += 1
+                return
+
+
+test_enter()

test/simple_source/stmts/04_withas.py

@@ -16,3 +16,25 @@ def withas_bug(self, nested, a, b):
     with self.assertRaises(ZeroDivisionError):
         with nested(a(), b()) as (x, y):
                 1 // 0
+
+# From 3.7.7 test_functools.py
+# Bug is a unreachable code after "return"
+def test_invalid_registrations(x):
+    return
+    with x:
+        x = 1
+
+# From 3.7.7 test_re.py
+# Bug was hooking in c_with.
+def test_re_tests(tests):
+    for t in tests:
+        with a:
+            continue
+
+# Adapted from 3.8 distutils/command/config.py
+# In 3.8 the problem was in handling "with .. as" code
+def _gen_temp_sourcefile(x, a, headers, lang):
+    with x as y:
+        if a:
+            y = 2
+    return 5

test/stdlib/.gitignore

@@ -1 +1,3 @@
 /.python-version
+/runun33.sh
+/runun7.sh

test/stdlib/2.4-exclude.sh

@@ -45,7 +45,6 @@ SKIP_TESTS=(
     [test_grammar.py]=1    # Too many stmts. Handle large stmts
     [test_grp.py]=1      # Long test - might work Control flow?
     [test_pep247.py]=1 # Long test - might work? Control flow?
-    [test_pwd.py]=1 # Long test - might work? Control flow?
     [test_socketserver.py]=1 # -- test takes too long to run: 40 seconds
     [test_threading.py]=1 # test takes too long to run: 11 seconds
     [test_thread.py]=1 # test takes too long to run: 36 seconds

test/stdlib/2.5-exclude.sh

@@ -55,12 +55,9 @@ SKIP_TESTS=(
     [test_ossaudiodev.py]=1 # it fails on its own
     [test_pdb.py]=1        # Line-number specific
     [test_pep277.py]=1    # it fails on its own
-    [test_pep352.py]=1     # Investigate
     [test_plistlib.py]=1 # it fails on its own
-    [test_pwd.py]=1 # Long test - might work? Control flow?
     [test_pyclbr.py]=1 # Investigate
     [test_rgbimg.py]=1 #  it fails on its own
-    [test_queue.py]=1 # Control flow?
     [test_scriptpackages.py]=1 # it fails on its own
     [test_socketserver.py]=1 # Too long to run - 42 seconds
     [test_sqlite.py]=1 # it fails on its own

test/stdlib/2.6-exclude.sh

@@ -1,4 +1,11 @@
 SKIP_TESTS=(
+    # ifelsestmt is borked in:
+    #   if filename == 'srcfile':
+    #      return srcfile
+    #   if filename == 'destfile':
+    #      return destfile
+    #   assert 0  # shouldn't reach here.
+    [test_shutil.py]=1
 
     [test___all__.py]=1  # it fails on its own
     [test___all__.py]=1 # it fails on its own
@@ -52,13 +59,12 @@ SKIP_TESTS=(
     [test_ossaudiodev.py]=1 # it fails on its own
 
     [test_pep277.py]=1 # it fails on its own
-    [test_pep352.py]=1     # Investigate
     [test_pyclbr.py]=1 # Investigate
-    [test_pwd.py]=1 # Long test - might work? Control flow?
     [test_py3kwarn.py]=1 # it fails on its own
 
     [test_scriptpackages.py]=1 # it fails on its own
     [test_select.py]=1 # test takes too long to run: 11 seconds
+
     [test_socket.py]=1 # test takes too long to run: 12 seconds
     [test_startfile.py]=1 # it fails on its own
     [test_structmembers.py]=1 # it fails on its own

test/stdlib/2.7-exclude.sh

@@ -19,7 +19,6 @@ SKIP_TESTS=(
     [test_modulefinder.py]=1 # FIX
     [test_multiprocessing.py]=1 # On uncompyle2, takes 24 secs
     [test_poll.py]=1  # test takes too long to run: 11 seconds
-    [test_pwd.py]=1     # Takes too long
     [test_regrtest.py]=1 #
     [test_runpy.py]=1   # Long and fails on its own
     [test_select.py]=1  # Runs okay but takes 11 seconds

test/stdlib/3.2-exclude.sh

@@ -17,8 +17,6 @@ SKIP_TESTS=(
     [test_peepholer.py]=1
     [test_pep352.py]=1
 
-    [test_quopri.py]=1 # TypeError: Can't convert 'bytes' object to str implicitly
-
     [test_runpy.py]=1
 
     [test_ssl.py]=1 # too installation specific
@@ -35,5 +33,4 @@ if (( BATCH )) ; then
     # Fails in crontab environment?
     # Figure out what's up here
     SKIP_TESTS[test_exception_variations.py]=1
-    SKIP_TESTS[test_quopri.py]=1
 fi

test/stdlib/3.3-exclude.sh

@@ -1,4 +1,15 @@
 SKIP_TESTS=(
+    # FIXME: Did this work sometime in the past ?
+    #   for elem in g(s):
+    #     if not tgt and isOdd(elem): continue
+    # is erroneously:
+    #   for elem in g(s):
+    #    if tgt or isOdd(elem):
+    #       pass
+    #    else:
+    #       tgt.append(elem)
+    [test_itertools.py]=1
+
     [test_buffer.py]=1  # FIXME: Works on c90ff51
     [test_cmath.py]=1  # FIXME: Works on c90ff51
 

test/stdlib/3.4-exclude.sh

@@ -1,4 +1,15 @@
 SKIP_TESTS=(
+    # FIXME: Did this work sometime in the past ?
+    #   for elem in g(s):
+    #     if not tgt and isOdd(elem): continue
+    # is erroneously:
+    #   for elem in g(s):
+    #    if tgt or isOdd(elem):
+    #       pass
+    #    else:
+    #       tgt.append(elem)
+    [test_itertools.py]=1
+
     [test_buffer.py]=1  # FIXME: Works on c90ff51
     [test_cmath.py]=1  # FIXME: Works on c90ff51
     [test_strftime.py]=1  # FIXME: Works on c90ff51
@@ -87,5 +98,4 @@ if (( batch )) ; then
     # Figure out what's up here
     SKIP_TESTS[test_exception_variations.py]=1
     SKIP_TESTS[test_mailbox.py]=1 # Takes to long on POWER; over 15 secs
-    SKIP_TESTS[test_quopri.py]=1
 fi

test/stdlib/3.5-exclude.sh

@@ -138,7 +138,6 @@ if (( BATCH )) ; then
     SKIP_TESTS[test_ioctl.py]=1 # it fails on its own
     SKIP_TESTS[test_poplib.py]=1 # May be a result of POWER installation
 
-    SKIP_TESTS[test_quopri.py]=1
     SKIP_TESTS[test_sysconfig.py]=1 # POWER extension fails
     SKIP_TESTS[test_tarfile.py]=1 # too long to run on POWER 15 secs
     SKIP_TESTS[test_venv.py]=1 # takes too long 11 seconds

test/stdlib/3.6-exclude.sh

@@ -123,8 +123,6 @@ SKIP_TESTS=(
     [test_pyclbr.py]=1 # it fails on its own
     [test_pydoc.py]=1 # it fails on its own
 
-    [test_quopri.py]=1      # AssertionError: b'123=four' != '123=four'
-
     [test_random.py]=1 # it fails on its own
     [test_range.py]=1
     [test_regrtest.py]=1 # test takes too long to run: 12 seconds

test/stdlib/3.7-exclude.sh

@@ -1,4 +1,19 @@
 SKIP_TESTS=(
+    # FIXME: Did this work sometime in the past ?
+    #   for elem in g(s):
+    #     if not tgt and isOdd(elem): continue
+    # is erroneously:
+    #   for elem in g(s):
+    #    if tgt or isOdd(elem):
+    #       pass
+    #    else:
+    #       tgt.append(elem)
+    [test_itertools.py]=1
+
+    # Fails on decompyle3 as well.
+    # complicated control flow and "and/or" expressions
+    [test_pickle.py]=1
+
     [test_builtin.py]=1 # FIXME works on decompyle6
     [test_context.py]=1 # FIXME works on decompyle6
     [test_doctest2.py]=1 # FIXME works on decompyle6
@@ -48,6 +63,7 @@ SKIP_TESTS=(
 
     [test_faulthandler.py]=1   # test takes too long before decompiling
     [test_fileinput.py]=1 # Test assertion failures
+    [test_finalization.py]=1 # if/else logic
     [test_frame.py]=1 # test assertion errors
     [test_ftplib.py]=1 # parse error
     [test_fstring.py]=1 # need to disambiguate leading fstrings from docstrings

test/stdlib/runtests.sh

@@ -78,7 +78,6 @@ case $PYVERSION in
 	    # Fails in crontab environment?
 	    # Figure out what's up here
 	    SKIP_TESTS[test_exception_variations.py]=1
-	    SKIP_TESTS[test_quopri.py]=1
 	fi
 	;;
     3.1)
@@ -91,7 +90,6 @@ case $PYVERSION in
 	    # Fails in crontab environment?
 	    # Figure out what's up here
 	    SKIP_TESTS[test_exception_variations.py]=1
-	    SKIP_TESTS[test_quopri.py]=1
 	fi
 	;;
     3.2)
@@ -131,6 +129,7 @@ fulldir=$(pwd)
 
 # DECOMPILER=uncompyle2
 DECOMPILER=${DECOMPILER:-"$fulldir/../../bin/uncompyle6"}
+OPTS=${OPTS:-""}
 TESTDIR=/tmp/test${PYVERSION}
 if [[ -e $TESTDIR ]] ; then
     rm -fr $TESTDIR
@@ -139,6 +138,8 @@ fi
 PYENV_ROOT=${PYENV_ROOT:-$HOME/.pyenv}
 pyenv_local=$(pyenv local)
 
+echo Python version is $pyenv_local
+
 # pyenv version update
 for dir in ../ ../../ ; do
     cp -v .python-version $dir
@@ -147,18 +148,25 @@ done
 
 mkdir $TESTDIR || exit $?
 cp -r ${PYENV_ROOT}/versions/${PYVERSION}.${MINOR}/lib/python${PYVERSION}/test $TESTDIR
-cd $TESTDIR/test
+if [[ $PYVERSION == 3.2 ]] ; then
+    cp ${PYENV_ROOT}/versions/${PYVERSION}.${MINOR}/lib/python${PYVERSION}/test/* $TESTDIR
+    cd $TESTDIR
+else
+    cd $TESTDIR/test
+fi
 pyenv local $FULLVERSION
 export PYTHONPATH=$TESTDIR
 export PATH=${PYENV_ROOT}/shims:${PATH}
 
+DONT_SKIP_TESTS=${DONT_SKIP_TESTS:-0}
+
 # Run tests
 typeset -i i=0
 typeset -i allerrs=0
 if [[ -n $1 ]] ; then
-    files=$1
-    typeset -a files_ary=( $(echo $1) )
-    if (( ${#files_ary[@]} == 1 )) ; then
+    files=$@
+    typeset -a files_ary=( $(echo $@) )
+    if (( ${#files_ary[@]} == 1 || DONT_SKIP_TESTS == 1 )) ; then
 	SKIP_TESTS=()
     fi
 else
@@ -190,7 +198,7 @@ for file in $files; do
     typeset -i ENDTIME=$(date +%s)
     typeset -i time_diff
     (( time_diff =  ENDTIME - STARTTIME))
-    if (( time_diff > 10 )) ; then
+    if (( time_diff > $timeout )) ; then
 	echo "Skipping test $file -- test takes too long to run: $time_diff seconds"
 	continue
     fi
@@ -202,7 +210,7 @@ for file in $files; do
     $fulldir/compile-file.py $file && \
     mv $file{,.orig} && \
     echo ==========  $(date +%X) Decompiling $file ===========
-    $DECOMPILER $decompiled_file > $file
+    $DECOMPILER $OPTS $decompiled_file > $file
     rc=$?
     if (( rc == 0 )) ; then
 	echo ========== $(date +%X) Running $file ===========

uncompyle6/__init__.py

@@ -28,20 +28,25 @@
 
 import sys
 
-__docformat__ = 'restructuredtext'
+__docformat__ = "restructuredtext"
 
-PYTHON3 = (sys.version_info >= (3, 0))
+from uncompyle6.version import VERSION
+
+# This ensures VERSION will appear in pydoc
+__version__  = VERSION
+
+PYTHON3 = sys.version_info >= (3, 0)
 
 # We do this crazy way to support Python 2.6 which
 # doesn't support version_major, and has a bug in
 # floating point so we can't divide 26 by 10 and get
 # 2.6
 PYTHON_VERSION = sys.version_info[0] + (sys.version_info[1] / 10.0)
-PYTHON_VERSION_STR  = "%s.%s" % (sys.version_info[0], sys.version_info[1])
+PYTHON_VERSION_STR = "%s.%s" % (sys.version_info[0], sys.version_info[1])
 
-IS_PYPY = '__pypy__' in sys.builtin_module_names
+IS_PYPY = "__pypy__" in sys.builtin_module_names
 
-if hasattr(sys, 'setrecursionlimit'):
+if hasattr(sys, "setrecursionlimit"):
     # pyston doesn't have setrecursionlimit
     sys.setrecursionlimit(5000)
 

uncompyle6/disas.py

@@ -34,8 +34,7 @@ from __future__ import print_function
 import sys
 from collections import deque
 
-from xdis import iscode
-from xdis.load import check_object_path, load_module
+from xdis import check_object_path, iscode, load_module
 from uncompyle6.scanner import get_scanner
 
 

uncompyle6/linenumbers.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016, 2818 by Rocky Bernstein
+#  Copyright (c) 2015-2016, 2818, 2020 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -15,14 +15,27 @@
 
 from collections import deque
 
-from xdis import iscode
-from xdis.load import load_file, load_module
-from xdis.main import get_opcode
-from xdis.bytecode import Bytecode, findlinestarts, offset2line
+from xdis import (
+    Bytecode,
+    iscode,
+    findlinestarts,
+    get_opcode,
+    offset2line,
+    load_file,
+    load_module,
+)
+
 
 def line_number_mapping(pyc_filename, src_filename):
-    (version, timestamp, magic_int, code1, is_pypy,
-     source_size, sip_hash) = load_module(pyc_filename)
+    (
+        version,
+        timestamp,
+        magic_int,
+        code1,
+        is_pypy,
+        source_size,
+        sip_hash,
+    ) = load_module(pyc_filename)
     try:
         code2 = load_file(src_filename)
     except SyntaxError as e:
@@ -44,7 +57,10 @@ def number_loop(queue, mappings, opc):
         assert code1.co_name == code2.co_name
         linestarts_orig = findlinestarts(code1)
         linestarts_uncompiled = list(findlinestarts(code2))
-        mappings += [[line, offset2line(offset, linestarts_uncompiled)] for offset, line in linestarts_orig]
+        mappings += [
+            [line, offset2line(offset, linestarts_uncompiled)]
+            for offset, line in linestarts_orig
+        ]
         bytecode1 = Bytecode(code1, opc)
         bytecode2 = Bytecode(code2, opc)
         instr2s = bytecode2.get_instructions(code2)

uncompyle6/main.py

@@ -16,8 +16,7 @@ from __future__ import print_function
 import datetime, py_compile, os, subprocess, sys, tempfile
 
 from uncompyle6 import verify, IS_PYPY, PYTHON_VERSION
-from xdis import iscode
-from xdis.magics import sysinfo2float
+from xdis import iscode, sysinfo2float
 from uncompyle6.disas import check_object_path
 from uncompyle6.semantics import pysource
 from uncompyle6.parser import ParserError
@@ -102,7 +101,7 @@ def decompile(
     )
     if PYTHON_VERSION < 3.0 and bytecode_version >= 3.0:
         write(
-            '# Warning: this version has problems handling the Python 3 "byte" type in constants properly.\n'
+            '# Warning: this version of Python has problems handling the Python 3 "byte" type in constants properly.\n'
         )
 
     if co.co_filename:
@@ -120,12 +119,10 @@ def decompile(
                 mapstream = _get_outstream(mapstream)
 
             deparsed = deparse_code_with_map(
-                bytecode_version,
                 co,
                 out,
-                showasm,
-                showast,
-                showgrammar,
+                bytecode_version,
+                debug_opts,
                 code_objects=code_objects,
                 is_pypy=is_pypy,
             )

uncompyle6/parser.py

@@ -23,14 +23,13 @@ from __future__ import print_function
 
 import sys
 
-from xdis import iscode
-from xdis.magics import py_str2float
+from xdis import iscode, py_str2float
 from spark_parser import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.show import maybe_show_asm
 
 
 class ParserError(Exception):
-    def __init__(self, token, offset, debug):
+    def __init__(self, token, offset, debug=PARSER_DEFAULT_DEBUG):
         self.token = token
         self.offset = offset
         self.debug = debug
@@ -624,7 +623,7 @@ class PythonParser(GenericASTBuilder):
         """
 
 
-def parse(p, tokens, customize):
+def parse(p, tokens, customize, code):
     p.customize_grammar_rules(tokens, customize)
     ast = p.parse(tokens)
     #  p.cleanup()
@@ -879,7 +878,7 @@ def python_parser(
     # parser_debug = {'rules': True, 'transition': True, 'reduce' : True,
     #                 'showstack': 'full'}
     p = get_python_parser(version, parser_debug)
-    return parse(p, tokens, customize)
+    return parse(p, tokens, customize, co)
 
 
 if __name__ == "__main__":

uncompyle6/parsers/parse10.py

@@ -7,7 +7,7 @@ from uncompyle6.parsers.parse11 import Python11Parser
 
 class Python10Parser(Python11Parser):
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python11Parser, self).__init__(debug_parser)
+        super(Python10Parser, self).__init__(debug_parser)
         self.customized = {}
 
 

uncompyle6/parsers/parse11.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019 Rocky Bernstein
+#  Copyright (c) 2019-2020 Rocky Bernstein
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.parser import PythonParserSingle
@@ -7,7 +7,7 @@ from uncompyle6.parsers.parse12 import Python12Parser
 
 class Python11Parser(Python12Parser):
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python12Parser, self).__init__(debug_parser)
+        super(Python11Parser, self).__init__(debug_parser)
         self.customized = {}
 
 

uncompyle6/parsers/parse21.py

@@ -8,7 +8,7 @@ from uncompyle6.parsers.parse22 import Python22Parser
 class Python21Parser(Python22Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python22Parser, self).__init__(debug_parser)
+        super(Python21Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_forstmt21(self, args):

uncompyle6/parsers/parse22.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2017 Rocky Bernstein
+#  Copyright (c) 2016-2017, 2020 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
@@ -8,7 +8,7 @@ from uncompyle6.parsers.parse23 import Python23Parser
 class Python22Parser(Python23Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python23Parser, self).__init__(debug_parser)
+        super(Python22Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_misc22(self, args):

uncompyle6/parsers/parse23.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2018 Rocky Bernstein
+#  Copyright (c) 2016-2018, 2020 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 #  Copyright (c) 1999 John Aycock
 
@@ -9,7 +9,7 @@ from uncompyle6.parsers.parse24 import Python24Parser
 class Python23Parser(Python24Parser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python24Parser, self).__init__(debug_parser)
+        super(Python23Parser, self).__init__(debug_parser)
         self.customized = {}
 
     def p_misc23(self, args):

uncompyle6/parsers/parse26.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2017-2019 Rocky Bernstein
+#  Copyright (c) 2017-2020 Rocky Bernstein
 """
 spark grammar differences over Python2 for Python 2.6.
 """
@@ -6,6 +6,7 @@ spark grammar differences over Python2 for Python 2.6.
 from uncompyle6.parser import PythonParserSingle
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.parsers.parse2 import Python2Parser
+from uncompyle6.parsers.reducecheck import (except_handler, tryelsestmt)
 
 class Python26Parser(Python2Parser):
 
@@ -24,7 +25,11 @@ class Python26Parser(Python2Parser):
         except_handler ::= JUMP_FORWARD COME_FROM except_stmts
                            come_froms_pop END_FINALLY come_froms
 
-        except_handler ::= JUMP_FORWARD COME_FROM except_stmts END_FINALLY
+        except_handler ::= JUMP_FORWARD COME_FROM except_stmts
+                           END_FINALLY
+
+        except_handler ::= JUMP_FORWARD COME_FROM except_stmts
+                           POP_TOP END_FINALLY
                            come_froms
 
         except_handler ::= jmp_abs COME_FROM except_stmts
@@ -33,6 +38,7 @@ class Python26Parser(Python2Parser):
         except_handler ::= jmp_abs COME_FROM except_stmts
                            END_FINALLY JUMP_FORWARD
 
+
         # Sometimes we don't put in COME_FROM to the next statement
         # like we do in 2.7. Perhaps we should?
         try_except     ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
@@ -87,8 +93,8 @@ class Python26Parser(Python2Parser):
 
         cf_jb_cf_pop ::= _come_froms JUMP_BACK come_froms POP_TOP
 
-        bp_come_from    ::= POP_BLOCK COME_FROM
-        jb_pb_come_from ::= JUMP_BACK bp_come_from
+        pb_come_from    ::= POP_BLOCK COME_FROM
+        jb_pb_come_from ::= JUMP_BACK pb_come_from
 
         _ifstmts_jump ::= c_stmts_opt JUMP_FORWARD COME_FROM POP_TOP
         _ifstmts_jump ::= c_stmts_opt JUMP_FORWARD come_froms POP_TOP COME_FROM
@@ -142,7 +148,7 @@ class Python26Parser(Python2Parser):
         while1stmt     ::= SETUP_LOOP l_stmts_opt CONTINUE _come_froms
 
         whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt jb_pop POP_BLOCK _come_froms
-        whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt jb_cf_pop bp_come_from
+        whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt jb_cf_pop pb_come_from
         whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt jb_cf_pop POP_BLOCK
         whilestmt      ::= SETUP_LOOP testexpr returns POP_BLOCK COME_FROM
 
@@ -232,7 +238,10 @@ class Python26Parser(Python2Parser):
 
         comp_for ::= SETUP_LOOP expr for_iter store comp_iter jb_pb_come_from
 
-        comp_body ::= gen_comp_body
+        comp_iter   ::= comp_if_not
+        comp_if_not ::= expr jmp_true comp_iter
+
+        comp_body   ::= gen_comp_body
 
         for_block ::= l_stmts_opt _come_froms POP_TOP JUMP_BACK
 
@@ -342,18 +351,30 @@ class Python26Parser(Python2Parser):
                 WITH_CLEANUP END_FINALLY
         """)
         super(Python26Parser, self).customize_grammar_rules(tokens, customize)
+        self.reduce_check_table = {
+            "except_handler": except_handler,
+            "tryelsestmt": tryelsestmt,
+            "tryelsestmtl": tryelsestmt,
+        }
+
+
         self.check_reduce['and'] = 'AST'
         self.check_reduce['assert_expr_and'] = 'AST'
+        self.check_reduce["except_handler"] = "tokens"
         self.check_reduce["ifstmt"] = "tokens"
         self.check_reduce["ifelsestmt"] = "AST"
+        self.check_reduce["forelselaststmtl"] = "tokens"
+        self.check_reduce["forelsestmt"] = "tokens"
         self.check_reduce['list_for'] = 'AST'
         self.check_reduce['try_except'] = 'tokens'
         self.check_reduce['tryelsestmt'] = 'AST'
+        self.check_reduce['tryelsestmtl'] = 'AST'
 
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
         invalid = super(Python26Parser,
                         self).reduce_is_invalid(rule, ast,
                                                 tokens, first, last)
+        lhs = rule[0]
         if invalid or tokens is None:
             return invalid
         if rule in (
@@ -386,6 +407,27 @@ class Python26Parser(Python2Parser):
             return not (jmp_target == tokens[test_index].offset or
                         tokens[last].pattr == jmp_false.pattr)
 
+        elif lhs in ("forelselaststmtl", "forelsestmt"):
+            # print("XXX", first, last)
+            # for t in range(first, last):
+            #     print(tokens[t])
+            # print("=" * 30)
+
+            # If the SETUP_LOOP jumps to the tokens[last] then
+            # this is a "for" not a "for else".
+
+            # However, in Python 2.2 and before there is a SET_LINENO
+            # instruction which might have gotten removed. So we need
+            # to account for that.  bytecode-1.4/anydbm.pyc exhibits
+            # this behavior.
+
+            # Also we need to use the setup_loop instruction (not opcode)
+            # since the operand can be a relative offset rather than
+            # an absolute offset.
+            setup_inst = self.insts[self.offset2inst_index[tokens[first].offset]]
+            if self.version <= 2.2 and tokens[last] == "COME_FROM":
+                last += 1
+            return tokens[last-1].off2int() > setup_inst.argval
         elif rule == ("ifstmt", ("testexpr", "_ifstmts_jump")):
             for i in range(last-1, last-4, -1):
                 t = tokens[i]
@@ -402,7 +444,7 @@ class Python26Parser(Python2Parser):
             # The JUMP_ABSOLUTE has to be to the last POP_TOP or this is invalid
             ja_attr = ast[4].attr
             return tokens[last].offset != ja_attr
-        elif rule[0] == 'try_except':
+        elif lhs == 'try_except':
             # We need to distingush try_except from tryelsestmt and we do that
             # by checking the jump before the END_FINALLY
             # If we have:
@@ -419,12 +461,12 @@ class Python26Parser(Python2Parser):
                 last -= 1
             if (tokens[last] == 'COME_FROM'
                 and tokens[last-1] == 'END_FINALLY'
-                    and tokens[last-2] == 'POP_TOP'):
+                   and tokens[last-2] == 'POP_TOP'):
                 # A jump of 2 is a jump around POP_TOP, END_FINALLY which
                 # would indicate try/else rather than try
                 return (tokens[last-3].kind not in frozenset(('JUMP_FORWARD', 'RETURN_VALUE'))
                         or (tokens[last-3] == 'JUMP_FORWARD' and tokens[last-3].attr != 2))
-        elif rule[0] == 'tryelsestmt':
+        elif lhs == 'tryelsestmt':
 
             # We need to distingush try_except from tryelsestmt and we do that
             # by making sure that the jump before the except handler jumps to

uncompyle6/parsers/parse27.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2019 Rocky Bernstein
+#  Copyright (c) 2016-2020 Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 
@@ -7,9 +7,10 @@ from xdis import next_offset
 from uncompyle6.parser import PythonParserSingle, nop_func
 from uncompyle6.parsers.parse2 import Python2Parser
 from uncompyle6.parsers.reducecheck import (
+    aug_assign1_check,
     or_check,
-    ifelsestmt,
     tryelsestmt,
+    except_handler,
 )
 
 class Python27Parser(Python2Parser):
@@ -92,7 +93,7 @@ class Python27Parser(Python2Parser):
         iflaststmtl     ::= testexpr c_stmts
 
         _ifstmts_jump   ::= c_stmts_opt JUMP_FORWARD come_froms
-        bp_come_from    ::= POP_BLOCK COME_FROM
+        pb_come_from    ::= POP_BLOCK COME_FROM
 
         # FIXME: Common with 3.0+
         jmp_false ::= POP_JUMP_IF_FALSE
@@ -164,7 +165,7 @@ class Python27Parser(Python2Parser):
         while1elsestmt    ::= SETUP_LOOP l_stmts JUMP_BACK
                               else_suitel COME_FROM
 
-        while1stmt        ::= SETUP_LOOP returns bp_come_from
+        while1stmt        ::= SETUP_LOOP returns pb_come_from
         while1stmt        ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_BLOCK COME_FROM
 
         whilestmt         ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK _come_froms
@@ -231,12 +232,15 @@ class Python27Parser(Python2Parser):
         # FIXME: Put more in this table
         self.reduce_check_table = {
             # "ifelsestmt": ifelsestmt,
+            "aug_assign1": aug_assign1_check,
+            "except_handler": except_handler,
             "or": or_check,
             "tryelsestmt": tryelsestmt,
             "tryelsestmtl": tryelsestmt,
         }
 
         self.check_reduce["and"] = "AST"
+        self.check_reduce["aug_assign1"] = "AST"
         self.check_reduce["if_exp"] = "AST"
 
         self.check_reduce["except_handler"] = "tokens"

uncompyle6/parsers/parse3.py

@@ -679,6 +679,7 @@ class Python3Parser(PythonParser):
                 "RAISE",
                 "SETUP",
                 "UNPACK",
+                "WITH",
             )
         )
 

uncompyle6/parsers/parse36.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2019 Rocky Bernstein
+#  Copyright (c) 2016-2020 Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -305,28 +305,25 @@ class Python36Parser(Python35Parser):
                 self.addRule(rule, nop_func)
                 # Check to combine assignment + annotation into one statement
                 self.check_reduce['assign'] = 'token'
+            elif opname == "WITH_CLEANUP_START":
+                rules_str = """
+                  stmt        ::= with_null
+                  with_null   ::= with_suffix
+                  with_suffix ::= WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                """
+                self.addRule(rules_str, nop_func)
             elif opname == 'SETUP_WITH':
                 rules_str = """
-                with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
-                               WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                  with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
+                                 with_suffix
 
-                # Removes POP_BLOCK LOAD_CONST from 3.6-
-                withasstmt ::= expr SETUP_WITH store suite_stmts_opt COME_FROM_WITH
-                               WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                  # Removes POP_BLOCK LOAD_CONST from 3.6-
+                  withasstmt ::= expr SETUP_WITH store suite_stmts_opt COME_FROM_WITH
+                                 with_suffix
+                  with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                                 BEGIN_FINALLY COME_FROM_WITH
+                                 with_suffix
                 """
-                if self.version < 3.8:
-                    rules_str += """
-                    with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
-                                   LOAD_CONST
-                                   WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
-                    """
-                else:
-                    rules_str += """
-                    with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
-                                   BEGIN_FINALLY COME_FROM_WITH
-                                   WITH_CLEANUP_START WITH_CLEANUP_FINISH
-                                   END_FINALLY
-                    """
                 self.addRule(rules_str, nop_func)
                 pass
             pass

uncompyle6/parsers/parse37.py

@@ -94,6 +94,9 @@ class Python37Parser(Python37BaseParser):
         else_suitec ::= c_stmts
         else_suitec ::= returns
 
+        else_suite_opt ::= else_suite
+        else_suite_opt ::= pass
+
         stmt ::= classdef
         stmt ::= call_stmt
 
@@ -635,6 +638,12 @@ class Python37Parser(Python37BaseParser):
         if_exp37                   ::= expr expr jf_cfs expr COME_FROM
         jf_cfs                     ::= JUMP_FORWARD _come_froms
         ifelsestmt                 ::= testexpr c_stmts_opt jf_cfs else_suite opt_come_from_except
+
+        # This is probably more realistically an "ifstmt" (with a null else)
+        # see _cmp() of python3.8/distutils/__pycache__/version.cpython-38.opt-1.pyc
+        ifelsestmt                 ::= testexpr stmts jf_cfs else_suite_opt opt_come_from_except
+
+
         expr_pjit                  ::= expr POP_JUMP_IF_TRUE
         expr_jit                   ::= expr JUMP_IF_TRUE
         expr_jt                    ::= expr jmp_true
@@ -930,8 +939,10 @@ class Python37Parser(Python37BaseParser):
 
         jitop_come_from_expr ::= JUMP_IF_TRUE_OR_POP come_froms expr
         jifop_come_from ::= JUMP_IF_FALSE_OR_POP come_froms
+        expr_jitop      ::= expr JUMP_IF_TRUE_OR_POP
+
         or        ::= and jitop_come_from_expr COME_FROM
-        or        ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
+        or        ::= expr_jitop  expr COME_FROM
         or        ::= expr_jit expr COME_FROM
         or        ::= expr_pjit expr POP_JUMP_IF_FALSE COME_FROM
 
@@ -954,8 +965,8 @@ class Python37Parser(Python37BaseParser):
         and  ::= expr JUMP_IF_FALSE_OR_POP expr come_from_opt
         and  ::= expr jifop_come_from expr
 
-        pjit_come_from ::= POP_JUMP_IF_TRUE COME_FROM
-        or  ::= expr pjit_come_from expr
+        expr_pjit_come_from ::= expr POP_JUMP_IF_TRUE COME_FROM
+        or  ::= expr_pjit_come_from expr
 
         ## Note that "jmp_false" is what we check on in the "and" reduce rule.
         and ::= expr jmp_false expr COME_FROM

uncompyle6/parsers/parse37base.py

@@ -2,10 +2,10 @@
 """
 Python 3.7 base code. We keep non-custom-generated grammar rules out of this file.
 """
-from uncompyle6.scanners.tok import Token
-from uncompyle6.parser import ParserError, PythonParser, PythonParserSingle, nop_func
+from uncompyle6.parser import ParserError, PythonParser, nop_func
 from uncompyle6.parsers.treenode import SyntaxTree
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser.spark import rule2str
 
 from uncompyle6.parsers.reducecheck import (
     and_check,
@@ -127,6 +127,7 @@ class Python37BaseParser(PythonParser):
                 "RAISE",
                 "SETUP",
                 "UNPACK",
+                "WITH",
             )
         )
 
@@ -614,7 +615,7 @@ class Python37BaseParser(PythonParser):
                                             JUMP_BACK COME_FROM
                                             POP_TOP POP_TOP POP_TOP POP_EXCEPT POP_TOP
                     list_comp_async     ::= BUILD_LIST_0 LOAD_FAST list_afor2
-                    get_aiter           ::= LOAD_DEREF GET_AITER
+                    get_aiter           ::= expr GET_AITER
                     list_afor           ::= get_aiter list_afor2
                     list_iter           ::= list_afor
                    """,
@@ -993,55 +994,70 @@ class Python37BaseParser(PythonParser):
                 )
                 custom_ops_processed.add(opname)
 
+            elif opname == "WITH_CLEANUP_START":
+                rules_str = """
+                  stmt        ::= with_null
+                  with_null   ::= with_suffix
+                  with_suffix ::= WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                """
+                self.addRule(rules_str, nop_func)
             elif opname == "SETUP_WITH":
                 rules_str = """
                   stmt       ::= with
                   stmt       ::= withasstmt
 
-                  with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                  with       ::= expr
+                                 SETUP_WITH POP_TOP
+                                 suite_stmts_opt
+                                 COME_FROM_WITH
+                                 with_suffix
                   withasstmt ::= expr SETUP_WITH store suite_stmts_opt COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 with_suffix
+
+                  with       ::= expr
+                                 SETUP_WITH POP_TOP
+                                 suite_stmts_opt
+                                 POP_BLOCK LOAD_CONST COME_FROM_WITH
+                                 with_suffix
+
+                  withasstmt ::= expr
+                                 SETUP_WITH store suite_stmts_opt
+                                 POP_BLOCK LOAD_CONST COME_FROM_WITH
+                                 with_suffix
 
                   with       ::= expr
                                  SETUP_WITH POP_TOP suite_stmts_opt
                                  POP_BLOCK LOAD_CONST COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 with_suffix
                   withasstmt ::= expr
                                  SETUP_WITH store suite_stmts_opt
                                  POP_BLOCK LOAD_CONST COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
-
-                  with       ::= expr
-                                 SETUP_WITH POP_TOP suite_stmts_opt
-                                 POP_BLOCK LOAD_CONST COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
-                  withasstmt ::= expr
-                                 SETUP_WITH store suite_stmts_opt
-                                 POP_BLOCK LOAD_CONST COME_FROM_WITH
-                                 WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                 with_suffix
                 """
                 if self.version < 3.8:
                     rules_str += """
                     with     ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                    LOAD_CONST
-                                   WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                   with_suffix
                     """
                 else:
                     rules_str += """
-                      with    ::= expr
+                      with       ::= expr
                                      SETUP_WITH POP_TOP suite_stmts_opt
                                      POP_BLOCK LOAD_CONST COME_FROM_WITH
-                                     WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+                                     with_suffix
 
                       withasstmt ::= expr
                                      SETUP_WITH store suite_stmts_opt
                                      POP_BLOCK LOAD_CONST COME_FROM_WITH
 
-                       with    ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
+                      withasstmt ::= expr
+                                     SETUP_WITH store suite_stmts
+                                     POP_BLOCK BEGIN_FINALLY COME_FROM_WITH with_suffix
+
+                      with       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK
                                      BEGIN_FINALLY COME_FROM_WITH
-                                     WITH_CLEANUP_START WITH_CLEANUP_FINISH
-                                     END_FINALLY
+                                     with_suffix
                     """
                 self.addRule(rules_str, nop_func)
 
@@ -1182,18 +1198,27 @@ class Python37BaseParser(PythonParser):
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
         lhs = rule[0]
         n = len(tokens)
-        last = min(last, n-1)
+        last = min(last, n - 1)
         fn = self.reduce_check_table.get(lhs, None)
         try:
             if fn:
                 return fn(self, lhs, n, rule, ast, tokens, first, last)
         except:
             import sys, traceback
-            print("Exception in %s %s\n" +
-                  "rule: %s\n" +
-                  "offsets %s .. %s" %
-                  (fn.__name__, sys.exc_info()[1], rule, tokens[first].offset, tokens[last].offset))
-            print(traceback.print_tb(sys.exc_info()[2],-1))
+
+            print(
+                ("Exception in %s %s\n"
+                 + "rule: %s\n"
+                 + "offsets %s .. %s")
+                % (
+                    fn.__name__,
+                    sys.exc_info()[1],
+                    rule2str(rule),
+                    tokens[first].offset,
+                    tokens[last].offset,
+                )
+            )
+            print(traceback.print_tb(sys.exc_info()[2], -1))
             raise ParserError(tokens[last], tokens[last].off2int(), self.debug["rules"])
 
         if lhs in ("aug_assign1", "aug_assign2") and ast[0][0] == "and":

uncompyle6/parsers/parse38.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2017-2019 Rocky Bernstein
+#  Copyright (c) 2017-2020 Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -39,14 +39,18 @@ class Python38Parser(Python37Parser):
         stmt               ::= forelselaststmtl38
         stmt               ::= tryfinally38stmt
         stmt               ::= tryfinally38rstmt
+        stmt               ::= tryfinally38rstmt2
+        stmt               ::= tryfinally38rstmt3
         stmt               ::= tryfinally38astmt
         stmt               ::= try_elsestmtl38
         stmt               ::= try_except_ret38
         stmt               ::= try_except38
+        stmt               ::= try_except_as
         stmt               ::= whilestmt38
         stmt               ::= whileTruestmt38
-        stmt               ::= call
+        stmt               ::= call_stmt
 
+        call_stmt          ::= call
         break ::= POP_BLOCK BREAK_LOOP
         break ::= POP_BLOCK POP_TOP BREAK_LOOP
         break ::= POP_TOP BREAK_LOOP
@@ -54,15 +58,12 @@ class Python38Parser(Python37Parser):
 
         # FIXME: this should be restricted to being inside a try block
         stmt               ::= except_ret38
+        stmt               ::= except_ret38a
 
         # FIXME: this should be added only when seeing GET_AITER or YIELD_FROM
-        async_for_stmt38   ::= expr
-                               GET_AITER
-                               SETUP_FINALLY
-                               GET_ANEXT
-                               LOAD_CONST
-                               YIELD_FROM
-                               POP_BLOCK
+        async_for          ::= GET_AITER _come_froms
+                               SETUP_FINALLY GET_ANEXT LOAD_CONST YIELD_FROM POP_BLOCK
+        async_for_stmt38   ::= expr async_for
                                store for_block
                                COME_FROM_FINALLY
                                END_ASYNC_FOR
@@ -81,7 +82,20 @@ class Python38Parser(Python37Parser):
                                END_ASYNC_FOR
                                else_suite
 
-        return             ::= ret_expr ROT_TWO POP_TOP RETURN_VALUE
+        # Seems to be used to discard values before a return in a "for" loop
+        discard_top        ::= ROT_TWO POP_TOP
+        discard_tops       ::= discard_top+
+
+        return             ::= ret_expr
+                               discard_tops RETURN_VALUE
+
+        return             ::= popb_return
+        return             ::= pop_return
+        return             ::= pop_ex_return
+        except_stmt        ::= pop_ex_return
+        pop_return         ::= POP_TOP ret_expr RETURN_VALUE
+        popb_return        ::= ret_expr POP_BLOCK RETURN_VALUE
+        pop_ex_return      ::= ret_expr ROT_FOUR POP_EXCEPT RETURN_VALUE
 
         # 3.8 can push a looping JUMP_BACK into into a JUMP_ from a statement that jumps to it
         lastl_stmt         ::= ifpoplaststmtl
@@ -120,6 +134,8 @@ class Python38Parser(Python37Parser):
         except_cond1       ::= DUP_TOP expr COMPARE_OP jmp_false
                                POP_TOP POP_TOP POP_TOP
                                POP_EXCEPT
+        except_cond_as     ::= DUP_TOP expr COMPARE_OP POP_JUMP_IF_FALSE
+                               POP_TOP STORE_FAST POP_TOP
 
         try_elsestmtl38    ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                except_handler38 COME_FROM
@@ -128,8 +144,18 @@ class Python38Parser(Python37Parser):
                                except_handler38
         try_except38       ::= SETUP_FINALLY POP_BLOCK POP_TOP suite_stmts_opt
                                except_handler38a
-        try_except_ret38   ::= SETUP_FINALLY expr POP_BLOCK
-                               RETURN_VALUE except_ret38a
+
+        # suite_stmts has a return
+        try_except38       ::= SETUP_FINALLY POP_BLOCK suite_stmts
+                               except_handler38b
+        try_except_as      ::= SETUP_FINALLY POP_BLOCK suite_stmts
+                               except_handler_as END_FINALLY COME_FROM
+        try_except_as      ::= SETUP_FINALLY suite_stmts
+                               except_handler_as END_FINALLY COME_FROM
+
+        try_except_ret38   ::= SETUP_FINALLY returns except_ret38a
+        try_except_ret38a  ::= SETUP_FINALLY returns except_handler38c
+                               END_FINALLY come_from_opt
 
         # Note: there is a suite_stmts_opt which seems
         # to be bookkeeping which is not expressed in source code
@@ -146,22 +172,50 @@ class Python38Parser(Python37Parser):
         except_handler38a  ::= COME_FROM_FINALLY POP_TOP POP_TOP POP_TOP
                                POP_EXCEPT POP_TOP stmts END_FINALLY
 
+        except_handler38c  ::= COME_FROM_FINALLY except_cond1a except_stmts
+                               POP_EXCEPT JUMP_FORWARD COME_FROM
+        except_handler_as  ::= COME_FROM_FINALLY except_cond_as tryfinallystmt
+                               POP_EXCEPT JUMP_FORWARD COME_FROM
+
         tryfinallystmt     ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                BEGIN_FINALLY COME_FROM_FINALLY suite_stmts_opt
                                END_FINALLY
-        tryfinally38rstmt  ::= SETUP_FINALLY POP_BLOCK CALL_FINALLY
+
+
+        lc_setup_finally   ::= LOAD_CONST SETUP_FINALLY
+        call_finally_pt    ::= CALL_FINALLY POP_TOP
+        cf_cf_finally      ::= come_from_opt COME_FROM_FINALLY
+        pop_finally_pt     ::= POP_FINALLY POP_TOP
+        ss_end_finally     ::= suite_stmts END_FINALLY
+        sf_pb_call_returns ::= SETUP_FINALLY POP_BLOCK CALL_FINALLY returns
+
+
+        # FIXME: DRY rules below
+        tryfinally38rstmt  ::= sf_pb_call_returns
+                               cf_cf_finally
+                               ss_end_finally
+        tryfinally38rstmt  ::= sf_pb_call_returns
+                               cf_cf_finally END_FINALLY
+                               suite_stmts
+        tryfinally38rstmt  ::= sf_pb_call_returns
+                               cf_cf_finally POP_FINALLY
+                               ss_end_finally
+        tryfinally38rstmt  ::= sf_bp_call_returns
+                               COME_FROM_FINALLY POP_FINALLY
+                               ss_end_finally
+
+        tryfinally38rstmt2 ::= lc_setup_finally POP_BLOCK call_finally_pt
                                returns
-                               COME_FROM_FINALLY END_FINALLY suite_stmts
-        tryfinally38rstmt  ::= SETUP_FINALLY POP_BLOCK CALL_FINALLY
-                               returns
-                               COME_FROM_FINALLY POP_FINALLY returns
-                               END_FINALLY
-        tryfinally38stmt   ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
-                               BEGIN_FINALLY COME_FROM_FINALLY
-                               POP_FINALLY suite_stmts_opt END_FINALLY
+                               cf_cf_finally pop_finally_pt
+                               ss_end_finally POP_TOP
+        tryfinally38rstmt3 ::= SETUP_FINALLY expr POP_BLOCK CALL_FINALLY RETURN_VALUE
+                               COME_FROM COME_FROM_FINALLY
+                               ss_end_finally
+
         tryfinally38stmt   ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                BEGIN_FINALLY COME_FROM_FINALLY
                                POP_FINALLY suite_stmts_opt END_FINALLY
+
         tryfinally38astmt  ::= LOAD_CONST SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                BEGIN_FINALLY COME_FROM_FINALLY
                                POP_FINALLY POP_TOP suite_stmts_opt END_FINALLY POP_TOP

uncompyle6/parsers/reducecheck/__init__.py

@@ -1,4 +1,6 @@
 from uncompyle6.parsers.reducecheck.and_check import *
+from uncompyle6.parsers.reducecheck.aug_assign import *
+from uncompyle6.parsers.reducecheck.except_handler import *
 from uncompyle6.parsers.reducecheck.except_handler_else import *
 from uncompyle6.parsers.reducecheck.ifelsestmt import *
 from uncompyle6.parsers.reducecheck.iflaststmt import *

uncompyle6/parsers/reducecheck/aug_assign.py

@@ -0,0 +1,10 @@
+#  Copyright (c) 2020 Rocky Bernstein
+
+def aug_assign1_check(self, lhs, n, rule, ast, tokens, first, last):
+    # print("XXX", first, last, rule)
+    # for t in range(first, last): print(tokens[t])
+    # print("="*40)
+
+    expr = ast[0]
+    return expr == "expr" and expr[0] == "or"
+    return False

uncompyle6/parsers/reducecheck/except_handler.py

@@ -0,0 +1,19 @@
+#  Copyright (c) 2020 Rocky Bernstein
+
+def except_handler(self, lhs, n, rule, ast, tokens, first, last):
+    end_token = tokens[last-1]
+
+    # print("XXX", first, last)
+    # for t in range(first, last):
+    #     print(tokens[t])
+    # print("=" * 30)
+
+    # FIXME: Figure out why this doesn't work on
+    # bytecode-1.4/anydbm.pyc
+    if self.version == 1.4:
+        return False
+
+    # Make sure come froms all come from within "except_handler".
+    if end_token != "COME_FROM":
+        return False
+    return end_token.attr < tokens[first].offset

uncompyle6/parsers/reducecheck/ifelsestmt.py

@@ -95,6 +95,34 @@ IFELSE_STMT_RULES = frozenset(
                 "else_suite",
             ),
         ),
+        (
+            "ifelsestmt",
+            (
+                "testexpr",
+                "stmts",
+                "jf_cfs",
+                "else_suite_opt",
+                "opt_come_from_except",
+            ),
+        ),
+        (
+            "ifelsestmt",
+            (
+                "testexpr",
+                "stmts",
+                "jf_cfs",
+                "\\e_else_suite_opt",
+                "\\e_opt_come_from_except")
+        ),
+        (
+            "ifelsestmt",
+            (
+                "testexpr",
+                "stmts",
+                "jf_cfs",
+                "\\e_else_suite_opt",
+                "opt_come_from_except")
+        ),
     ])
 
 def ifelsestmt(self, lhs, n, rule, ast, tokens, first, last):
@@ -103,12 +131,17 @@ def ifelsestmt(self, lhs, n, rule, ast, tokens, first, last):
         # ifelsestmt jumped outside of loop. No good.
         return True
 
+    # print("XXX", first, last)
+    # for t in range(first, last):
+    #     print(tokens[t])
+    # print("=" * 30)
+
     if rule not in IFELSE_STMT_RULES:
         # print("XXX", rule)
         return False
 
     # Avoid if/else where the "then" is a "raise_stmt1" for an
-    # assert statemetn. Parse this as an "assert" instead.
+    # assert statement. Parse this as an "assert" instead.
     stmts = ast[1]
     if stmts in ("c_stmts",) and len(stmts) == 1:
         raise_stmt1 = stmts[0]
@@ -176,9 +209,7 @@ def ifelsestmt(self, lhs, n, rule, ast, tokens, first, last):
             if jump_else_end == "jf_cf_pop":
                 jump_else_end = jump_else_end[0]
 
-            jump_to_jump = False
             if jump_else_end == "JUMP_FORWARD":
-                jump_to_jump = True
                 endif_target = int(jump_else_end.pattr)
                 last_offset = tokens[last].off2int()
                 if endif_target != last_offset:

uncompyle6/parsers/reducecheck/or_check.py

@@ -8,9 +8,6 @@ def or_check(self, lhs, n, rule, ast, tokens, first, last):
     # for t in range(first, last): print(tokens[t])
     # print("="*40)
 
-    if (first, last) == (2, 6):
-        from trepan.api import debug; debug()
-
     if rhs[0:2] in (("expr_jt", "expr"),
                     ("expr_jitop", "expr"),
                     ("expr_jit", "expr")):

uncompyle6/parsers/reducecheck/tryelsestmt.py

@@ -6,7 +6,13 @@ def tryelsestmt(self, lhs, n, rule, ast, tokens, first, last):
     # Check the end of the except handler that there isn't a jump from
     # inside the except handler to the end. If that happens
     # then this is a "try" with no "else".
+
+    # for t in range(first, last):
+    #     print(tokens[t])
+    # print("=" * 30)
+
     except_handler = ast[3]
+
     if except_handler == "except_handler_else":
         except_handler = except_handler[0]
     if except_handler == "except_handler":
@@ -32,5 +38,8 @@ def tryelsestmt(self, lhs, n, rule, ast, tokens, first, last):
             except_handler_first_offset = leading_jump.first_child().off2int()
         else:
             except_handler_first_offset = leading_jump.off2int()
+
+        if first_come_from.attr < tokens[first].offset:
+            return True
         return first_come_from.attr > except_handler_first_offset
     return False

uncompyle6/scanner.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016, 2018-2019 by Rocky Bernstein
+#  Copyright (c) 2016, 2018-2020 by Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
@@ -30,9 +30,7 @@ import sys
 from uncompyle6 import PYTHON3, IS_PYPY
 from uncompyle6.scanners.tok import Token
 import xdis
-from xdis.bytecode import Bytecode, instruction_size, extended_arg_val, next_offset
-from xdis.magics import canonic_python_version
-from xdis.util import code2num
+from xdis import Bytecode, canonic_python_version, code2num, instruction_size, extended_arg_val, next_offset
 
 # The byte code versions we support.
 # Note: these all have to be floats

uncompyle6/scanners/pypy33.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019 by Rocky Bernstein
+#  Copyright (c) 2019-2020 by Rocky Bernstein
 """
 Python PyPy 3.3 decompiler scanner.
 
@@ -10,6 +10,7 @@ import uncompyle6.scanners.scanner33 as scan
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_33pypy as opc
+
 JUMP_OPs = map(lambda op: opc.opname[op], opc.hasjrel + opc.hasjabs)
 
 # We base this off of 3.3

uncompyle6/scanners/scanner3.py

@@ -35,8 +35,8 @@ Finally we save token information.
 
 from __future__ import print_function
 
-from xdis import iscode
-from xdis.bytecode import instruction_size, _get_const_info
+from xdis import iscode, instruction_size
+from xdis.bytecode import _get_const_info
 
 from uncompyle6.scanner import Token, parse_fn_counts
 import xdis
@@ -888,6 +888,7 @@ class Scanner3(Scanner):
                     start, self.next_stmt[offset], self.opc.POP_JUMP_IF_FALSE, target
                 )
 
+                # FIXME: Remove this whole "if" block
                 # If we still have any offsets in set, start working on it
                 if match:
                     is_jump_forward = self.is_jump_forward(pre_rtarget)
@@ -956,7 +957,7 @@ class Scanner3(Scanner):
                             )
                         ):
                             pass
-                        else:
+                        elif self.version <= 3.2:
                             fix = None
                             jump_ifs = self.inst_matches(
                                 start,

uncompyle6/scanners/scanner30.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016, 2017 by Rocky Bernstein
+#  Copyright (c) 2016-2017, 2020 by Rocky Bernstein
 """
 Python 3.0 bytecode scanner/deparser
 
@@ -10,17 +10,19 @@ from __future__ import print_function
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_30 as opc
-from xdis.bytecode import instruction_size
+from xdis import instruction_size
 import xdis
 
 JUMP_TF = frozenset([opc.JUMP_IF_FALSE, opc.JUMP_IF_TRUE])
 
 from uncompyle6.scanners.scanner3 import Scanner3
-class Scanner30(Scanner3):
 
+
+class Scanner30(Scanner3):
     def __init__(self, show_asm=None, is_pypy=False):
         Scanner3.__init__(self, 3.0, show_asm, is_pypy)
         return
+
     pass
 
     def detect_control_flow(self, offset, targets, inst_index):
@@ -35,17 +37,18 @@ class Scanner30(Scanner3):
 
         # Detect parent structure
         parent = self.structs[0]
-        start  = parent['start']
-        end    = parent['end']
+        start = parent["start"]
+        end = parent["end"]
 
         # Pick inner-most parent for our offset
         for struct in self.structs:
-            current_start = struct['start']
-            current_end   = struct['end']
-            if ((current_start <= offset < current_end)
-                and (current_start >= start and current_end <= end)):
-                start  = current_start
-                end    = current_end
+            current_start = struct["start"]
+            current_end = struct["end"]
+            if (current_start <= offset < current_end) and (
+                current_start >= start and current_end <= end
+            ):
+                start = current_start
+                end = current_end
                 parent = struct
 
         if op == self.opc.SETUP_LOOP:
@@ -56,28 +59,35 @@ class Scanner30(Scanner3):
 
             start += instruction_size(op, self.opc)
             target = self.get_target(offset)
-            end    = self.restrict_to_parent(target, parent)
+            end = self.restrict_to_parent(target, parent)
             self.setup_loops[target] = offset
 
             if target != end:
                 self.fixed_jumps[offset] = end
 
             (line_no, next_line_byte) = self.lines[offset]
-            jump_back = self.last_instr(start, end, self.opc.JUMP_ABSOLUTE,
-                                            next_line_byte, False)
+            jump_back = self.last_instr(
+                start, end, self.opc.JUMP_ABSOLUTE, next_line_byte, False
+            )
 
             if jump_back:
-                jump_forward_offset = xdis.next_offset(code[jump_back], self.opc, jump_back)
+                jump_forward_offset = xdis.next_offset(
+                    code[jump_back], self.opc, jump_back
+                )
             else:
                 jump_forward_offset = None
 
             return_val_offset1 = self.prev[self.prev[end]]
 
-            if (jump_back and jump_back != self.prev_op[end]
-                and self.is_jump_forward(jump_forward_offset)):
-                if (code[self.prev_op[end]] == self.opc.RETURN_VALUE or
-                    (code[self.prev_op[end]] == self.opc.POP_BLOCK
-                     and code[return_val_offset1] == self.opc.RETURN_VALUE)):
+            if (
+                jump_back
+                and jump_back != self.prev_op[end]
+                and self.is_jump_forward(jump_forward_offset)
+            ):
+                if code[self.prev_op[end]] == self.opc.RETURN_VALUE or (
+                    code[self.prev_op[end]] == self.opc.POP_BLOCK
+                    and code[return_val_offset1] == self.opc.RETURN_VALUE
+                ):
                     jump_back = None
             if not jump_back:
                 # loop suite ends in return
@@ -92,56 +102,63 @@ class Scanner30(Scanner3):
                 if code[self.prev_op[next_line_byte]] not in JUMP_TF:
                     if_offset = self.prev[next_line_byte]
                 if if_offset:
-                    loop_type = 'while'
+                    loop_type = "while"
                     self.ignore_if.add(if_offset)
                 else:
-                    loop_type = 'for'
+                    loop_type = "for"
                 target = next_line_byte
                 end = jump_back + 3
             else:
                 if self.get_target(jump_back) >= next_line_byte:
-                    jump_back = self.last_instr(start, end, self.opc.JUMP_ABSOLUTE, start, False)
+                    jump_back = self.last_instr(
+                        start, end, self.opc.JUMP_ABSOLUTE, start, False
+                    )
 
                 jb_inst = self.get_inst(jump_back)
 
                 jb_next_offset = self.next_offset(jb_inst.opcode, jump_back)
                 if end > jb_next_offset and self.is_jump_forward(end):
                     if self.is_jump_forward(jb_next_offset):
-                        if self.get_target(jump_back+4) == self.get_target(end):
-                            self.fixed_jumps[offset] = jump_back+4
+                        if self.get_target(jump_back + 4) == self.get_target(end):
+                            self.fixed_jumps[offset] = jump_back + 4
                             end = jb_next_offset
                 elif target < offset:
-                    self.fixed_jumps[offset] = jump_back+4
+                    self.fixed_jumps[offset] = jump_back + 4
                     end = jb_next_offset
 
                 target = self.get_target(jump_back)
 
                 if code[target] in (self.opc.FOR_ITER, self.opc.GET_ITER):
-                    loop_type = 'for'
+                    loop_type = "for"
                 else:
-                    loop_type = 'while'
+                    loop_type = "while"
                     test = self.prev_op[next_line_byte]
 
                     if test == offset:
-                        loop_type = 'while 1'
+                        loop_type = "while 1"
                     elif self.code[test] in self.opc.JUMP_OPs:
                         self.ignore_if.add(test)
                         test_target = self.get_target(test)
-                        if test_target > (jump_back+3):
+                        if test_target > (jump_back + 3):
                             jump_back = test_target
                 self.not_continue.add(jump_back)
             self.loops.append(target)
-            self.structs.append({'type': loop_type + '-loop',
-                                 'start': target,
-                                 'end':   jump_back})
+            self.structs.append(
+                {"type": loop_type + "-loop", "start": target, "end": jump_back}
+            )
             after_jump_offset = xdis.next_offset(code[jump_back], self.opc, jump_back)
-            if (self.get_inst(after_jump_offset).opname == 'POP_TOP'):
-                after_jump_offset = xdis.next_offset(code[after_jump_offset], self.opc,
-                                                     after_jump_offset)
+            if self.get_inst(after_jump_offset).opname == "POP_TOP":
+                after_jump_offset = xdis.next_offset(
+                    code[after_jump_offset], self.opc, after_jump_offset
+                )
             if after_jump_offset != end:
-                self.structs.append({'type': loop_type + '-else',
-                                     'start': after_jump_offset,
-                                     'end':   end})
+                self.structs.append(
+                    {
+                        "type": loop_type + "-else",
+                        "start": after_jump_offset,
+                        "end": end,
+                    }
+                )
         elif op in self.pop_jump_tf:
             start = offset + instruction_size(op, self.opc)
             target = self.get_target(offset)
@@ -149,7 +166,7 @@ class Scanner30(Scanner3):
             prev_op = self.prev_op
 
             # Do not let jump to go out of parent struct bounds
-            if target != rtarget and parent['type'] == 'and/or':
+            if target != rtarget and parent["type"] == "and/or":
                 self.fixed_jumps[offset] = rtarget
                 return
 
@@ -158,12 +175,15 @@ class Scanner30(Scanner3):
             # rocky: if we have a conditional jump to the next instruction, then
             # possibly I am "skipping over" a "pass" or null statement.
 
-            if ((code[prev_op[target]] in self.pop_jump_if_pop) and
-                (target > offset) and prev_op[target] != offset):
+            if (
+                (code[prev_op[target]] in self.pop_jump_if_pop)
+                and (target > offset)
+                and prev_op[target] != offset
+            ):
                 self.fixed_jumps[offset] = prev_op[target]
-                self.structs.append({'type': 'and/or',
-                                     'start': start,
-                                     'end': prev_op[target]})
+                self.structs.append(
+                    {"type": "and/or", "start": start, "end": prev_op[target]}
+                )
                 return
 
             # The op offset just before the target jump offset is important
@@ -176,35 +196,80 @@ class Scanner30(Scanner3):
                 # Search for another JUMP_IF_FALSE targetting the same op,
                 # in current statement, starting from current offset, and filter
                 # everything inside inner 'or' jumps and midline ifs
-                match = self.rem_or(start, self.next_stmt[offset],
-                                    opc.JUMP_IF_FALSE, target)
+                match = self.rem_or(
+                    start, self.next_stmt[offset], opc.JUMP_IF_FALSE, target
+                )
 
                 # If we still have any offsets in set, start working on it
                 if match:
                     is_jump_forward = self.is_jump_forward(pre_rtarget)
-                    if (is_jump_forward and pre_rtarget not in self.stmts and
-                        self.restrict_to_parent(self.get_target(pre_rtarget), parent) == rtarget):
-                        if (code[prev_op[pre_rtarget]] == self.opc.JUMP_ABSOLUTE
-                            and self.remove_mid_line_ifs([offset]) and
-                            target == self.get_target(prev_op[pre_rtarget]) and
-                            (prev_op[pre_rtarget] not in self.stmts or
-                             self.get_target(prev_op[pre_rtarget]) > prev_op[pre_rtarget]) and
-                            1 == len(self.remove_mid_line_ifs(self.rem_or(start, prev_op[pre_rtarget], JUMP_TF, target)))):
+                    if (
+                        is_jump_forward
+                        and pre_rtarget not in self.stmts
+                        and self.restrict_to_parent(
+                            self.get_target(pre_rtarget), parent
+                        )
+                        == rtarget
+                    ):
+                        if (
+                            code[prev_op[pre_rtarget]] == self.opc.JUMP_ABSOLUTE
+                            and self.remove_mid_line_ifs([offset])
+                            and target == self.get_target(prev_op[pre_rtarget])
+                            and (
+                                prev_op[pre_rtarget] not in self.stmts
+                                or self.get_target(prev_op[pre_rtarget])
+                                > prev_op[pre_rtarget]
+                            )
+                            and 1
+                            == len(
+                                self.remove_mid_line_ifs(
+                                    self.rem_or(
+                                        start, prev_op[pre_rtarget], JUMP_TF, target
+                                    )
+                                )
+                            )
+                        ):
                             pass
-                        elif (code[prev_op[pre_rtarget]] == self.opc.RETURN_VALUE
-                              and self.remove_mid_line_ifs([offset]) and
-                              1 == (len(set(self.remove_mid_line_ifs(self.rem_or(start, prev_op[pre_rtarget],
-                                                                                 JUMP_TF, target))) |
-                                    set(self.remove_mid_line_ifs(self.rem_or(start, prev_op[pre_rtarget],
-                                                                             (opc.JUMP_IF_FALSE,
-                                                                              opc.JUMP_IF_TRUE,
-                                                                              opc.JUMP_ABSOLUTE),
-                                                                             pre_rtarget, True)))))):
+                        elif (
+                            code[prev_op[pre_rtarget]] == self.opc.RETURN_VALUE
+                            and self.remove_mid_line_ifs([offset])
+                            and 1
+                            == (
+                                len(
+                                    set(
+                                        self.remove_mid_line_ifs(
+                                            self.rem_or(
+                                                start,
+                                                prev_op[pre_rtarget],
+                                                JUMP_TF,
+                                                target,
+                                            )
+                                        )
+                                    )
+                                    | set(
+                                        self.remove_mid_line_ifs(
+                                            self.rem_or(
+                                                start,
+                                                prev_op[pre_rtarget],
+                                                (
+                                                    opc.JUMP_IF_FALSE,
+                                                    opc.JUMP_IF_TRUE,
+                                                    opc.JUMP_ABSOLUTE,
+                                                ),
+                                                pre_rtarget,
+                                                True,
+                                            )
+                                        )
+                                    )
+                                )
+                            )
+                        ):
                             pass
                         else:
                             fix = None
-                            jump_ifs = self.inst_matches(start, self.next_stmt[offset],
-                                                         opc.JUMP_IF_FALSE)
+                            jump_ifs = self.inst_matches(
+                                start, self.next_stmt[offset], opc.JUMP_IF_FALSE
+                            )
                             last_jump_good = True
                             for j in jump_ifs:
                                 if target == self.get_target(j):
@@ -226,14 +291,19 @@ class Scanner30(Scanner3):
                     pass
                 elif self.is_jump_forward(next) and target == self.get_target(next):
                     if code[prev_op[next]] == opc.JUMP_IF_FALSE:
-                        if (code[next] == self.opc.JUMP_FORWARD
+                        if (
+                            code[next] == self.opc.JUMP_FORWARD
                             or target != rtarget
-                            or code[prev_op[pre_rtarget]] not in
-                            (self.opc.JUMP_ABSOLUTE, self.opc.RETURN_VALUE)):
+                            or code[prev_op[pre_rtarget]]
+                            not in (self.opc.JUMP_ABSOLUTE, self.opc.RETURN_VALUE)
+                        ):
                             self.fixed_jumps[offset] = prev_op[next]
                             return
-                elif (code[next] == self.opc.JUMP_ABSOLUTE and self.is_jump_forward(target) and
-                      self.get_target(target) == self.get_target(next)):
+                elif (
+                    code[next] == self.opc.JUMP_ABSOLUTE
+                    and self.is_jump_forward(target)
+                    and self.get_target(target) == self.get_target(next)
+                ):
                     self.fixed_jumps[offset] = prev_op[next]
                     return
 
@@ -241,13 +311,17 @@ class Scanner30(Scanner3):
             if offset in self.ignore_if:
                 return
 
-            if (code[pre_rtarget] == self.opc.JUMP_ABSOLUTE and
-                pre_rtarget in self.stmts and
-                pre_rtarget != offset and
-                prev_op[pre_rtarget] != offset and
-                not (code[rtarget] == self.opc.JUMP_ABSOLUTE and
-                     code[rtarget+3] == self.opc.POP_BLOCK and
-                     code[prev_op[pre_rtarget]] != self.opc.JUMP_ABSOLUTE)):
+            if (
+                code[pre_rtarget] == self.opc.JUMP_ABSOLUTE
+                and pre_rtarget in self.stmts
+                and pre_rtarget != offset
+                and prev_op[pre_rtarget] != offset
+                and not (
+                    code[rtarget] == self.opc.JUMP_ABSOLUTE
+                    and code[rtarget + 3] == self.opc.POP_BLOCK
+                    and code[prev_op[pre_rtarget]] != self.opc.JUMP_ABSOLUTE
+                )
+            ):
                 rtarget = pre_rtarget
 
             # Does the "jump if" jump beyond a jump op?
@@ -268,16 +342,17 @@ class Scanner30(Scanner3):
                 if_end = self.get_target(pre_rtarget, 0)
 
                 # If the jump target is back, we are looping
-                if (if_end < pre_rtarget and
-                    (code[prev_op[if_end]] == self.opc.SETUP_LOOP)):
-                    if (if_end > start):
+                if if_end < pre_rtarget and (
+                    code[prev_op[if_end]] == self.opc.SETUP_LOOP
+                ):
+                    if if_end > start:
                         return
 
                 end = self.restrict_to_parent(if_end, parent)
 
-                self.structs.append({'type': 'if-then',
-                                     'start': start,
-                                     'end': pre_rtarget})
+                self.structs.append(
+                    {"type": "if-then", "start": start, "end": pre_rtarget}
+                )
                 self.not_continue.add(pre_rtarget)
 
                 # if rtarget < end and (
@@ -291,20 +366,17 @@ class Scanner30(Scanner3):
                 #     self.else_start[rtarget] = end
             elif self.is_jump_back(pre_rtarget, 0):
                 if_end = rtarget
-                self.structs.append({'type': 'if-then',
-                                     'start': start,
-                                     'end': pre_rtarget})
+                self.structs.append(
+                    {"type": "if-then", "start": start, "end": pre_rtarget}
+                )
                 self.not_continue.add(pre_rtarget)
-            elif code[pre_rtarget] in (self.opc.RETURN_VALUE,
-                                       self.opc.BREAK_LOOP):
-                self.structs.append({'type': 'if-then',
-                                     'start': start,
-                                     'end': rtarget})
+            elif code[pre_rtarget] in (self.opc.RETURN_VALUE, self.opc.BREAK_LOOP):
+                self.structs.append({"type": "if-then", "start": start, "end": rtarget})
                 # It is important to distingish if this return is inside some sort
                 # except block return
                 jump_prev = prev_op[offset]
                 if self.is_pypy and code[jump_prev] == self.opc.COMPARE_OP:
-                    if self.opc.cmp_op[code[jump_prev+1]] == 'exception-match':
+                    if self.opc.cmp_op[code[jump_prev + 1]] == "exception-match":
                         return
                 if self.version >= 3.5:
                     # Python 3.5 may remove as dead code a JUMP
@@ -332,7 +404,10 @@ class Scanner30(Scanner3):
                         if code[next_op] == self.opc.POP_TOP:
                             next_op = rtarget
                         for block in self.structs:
-                            if block['type'] == 'while-loop' and block['end'] == next_op:
+                            if (
+                                block["type"] == "while-loop"
+                                and block["end"] == next_op
+                            ):
                                 return
                         next_op += instruction_size(self.code[next_op], self.opc)
                         if code[next_op] == self.opc.POP_BLOCK:
@@ -342,20 +417,21 @@ class Scanner30(Scanner3):
                     self.fixed_jumps[offset] = rtarget
                     self.not_continue.add(pre_rtarget)
 
-
         elif op == self.opc.SETUP_EXCEPT:
             target = self.get_target(offset)
-            end    = self.restrict_to_parent(target, parent)
+            end = self.restrict_to_parent(target, parent)
             self.fixed_jumps[offset] = end
         elif op == self.opc.SETUP_FINALLY:
             target = self.get_target(offset)
-            end    = self.restrict_to_parent(target, parent)
+            end = self.restrict_to_parent(target, parent)
             self.fixed_jumps[offset] = end
         elif op in self.jump_if_pop:
             target = self.get_target(offset)
             if target > offset:
-                unop_target = self.last_instr(offset, target, self.opc.JUMP_FORWARD, target)
-                if unop_target and code[unop_target+3] != self.opc.ROT_TWO:
+                unop_target = self.last_instr(
+                    offset, target, self.opc.JUMP_FORWARD, target
+                )
+                if unop_target and code[unop_target + 3] != self.opc.ROT_TWO:
                     self.fixed_jumps[offset] = unop_target
                 else:
                     self.fixed_jumps[offset] = self.restrict_to_parent(target, parent)
@@ -366,8 +442,11 @@ class Scanner30(Scanner3):
             # misclassified as RETURN_END_IF. Handle that here.
             # In RETURN_VALUE, JUMP_ABSOLUTE, RETURN_VALUE is never RETURN_END_IF
             if op == self.opc.RETURN_VALUE:
-                if (offset+1 < len(code) and code[offset+1] == self.opc.JUMP_ABSOLUTE and
-                    offset in self.return_end_ifs):
+                if (
+                    offset + 1 < len(code)
+                    and code[offset + 1] == self.opc.JUMP_ABSOLUTE
+                    and offset in self.return_end_ifs
+                ):
                     self.return_end_ifs.remove(offset)
                     pass
                 pass
@@ -377,8 +456,10 @@ class Scanner30(Scanner3):
                 # then RETURN_VALUE is not RETURN_END_IF
                 rtarget = self.get_target(offset)
                 rtarget_prev = self.prev[rtarget]
-                if (code[rtarget_prev] == self.opc.RETURN_VALUE and
-                    rtarget_prev in self.return_end_ifs):
+                if (
+                    code[rtarget_prev] == self.opc.RETURN_VALUE
+                    and rtarget_prev in self.return_end_ifs
+                ):
                     i = rtarget_prev
                     while i != offset:
                         if code[i] in [opc.JUMP_FORWARD, opc.JUMP_ABSOLUTE]:
@@ -388,15 +469,17 @@ class Scanner30(Scanner3):
                 pass
         return
 
+
 if __name__ == "__main__":
     from uncompyle6 import PYTHON_VERSION
+
     if PYTHON_VERSION == 3.0:
         import inspect
+
         co = inspect.currentframe().f_code
         tokens, customize = Scanner30().ingest(co)
         for t in tokens:
             print(t)
         pass
     else:
-        print("Need to be Python 3.0 to demo; I am %s." %
-              PYTHON_VERSION)
+        print("Need to be Python 3.0 to demo; I am %s." % PYTHON_VERSION)

uncompyle6/scanners/scanner37base.py

@@ -29,8 +29,8 @@ For example:
 Finally we save token information.
 """
 
-from xdis import iscode
-from xdis.bytecode import instruction_size, _get_const_info, Instruction
+from xdis import iscode, instruction_size, Instruction
+from xdis.bytecode import _get_const_info
 
 from uncompyle6.scanner import Token
 import xdis

uncompyle6/semantics/aligner.py

@@ -142,7 +142,7 @@ def code_deparse_align(co, out=sys.stderr, version=None, is_pypy=None,
                             is_pypy = is_pypy)
 
     isTopLevel = co.co_name == '<module>'
-    deparsed.ast = deparsed.build_ast(tokens, customize, isTopLevel=isTopLevel)
+    deparsed.ast = deparsed.build_ast(tokens, customize, co, isTopLevel=isTopLevel)
 
     assert deparsed.ast == 'stmts', 'Should have parsed grammar start'
 

uncompyle6/semantics/check_ast.py

@@ -8,22 +8,29 @@ FIXME idea: extend parsing system to do same kinds of checks or nonterminal
 before reduction and don't reduce when there is a problem.
 """
 
+
 def checker(ast, in_loop, errors):
     if ast is None:
         return
-    in_loop = (in_loop or (ast.kind in ('while1stmt', 'whileTruestmt',
-                                      'whilestmt', 'whileelsestmt', 'while1elsestmt',
-                                      'for_block'))
-               or ast.kind.startswith('async_for'))
-    if ast.kind in ('aug_assign1', 'aug_assign2') and ast[0][0] == 'and':
+    in_loop = (
+        in_loop
+        or ast.kind.startswith("for")
+        or ast.kind.startswith("while")
+        or ast.kind.startswith("async_for")
+    )
+    if ast.kind in ("aug_assign1", "aug_assign2") and ast[0][0] == "and":
         text = str(ast)
-        error_text = '\n# improper augmented assigment (e.g. +=, *=, ...):\n#\t' + '\n# '.join(text.split("\n")) + '\n'
+        error_text = (
+            "\n# improper augmented assigment (e.g. +=, *=, ...):\n#\t"
+            + "\n# ".join(text.split("\n"))
+            + "\n"
+        )
         errors.append(error_text)
 
     for node in ast:
-        if not in_loop and node.kind in ('continue', 'break'):
+        if not in_loop and node.kind in ("continue", "break"):
             text = str(node)
-            error_text = '\n# not in loop:\n#\t' + '\n# '.join(text.split("\n"))
+            error_text = "\n# not in loop:\n#\t" + "\n# ".join(text.split("\n"))
             errors.append(error_text)
-        if hasattr(node, '__repr1__'):
+        if hasattr(node, "__repr1__"):
             checker(node, in_loop, errors)

uncompyle6/semantics/consts.py

@@ -142,17 +142,12 @@ PASS = SyntaxTree(
 )
 
 ASSIGN_DOC_STRING = lambda doc_string, doc_load: SyntaxTree(
-    "stmt",
+    "assign",
     [
         SyntaxTree(
-            "assign",
-            [
-                SyntaxTree(
-                    "expr", [Token(doc_load, pattr=doc_string, attr=doc_string)]
-                ),
-                SyntaxTree("store", [Token("STORE_NAME", pattr="__doc__")]),
-            ],
-        )
+            "expr", [Token(doc_load, pattr=doc_string, attr=doc_string)]
+        ),
+        SyntaxTree("store", [Token("STORE_NAME", pattr="__doc__")]),
     ],
 )
 
@@ -339,7 +334,11 @@ TABLE_DIRECT = {
     "raise_stmt1": ("%|raise %c\n", 0),
     "raise_stmt3": ("%|raise %c, %c, %c\n", 0, 1, 2),
     #    "yield":	        ( "yield %c", 0),
-    #    "return":	        ( "%|return %c\n", 0),
+
+    # Note: we have a custom rule, which calls when we don't
+    # have "return None"
+    "return":	( "%|return %c\n", 0),
+
     "return_if_stmt": ("return %c\n", 0),
     "ifstmt": (
         "%|if %c:\n%+%c%-",

uncompyle6/semantics/customize3.py

@@ -17,9 +17,8 @@
 """
 
 from uncompyle6.semantics.consts import TABLE_DIRECT
-from xdis.util import co_flags_is_async
 
-from xdis import iscode
+from xdis import co_flags_is_async, iscode
 from uncompyle6.scanner import Code
 from uncompyle6.semantics.helper import (
     find_code_node,
@@ -91,7 +90,7 @@ def customize_for_version3(self, version):
         code_obj = node[1].attr
         assert iscode(code_obj)
         code = Code(code_obj, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
 
         # skip over: sstmt, stmt, return, ret_expr

uncompyle6/semantics/customize35.py

@@ -15,8 +15,7 @@
 """Isolate Python 3.5 version-specific semantic actions here.
 """
 
-from xdis import iscode
-from xdis.util import co_flags_is_async
+from xdis import co_flags_is_async, iscode
 from uncompyle6.semantics.consts import (
     INDENT_PER_LEVEL,
     PRECEDENCE,

uncompyle6/semantics/customize37.py

@@ -143,6 +143,12 @@ def customize_for_version37(self, version):
 
             "importattr37": ("%c", (0, "IMPORT_NAME_ATTR")),
             "importlist37": ("%C", (0, maxint, ", ")),
+
+            "list_afor": (
+                " async for %[1]{%c} in %c%[1]{%c}",
+                (1, "store"), (0, "get_aiter"), (3, "list_iter"),
+            ),
+
             "list_if37": (" if %p%c", (0, 27), 1),
             "list_if37_not": (" if not %p%c", (0, 27), 1),
             "testfalse_not_or": ("not %c or %c", (0, "expr"), (2, "expr")),

uncompyle6/semantics/customize38.py

@@ -29,9 +29,9 @@ def customize_for_version38(self, version):
     #     del TABLE_DIRECT[lhs]
 
     TABLE_DIRECT.update({
-        'async_for_stmt38':  (
-            '%|async for %c in %c:\n%+%c%-%-\n\n',
-            (7, 'store'), (0, 'expr'), (8, 'for_block') ),
+        "async_for_stmt38":  (
+            "%|async for %c in %c:\n%+%c%-%-\n\n",
+            (2, "store"), (0, "expr"), (3, "for_block") ),
 
         'async_forelse_stmt38':  (
             '%|async for %c in %c:\n%+%c%-%|else:\n%+%c%-\n\n',
@@ -46,12 +46,28 @@ def customize_for_version38(self, version):
             (0, 'expr'), (6, 'store'),
             (7, 'suite_stmts') ),
 
+        "call_stmt": (
+            "%|%c\n", 0
+        ),
+
+        "except_cond_as": (
+            "%|except %c as %c:\n",
+            (1, "expr"),
+            (-2, "STORE_FAST"),
+            ),
+
         'except_handler38': (
             '%c', (2, 'except_stmts') ),
 
         'except_handler38a': (
             '%c', (-2, 'stmts') ),
 
+        "except_handler_as": (
+            "%c%+\n%+%c%-",
+            (1, "except_cond_as"),
+            (2, "tryfinallystmt"),
+        ),
+
         'except_ret38a': (
             'return %c', (4, 'expr') ),
 
@@ -101,12 +117,33 @@ def customize_for_version38(self, version):
         'try_except38': (
             '%|try:\n%+%c\n%-%|except:\n%|%-%c\n\n',
                    (-2, 'suite_stmts_opt'), (-1, 'except_handler38a') ),
-        'try_except_ret38': (
-            '%|try:\n%+%|return %c%-\n%|except:\n%+%|%c%-\n\n',
-                   (1, 'expr'), (-1, 'except_ret38a') ),
+
+        "try_except_as": (
+            "%|try:\n%+%c%-\n%|%-%c\n\n",
+            (-4, "suite_stmts"),  # Go from the end because of POP_BLOCK variation
+            (-3, "except_handler_as"),
+        ),
+
+        "try_except_ret38": (
+            "%|try:\n%+%c%-\n%|except:\n%+%|%c%-\n\n",
+            (1, "returns"),
+            (2, "except_ret38a"),
+        ),
         'tryfinally38rstmt': (
             '%|try:\n%+%c%-%|finally:\n%+%c%-\n\n',
-                   (3, 'returns'), 6 ),
+            (0, "sf_pb_call_returns"),
+            (-1, ("ss_end_finally", "suite_stmts")),
+        ),
+        "tryfinally38rstmt2": (
+            "%|try:\n%+%c%-%|finally:\n%+%c%-\n\n",
+            (4, "returns"),
+            -2, "ss_end_finally"
+        ),
+        "tryfinally38rstmt3": (
+            "%|try:\n%+%|return %c%-\n%|finally:\n%+%c%-\n\n",
+            (1, "expr"),
+            (-1, "ss_end_finally")
+        ),
         'tryfinally38stmt': (
             '%|try:\n%+%c%-%|finally:\n%+%c%-\n\n',
             (1, "suite_stmts_opt"),

uncompyle6/semantics/fragments.py

@@ -67,8 +67,7 @@ from __future__ import print_function
 
 import re
 
-from xdis import iscode
-from xdis.magics import sysinfo2float
+from xdis import iscode, sysinfo2float
 from uncompyle6.semantics import pysource
 from uncompyle6 import parser
 from uncompyle6.scanner import Token, Code, get_scanner
@@ -682,7 +681,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
         assert iscode(cn.attr)
 
         code = Code(cn.attr, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
         ast = ast[0][0][0]
 
@@ -729,7 +728,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
         code_name = code.co_name
         code = Code(code, self.scanner, self.currentclass)
 
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
 
         self.customize(code._customize)
         if ast[0] == "sstmt":
@@ -851,7 +850,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
         self.prec = 27
 
         code = Code(node[1].attr, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
         if node == "set_comp":
             ast = ast[0][0][0]
@@ -963,7 +962,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
 
         self.prune()
 
-    def n_listcomp(self, node):
+    def n_list_comp(self, node):
         self.write("[")
         if node[0].kind == "load_closure":
             self.listcomprehension_walk2(node)
@@ -993,7 +992,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
         self.prec = 27
 
         code = Code(node[1].attr, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
         ast = ast[0][0][0]
         store = ast[3]
@@ -1143,9 +1142,8 @@ class FragmentsWalker(pysource.SourceWalker, object):
         self.name = old_name
         self.return_none = rn
 
-    def build_ast(
-        self, tokens, customize, is_lambda=False, noneInNames=False, isTopLevel=False
-    ):
+    def build_ast(self, tokens, customize, code, is_lambda=False,
+                  noneInNames=False, isTopLevel=False):
 
         # FIXME: DRY with pysource.py
 
@@ -1163,6 +1161,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
                 # modularity is broken here
                 p_insts = self.p.insts
                 self.p.insts = self.scanner.insts
+                self.p.offset2inst_index = self.scanner.offset2inst_index
                 ast = python_parser.parse(self.p, tokens, customize)
                 self.p.insts = p_insts
             except (python_parser.ParserError, AssertionError) as e:
@@ -1200,10 +1199,11 @@ class FragmentsWalker(pysource.SourceWalker, object):
             # modularity is broken here
             p_insts = self.p.insts
             self.p.insts = self.scanner.insts
+            self.p.offset2inst_index = self.scanner.offset2inst_index
             ast = parser.parse(self.p, tokens, customize)
             self.p.insts = p_insts
         except (parser.ParserError, AssertionError) as e:
-            raise ParserError(e, tokens)
+            raise ParserError(e, tokens, {})
 
         maybe_show_tree(self, ast)
 

uncompyle6/semantics/make_function2.py

@@ -18,9 +18,7 @@ All the crazy things we have to do to handle Python functions in Python before 3
 The saga of changes continues in 3.0 and above and in other files.
 """
 from xdis import iscode, code_has_star_arg, code_has_star_star_arg
-from xdis.util import CO_GENERATOR
 from uncompyle6.scanner import Code
-from uncompyle6.parsers.treenode import SyntaxTree
 from uncompyle6 import PYTHON3
 from uncompyle6.semantics.parser_error import ParserError
 from uncompyle6.parser import ParserError as ParserError2
@@ -102,6 +100,7 @@ def make_function2(self, node, is_lambda, nested=1, code_node=None):
         ast = self.build_ast(
             code._tokens,
             code._customize,
+            code,
             is_lambda=is_lambda,
             noneInNames=("None" in code.co_names),
         )

uncompyle6/semantics/make_function3.py

@@ -16,8 +16,7 @@
 All the crazy things we have to do to handle Python functions in 3.0-3.5 or so.
 The saga of changes before and after is in other files.
 """
-from xdis import iscode, code_has_star_arg, code_has_star_star_arg
-from xdis.util import CO_GENERATOR
+from xdis import iscode, code_has_star_arg, code_has_star_star_arg, CO_GENERATOR
 from uncompyle6.scanner import Code
 from uncompyle6.parsers.treenode import SyntaxTree
 from uncompyle6 import PYTHON3
@@ -39,6 +38,7 @@ from uncompyle6.show import maybe_show_tree_param_default
 
 # FIXME: DRY the below code...
 
+
 def make_function3_annotate(
     self, node, is_lambda, nested=1, code_node=None, annotate_last=-1
 ):
@@ -132,6 +132,7 @@ def make_function3_annotate(
         ast = self.build_ast(
             code._tokens,
             code._customize,
+            code,
             is_lambda=is_lambda,
             noneInNames=("None" in code.co_names),
         )
@@ -269,8 +270,8 @@ def make_function3_annotate(
                 self.write("\n" + indent)
                 line_number = self.line_number
             self.write(" -> ")
-            if 'return' in annotate_dict:
-                self.write(annotate_dict['return'])
+            if "return" in annotate_dict:
+                self.write(annotate_dict["return"])
             else:
                 # value, string = annotate_args['return']
                 # if string:
@@ -427,9 +428,7 @@ def make_function3(self, node, is_lambda, nested=1, code_node=None):
             lc_index = -3
             pass
 
-        if (len(node) > 2
-            and (have_kwargs or node[lc_index].kind != "load_closure")
-        ):
+        if len(node) > 2 and (have_kwargs or node[lc_index].kind != "load_closure"):
 
             # Find the index in "node" where the first default
             # parameter value is located. Note this is in contrast to
@@ -480,7 +479,7 @@ def make_function3(self, node, is_lambda, nested=1, code_node=None):
         if is_lambda:
             kwargs = []
             for i in range(kwonlyargcount):
-                paramnames.append(scanner_code.co_varnames[argc+i])
+                paramnames.append(scanner_code.co_varnames[argc + i])
             pass
         else:
             kwargs = list(scanner_code.co_varnames[argc : argc + kwonlyargcount])
@@ -493,6 +492,7 @@ def make_function3(self, node, is_lambda, nested=1, code_node=None):
         ast = self.build_ast(
             scanner_code._tokens,
             scanner_code._customize,
+            scanner_code,
             is_lambda=is_lambda,
             noneInNames=("None" in code.co_names),
         )
@@ -687,5 +687,5 @@ def make_function3(self, node, is_lambda, nested=1, code_node=None):
         if need_bogus_yield:
             self.template_engine(("%|if False:\n%+%|yield None%-",), node)
 
-    scanner_code._tokens = None # save memory
+    scanner_code._tokens = None  # save memory
     scanner_code._customize = None  # save memory

uncompyle6/semantics/make_function36.py

@@ -16,8 +16,13 @@
 All the crazy things we have to do to handle Python functions in 3.6 and above.
 The saga of changes before 3.6 is in other files.
 """
-from xdis import iscode, code_has_star_arg, code_has_star_star_arg
-from xdis.util import CO_GENERATOR, CO_ASYNC_GENERATOR
+from xdis import (
+    iscode,
+    code_has_star_arg,
+    code_has_star_star_arg,
+    CO_GENERATOR,
+    CO_ASYNC_GENERATOR,
+)
 from uncompyle6.scanner import Code
 from uncompyle6.parsers.treenode import SyntaxTree
 from uncompyle6.semantics.parser_error import ParserError
@@ -107,9 +112,7 @@ def make_function36(self, node, is_lambda, nested=1, code_node=None):
             if annotate_node == "dict" and annotate_name_node.kind.startswith(
                 "BUILD_CONST_KEY_MAP"
             ):
-                types = [
-                    self.traverse(n, indent="") for n in annotate_node[:-2]
-                ]
+                types = [self.traverse(n, indent="") for n in annotate_node[:-2]]
                 names = annotate_node[-2].attr
                 l = len(types)
                 assert l == len(names)
@@ -170,6 +173,7 @@ def make_function36(self, node, is_lambda, nested=1, code_node=None):
         ast = self.build_ast(
             scanner_code._tokens,
             scanner_code._customize,
+            scanner_code,
             is_lambda=is_lambda,
             noneInNames=("None" in code.co_names),
         )
@@ -329,9 +333,7 @@ def make_function36(self, node, is_lambda, nested=1, code_node=None):
             self.write(" -> %s" % annotate_dict["return"])
         self.println(":")
 
-    if (
-        node[-2] == "docstring" and not is_lambda
-    ):
+    if node[-2] == "docstring" and not is_lambda:
         # docstring exists, dump it
         self.println(self.traverse(node[-2]))
 
@@ -370,5 +372,5 @@ def make_function36(self, node, is_lambda, nested=1, code_node=None):
         if need_bogus_yield:
             self.template_engine(("%|if False:\n%+%|yield None%-",), node)
 
-    scanner_code._tokens = None # save memory
+    scanner_code._tokens = None  # save memory
     scanner_code._customize = None  # save memory

uncompyle6/semantics/pysource.py

@@ -80,7 +80,7 @@ Python.
 #
 #   Escapes in the format string are:
 #
-#     %c  evaluate the node recursively. Its argument is a single
+#     %c  evaluate/traverse the node recursively. Its argument is a single
 #         integer or tuple representing a node index.
 #         If a tuple is given, the first item is the node index while
 #         the second item is a string giving the node/noterminal name.
@@ -91,7 +91,7 @@ Python.
 #         index and the precedence value, an integer. If 3 items are given,
 #         the second item is the nonterminal name and the precedence is given last.
 #
-#     %C  evaluate children recursively, with sibling children separated by the
+#     %C  evaluate/travers children recursively, with sibling children separated by the
 #         given string.  It needs a 3-tuple: a starting node, the maximimum
 #         value of an end node, and a string to be inserted between sibling children
 #
@@ -99,7 +99,7 @@ Python.
 #         on the LHS of an assignment statement since BUILD_TUPLE_n pretty-prints
 #         other tuples. The specifier takes no arguments
 #
-#     %P same as %C but sets operator precedence.  Its argument is a 4-tuple:
+#     %P  same as %C but sets operator precedence.  Its argument is a 4-tuple:
 #         the node low and high indices, the separator, a string the precidence
 #         value, an integer.
 #
@@ -115,7 +115,13 @@ Python.
 #
 #     %- decrease current indentation level. Takes no arguments.
 #
-#     %{...} evaluate ... in context of N
+#     %{EXPR} Python eval(EXPR) in context of node. Takes no arguments
+#
+#     %[N]{EXPR} Python eval(EXPR) in context of node[N]. Takes no arguments
+#
+#     %[N]{%X} evaluate/recurse on child node[N], using specifier %X.
+#     %X can be one of the above, e.g. %c, %p, etc. Takes the arguemnts
+#     that the specifier uses.
 #
 #     %% literal '%'. Takes no arguments.
 #
@@ -129,8 +135,7 @@ import sys
 IS_PYPY = "__pypy__" in sys.builtin_module_names
 PYTHON3 = sys.version_info >= (3, 0)
 
-from xdis import iscode
-from xdis.util import COMPILER_FLAG_BIT
+from xdis import iscode, COMPILER_FLAG_BIT
 
 from uncompyle6.parser import get_python_parser
 from uncompyle6.parsers.treenode import SyntaxTree
@@ -159,7 +164,6 @@ from uncompyle6.semantics.consts import (
     NONE,
     RETURN_NONE,
     PASS,
-    ASSIGN_DOC_STRING,
     NAME_MODULE,
     TAB,
     INDENT_PER_LEVEL,
@@ -504,14 +508,14 @@ class SourceWalker(GenericASTTraversal, object):
             self.preorder(node[0])
             self.prune()
         else:
-            self.write(self.indent, "return")
             # One reason we worry over whether we use "return None" or "return"
             # is that inside a generator, "return None" is illegal.
             # Thank you, Python!
             if self.return_none or not self.is_return_none(node):
-                self.write(" ")
-                self.preorder(node[0])
-            self.println()
+                self.default(node)
+            else:
+                self.template_engine(("%|return\n",), node)
+
             self.prune()  # stop recursing
 
     def n_return_if_stmt(self, node):
@@ -583,8 +587,7 @@ class SourceWalker(GenericASTTraversal, object):
         if n == "LOAD_CONST" and repr(n.pattr)[0] == "-":
             self.prec = 6
 
-        # print(n.kind, p, "<", self.prec)
-        # print(self.f.getvalue())
+        # print("XXX", n.kind, p, "<", self.prec)
 
         if p < self.prec:
             self.write("(")
@@ -600,7 +603,6 @@ class SourceWalker(GenericASTTraversal, object):
             # If expr is yield we want parens.
             self.prec = PRECEDENCE["yield"] - 1
             self.n_expr(node[0])
-            p = self.prec
         else:
             self.n_expr(node)
 
@@ -890,6 +892,12 @@ class SourceWalker(GenericASTTraversal, object):
         doc_node = node[0]
         if doc_node.attr:
             docstring = doc_node.attr
+            if not isinstance(docstring, str):
+                # FIXME: we have mistakenly tagged something as a doc
+                # string in transform when it isn't one.
+                # The rule in n_mkfunc is pretty flaky.
+                self.prune()
+                return
         else:
             docstring = node[0].pattr
 
@@ -1109,7 +1117,7 @@ class SourceWalker(GenericASTTraversal, object):
         assert iscode(cn.attr)
 
         code = Code(cn.attr, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
 
         # Remove single reductions as in ("stmts", "sstmt"):
@@ -1193,7 +1201,7 @@ class SourceWalker(GenericASTTraversal, object):
         assert iscode(code), node[code_index]
         code = Code(code, self.scanner, self.currentclass)
 
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
 
         # skip over: sstmt, stmt, return, ret_expr
@@ -1205,7 +1213,7 @@ class SourceWalker(GenericASTTraversal, object):
             ast = ast[0]
 
         # Pick out important parts of the comprehension:
-        # * the variable we interate over: "store"
+        # * the variable we iterate over: "store"
         # * the results we accumulate: "n"
 
         is_30_dict_comp = False
@@ -1265,17 +1273,25 @@ class SourceWalker(GenericASTTraversal, object):
         # Iterate to find the innermost store
         # We'll come back to the list iteration below.
 
-        while n in ("list_iter", "comp_iter"):
+        while n in ("list_iter", "list_afor", "list_afor2", "comp_iter"):
             # iterate one nesting deeper
             if self.version == 3.0 and len(n) == 3:
                 assert n[0] == "expr" and n[1] == "expr"
                 n = n[1]
+            elif n == "list_afor":
+                n = n[1]
+            elif n == "list_afor2":
+                if n[1] == "store":
+                    store = n[1]
+                n = n[3]
             else:
                 n = n[0]
 
             if n in ("list_for", "comp_for"):
                 if n[2] == "store" and not store:
                     store = n[2]
+                    if not comp_store:
+                        comp_store = store
                 n = n[3]
             elif n in ("list_if", "list_if_not",
                        "list_if37", "list_if37_not",
@@ -1380,7 +1396,7 @@ class SourceWalker(GenericASTTraversal, object):
         self.prec = 27
 
         code = Code(node[1].attr, self.scanner, self.currentclass)
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         self.customize(code._customize)
 
         # Remove single reductions as in ("stmts", "sstmt"):
@@ -2039,10 +2055,17 @@ class SourceWalker(GenericASTTraversal, object):
             elif typ == "c":
                 index = entry[arg]
                 if isinstance(index, tuple):
-                    assert node[index[0]] == index[1], (
-                        "at %s[%d], expected '%s' node; got '%s'"
-                        % (node.kind, arg, index[1], node[index[0]].kind)
-                    )
+                    if isinstance(index[1], str):
+                        assert node[index[0]] == index[1], (
+                            "at %s[%d], expected '%s' node; got '%s'"
+                            % (node.kind, arg, index[1], node[index[0]].kind)
+                        )
+                    else:
+                        assert node[index[0]] in index[1], (
+                            "at %s[%d], expected to be in '%s' node; got '%s'"
+                            % (node.kind, arg, index[1], node[index[0]].kind)
+                        )
+
                     index = index[0]
                 assert isinstance(
                     index, int
@@ -2051,6 +2074,16 @@ class SourceWalker(GenericASTTraversal, object):
                     arg,
                     type(index),
                 )
+
+                try:
+                    node[index]
+                except IndexError:
+                    raise RuntimeError(
+                        """
+                        Expanding '%' in template '%s[%s]':
+                        %s is invalid; has only %d entries
+                        """ % (node,kind, enty, arg, index, len(node))
+                    )
                 self.preorder(node[index])
 
                 arg += 1
@@ -2112,7 +2145,6 @@ class SourceWalker(GenericASTTraversal, object):
                 self.prec = p
                 arg += 1
             elif typ == "{":
-                d = node.__dict__
                 expr = m.group("expr")
 
                 # Line mapping stuff
@@ -2123,10 +2155,16 @@ class SourceWalker(GenericASTTraversal, object):
                 ):
                     self.source_linemap[self.current_line_number] = node.linestart
 
-                try:
-                    self.write(eval(expr, d, d))
-                except:
-                    raise
+                if expr[0] == "%":
+                    index = entry[arg]
+                    self.template_engine((expr, index), node)
+                    arg += 1
+                else:
+                    d = node.__dict__
+                    try:
+                        self.write(eval(expr, d, d))
+                    except:
+                        raise
             m = escape.search(fmt, i)
         self.write(fmt[i:])
 
@@ -2272,7 +2310,7 @@ class SourceWalker(GenericASTTraversal, object):
 
         indent = self.indent
         # self.println(indent, '#flags:\t', int(code.co_flags))
-        ast = self.build_ast(code._tokens, code._customize)
+        ast = self.build_ast(code._tokens, code._customize, code)
         code._tokens = None  # save memory
         assert ast == "stmts"
 
@@ -2283,6 +2321,7 @@ class SourceWalker(GenericASTTraversal, object):
         if ast[0] == "docstring":
             self.println(self.traverse(ast[0]))
             del ast[0]
+            first_stmt = ast[0]
 
         if 3.0 <= self.version <= 3.3:
             try:
@@ -2346,10 +2385,10 @@ class SourceWalker(GenericASTTraversal, object):
         # if docstring exists, dump it
         if code.co_consts and code.co_consts[0] is not None and len(ast) > 0:
             do_doc = False
-            if is_docstring(ast[0]):
+            if is_docstring(ast[0], self.version, code.co_consts):
                 i = 0
                 do_doc = True
-            elif len(ast) > 1 and is_docstring(ast[1]):
+            elif len(ast) > 1 and is_docstring(ast[1], self.version, code.co_consts):
                 i = 1
                 do_doc = True
             if do_doc and self.hide_internal:
@@ -2425,7 +2464,7 @@ class SourceWalker(GenericASTTraversal, object):
         self.return_none = rn
 
     def build_ast(
-        self, tokens, customize, is_lambda=False, noneInNames=False, isTopLevel=False
+        self, tokens, customize, code, is_lambda=False, noneInNames=False, isTopLevel=False
     ):
 
         # FIXME: DRY with fragments.py
@@ -2445,13 +2484,13 @@ class SourceWalker(GenericASTTraversal, object):
                 p_insts = self.p.insts
                 self.p.insts = self.scanner.insts
                 self.p.offset2inst_index = self.scanner.offset2inst_index
-                ast = python_parser.parse(self.p, tokens, customize)
+                ast = python_parser.parse(self.p, tokens, customize, code)
                 self.customize(customize)
                 self.p.insts = p_insts
 
             except (python_parser.ParserError, AssertionError) as e:
                 raise ParserError(e, tokens, self.p.debug['reduce'])
-            transform_ast = self.treeTransform.transform(ast)
+            transform_ast = self.treeTransform.transform(ast, code)
             self.maybe_show_tree(ast)
             del ast  # Save memory
             return transform_ast
@@ -2483,7 +2522,7 @@ class SourceWalker(GenericASTTraversal, object):
             self.p.insts = self.scanner.insts
             self.p.offset2inst_index = self.scanner.offset2inst_index
             self.p.opc = self.scanner.opc
-            ast = python_parser.parse(self.p, tokens, customize)
+            ast = python_parser.parse(self.p, tokens, customize, code)
             self.p.insts = p_insts
         except (python_parser.ParserError, AssertionError) as e:
             raise ParserError(e, tokens, self.p.debug['reduce'])
@@ -2491,7 +2530,7 @@ class SourceWalker(GenericASTTraversal, object):
         checker(ast, False, self.ast_errors)
 
         self.customize(customize)
-        transform_ast = self.treeTransform.transform(ast)
+        transform_ast = self.treeTransform.transform(ast, code)
 
         self.maybe_show_tree(ast)
 
@@ -2553,7 +2592,7 @@ def code_deparse(
     )
 
     isTopLevel = co.co_name == "<module>"
-    deparsed.ast = deparsed.build_ast(tokens, customize, isTopLevel=isTopLevel)
+    deparsed.ast = deparsed.build_ast(tokens, customize, co, isTopLevel=isTopLevel)
 
     #### XXX workaround for profiling
     if deparsed.ast is None:

uncompyle6/semantics/transform.py

@@ -13,7 +13,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-from xdis import iscode
 from uncompyle6.show import maybe_show_tree
 from copy import copy
 from spark_parser import GenericASTTraversal, GenericASTTraversalPruningException
@@ -21,16 +20,39 @@ from spark_parser import GenericASTTraversal, GenericASTTraversalPruningExceptio
 from uncompyle6.semantics.helper import find_code_node
 from uncompyle6.parsers.treenode import SyntaxTree
 from uncompyle6.scanners.tok import NoneToken, Token
-from uncompyle6.semantics.consts import RETURN_NONE
+from uncompyle6.semantics.consts import RETURN_NONE, ASSIGN_DOC_STRING
 
 
-def is_docstring(node):
+def is_docstring(node, version, co_consts):
     if node == "sstmt":
         node = node[0]
-    try:
-        return node.kind == "assign" and node[1][0].pattr == "__doc__"
-    except:
-        return False
+    # TODO: the test below on 2.7 succeeds for
+    #   class OldClass:
+    #     __doc__ = DocDescr()
+    # which produces:
+    #
+    #   assign (2)
+    #      0. expr
+    #         call (2)
+    #              0. expr
+    #                  L.  16         6  LOAD_DEREF            0  'DocDescr'
+    #              1.                 9  CALL_FUNCTION_0       0  None
+    #      1. store
+    #
+    # See Python 2.7 test_descr.py
+
+    # If ASSIGN_DOC_STRING doesn't work we need something like the below
+    # but more elaborate to address the above.
+
+    # try:
+    #     return node.kind == "assign" and node[1][0].pattr == "__doc__"
+    # except:
+    #     return False
+    if version <= 2.7:
+        doc_load = "LOAD_CONST"
+    else:
+        doc_load = "LOAD_STR"
+    return node == ASSIGN_DOC_STRING(co_consts[0], doc_load)
 
 
 def is_not_docstring(call_stmt_node):
@@ -95,10 +117,18 @@ class TreeTransform(GenericASTTraversal, object):
 
         code = find_code_node(node, code_index).attr
 
+        mkfunc_pattr = node[-1].pattr
+        if isinstance(mkfunc_pattr, tuple):
+            assert len(mkfunc_pattr, 4) and isinstance(mkfunc_pattr, int)
+            is_closure = node[-1].pattr[3] != 0
+        else:
+            # FIXME: This is what we had before. It is hoaky and probably wrong.
+            is_closure = mkfunc_pattr == "closure"
+
         if (
-            node[-1].pattr != "closure"
+            (not is_closure)
             and len(code.co_consts) > 0
-            and code.co_consts[0] is not None
+            and isinstance(code.co_consts[0], str)
         ):
             docstring_node = SyntaxTree(
                 "docstring", [Token("LOAD_STR", has_arg=True, pattr=code.co_consts[0])]
@@ -409,7 +439,7 @@ class TreeTransform(GenericASTTraversal, object):
         node = self.preorder(node)
         return node
 
-    def transform(self, ast):
+    def transform(self, ast, code):
         self.maybe_show_tree(ast)
         self.ast = copy(ast)
         self.ast = self.traverse(self.ast, is_lambda=False)
@@ -430,9 +460,10 @@ class TreeTransform(GenericASTTraversal, object):
             for i in range(len(self.ast)):
                 sstmt = ast[i]
                 if len(sstmt) == 1 and sstmt == "sstmt":
-                    ast[i] = ast[i][0]
+                    self.ast[i] = self.ast[i][0]
 
-                if is_docstring(self.ast[i]):
+                if is_docstring(self.ast[i], self.version, code.co_consts):
+                    load_const = self.ast[i].first_child()
                     docstring_ast = SyntaxTree(
                         "docstring",
                         [
@@ -440,8 +471,8 @@ class TreeTransform(GenericASTTraversal, object):
                                 "LOAD_STR",
                                 has_arg=True,
                                 offset=0,
-                                attr=self.ast[i][0][0].attr,
-                                pattr=self.ast[i][0][0].pattr,
+                                attr=load_const.attr,
+                                pattr=load_const.pattr,
                             )
                         ],
                     )