Remove setuptools pin

Change copyright year to 2018

.circleci/config.yml

@@ -64,7 +64,7 @@ jobs:
       # Test
       #   This would typically be a build job when using workflows, possibly combined with build
       # This is based on your 1.0 configuration file or project settings
-      - run: sudo python ./setup.py develop && make check-3.6
+      - run: sudo pip install -e . && make check-3.6
       - run: cd ./test/stdlib && bash ./runtests.sh 'test_[p-z]*.py'
       # Teardown
       #   If you break your build into multiple jobs with workflows, you will probably want to do the parts of this that are relevant in each

.github/workflows/osx.yml

@@ -12,7 +12,7 @@ jobs:
     strategy:
       matrix:
         os: [macOS]
-        python-version: [3.7, 3.8]
+        python-version: [3.8]
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}

.github/workflows/ubuntu.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.7, 3.8]
+        python-version: [3.8]
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}

.github/workflows/windows.yml

@@ -12,7 +12,7 @@ jobs:
     strategy:
       matrix:
         os: [windows]
-        python-version: [3.7, 3.8]
+        python-version: [3.8]
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}

.pre-commit-config.yaml

@@ -6,17 +6,17 @@ repos:
     hooks:
     - id: check-merge-conflict
     - id: debug-statements
-      stages: [commit]
+      stages: [pre-commit]
     - id: end-of-file-fixer
-      stages: [commit]
+      stages: [pre-commit]
 -   repo: https://github.com/pycqa/isort
     rev: 5.13.2
     hooks:
       - id: isort
-        stages: [commit]
+        stages: [pre-commit]
 -   repo: https://github.com/psf/black
     rev: 23.12.1
     hooks:
     - id: black
       language_version: python3
-      stages: [commit]
+      stages: [pre-commit]

HOW-TO-REPORT-A-BUG.md

@@ -19,17 +19,17 @@
 
 TL;DR (too long; didn't read)
 
-* Don't do something illegal. And don't ask me to do something illegal or help you do something illegal
-* We already have an infinite supply of decompilation bugs that need fixing, and an automated mechanism for finding more. Decompilation bugs get addressed by easiness to fix and by whim. If you expect yours to be fixed ahead of those, you need to justify why.
+* Don't do something illegal. And don't ask me to do something illegal or help you do something illegal.
+* We already have an infinite supply of decompilation bugs that need fixing, and an automated mechanism for finding more. Decompilation bugs get addressed by easiness to fix and by whim. If you expect yours to be fixed ahead of those, you need to justify why. You can ask for a hand-assisted decompilation, but that is expensive and beyond what most are willing to spend. A $100 fee is needed just to look at the bytecode.
 * When asking for help, you may be asked for what you've tried on your own first. There are plenty of sources of information about this code.
-* If you are looking for *timely* help or support, well, that is typically known as a _paid_ service. I don't really have a mechanism for that since I have a full-time job. But supporting the project is an approximation.
-* Submitting a bug or issue report that is likely to get acted upon may require a bit of effort on your part to make it easy for the problem solver. If you are not willing to do that, please don't waste our time. As indicated above, supporting the project will increase the likelihood of your issue getting noticed and acted upon.
+* Bugs get fixed, slowly. Sometimes on the order of months or years. If you are looking for *timely* help or support, that is typically known as a _paid_ service.
+* Submitting a bug or issue report that is likely to get acted upon may require a bit of effort on your part to make it easy for the problem solver. If you are not willing to do that, please don't waste your or our time. Bug report may be closed with about as much thought and care as apparent in the effort to create the bug. Supporting the project however, does increase the likelihood of your issue getting noticed and acted upon.
 
 # Ethics
 
-I do not condone using this program for unethical or illegal purposes. More detestable, at least to me, is asking for help to assist you in something that might not legitimate.
+Do not use this program for unethical or illegal purposes. More detestable, at least to me, is asking for help to assist you in something that might not legitimate.
 
-Don't use the issue tracker for such solicitations. To try to stave off illegitimate behavior, you should note that the issue tracker, the code, and bugs mentioned in that are in the open: there is no
+Don't use the issue tracker for such unethical or illegal solicitations. To try to stave off illegitimate behavior, you should note that the issue tracker, the code, and bugs mentioned in that are in the open: there is no
 confidentiality. You may be asked about the authorship or claimed ownership of the bytecode. If I think something is not quite right, I may label the issue questionable which may make the it easier those who are looking for illegal activity.
 
 
@@ -37,13 +37,13 @@ confidentiality. You may be asked about the authorship or claimed ownership of t
 
 For many open-source projects bugs where the expectation is that bugs are rare, reporting bugs in a *thoughtful* way can be helpful. See also [How to Ask Questions the Smart Way](http://www.catb.org/~esr/faqs/smart-questions.html).
 
-In this project though, most of the bug reports boil down to the something like: I am trying to reverse engineer some code that I am not the author/owner and that person doesn't want me to have access to. I am hitting a problem somewhere along the line which might have to do with decompilation, but it could be something else like how the bytecode was extracted, some problem in deliberately obfuscated code, or the use some kind of Python bytecode version that isn't supported by the decompiler.
+In this project though, most of the bug reports boil down to the something like: I am trying to reverse engineer some code that I am not the author/owner and that person doesn't want me to have access to. I am hitting a problem somewhere along the line which might have to do with decompilation. But it could be something else like how the bytecode was extracted, some problem in deliberately obfuscated code, or the use some kind of Python bytecode version that isn't supported by the decompiler. Gee this stuff is complicated, here's an open source project, so maybe someone there will help me figure stuff out.
 
-While you are free to report these, unless you sponsor the project, I may close them with about the same amount of effort spent that I think was used to open the report for them. And if you spent a considerable amount of time to create the bug report but didn't follow instructions given here and in the issue template, I am sorry in advance. Just go back, read, and follow instructions.
+While you are free to report bugs, unless you sponsor the project, I may close them with about the same amount of effort spent that I think was used to open the report for them. And if you spent a considerable amount of time to create the bug report but didn't follow instructions given here and in the issue template, I am sorry in advance. Just go back, read, and follow instructions.
 
 This project already has an infinite supply of bugs that have been narrowed to the most minimal form and where I have source code to compare against. And in the unlikely event this supply runs out, I have automated means for generating *another* infinite supply.
 
-In this project the task of justifying why addressing your bug is of use to the community, and why it should be prioritized over the others, is the bug reporter's responsibility.
+The task of justifying why addressing your bug is of use to the community, and why it should be prioritized over the others, is the bug reporter's responsibility.
 
 While in the abstract, I have no problem answering questions about how to read a Python traceback or install Python software, or trying to understand what is going wrong in your particular setup, I am not a paid support person and there other things I'd rather be doing with my limited volunteer time. So save us both time, effort, and aggravation: use other avenues like StackOverflow. Again, justifying why you should receive unpaid help is the help requester's responsibility.
 

NEWS.md

@@ -1,3 +1,9 @@
+3.9.2: 2024-07-21
+=================
+
+- track xdis API changes
+- Bug fixes and lint
+
 3.9.1: 2024-05-15
 =================
 

README.rst

@@ -10,6 +10,8 @@ uncompyle6
 A native Python cross-version decompiler and fragment decompiler.
 The successor to decompyle, uncompyle, and uncompyle2.
 
+I gave a talk on this at `BlackHat Asia 2024 <https://youtu.be/H-7ZNrpsV50?si=nOaixgYHr7RbILVS>`_.
+
 
 Introduction
 ------------
@@ -162,7 +164,7 @@ the right bytecode for the Python interpreter that will be checking
 the syntax.
 
 You can also cross compare the results with another version of
-`uncompyle6` since there are sometimes regressions in decompiling
+*uncompyle6* since there are sometimes regressions in decompiling
 specific bytecode as the overall quality improves.
 
 For Python 3.7 and 3.8, the code in decompyle3_ is generally
@@ -265,7 +267,7 @@ be solved if one were to put in the time to do so. The problem is that
 there aren't that many people who have been working on bug fixing.
 
 Some of the bugs in 3.7 and 3.8 are simply a matter of back-porting
-the fixes in decompyle3. Volunteers are welcome to do so.
+the fixes in *decompyle3*. Any volunteers?
 
 You may run across a bug, that you want to report. Please do so after
 reading `How to report a bug
@@ -274,11 +276,15 @@ follow the `instructions when opening an issue <https://github.com/rocky/python-
 
 Be aware that it might not get my attention for a while. If you
 sponsor or support the project in some way, I'll prioritize your
-issues above the queue of other things I might be doing instead.
+issues above the queue of other things I might be doing instead. In
+rare situtations, I can do a hand decompilation of bytecode for a fee.
+However this is expansive, usually beyond what most people are willing
+to spend.
 
 See Also
 --------
 
+* https://rocky.github.io/blackhat-asia-2024-additional/all-notes-print.html :  How to Read and Write a High-Level Bytecode Decompiler: ``uncompyle6`` ``decompyle3`` -- BlackHat 2024 Asia (`video <https://www.youtube.com/watch?v=NA77SFncppE>`_). A big thanks to the Organizers and Reviewers for letting me speak. This kind of thing encourages me to work on projects like this.
 * https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7 and 3.8. Changes in that will get migrated back here.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
@@ -303,8 +309,8 @@ See Also
 .. _uncompyle2: https://github.com/wibiti/uncompyle2
 .. _unpyc37: https://github.com/andrew-tavera/unpyc37
 .. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
-.. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
-		 :target: https://travis-ci.org/rocky/python-uncompyle6
+.. |buildstatus| image:: https://circleci.com/gh/rocky/python-uncompyle6.svg?style=svg
+		 :target: https://app.circleci.com/pipelines/github/rocky/python-uncompyle6
 .. |packagestatus| image:: https://repology.org/badge/vertical-allrepos/python:uncompyle6.svg
 		 :target: https://repology.org/project/python:uncompyle6/versions
 .. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84

__pkginfo__.py

@@ -79,7 +79,7 @@ entry_points = {
     ]
 }
 ftp_url = None
-install_requires = ["click", "spark-parser >= 1.8.9, < 1.9.0", "xdis >= 6.0.8, < 6.2.0"]
+install_requires = ["click", "spark-parser >= 1.8.9, < 1.9.2", "xdis >= 6.1.1, < 6.2.0"]
 
 license = "GPL3"
 mailing_list = "python-debugger@googlegroups.com"

admin-tools/.gitignore

@@ -0,0 +1 @@
+/.python-version

admin-tools/checkout_common.sh

@@ -0,0 +1,21 @@
+# Common checkout routine
+export PATH=$HOME/.pyenv/bin/pyenv:$PATH
+bs=${BASH_SOURCE[0]}
+mydir=$(dirname $bs)
+fulldir=$(readlink -f $mydir)
+
+function setup_version {
+    local repo=$1
+    version=$2
+    echo Running setup $version on $repo ...
+    (cd ../$repo && . ./admin-tools/setup-${version}.sh)
+    return $?
+}
+
+function checkout_finish {
+    branch=$1
+    cd $uncompyle6_owd
+    git checkout $branch && pyenv local $PYTHON_VERSION && git pull
+    rc=$?
+    return $rc
+}

admin-tools/merge-for-3.3.sh

@@ -2,6 +2,6 @@
 uncompyle6_merge_33_owd=$(pwd)
 cd $(dirname ${BASH_SOURCE[0]})
 if . ./setup-python-3.3.sh; then
-    git merge master
+    git merge python-3.6-to-3.10
 fi
 cd $uncompyle6_merge_33_owd

admin-tools/merge-for-3.6.sh

@@ -0,0 +1,7 @@
+#/bin/bash
+uncompyle6_merge_36_owd=$(pwd)
+cd $(dirname ${BASH_SOURCE[0]})
+if . ./setup-python-3.6.sh; then
+    git merge master
+fi
+cd $uncompyle6_merge_36_owd

admin-tools/setup-master.sh

@@ -1,32 +1,20 @@
 #!/bin/bash
 # Check out master branch and dependent development master branches
-PYTHON_VERSION=3.8.18
-
 bs=${BASH_SOURCE[0]}
 if [[ $0 == $bs ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
 
-function checkout_version {
-    local repo=$1
-    version=${2:-master}
-    echo Checking out $version on $repo ...
-    (cd ../$repo && git checkout $version && pyenv local $PYTHON_VERSION) && \
-	git pull
-    return $?
-}
-
-owd=$(pwd)
-
-export PATH=$HOME/.pyenv/bin/pyenv:$PATH
+PYTHON_VERSION=3.12
 
+uncompyle6_owd=$(pwd)
 mydir=$(dirname $bs)
 fulldir=$(readlink -f $mydir)
+cd $mydir
+. ./checkout_common.sh
 cd $fulldir/..
-(cd $fulldir/.. && checkout_version python-spark && checkout_version python-xdis &&
-     checkout_version python-uncompyle6)
-
-git pull
-rm -v */.python-version || true
-cd $owd
+(cd $fulldir/.. && \
+     setup_version python-spark master && \
+     setup_version python-xdis master )
+checkout_finish master

admin-tools/setup-python-2.4.sh

@@ -1,32 +1,23 @@
 #!/bin/bash
 # Check out python-2.4-to-2.7 and dependent development branches.
 
-PYTHON_VERSION=2.4.6
-
 bs=${BASH_SOURCE[0]}
 if [[ $0 == $bs ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
 
-function checkout_version {
-    local repo=$1
-    version=${2:-python-2.4-to-2.7}
-    echo Checking out $version on $repo ...
-    (cd ../$repo && git checkout $version && pyenv local $PYTHON_VERSION) && \
-	git pull
-    return $?
-}
-
-owd=$(pwd)
-
-export PATH=$HOME/.pyenv/bin/pyenv:$PATH
+PYTHON_VERSION=2.4
 
+uncompyle6_owd=$(pwd)
 mydir=$(dirname $bs)
 fulldir=$(readlink -f $mydir)
-(cd $fulldir/.. && checkout_version python-spark && checkout_version python-xdis python-2.4-to-2.7 &&
-     checkout_version python-uncompyle6)
+cd $mydir
+. ./checkout_common.sh
 
-git pull
-rm -v */.python-version || true
-cd $owd
+(cd $fulldir/.. && \
+     setup_version python-spark python-2.4 && \
+     setup_version python-xdis python-2.4)
+
+
+checkout_finish python-2.4-to-2.7

admin-tools/setup-python-3.0.sh

@@ -1,35 +1,20 @@
 #!/bin/bash
 # Check out python-3.0-to-3.2 and dependent development branches.
-
-PYTHON_VERSION=3.0.1
-
 bs=${BASH_SOURCE[0]}
 if [[ $0 == $bs ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
 
-# FIXME put some of the below in a common routine
-function checkout_version {
-    local repo=$1
-    version=${2:-python-3.0-to-3.2}
-    echo Checking out $version on $repo ...
-    (cd ../$repo && git checkout $version && pyenv local $PYTHON_VERSION) && \
-	git pull
-    return $?
-}
-
-owd=$(pwd)
-trap finish EXIT
-
-export PATH=$HOME/.pyenv/bin/pyenv:$PATH
+PYTHON_VERSION=3.0
 
+uncompyle6_owd=$(pwd)
 mydir=$(dirname $bs)
 fulldir=$(readlink -f $mydir)
-cd $fulldir/..
-(cd $fulldir/.. && checkout_version python-spark master && checkout_version python-xdis &&
-      checkout_version python-uncompyle6)
+cd $mydir
+. ./checkout_common.sh
+(cd $fulldir/.. && \
+     setup_version python-spark python-3.0 && \
+     setup_version python-xdis python-3.0)
 
-git pull
-rm -v */.python-version || true
-cd $owd
+checkout_finish python-3.0-to-3.2

admin-tools/setup-python-3.3.sh

@@ -1,34 +1,21 @@
 #!/bin/bash
 # Check out python-3.3-to-3.5 and dependent development branches.
-PYTHON_VERSION=3.3.7
-
 bs=${BASH_SOURCE[0]}
 if [[ $0 == $bs ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
 
-# FIXME put some of the below in a common routine
-function checkout_version {
-    local repo=$1
-    version=${2:-python-3.3-to-3.5}
-    echo Checking out $version on $repo ...
-    (cd ../$repo && git checkout $version && pyenv local $PYTHON_VERSION) && \
-	git pull
-    return $?
-}
-
-owd=$(pwd)
-
-export PATH=$HOME/.pyenv/bin/pyenv:$PATH
+PYTHON_VERSION=3.3
 
+uncompyle6_owd=$(pwd)
 mydir=$(dirname $bs)
+cd $mydir
 fulldir=$(readlink -f $mydir)
+. ./checkout_common.sh
 cd $fulldir/..
-(cd $fulldir/.. && checkout_version python-spark master && checkout_version python-xdis &&
-     checkout_version python-uncompyle6)
-rm -v */.python-version || true
+(cd $fulldir/.. && \
+     setup_version python-spark python-3.3 && \
+     setup_version python-xdis python-3.3 )
 
-git pull
-rm -v */.python-version || true
-cd $owd
+checkout_finish python-3.3-to-3.5

admin-tools/setup-python-3.6.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+# Check out python-3.6-to-3.10 and dependent development branches.
+bs=${BASH_SOURCE[0]}
+if [[ $0 == $bs ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+
+PYTHON_VERSION=3.6
+
+uncompyle6_owd=$(pwd)
+mydir=$(dirname $bs)
+cd $mydir
+fulldir=$(readlink -f $mydir)
+. ./checkout_common.sh
+cd $fulldir/..
+(cd $fulldir/.. && \
+     setup_version python-spark python-3.6 && \
+     setup_version python-xdis python-3.6 )
+
+checkout_finish python-3.6-to-3.10

pyproject.toml

@@ -1,6 +1,7 @@
 [build-system]
 requires = [
-    "setuptools>=61.2",
+    "setuptools",
+    # "setuptools>=59.6.0", # for 3.6
 ]
 
 build-backend = "setuptools.build_meta"
@@ -14,8 +15,8 @@ name = "uncompyle6"
 description = "Python cross-version byte-code library and disassembler"
 dependencies = [
     "click",
-    "spark-parser >= 1.8.9, < 1.9.0",
-    "xdis >= 6.0.8, < 6.2.0",
+    "spark-parser >= 1.8.9, < 1.9.2",
+    "xdis >= 6.1.0, < 6.2.0",
 ]
 readme = "README.rst"
 license = {text = "GPL"}

requirements.txt

@@ -6,4 +6,4 @@ pytest
 Click~=7.0
 xdis>=6.0.4
 configobj~=5.0.6
-setuptools~=65.5.1
+setuptools

setup.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-"""Setup script for the 'xdis' distribution."""
+"""Setup script for the 'uncompyle6' distribution."""
 
 from setuptools import setup
 

test/ok_lib2.7/bsddb/dbshelve.py

@@ -29,6 +29,7 @@ storage.
 #------------------------------------------------------------------------
 
 import sys
+
 absolute_import = (sys.version_info[0] >= 3)
 if absolute_import :
     # Because this syntaxis is not valid before Python 2.5
@@ -229,7 +230,7 @@ class DBShelf(MutableMapping):
 
     def associate(self, secondaryDB, callback, flags=0):
         def _shelf_callback(priKey, priData, realCallback=callback):
-            # Safe in Python 2.x because expresion short circuit
+            # Safe in Python 2.x because expression short circuit
             if sys.version_info[0] < 3 or isinstance(priData, bytes) :
                 data = cPickle.loads(priData)
             else :
@@ -366,7 +367,7 @@ class DBShelfCursor:
             return None
         else:
             key, data = rec
-            # Safe in Python 2.x because expresion short circuit
+            # Safe in Python 2.x because expression short circuit
             if sys.version_info[0] < 3 or isinstance(data, bytes) :
                 return key, cPickle.loads(data)
             else :

test/simple_source/bug25/06_if_and_bugs.py

@@ -0,0 +1,18 @@
+# 2.5 Bug is from nose/plugins/cover.py
+def wantFile(self, file, package=None):
+    if self.coverInclusive:
+        if file.endswith(".py"):
+            if package and self.coverPackages:
+                for want in self.coverPackages:
+                    if package.startswith(want):
+                        return True
+            else:
+                return True
+    return None
+
+
+# 2.5 bug is from nose/plugins/doctests.py
+def wantFile2(self, file):
+    if self and (self.conf or [exc.search(file) for exc in self.conf]):
+        return True
+    return None

test/simple_source/bug26/.gitignore

@@ -0,0 +1 @@
+/.python-version

test/simple_source/bug26/03_weird26.py

@@ -3,10 +3,10 @@
 # Grammar allows multiple adjacent 'if's in listcomps and genexps,
 # even though it's silly. Make sure it works (ifelse broke this.)
 
-[ x for x in range(10) if x % 2 if x % 3 ]
+[x for x in range(10) if x % 2 if x % 3]
 list(x for x in range(10) if x % 2 if x % 3)
 
-# expresion which evaluates True unconditionally,
+# expression which evaluates True unconditionally,
 # but leave dead code or junk around that we have to match on.
 # Tests "if_exp_true" rule
 5 if 1 else 2

test/simple_source/bug26/07_try_except.py

@@ -0,0 +1,34 @@
+# Bug portion of Issue #405 https://github.com/rocky/python-uncompyle6/issues/405
+# Bug was detecting if/else as the last item in a "try: .. except" block.
+class Saveframe(object):
+    """A saveframe. Use the classmethod from_scratch to create one."""
+
+    frame_list = {}
+
+    def frame_dict(self):
+        return
+
+    # Next line is 1477
+    def __setitem__(self, key, item):
+        # Next line is 1481
+        if isinstance(item, Saveframe):
+            try:
+                self.frame_list[key] = item
+            except TypeError:
+                if key in (self.frame_dict()):
+                    dict((frame.name, frame) for frame in self.frame_list)
+                    for pos, frame in enumerate(self.frame_list):
+                        if frame.name == key:
+                            self.frame_list[pos] = item
+                else:
+                    raise KeyError(
+                        "Saveframe with name '%s' does not exist and "
+                        "therefore cannot be written to. Use the add_saveframe method to add new saveframes."
+                        % key
+                    )
+        # Next line is 1498
+        raise ValueError("You can only assign an entry to a saveframe splice.")
+
+
+x = Saveframe()
+x.__setitem__("foo", 5)

test/simple_source/bug27+/01_module_doc.py

@@ -1,8 +1,8 @@
 # From 2.7.17 test_bdb.py
-# The problem was detecting a docstring at the begining of the module
+# The problem was detecting a docstring at the beginning of the module
 # It must be detected and change'd or else the "from __future__" below
 # is invalid.
-# Note that this has to be compiled with optimation < 2 or else optimization
+# Note that this has to be compiled with optimization < 2 or else optimization
 # will remove the docstring
 """Rational, infinite-precision, real numbers."""
 

test/simple_source/bug30/01_ops.py

@@ -1,20 +1,20 @@
 # Statements to beef up grammar coverage rules
 # Force "inplace" ops
 # Note this is like simple_source/bug22/01_ops.py
-# But we don't ahve the UNARY_CONVERT which dropped
+# But we don't have the UNARY_CONVERT which dropped
 # out around 2.7
 y = +10  # UNARY_POSITIVE
-y /= 1   # INPLACE_DIVIDE
-y %= 4   # INPLACE_MODULO
+y /= 1  # INPLACE_DIVIDE
+y %= 4  # INPLACE_MODULO
 y **= 1  # INPLACE POWER
 y >>= 2  # INPLACE_RSHIFT
 y <<= 2  # INPLACE_LSHIFT
 y //= 1  # INPLACE_TRUE_DIVIDE
-y &= 1   # INPLACE_AND
-y ^= 1   # INPLACE_XOR
+y &= 1  # INPLACE_AND
+y ^= 1  # INPLACE_XOR
 
 # Beef up aug_assign and STORE_SLICE+3
-x = [1,2,3,4,5]
+x = [1, 2, 3, 4, 5]
 x[0:1] = 1
 x[0:3] += 1, 2, 3
 

test/simple_source/bug33/04_lambda_star_default.py

@@ -1,18 +1,20 @@
 # From 3.x test_audiop.py
 
 # Bug is handling default value after * argument in a lambda.
-# That's a mouthful of desciption; I am not sure if the really
+# That's a mouthful of description; I am not sure if the really
 # hacky fix to the code is even correct.
 
 #
 # FIXME: try and test with more than one default argument.
 
+
 # RUNNABLE
 def pack(width, data):
     return (width, data)
 
+
 packs = {w: (lambda *data, width=w: pack(width, data)) for w in (1, 2, 4)}
 
-assert packs[1]('a') == (1, ('a',))
-assert packs[2]('b') == (2, ('b',))
-assert packs[4]('c') == (4, ('c',))
+assert packs[1]("a") == (1, ("a",))
+assert packs[2]("b") == (2, ("b",))
+assert packs[4]("c") == (4, ("c",))

test/simple_source/bug33/08_if_else.py

@@ -1,16 +1,19 @@
 # From python 3.3.7 trace
 # Bug was not having not having semantic rule for conditional not
 
+
 # RUNNABLE!
 def init(modules=None):
     mods = set() if not modules else set(modules)
     return mods
 
+
 assert init() == set()
 assert init([1, 2, 3]) == set([1, 2, 3])
 
+
 # From 3.6 sre_parse
-# Bug was in handling multple COME_FROMS from nested if's
+# Bug was in handling multiple COME_FROMS from nested if's
 def _escape(a, b, c, d, e):
     if a:
         if b:
@@ -24,15 +27,16 @@ def _escape(a, b, c, d, e):
             return
         raise
 
-assert _escape(False, True, True,  True,  True) is None
-assert _escape(True,  True, True,  False, True) is None
-assert _escape(True,  True, False, False, True) is None
+
+assert _escape(False, True, True, True, True) is None
+assert _escape(True, True, True, False, True) is None
+assert _escape(True, True, False, False, True) is None
 
 for args in (
-        (True, True,  True, False, True),
-        (True, False, True, True,  True),
-        (True, False, True, True,  False),
-        ):
+    (True, True, True, False, True),
+    (True, False, True, True, True),
+    (True, False, True, True, False),
+):
     try:
         _escape(*args)
         assert False, args

test/simple_source/bug34/03_ifelse_in_lambda.py

@@ -0,0 +1,4 @@
+# Next line is 1164
+def foo():
+    name = "bar"
+    lambda x: compile(x, "<register %s's commit>" % name, "exec") if x else None

test/simple_source/bug35/02_for_else_bug.py

@@ -0,0 +1,10 @@
+# Adapted 3.5 from _bootstrap_external.py
+
+
+def spec_from_file_location(loader, location):
+    if loader:
+        for _ in __file__:
+            if location:
+                break
+        else:
+            return None

test/simple_source/bug35/06_while_return.py

@@ -1,8 +1,9 @@
 # From Python 3.4 asynchat.py
-# Tests presence or absense of
+# Tests presence or absence of
 # SETUP_LOOP testexpr return_stmts POP_BLOCK COME_FROM_LOOP
 # Note: that there is no JUMP_BACK because of the return_stmts.
 
+
 def initiate_send(a, b, c, num_sent):
     while a and b:
         try:
@@ -24,6 +25,7 @@ def initiate_send2(a, b):
 
         return 2
 
+
 assert initiate_send(1, 1, 2, False) == 1
 assert initiate_send(1, 2, 3, False) == 3
 assert initiate_send(1, 2, 3, True) == 2

test/simple_source/bug36/03_fn_defaults.py

@@ -1,13 +1,20 @@
-# Python 3.6 changes, yet again, the way deafult pairs are handled
+# Python 3.6 changes, yet again, the way default pairs are handled
 def foo1(bar, baz=1):
     return 1
+
+
 def foo2(bar, baz, qux=1):
     return 2
+
+
 def foo3(bar, baz=1, qux=2):
     return 3
+
+
 def foo4(bar, baz, qux=1, quux=2):
     return 4
 
+
 # From 3.6 compileall.
 # Bug was in omitting default which when used in an "if"
 # are treated as False would be

test/simple_source/bug36/04_class_kwargs.py

@@ -1,17 +1,23 @@
 # From 3.6 test_abc.py
-# Bug was Reciever() class definition
+# Bug was Receiver() class definition
 import abc
 import unittest
+
+
 class TestABCWithInitSubclass(unittest.TestCase):
     def test_works_with_init_subclass(self):
         class ReceivesClassKwargs:
             def __init_subclass__(cls, **kwargs):
                 super().__init_subclass__()
+
         class Receiver(ReceivesClassKwargs, abc.ABC, x=1, y=2, z=3):
             pass
 
+
 def test_abstractmethod_integration(self):
     for abstractthing in [abc.abstractmethod]:
+
         class C(metaclass=abc.ABCMeta):
             @abstractthing
-            def foo(self): pass  # abstract
+            def foo(self):
+                pass  # abstract

test/simple_source/bug36/05_if_and_comp.py

@@ -1,12 +1,12 @@
 # From 3.6 base64.py
-# Bug was handling "and" condition in the presense of POP_JUMP_IF_FALSE
+# Bug was handling "and" condition in the presence of POP_JUMP_IF_FALSE
 # locations
 def _85encode(foldnuls, words):
-    return ['z' if foldnuls and word
-                else  'y'
-                for word in words]
+    return ["z" if foldnuls and word else "y" for word in words]
+
 
 # From Python 3.6 enum.py
 
+
 def __new__(metacls, cls, bases, classdict):
     {k: classdict[k] for k in classdict._member_names}

test/simple_source/bug36/09_long_whilestmt.py

@@ -0,0 +1,74 @@
+# From https://github.com/rocky/python-uncompyle6/issues/420
+# Related to EXTENDED_ARG in whilestmt
+ERRPR_CODE_DEFINE = {}  # Remove this and things works
+
+try:
+    print()
+except Exception:
+    var1 = 0
+    var2 = 1
+    if var1 or var2:
+        times = 1
+        while times != False and self.scanner.is_open():
+            try:
+                try:
+                    print()
+                except Exception:
+                    print()
+
+                out = 0
+                count = 1
+                if out == 1:
+                    break
+                elif out == 2:
+                    count += 1
+                    if times == 3:
+                        self.func.emit({})
+                        break
+                    else:
+                        continue
+                if out == 3 or out == b"":
+                    if self.times == 3:
+                        break
+                    count += 1
+                    if count == 3:
+                        count = 0
+                        if out == 4:
+                            self.func.emit(ERRPR_CODE_DEFINE.ReceiedError())
+                        else:
+                            print()
+                        break
+                    continue
+                else:
+                    count = 0
+            except Exception:
+                print("upper exception")
+    else:
+        try:
+            print("jump forward")
+            while True:
+                out = self.func.read(count)
+                if out == b"":
+                    self.func.emit(ERRPR_CODE_DEFINE.ReceiedError())
+                    break
+                    continue
+                imagedata = out[0]
+                if imagedata == b"\x05":
+                    self.func.emit(INFORMATION.UnsupportedImage())
+                    break
+                    continue
+                if imagedata == b"\x15":
+                    self.func.emit(INFORMATION.NoneImage())
+                    break
+                    continue
+                if out[1] == False:
+                    start_index = imagedata.find(b"BM6")
+                    self.func.emit(imagedata[start_index:], False)
+                    continue
+                (imagedata, all_code) = imagedata
+                self.func.emit({})
+                self.func.emit({})
+                self.func.emit({})  # remove {} and this works
+                break
+        except Exception:
+            pass

test/simple_source/bug36/10_fstring.py

@@ -14,6 +14,7 @@ assert (
 assert "def0" == f"{abc}0"
 assert "defdef" == f"{abc}{abc!s}"
 
+
 # From 3.8 test/test_string.py
 # We had the precedence of yield vs. lambda incorrect.
 def fn(x):
@@ -97,9 +98,10 @@ else:
 (x, y, width) = ("foo", 2, 10)
 assert f"x={x*y:{width}}" == "x=foofoo    "
 
+
 # Why the fact that the distinction of docstring versus stmt is a
 # string expression is important academic, but we will decompile an
-# equivalent thing. For compatiblity with older Python we'll use "%"
+# equivalent thing. For compatibility with older Python we'll use "%"
 # instead of a format string
 def f():
     f"""Not a docstring"""  # noqa

test/simple_source/bug36/10_long_pop_jump.py

@@ -1,26 +1,27 @@
 # From 3.6 _markupbase.py
 
-# Bug is that the routine is long enough that POP_JUMP_IF_FALSE instruciton has an
-# EXTENDED_ARG intruction before it and we weren't picking out the jump offset properly
+# Bug is that the routine is long enough that POP_JUMP_IF_FALSE instruction has an
+# EXTENDED_ARG instruction before it and we weren't picking out the jump offset properly
+
 
 def parse_declaration(self, i):
     if rawdata[j:j] in ("-", ""):
         return -1
     n = len(rawdata)
-    if rawdata[j:j+2] == '-':
+    if rawdata[j : j + 2] == "-":
         return self.parse_comment(i)
-    elif rawdata[j] == '[':
+    elif rawdata[j] == "[":
         return self.parse_marked_section(i)
     else:
         decltype, j = self._scan_name(j, i)
     if j < 0:
         return j
     if decltype == "d":
-        self._decl_otherchars = ''
+        self._decl_otherchars = ""
     while j < n:
         c = rawdata[j]
         if c == ">":
-            data = rawdata[i+2:j]
+            data = rawdata[i + 2 : j]
             if decltype == "d":
                 self.handle_decl(data)
             else:
@@ -43,8 +44,7 @@ def parse_declaration(self, i):
             else:
                 self.error("unexpected '[' char in declaration")
         else:
-            self.error(
-                "unexpected %r char in declaration" % rawdata[j])
+            self.error("unexpected %r char in declaration" % rawdata[j])
         if j < 0:
             return j
     return -1

test/simple_source/bug38/03_while_bug.py

@@ -0,0 +1,9 @@
+# See https://github.com/rocky/python-uncompyle6/issues/498
+# Bug was in not allowing _stmts in whilestmt38
+import time
+
+r = 0
+while r == 1:
+    print(time.time())
+    if r == 1:
+        r = 0

test/simple_source/calls/01_positional.py

@@ -1,5 +1,5 @@
 # Tests custom added grammar rule:
 #   expr ::= expr {expr}^n CALL_FUNCTION_n
-# which in the specifc case below is:
+# which in the specific case below is:
 #   expr ::= expr expr expr CALL_FUNCTION_2
 max(1, 2)

test/simple_source/looping/04_while1_while1.py

@@ -27,7 +27,7 @@ while 1:
     else:
         raise RuntimeError
 
-# Degenerate case. Note: we can't run becase this causes an infinite loop.
+# Degenerate case. Note: we can't run because this causes an infinite loop.
 # Suggested in issue #172
 while 1:
     pass

test/simple_source/operation_logic/05_control_flow_bugs.py

@@ -1,7 +1,8 @@
 # From 3.6.10 test_binascii.py
-# Bug was getting "while c and noise" parsed correclty
+# Bug was getting "while c and noise" parsed correctly
 # and not put into the "ifelsesmt"
 
+
 # RUNNABLE!
 def addnoise(c, noise):
     while c and noise:
@@ -12,6 +13,7 @@ def addnoise(c, noise):
         noise = False
     return c
 
+
 assert addnoise(0, True) == 0
 assert addnoise(1, False) == 1
 assert addnoise(2, True) == 2
@@ -19,9 +21,10 @@ assert addnoise(3, True) == 3
 assert addnoise(4, True) == 3
 assert addnoise(5, False) == 5
 
+
 # From 3.6.10 test_dbm_dumb.py
 # Bug was getting attaching "else" to the right "if" in the
-# presense of a loop.
+# presence of a loop.
 def test_random(a, r):
     x = 0
     for dummy in r:
@@ -32,11 +35,13 @@ def test_random(a, r):
             x += 1
     return x
 
+
 assert test_random(True, [1]) == 2
 assert test_random(True, [1, 1]) == 4
 assert test_random(False, [1]) == 0
 assert test_random(False, [1, 1]) == 0
 
+
 # From 2.7.17 test_frozen.py
 # Bug was getting making sure we have "try" not
 # "try"/"else"
@@ -53,11 +58,13 @@ def test_frozen(a, b):
 
     return x
 
+
 assert test_frozen(1, 1) == 4.0
 assert test_frozen(0, 1) == 5.0
 assert test_frozen(0.5, 0) == 6.0
 assert test_frozen(0, 0.5) == 8.0
 
+
 # From 3.6.10 test_binop.py
 # Bug was getting "other += 3" outside of "if"/"else.
 def __floordiv__(a, b):
@@ -70,6 +77,7 @@ def __floordiv__(a, b):
     other += 3
     return other
 
+
 assert __floordiv__(True, True) == 4
 assert __floordiv__(True, False) == 4
 assert __floordiv__(False, True) == 3

test/simple_source/operation_logic/10_mixed_boolean.py

@@ -1,19 +1,19 @@
 # Self-checking test.
-# Mixed boolean expresions
+# Mixed boolean expressions
 
 b = True
-assert b, 'b = True'
+assert b, "b = True"
 c = False
-assert not c, 'c = False'
+assert not c, "c = False"
 d = True
 a = b and c or d
-assert a, 'b and c or d'
+assert a, "b and c or d"
 a = (b or c) and d
-assert a, '(b or c) and d'
+assert a, "(b or c) and d"
 a = b or c or d
-assert  a, 'b or c or d'
+assert a, "b or c or d"
 a = b and c and d
-assert not a, 'b and c and d'
+assert not a, "b and c and d"
 a = b or c and d
 assert a
 a = b and (c or d)

test/simple_source/stmts/03_if_elif.py

@@ -1,5 +1,5 @@
 # 2.6.9 symbols.py
-# Bug in 2.6 is having multple COME_FROMs due to the
+# Bug in 2.6 is having multiple COME_FROMs due to the
 # "and" in the "if" clause
 
 # RUNNABLE
@@ -10,7 +10,7 @@ if __name__:
         assert False
 
 # 2.6.9 transformer.py
-# Bug in 2.6 is multple COME_FROMs as a result
+# Bug in 2.6 is multiple COME_FROMs as a result
 # of the "or" in the "assert"
 
 # In PyPy the assert is handled via PyPy's unique JUMP_IF_NOT_DEBUG
@@ -24,6 +24,7 @@ elif __file__:
 else:
     pass
 
+
 # From 3.3.7 test_binop.py
 # Bug was in ifelsestmt(c) ensuring b+=5 is not in "else"
 # Also note: ifelsetmtc should not have been used since this
@@ -36,6 +37,7 @@ def __floordiv__(a, b):
     b += 5
     return b
 
+
 assert __floordiv__(1, 1) == 7
 assert __floordiv__(1, 0) == 6
 assert __floordiv__(0, 3) == 8

test/simple_source/stmts/16_bytestring_docstring.py

@@ -0,0 +1,45 @@
+"""Module docstring"""
+class A:
+    b"""Got \xe7\xfe Bytes?"""
+    assert __doc__ == b"""Got \xe7\xfe Bytes?"""
+
+    def class_func(self):
+       b"""Got \xe7\xfe Bytes?"""
+       assert __doc__ == """Module docstring"""
+
+class B:
+    """Got no Bytes?"""
+    assert __doc__ == """Got no Bytes?"""
+
+    def class_func(self):
+        """Got no Bytes?"""
+        assert __doc__ == """Module docstring"""
+
+def single_func():
+    """single docstring?"""
+    assert __doc__ == """Module docstring"""
+
+def single_byte_func():
+    b"""Got \xe7\xfe Bytes?"""
+    assert __doc__ == """Module docstring"""
+
+assert __doc__ == """Module docstring"""
+
+assert single_func.__doc__ == """single docstring?"""
+single_func()
+
+assert single_byte_func.__doc__ == b"""Got \xe7\xfe Bytes?"""
+single_byte_func()
+
+assert A.__doc__ == b"""Got \xe7\xfe Bytes?"""
+assert A.class_func.__doc__ == b"""Got \xe7\xfe Bytes?"""
+a = A()
+assert a.class_func.__doc__ == b"""Got \xe7\xfe Bytes?"""
+a.class_func()
+
+assert B.__doc__ == """Got no Bytes?"""
+assert B.class_func.__doc__ == """Got no Bytes?"""
+b = B()
+assert b.class_func.__doc__ == """Got no Bytes?"""
+b.class_func()
+

test/simple_source/stmts/16_no_bytestring_docstring.py

@@ -0,0 +1,45 @@
+"""Module docstring"""
+class A:
+    b"""Got \xe7\xfe Bytes?"""
+    assert __doc__ == """Module docstring"""
+
+    def class_func(self):
+       b"""Got \xe7\xfe Bytes?"""
+       assert __doc__ == """Module docstring"""
+
+class B:
+    """Got no Bytes?"""
+    assert __doc__ == """Got no Bytes?"""
+
+    def class_func(self):
+        """Got no Bytes?"""
+        assert __doc__ == """Module docstring"""
+
+def single_func():
+    """single docstring?"""
+    assert __doc__ == """Module docstring"""
+
+def single_byte_func():
+    b"""Got \xe7\xfe Bytes?"""
+    assert __doc__ == """Module docstring"""
+
+assert __doc__ == """Module docstring"""
+
+assert single_func.__doc__ == """single docstring?"""
+single_func()
+
+assert single_byte_func.__doc__ is None
+single_byte_func()
+
+assert A.__doc__ is None
+assert A.class_func.__doc__ is None
+a = A()
+assert a.class_func.__doc__ is None
+a.class_func()
+
+assert B.__doc__ == """Got no Bytes?"""
+assert B.class_func.__doc__ == """Got no Bytes?"""
+b = B()
+assert b.class_func.__doc__ == """Got no Bytes?"""
+b.class_func()
+

test/stdlib/2.4-exclude.sh

@@ -38,17 +38,14 @@ SKIP_TESTS=(
     [test_winreg.py]=1 # it fails on its own
     [test_winsound.py]=1 # it fails on its own
     [test_zlib.py]=1 # it fails on its own
-
-    [test_decimal.py]=1  #
+    [test_decimal.py]=1  # fails on its own - no module named test_support
     [test_dis.py]=1   # We change line numbers - duh!
-    [test_generators.py]=1  # Investigate
+    [test_generators.py]=1  # fails on its own - no module named test_support
     # [test_grammar.py]=1    # fails on its own - no module tests.test_support
     [test_grp.py]=1      # Long test - might work Control flow?
-    [test_pep247.py]=1 # Long test - might work? Control flow?
     [test_socketserver.py]=1 # -- test takes too long to run: 40 seconds
     [test_threading.py]=1 # test takes too long to run: 11 seconds
     [test_thread.py]=1 # test takes too long to run: 36 seconds
     [test_trace.py]=1 # Long test - works
-    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
 )
  # About 243 files, 0 in 19 minutes

test/test_pyenvlib.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 # emacs-mode: -*-python-*-
 """
-test_pyenvlib -- uncompyle and verify Python libraries
+test_pyenvlib -- decompile and verify Python libraries
 
 Usage-Examples:
 
@@ -20,14 +20,19 @@ Step 2: Run the test:
 	  test_pyenvlib --mylib --verify # decompile verify 'mylib'
 """
 
-from __future__ import print_function
-
-import os, time, re, shutil, sys
+# Does not work on 2.5.9 or before
+# from __future__ import print_function
+import os
+import re
+import shutil
+import sys
+import time
 from fnmatch import fnmatch
 
-from uncompyle6 import main
 import xdis.magics as magics
 
+from uncompyle6 import main
+
 # ----- configure this for your needs
 
 python_versions = [v for v in magics.python_versions if re.match("^[0-9.]+$", v)]
@@ -82,6 +87,7 @@ for vers in TEST_VERSIONS:
         if vers == "native":
             short_vers = os.path.basename(sys.path[-1])
             from xdis.version_info import PYTHON_VERSION_TRIPLE, version_tuple_to_str
+
             if PYTHON_VERSION_TRIPLE > (3, 0):
                 version = version_tuple_to_str(end=2)
                 PYC = f"*.cpython-{version}.pyc"
@@ -133,8 +139,17 @@ def do_tests(
             pass
 
     if len(files) > max_files:
-        files = [file for file in files if not "site-packages" in file and (file.endswith(".pyo") or file.endswith(".pyc"))]
-        files = [file for file in files if not "test" in file and (file.endswith(".pyo") or file.endswith(".pyc"))]
+        files = [
+            file
+            for file in files
+            if not "site-packages" in file
+            and (file.endswith(".pyo") or file.endswith(".pyc"))
+        ]
+        files = [
+            file
+            for file in files
+            if not "test" in file and (file.endswith(".pyo") or file.endswith(".pyc"))
+        ]
         if len(files) > max_files:
             # print("Number of files %d - truncating to last 200" % len(files))
             print(
@@ -151,7 +166,8 @@ def do_tests(
 
 
 if __name__ == "__main__":
-    import getopt, sys
+    import getopt
+    import sys
 
     do_coverage = do_verify = False
     test_dirs = []

uncompyle6/bin/uncompile.py

@@ -30,7 +30,7 @@ def usage():
 # __doc__ = """
 # Usage:
 #   %s [OPTIONS]... [ FILE | DIR]...
-#   %s [--help | -h | --V | --version]
+#   %s [--help | --version]
 
 # Examples:
 #   %s      foo.pyc bar.pyc       # decompile foo.pyc, bar.pyc to stdout

uncompyle6/code_fns.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016, 2818-2022 by Rocky Bernstein
+#  Copyright (c) 2015-2016, 2818-2022, 2024 by Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
@@ -17,10 +17,12 @@
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 """
-CPython magic- and version- independent disassembly routines
+CPython magic- and version-independent disassembly routines
 
 There are two reasons we can't use Python's built-in routines
-from dis. First, the bytecode we are extracting may be from a different
+from ``dis``.
+
+First, the bytecode we are extracting may be from a different
 version of Python (different magic number) than the version of Python
 that is doing the extraction.
 
@@ -39,12 +41,12 @@ from uncompyle6.scanner import get_scanner
 
 def disco(version, co, out=None, is_pypy=False):
     """
-    diassembles and deparses a given code block 'co'
+    disassembles and deparses a given code block ``co``.
     """
 
     assert iscode(co)
 
-    # store final output stream for case of error
+    # Store final output stream in case there is an error.
     real_out = out or sys.stdout
     print("# Python %s" % version_tuple_to_str(version), file=real_out)
     if co.co_filename:
@@ -99,7 +101,7 @@ def disco_loop(disasm, queue, real_out):
 
 def disassemble_file(filename, outstream=None):
     """
-    disassemble Python byte-code file (.pyc)
+    Disassemble Python byte-code file (.pyc).
 
     If given a Python source file (".py") file, we'll
     try to find the corresponding compiled object.
@@ -113,7 +115,6 @@ def disassemble_file(filename, outstream=None):
             disco(version, con, outstream)
     else:
         disco(version, co, outstream, is_pypy=is_pypy)
-    co = None
 
 
 def _test():

uncompyle6/linenumbers.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016, 2818, 2020 by Rocky Bernstein
+#  Copyright (c) 2015-2016, 2018, 2020 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by

uncompyle6/parser.py

@@ -391,6 +391,10 @@ class PythonParser(GenericASTBuilder):
 
         returns ::= return
         returns ::= _stmts return
+        
+        # NOP
+        stmt     ::= nop_stmt
+        nop_stmt ::= NOP
 
         """
         pass
@@ -669,6 +673,8 @@ def get_python_parser(
 
     version = version[:2]
 
+    p = None
+
     # FIXME: there has to be a better way...
     # We could do this as a table lookup, but that would force us
     # in import all of the parsers all of the time. Perhaps there is

uncompyle6/parsers/parse24.py

@@ -120,19 +120,6 @@ class Python24Parser(Python25Parser):
             token_len = len(tokens)
             if 0 <= token_len < len(tokens):
                 return not int(tokens[first].pattr) == tokens[last].offset
-        elif lhs == "try_except":
-            if last == len(tokens):
-                last -= 1
-            if tokens[last] != "COME_FROM" and tokens[last - 1] == "COME_FROM":
-                last -= 1
-            return (
-                tokens[last] == "COME_FROM"
-                and tokens[last - 1] == "END_FINALLY"
-                and tokens[last - 2] == "POP_TOP"
-                and tokens[last - 3].kind != "JUMP_FORWARD"
-            )
-
-        return False
 
 
 class Python24ParserSingle(Python24Parser, PythonParserSingle):

uncompyle6/parsers/parse25.py

@@ -36,6 +36,8 @@ class Python25Parser(Python26Parser):
         with_as ::= expr setupwithas store suite_stmts_opt
                        POP_BLOCK LOAD_CONST COME_FROM with_cleanup
 
+        # The last except of a "try: ... except" can do this...
+        except_suite ::= c_stmts_opt COME_FROM JUMP_ABSOLUTE POP_TOP
 
         store ::= STORE_NAME
         store ::= STORE_FAST
@@ -58,6 +60,13 @@ class Python25Parser(Python26Parser):
 
         kvlist ::= kvlist kv
         kv     ::= DUP_TOP expr ROT_TWO expr STORE_SUBSCR
+
+        _ifstmts_jump ::= c_stmts_opt COME_FROM JUMP_ABSOLUTE COME_FROM POP_TOP
+
+
+        # "and_then" is a hack around the fact we have THEN detection.
+        and_then ::= expr JUMP_IF_FALSE THEN POP_TOP expr JUMP_IF_FALSE THEN POP_TOP
+        testexpr_then ::= and_then
         """
 
     def customize_grammar_rules(self, tokens, customize):

uncompyle6/parsers/parse26.py

@@ -130,6 +130,13 @@ class Python26Parser(Python2Parser):
         # Semantic actions want else_suitel to be at index 3
         ifelsestmtl ::= testexpr c_stmts_opt cf_jb_cf_pop else_suitel
         ifelsestmtc ::= testexpr c_stmts_opt ja_cf_pop    else_suitec
+        ifelsestmt ::=  testexpr stmts_opt ja_cf_pop      else_suite
+
+        stmts_opt ::= stmts
+        stmts_opt ::=
+
+        # The last except of a "try: ... except" can do this...
+        except_suite ::= stmts_opt COME_FROM JUMP_ABSOLUTE POP_TOP
 
         # Semantic actions want suite_stmts_opt to be at index 3
         with        ::= expr setupwith SETUP_FINALLY suite_stmts_opt
@@ -466,14 +473,16 @@ class Python26Parser(Python2Parser):
             return tokens[last].offset != ja_attr
         elif lhs == "try_except":
             # We need to distinguish "try_except" from "tryelsestmt"; we do that
-            # by checking the jump before the END_FINALLY
+            # by looking for a jump before the END_FINALLY to the "else" clause of
+            # "try else".
+            #
             # If we have:
-            #    insn
+            #    <insn>
             #    POP_TOP
             #    END_FINALLY
             #    COME_FROM
-            # then insn has to be either a JUMP_FORWARD or a RETURN_VALUE
-            # and if it is JUMP_FORWARD, then it has to be a JUMP_FORWARD to right after
+            # then <insn> has to be either a a jump of some sort (JUMP_FORWARD, BREAK_LOOP, JUMP_BACK, or RETURN_VALUE).
+            # Furthermore, if it is JUMP_FORWARD, then it has to be a JUMP_FORWARD to right after
             # COME_FROM
             if last == len(tokens):
                 last -= 1
@@ -487,53 +496,8 @@ class Python26Parser(Python2Parser):
                 # A jump of 2 is a jump around POP_TOP, END_FINALLY which
                 # would indicate try/else rather than try
                 return tokens[last - 3].kind not in frozenset(
-                    ("JUMP_FORWARD", "RETURN_VALUE")
+                    ("JUMP_FORWARD", "JUMP_BACK", "BREAK_LOOP", "RETURN_VALUE")
                 ) or (tokens[last - 3] == "JUMP_FORWARD" and tokens[last - 3].attr != 2)
-        elif lhs == "tryelsestmt":
-            # We need to distinguish "try_except" from "tryelsestmt"; we do that
-            # by making sure that the jump before the except handler jumps to
-            # code somewhere before the end of the construct.
-            # This AST method is slower, but the token-only based approach
-            # didn't work as it failed with a "try" embedded inside a "try/else"
-            # since we can't detect COME_FROM boundaries.
-
-            if ast[3] == "except_handler":
-                except_handler = ast[3]
-                if except_handler[0] == "JUMP_FORWARD":
-                    else_start = int(except_handler[0].pattr)
-                    if last == len(tokens):
-                        last -= 1
-                    if tokens[last] == "COME_FROM" and isinstance:
-                        last_offset = int(tokens[last].offset.split("_")[0])
-                        return else_start >= last_offset
-
-            # The above test apparently isn't good enough, so we have additional
-            # checks distinguish "try_except" from "tryelsestmt". we do that
-            # by checking the jump before the "END_FINALLY".
-            # If we have:
-            #    insn
-            #    POP_TOP
-            #    END_FINALLY
-            #    COME_FROM
-            # then insn is neither a JUMP_FORWARD nor RETURN_VALUE,
-            # or if it is JUMP_FORWARD, then it can't be a JUMP_FORWARD to right after
-            # COME_FROM
-            if last == len(tokens):
-                last -= 1
-            while tokens[last - 1] == "COME_FROM" and tokens[last - 2] == "COME_FROM":
-                last -= 1
-            if tokens[last] == "COME_FROM" and tokens[last - 1] == "COME_FROM":
-                last -= 1
-            if (
-                tokens[last] == "COME_FROM"
-                and tokens[last - 1] == "END_FINALLY"
-                and tokens[last - 2] == "POP_TOP"
-            ):
-                # A jump of 2 is a jump around POP_TOP, END_FINALLY which
-                # would indicate try/else rather than try
-                return tokens[last - 3].kind in frozenset(
-                    ("JUMP_FORWARD", "RETURN_VALUE")
-                ) and (tokens[last - 3] != "JUMP_FORWARD" or tokens[last - 3].attr == 2)
 
         return False
 

uncompyle6/parsers/parse27.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2020, 2023 Rocky Bernstein
+#  Copyright (c) 2016-2020, 2023-2024 Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 

uncompyle6/parsers/parse3.py

@@ -706,7 +706,7 @@ class Python3Parser(PythonParser):
         # Note: BUILD_TUPLE_UNPACK_WITH_CALL gets considered by
         # default because it starts with BUILD. So we'll set to ignore it from
         # the start.
-        custom_ops_processed = set(("BUILD_TUPLE_UNPACK_WITH_CALL",))
+        custom_ops_processed = {"BUILD_TUPLE_UNPACK_WITH_CALL"}
 
         # A set of instruction operation names that exist in the token stream.
         # We use this customize the grammar that we create.

uncompyle6/parsers/parse31.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2017, 2022 Rocky Bernstein
+#  Copyright (c) 2016-2017, 2022, 2024 Rocky Bernstein
 """
 spark grammar differences over Python 3.2 for Python 3.1.
 """

uncompyle6/parsers/parse32.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2017, 2022 Rocky Bernstein
+#  Copyright (c) 2016-2017, 2022-2024 Rocky Bernstein
 """
 spark grammar differences over Python 3 for Python 3.2.
 """
@@ -84,7 +84,7 @@ class Python32Parser(Python3Parser):
         for i, token in enumerate(tokens):
             opname = token.kind
             if opname.startswith("MAKE_FUNCTION_A"):
-                args_pos, args_kw, annotate_args = token.attr
+                args_pos, _, annotate_args = token.attr
                 # Check that there are 2 annotated params?
                 rule = (
                     "mkfunc_annotate ::= %s%sannotate_tuple "

uncompyle6/parsers/parse34.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2017-2018 Rocky Bernstein
+#  Copyright (c) 2017-2018, 2022-2024 Rocky Bernstein
 
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -21,7 +21,6 @@ from uncompyle6.parsers.parse33 import Python33Parser
 
 
 class Python34Parser(Python33Parser):
-
     def p_misc34(self, args):
         """
         expr ::= LOAD_ASSERT
@@ -54,39 +53,52 @@ class Python34Parser(Python33Parser):
         _ifstmts_jump ::= c_stmts_opt JUMP_ABSOLUTE JUMP_FORWARD COME_FROM
 
         genexpr_func ::= LOAD_ARG _come_froms FOR_ITER store comp_iter JUMP_BACK
+
+        if_exp_lambda      ::= expr jmp_false expr return_if_lambda return_stmt_lambda LAMBDA_MARKER
+        return_if_lambda   ::= RETURN_END_IF_LAMBDA come_froms
+        return_if_stmt     ::= return_expr RETURN_END_IF POP_BLOCK
         """
 
     def customize_grammar_rules(self, tokens, customize):
-        self.remove_rules("""
+        self.remove_rules(
+            """
         yield_from    ::= expr expr YIELD_FROM
         # 3.4.2 has this. 3.4.4 may now
         # while1stmt ::= SETUP_LOOP l_stmts COME_FROM JUMP_BACK COME_FROM_LOOP
-        """)
+        """
+        )
         super(Python34Parser, self).customize_grammar_rules(tokens, customize)
         return
 
+
 class Python34ParserSingle(Python34Parser, PythonParserSingle):
     pass
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     # Check grammar
     p = Python34Parser()
     p.check_grammar()
     from xdis.version_info import IS_PYPY, PYTHON_VERSION_TRIPLE
+
     if PYTHON_VERSION_TRIPLE[:2] == (3, 4):
         lhs, rhs, tokens, right_recursive, dup_rhs = p.check_sets()
         from uncompyle6.scanner import get_scanner
+
         s = get_scanner(PYTHON_VERSION_TRIPLE, IS_PYPY)
-        opcode_set = set(s.opc.opname).union(set(
-            """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
+        opcode_set = set(s.opc.opname).union(
+            set(
+                """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
                LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_CLASSNAME
                LAMBDA_MARKER RETURN_LAST
-            """.split()))
+            """.split()
+            )
+        )
         remain_tokens = set(tokens) - opcode_set
         import re
-        remain_tokens = set([re.sub(r'_\d+$', '',  t) for t in remain_tokens])
-        remain_tokens = set([re.sub('_CONT$', '', t) for t in remain_tokens])
+
+        remain_tokens = set([re.sub(r"_\d+$", "", t) for t in remain_tokens])
+        remain_tokens = set([re.sub("_CONT$", "", t) for t in remain_tokens])
         remain_tokens = set(remain_tokens) - opcode_set
         print(remain_tokens)
         # print(sorted(p.rule2name.items()))

uncompyle6/parsers/parse35.py

@@ -108,9 +108,10 @@ class Python35Parser(Python34Parser):
         # Python 3.5+ does jump optimization
         # In <.3.5 the below is a JUMP_FORWARD to a JUMP_ABSOLUTE.
 
-        return_if_stmt    ::= return_expr RETURN_END_IF POP_BLOCK
         return_if_lambda  ::= RETURN_END_IF_LAMBDA COME_FROM
 
+        return            ::= return_expr RETURN_END_IF
+
         jb_else     ::= JUMP_BACK ELSE
         ifelsestmtc ::= testexpr c_stmts_opt JUMP_FORWARD else_suitec
         ifelsestmtl ::= testexpr c_stmts_opt jb_else else_suitel

uncompyle6/parsers/parse36.py

@@ -53,6 +53,8 @@ class Python36Parser(Python35Parser):
         for_block       ::= l_stmts_opt come_from_loops JUMP_BACK
         come_from_loops ::= COME_FROM_LOOP*
 
+        whilestmt       ::= SETUP_LOOP testexpr l_stmts_opt
+                            JUMP_BACK come_froms POP_BLOCK
         whilestmt       ::= SETUP_LOOP testexpr l_stmts_opt
                             JUMP_BACK come_froms POP_BLOCK COME_FROM_LOOP
         whilestmt       ::= SETUP_LOOP testexpr l_stmts_opt

uncompyle6/parsers/parse37base.py

@@ -138,7 +138,7 @@ class Python37BaseParser(PythonParser):
         # Note: BUILD_TUPLE_UNPACK_WITH_CALL gets considered by
         # default because it starts with BUILD. So we'll set to ignore it from
         # the start.
-        custom_ops_processed = set(("BUILD_TUPLE_UNPACK_WITH_CALL",))
+        custom_ops_processed = {"BUILD_TUPLE_UNPACK_WITH_CALL"}
 
         # A set of instruction operation names that exist in the token stream.
         # We use this customize the grammar that we create.

uncompyle6/parsers/parse38.py

@@ -417,7 +417,7 @@ class Python38Parser(Python37Parser):
             [opname[: opname.rfind("_")] for opname in self.seen_ops]
         )
 
-        custom_ops_processed = set(["DICT_MERGE"])
+        custom_ops_processed = {"DICT_MERGE"}
 
         # Loop over instructions adding custom grammar rules based on
         # a specific instruction seen.

uncompyle6/parsers/reducecheck/except_handler.py

@@ -1,7 +1,8 @@
 #  Copyright (c) 2020 Rocky Bernstein
 
+
 def except_handler(self, lhs, n, rule, ast, tokens, first, last):
-    end_token = tokens[last-1]
+    end_token = tokens[last - 1]
 
     # print("XXX", first, last)
     # for t in range(first, last):
@@ -13,7 +14,7 @@ def except_handler(self, lhs, n, rule, ast, tokens, first, last):
     if self.version[:2] == (1, 4):
         return False
 
-    # Make sure come froms all come from within "except_handler".
+    # Make sure COME_FROMs froms come from within "except_handler".
     if end_token != "COME_FROM":
         return False
     return end_token.attr < tokens[first].offset

uncompyle6/parsers/reducecheck/ifelsestmt.py

@@ -82,14 +82,8 @@ IFELSE_STMT_RULES = frozenset(
             ),
         ),
         (
-            'ifelsestmtc',
-            (
-                'testexpr',
-                'c_stmts_opt',
-                'JUMP_FORWARD',
-                'else_suite',
-                'come_froms'
-            ),
+            "ifelsestmtc",
+            ("testexpr", "c_stmts_opt", "JUMP_FORWARD", "else_suite", "come_froms"),
         ),
         (
             "ifelsestmt",
@@ -155,7 +149,6 @@ IFELSE_STMT_RULES = frozenset(
 
 
 def ifelsestmt(self, lhs, n, rule, tree, tokens, first, last):
-
     if (last + 1) < n and tokens[last + 1] == "COME_FROM_LOOP" and lhs != "ifelsestmtc":
         # ifelsestmt jumped outside of loop. No good.
         return True
@@ -176,13 +169,10 @@ def ifelsestmt(self, lhs, n, rule, tree, tokens, first, last):
     stmts = tree[1]
     if stmts in ("c_stmts",) and len(stmts) == 1:
         raise_stmt1 = stmts[0]
-        if (
-                raise_stmt1 == "raise_stmt1" and
-                raise_stmt1[0] in ("LOAD_ASSERT",)
-        ):
+        if raise_stmt1 == "raise_stmt1" and raise_stmt1[0] in ("LOAD_ASSERT",):
             return True
 
-    # Make sure all the offsets from the "come froms" at the
+    # Make sure all the offsets from the "COME_FROMs" at the
     # end of the "if" come from somewhere inside the "if".
     # Since the come_froms are ordered so that lowest
     # offset COME_FROM is last, it is sufficient to test
@@ -277,17 +267,18 @@ def ifelsestmt(self, lhs, n, rule, tree, tokens, first, last):
                 # only if we are trying to match or reduce an "if"
                 # statement of the kind that can occur only inside a
                 # loop construct.
+
                 if lhs in ("ifelsestmtl", "ifelsestmtc"):
                     jump_false = jmp
                     if (
-                        tree[2].kind == "JUMP_FORWARD"
+                        tree[2].kind in ("JUMP_FORWARD", "JUMP_ABSOLUTE")
                         and jump_false == "jmp_false"
                         and len(else_suite) == 1
                     ):
                         suite_stmts = else_suite[0]
                         continue_stmt = suite_stmts[0]
                         if (
-                            suite_stmts == "suite_stmts"
+                            suite_stmts in ("suite_stmts", "c_stmts")
                             and len(suite_stmts) == 1
                             and continue_stmt == "continue"
                             and jump_false[0].attr == continue_stmt[0].attr

uncompyle6/parsers/reducecheck/tryexcept.py

@@ -1,16 +1,17 @@
-#  Copyright (c) 2020, 2022 Rocky Bernstein
+#  Copyright (c) 2020, 2022, 2024 Rocky Bernstein
+
 
 def tryexcept(self, lhs, n, rule, ast, tokens, first, last):
     come_from_except = ast[-1]
     if rule == (
-            "try_except",
-            (
-                "SETUP_EXCEPT",
-                "suite_stmts_opt",
-                "POP_BLOCK",
-                "except_handler",
-                "opt_come_from_except",
-            ),
+        "try_except",
+        (
+            "SETUP_EXCEPT",
+            "suite_stmts_opt",
+            "POP_BLOCK",
+            "except_handler",
+            "opt_come_from_except",
+        ),
     ):
         if come_from_except[0] == "COME_FROM":
             # There should be at least two COME_FROMs, one from an
@@ -20,31 +21,31 @@ def tryexcept(self, lhs, n, rule, ast, tokens, first, last):
         pass
 
     elif rule == (
-            "try_except",
-            (
-                "SETUP_EXCEPT",
-                "suite_stmts_opt",
-                "POP_BLOCK",
-                "except_handler",
-                "COME_FROM",
-            ),
+        "try_except",
+        (
+            "SETUP_EXCEPT",
+            "suite_stmts_opt",
+            "POP_BLOCK",
+            "except_handler",
+            "COME_FROM",
+        ),
     ):
         return come_from_except.attr < tokens[first].offset
 
     elif rule == (
-            'try_except',
-            (
-                'SETUP_EXCEPT',
-                'suite_stmts_opt',
-                'POP_BLOCK',
-                'except_handler',
-                '\\e_opt_come_from_except'
-            ),
+        "try_except",
+        (
+            "SETUP_EXCEPT",
+            "suite_stmts_opt",
+            "POP_BLOCK",
+            "except_handler",
+            "\\e_opt_come_from_except",
+        ),
     ):
         # Find END_FINALLY.
         for i in range(last, first, -1):
             if tokens[i] == "END_FINALLY":
-                jump_before_finally = tokens[i-1]
+                jump_before_finally = tokens[i - 1]
                 if jump_before_finally.kind.startswith("JUMP"):
                     if jump_before_finally == "JUMP_FORWARD":
                         # If there is a JUMP_FORWARD before
@@ -52,7 +53,9 @@ def tryexcept(self, lhs, n, rule, ast, tokens, first, last):
                         # beyond tokens[last].off2int() then
                         # this is a try/else rather than an
                         # try (no else).
-                        return tokens[i-1].attr > tokens[last].off2int(prefer_last=True)
+                        return tokens[i - 1].attr > tokens[last].off2int(
+                            prefer_last=True
+                        )
                     elif jump_before_finally == "JUMP_BACK":
                         # If there is a JUMP_BACK before the
                         # END_FINALLY then this is a looping
@@ -61,8 +64,10 @@ def tryexcept(self, lhs, n, rule, ast, tokens, first, last):
                         # jump or this is a try/else rather
                         # than an try (no else).
                         except_handler = ast[3]
-                        if (except_handler == "except_handler" and
-                            except_handler[0] == "JUMP_FORWARD"):
+                        if (
+                            except_handler == "except_handler"
+                            and except_handler[0] == "JUMP_FORWARD"
+                        ):
                             return True
                         return False
                     pass

uncompyle6/scanner.py

@@ -605,6 +605,10 @@ class Scanner(ABC):
 
 
 def get_scanner(version: Union[str, tuple], is_pypy=False, show_asm=None) -> Scanner:
+    """
+    Import the right scanner module for ``version`` and return the Scanner class
+    in that module.
+    """
     # If version is a string, turn that into the corresponding float.
     if isinstance(version, str):
         if version not in canonic_python_version:

uncompyle6/scanners/scanner2.py

@@ -213,7 +213,7 @@ class Scanner2(Scanner):
                 names=co.co_names,
                 constants=co.co_consts,
                 cells=bytecode._cell_names,
-                linestarts=bytecode._linestarts,
+                line_starts=bytecode._linestarts,
                 asm_format="extended",
             )
 
@@ -495,7 +495,8 @@ class Scanner2(Scanner):
 
         if show_asm in ("both", "after"):
             print("\n# ---- tokenization:")
-            for t in new_tokens:
+            # FIXME: t.format() is changing tokens!
+            for t in new_tokens.copy():
                 print(t.format(line_prefix=""))
             print()
         return new_tokens, customize

uncompyle6/scanners/scanner26.py

@@ -81,9 +81,15 @@ class Scanner26(scan.Scanner2):
         # show_asm = 'after'
         if show_asm in ("both", "before"):
             print("\n# ---- disassembly:")
-            for instr in bytecode.get_instructions(co):
-                print(instr.disassemble(self.opc))
-
+            bytecode.disassemble_bytes(
+                co.co_code,
+                varnames=co.co_varnames,
+                names=co.co_names,
+                constants=co.co_consts,
+                cells=bytecode._cell_names,
+                line_starts=bytecode._linestarts,
+                asm_format="extended",
+            )
         # Container for tokens
         tokens = []
 
@@ -347,7 +353,8 @@ class Scanner26(scan.Scanner2):
 
         if show_asm in ("both", "after"):
             print("\n# ---- tokenization:")
-            for t in tokens:
+            # FIXME: t.format() is changing tokens!
+            for t in tokens.copy():
                 print(t.format(line_prefix=""))
             print()
         return tokens, customize

uncompyle6/scanners/scanner3.py

@@ -216,7 +216,7 @@ class Scanner3(Scanner):
         collection_type: str,
     ) -> Optional[list]:
         """
-        Try to a replace sequence of instruction that ends with a
+        Try to replace a sequence of instruction that ends with a
         BUILD_xxx with a sequence that can be parsed much faster, but
         inserting the token boundary at the beginning of the sequence.
         """
@@ -298,8 +298,9 @@ class Scanner3(Scanner):
         )
         return new_tokens
 
-    def bound_map_from_inst(
-        self, insts: list, next_tokens: list, inst: Instruction, t: Token, i: int
+    # Move to scanner35?
+    def bound_map_from_inst_35(
+        self, insts: list, next_tokens: list, t: Token, i: int
     ) -> Optional[list]:
         """
         Try to a sequence of instruction that ends with a BUILD_MAP into
@@ -315,6 +316,8 @@ class Scanner3(Scanner):
         if count < 5:
             return None
 
+        # Newer Python BUILD_MAP argument's count is a
+        # key and value pair so it is multiplied by two.
         collection_start = i - (count * 2)
         assert (count * 2) <= i
 
@@ -338,7 +341,7 @@ class Scanner3(Scanner):
                 attr=collection_enum,
                 pattr="CONST_MAP",
                 offset=f"{start_offset}_0",
-                linestart=False,
+                linestart=insts[collection_start].starts_line,
                 has_arg=True,
                 has_extended_arg=False,
                 opc=self.opc,
@@ -356,6 +359,7 @@ class Scanner3(Scanner):
                     has_arg=True,
                     has_extended_arg=False,
                     opc=self.opc,
+                    optype="pseudo",
                 )
             )
             new_tokens.append(
@@ -368,7 +372,7 @@ class Scanner3(Scanner):
                     has_arg=True,
                     has_extended_arg=False,
                     opc=self.opc,
-                    optype=insts[j + 1].optype,
+                    optype="pseudo",
                 )
             )
         new_tokens.append(
@@ -381,7 +385,7 @@ class Scanner3(Scanner):
                 has_arg=t.has_arg,
                 has_extended_arg=False,
                 opc=t.opc,
-                optype=t.optype,
+                optype="pseudo",
             )
         )
         return new_tokens
@@ -425,7 +429,7 @@ class Scanner3(Scanner):
                 names=co.co_names,
                 constants=co.co_consts,
                 cells=bytecode._cell_names,
-                linestarts=bytecode._linestarts,
+                line_starts=bytecode._linestarts,
                 asm_format="extended",
             )
 
@@ -490,7 +494,16 @@ class Scanner3(Scanner):
         last_op_was_break = False
         new_tokens = []
 
+        skip_end_offset = None
+
         for i, inst in enumerate(self.insts):
+            # BUILD_MAP for < 3.5 can skip *forward* in instructions and
+            # replace them. So we use the below to get up to the position
+            # scanned and replaced forward
+            if skip_end_offset and inst.offset <= skip_end_offset:
+                continue
+            skip_end_offset = None
+
             opname = inst.opname
             argval = inst.argval
             pattr = inst.argrepr
@@ -524,17 +537,21 @@ class Scanner3(Scanner):
                 if try_tokens is not None:
                     new_tokens = try_tokens
                     continue
+
             elif opname in ("BUILD_MAP",):
-                try_tokens = self.bound_map_from_inst(
-                    self.insts,
-                    new_tokens,
-                    inst,
-                    t,
-                    i,
-                )
-                if try_tokens is not None:
-                    new_tokens = try_tokens
-                    continue
+                if self.version >= (3, 5):
+                    try_tokens = self.bound_map_from_inst_35(
+                        self.insts,
+                        new_tokens,
+                        t,
+                        i,
+                    )
+                    if try_tokens is not None:
+                        new_tokens = try_tokens
+                        continue
+                        pass
+                    pass
+                pass
 
             argval = inst.argval
             op = inst.opcode
@@ -789,7 +806,8 @@ class Scanner3(Scanner):
 
         if show_asm in ("both", "after"):
             print("\n# ---- tokenization:")
-            for t in new_tokens:
+            # FIXME: t.format() is changing tokens!
+            for t in new_tokens.copy():
                 print(t.format(line_prefix=""))
             print()
         return new_tokens, customize

uncompyle6/scanners/scanner37base.py

@@ -228,13 +228,13 @@ class Scanner37Base(Scanner):
 
         if show_asm in ("both", "before"):
             print("\n# ---- disassembly:")
-            self.insts = bytecode.disassemble_bytes(
+            bytecode.disassemble_bytes(
                 co.co_code,
                 varnames=co.co_varnames,
                 names=co.co_names,
                 constants=co.co_consts,
                 cells=bytecode._cell_names,
-                linestarts=bytecode._linestarts,
+                line_starts=bytecode._linestarts,
                 asm_format="extended",
                 filename=co.co_filename,
                 show_source=True,
@@ -481,12 +481,17 @@ class Scanner37Base(Scanner):
                     next_opname = self.insts[i + 1].opname
 
                     # 'Continue's include jumps to loops that are not
-                    # and the end of a block which follow with POP_BLOCK and COME_FROM_LOOP.
-                    # If the JUMP_ABSOLUTE is to a FOR_ITER and it is followed by another JUMP_FORWARD
-                    # then we'll take it as a "continue".
-                    is_continue = (
-                        self.insts[self.offset2inst_index[target]].opname == "FOR_ITER"
-                        and self.insts[i + 1].opname == "JUMP_FORWARD"
+                    # and the end of a block which follow with
+                    # POP_BLOCK and COME_FROM_LOOP.  If the
+                    # JUMP_ABSOLUTE is to a FOR_ITER, and it is
+                    # followed by another JUMP_FORWARD then we'll take
+                    # it as a "continue".
+                    next_inst = self.insts[i + 1]
+                    is_continue = self.insts[
+                        self.offset2inst_index[target]
+                    ].opname == "FOR_ITER" and next_inst.opname in (
+                        "JUMP_FORWARD",
+                        "JUMP_ABSOLUTE",
                     )
 
                     if self.version < (3, 8) and (
@@ -501,21 +506,65 @@ class Scanner37Base(Scanner):
                     ):
                         opname = "CONTINUE"
                     else:
+                        # "continue" versus "break_loop" dectction is more complicated
+                        # because "continue" to an outer loop is really a "break loop"
                         opname = "JUMP_BACK"
+
                         # FIXME: this is a hack to catch stuff like:
                         #   if x: continue
                         # the "continue" is not on a new line.
-                        # There are other situations where we don't catch
-                        # CONTINUE as well.
-                        if tokens[-1].kind == "JUMP_BACK" and tokens[-1].attr <= argval:
+                        #
+                        # Another situation is where we have
+                        #   for method in methods:
+                        #      for B in method:
+                        #         if c:
+                        #           return
+                        #        break  # A "continue" but not the innermost one
+                        if tokens[-1].kind == "JUMP_LOOP" and tokens[-1].attr <= argval:
                             if tokens[-2].kind == "BREAK_LOOP":
                                 del tokens[-1]
+                                j -= 1
                             else:
-                                # intern is used because we are changing the *previous* token
-                                tokens[-1].kind = sys.intern("CONTINUE")
-                    if last_op_was_break and opname == "CONTINUE":
-                        last_op_was_break = False
-                        continue
+                                # "intern" is used because we are
+                                # changing the *previous* token.  A
+                                # POP_TOP suggests a "break" rather
+                                # than a "continue"?
+                                if tokens[-2] == "POP_TOP" and (
+                                    is_continue and next_inst.argval != tokens[-1].attr
+                                ):
+                                    tokens[-1].kind = sys.intern("BREAK_LOOP")
+                                else:
+                                    tokens[-1].kind = sys.intern("CONTINUE")
+                                    last_continue = tokens[-1]
+                                    pass
+                                pass
+                            pass
+                    #     elif (
+                    #         last_continue is not None
+                    #         and tokens[-1].kind == "JUMP_LOOP"
+                    #         and last_continue.attr <= tokens[-1].attr
+                    #         and last_continue.offset > tokens[-1].attr
+                    #     ):
+                    #         # Handle mis-characterized "CONTINUE"
+                    #         # We have a situation like:
+                    #         # loop ... for or while)
+                    #         #   loop
+                    #         #     if ...:   # code below starts here
+                    #         #       break  # not continue
+                    #         #
+                    #         #   POP_JUMP_IF_FALSE_LOOP   # to outer loop
+                    #         #   JUMP_LOOP                # to inner loop
+                    #         #   ...
+                    #         #   JUMP_LOOP                # to outer loop
+                    #         tokens[-2].kind = sys.intern("BREAK_LOOP")
+                    #         pass
+
+                    # if last_op_was_break and opname == "CONTINUE":
+                    #     last_op_was_break = False
+                    #     continue
+                    pass
+                else:
+                    opname = "JUMP_FORWARD"
 
             elif inst.offset in self.load_asserts:
                 opname = "LOAD_ASSERT"
@@ -538,9 +587,10 @@ class Scanner37Base(Scanner):
             )
             pass
 
-        if show_asm in ("both", "after"):
+        if show_asm in ("both", "after") and self.version < (3, 8):
             print("\n# ---- tokenization:")
-            for t in tokens:
+            # FIXME: t.format() is changing tokens!
+            for t in tokens.copy():
                 print(t.format(line_prefix=""))
             print()
         return tokens, customize

uncompyle6/scanners/scanner38.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019-2022 by Rocky Bernstein
+#  Copyright (c) 2019-2022, 2024 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -24,13 +24,13 @@ scanner routine for Python 3.7 and up.
 
 from typing import Dict, Tuple
 
-from uncompyle6.scanners.tok import off2int
-from uncompyle6.scanners.scanner37 import Scanner37
-from uncompyle6.scanners.scanner37base import Scanner37Base
-
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_38 as opc
 
+from uncompyle6.scanners.scanner37 import Scanner37
+from uncompyle6.scanners.scanner37base import Scanner37Base
+from uncompyle6.scanners.tok import off2int
+
 # bytecode verification, verify(), uses JUMP_OPS from here
 JUMP_OPs = opc.JUMP_OPS
 
@@ -121,35 +121,26 @@ class Scanner38(Scanner37):
                     new_tokens.append(token)
                     continue
 
-                # We also want to avoid confusing BREAK_LOOPS with parts of the
-                # grammar rules for loops. (Perhaps we should change the grammar.)
-                # Try to find an adjacent JUMP_BACK which is part of the normal loop end.
+                j = i
+                while tokens[j - 1] in ("POP_TOP", "POP_BLOCK", "POP_EXCEPT"):
+                    j -= 1
+                    if tokens[j].linestart:
+                        break
+                token_with_linestart = tokens[j]
 
-                if i + 1 < len(tokens) and tokens[i + 1] == "JUMP_BACK":
-                    # Sometimes the jump back is after the "break" instruction..
-                    jump_back_index = i + 1
-                else:
-                    # and sometimes, because of jump-to-jump optimization, it is before the
-                    # jump target instruction.
-                    jump_back_index = self.offset2tok_index[jump_target] - 1
-                    while tokens[jump_back_index].kind.startswith("COME_FROM_"):
-                        jump_back_index -= 1
-                        pass
-                    pass
-                jump_back_token = tokens[jump_back_index]
-
-                # Is this a forward jump not next to a JUMP_BACK ? ...
-                break_loop = token.linestart and jump_back_token != "JUMP_BACK"
-
-                # or if there is looping jump back, then that loop
-                # should start before where the "break" instruction sits.
-                if break_loop or (
-                    jump_back_token == "JUMP_BACK"
-                    and jump_back_token.attr < token.off2int()
-                ):
+                if token_with_linestart.linestart:
                     token.kind = "BREAK_LOOP"
+
                 pass
             new_tokens.append(token)
+
+        if show_asm in ("both", "after"):
+            print("\n# ---- tokenization:")
+            # FIXME: t.format() is changing tokens!
+            for t in new_tokens.copy():
+                print(t.format(line_prefix=""))
+            print()
+
         return new_tokens, customize
 
 

uncompyle6/scanners/tok.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2021, 2023 by Rocky Bernstein
+#  Copyright (c) 2016-2021, 2023-2024 by Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
 #

uncompyle6/semantics/aligner.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2018, 2022-2023 by Rocky Bernstein
+#  Copyright (c) 2018, 2022-2024 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -18,11 +18,14 @@ import sys
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from xdis import iscode
 
-from xdis.version_info import IS_PYPY
+from xdis.version_info import IS_PYPY, PYTHON_VERSION_TRIPLE
+
 from uncompyle6.scanner import get_scanner
+from uncompyle6.semantics.consts import  ASSIGN_DOC_STRING
+
 from uncompyle6.semantics.pysource import (
-    ASSIGN_DOC_STRING,
     RETURN_NONE,
+    TREE_DEFAULT_DEBUG,
     SourceWalker,
     SourceWalkerError,
     find_globals_and_nonlocals
@@ -38,7 +41,7 @@ class AligningWalker(SourceWalker, object):
         version,
         out,
         scanner,
-        showast=False,
+        showast=TREE_DEFAULT_DEBUG,
         debug_parser=PARSER_DEFAULT_DEBUG,
         compile_mode="exec",
         is_pypy=False,
@@ -48,6 +51,7 @@ class AligningWalker(SourceWalker, object):
         )
         self.desired_line_number = 0
         self.current_line_number = 0
+        self.showast = showast
 
     def println(self, *data):
         if data and not (len(data) == 1 and data[0] == ""):
@@ -113,12 +117,12 @@ class AligningWalker(SourceWalker, object):
             key = key[i]
             pass
 
-        if key.type in table:
-            self.template_engine(table[key.type], node)
+        if key.kind in table:
+            self.template_engine(table[key.kind], node)
             self.prune()
 
 
-DEFAULT_DEBUG_OPTS = {"asm": False, "tree": False, "grammar": False}
+DEFAULT_DEBUG_OPTS = {"asm": False, "tree": TREE_DEFAULT_DEBUG, "grammar": False}
 
 
 def code_deparse_align(
@@ -137,7 +141,7 @@ def code_deparse_align(
     assert iscode(co)
 
     if version is None:
-        version = float(sys.version[0:3])
+        version = PYTHON_VERSION_TRIPLE
     if is_pypy is None:
         is_pypy = IS_PYPY
 
@@ -156,11 +160,11 @@ def code_deparse_align(
         debug_parser["errorstack"] = True
 
     #  Build a parse tree from tokenized and massaged disassembly.
-    show_ast = debug_opts.get("ast", None)
+    show_ast = debug_opts.get("ast", TREE_DEFAULT_DEBUG)
     deparsed = AligningWalker(
         version,
-        scanner,
         out,
+        scanner,
         showast=show_ast,
         debug_parser=debug_parser,
         compile_mode=compile_mode,
@@ -210,4 +214,4 @@ if __name__ == "__main__":
         print(deparsed.text)
         return
 
-    deparse_test(deparse_test.__code__)
+    deparse_test(deparse_test.func_code)

uncompyle6/semantics/consts.py

@@ -44,7 +44,7 @@ maxint = sys.maxsize
 # children. For example, "call" has precedence 2 so we don't get
 # additional the additional parenthesis of: ".. op (call())".  However
 # for call's children, it parameters, we set the the precedence high,
-# say to 100, to make sure we avoid additional prenthesis in
+# say to 100, to make sure we avoid additional parenthesis in
 # call((.. op ..)).
 
 NO_PARENTHESIS_EVER = 100
@@ -57,6 +57,8 @@ PRECEDENCE = {
     "list_unpack":            38,  # *args
     "yield_from":             38,
     "tuple_list_starred":     38,  # *x, *y, *z - about at the level of yield?
+    "unpack":                 38,  # A guess. Used in "async with ... as ...
+                                   # This might also get used in tuple assignment?
 
     "_lambda_body":           30,
     "lambda_body":            32,  # lambda ... : lambda_body
@@ -130,7 +132,7 @@ LINE_LENGTH = 80
 # Some parse trees created below are used for comparing code
 # fragments (like "return None" at the end of functions).
 
-ASSIGN_DOC_STRING = lambda doc_string, doc_load: SyntaxTree(
+ASSIGN_DOC_STRING = lambda doc_string, doc_load: SyntaxTree(    # noqa
     "assign",
     [
         SyntaxTree(
@@ -156,6 +158,7 @@ NAME_MODULE = SyntaxTree(
     ],
 )
 
+NEWLINE = SyntaxTree("newline", [])
 NONE = SyntaxTree("expr", [NoneToken])
 
 RETURN_NONE = SyntaxTree("stmt", [SyntaxTree("return", [NONE, Token("RETURN_VALUE")])])
@@ -240,149 +243,86 @@ TABLE_DIRECT = {
     "UNARY_NOT":                ( "not ", ),
     "UNARY_POSITIVE":           ( "+",),
 
-    # bin_op (formerly "binary_expr") is the Python AST BinOp
-    "bin_op": ("%c %c %c", 0, (-1, "binary_operator"), (1, "expr")),
-    # unary_op (formerly "unary_expr") is the Python AST UnaryOp
-    "unary_op": ("%c%c", (1, "unary_operator"), (0, "expr")),
-    "unary_not": ("not %c", (0, "expr")),
-    "unary_convert": ("`%c`", (0, "expr"),),
-    "get_iter": ("iter(%c)", (0, "expr"),),
+    "and": ("%c and %c", 0, 2),
+    "and2": ("%c", 3),
 
-    "set_iter":        ( "%c", 0 ),
+    "assert_expr_or": ("%c or %c", 0, 2),
+    "assert_expr_and": ("%c and %c", 0, 2),
 
-    "slice0": (
-        "%c[:]",
-        (0, "expr"),
-        ),
-    "slice1": (
-        "%c[%p:]",
-        (0, "expr"),
-        (1, NO_PARENTHESIS_EVER)
-        ),
-
-    "slice2": ( "%c[:%p]",
-        (0, "expr"),
-        (1, NO_PARENTHESIS_EVER)
-        ),
-
-    "slice3": (
-        "%c[%p:%p]",
-        (0, "expr"),
-        (1, NO_PARENTHESIS_EVER),
-        (2, NO_PARENTHESIS_EVER)
+    "assign": (
+        "%|%c = %p\n",
+        -1,
+        (0, ("expr", "branch_op"), PRECEDENCE["tuple_list_starred"] + 1)
         ),
 
     "attribute": ("%c.%[1]{pattr}", (0, "expr")),
-    "delete_subscript": (
-        "%|del %p[%c]\n",
-        (0, "expr", PRECEDENCE["subscript"]),
-        (1, "expr"),
-    ),
 
-    "subscript": (
-        "%p[%p]",
-        (0, "expr", PRECEDENCE["subscript"]),
-        (1, "expr", NO_PARENTHESIS_EVER)
-    ),
-
-    "subscript2": (
-        "%p[%p]",
-        (0, "expr", PRECEDENCE["subscript"]),
-        (1, "expr", NO_PARENTHESIS_EVER)
-    ),
-
-    "store_subscript": ("%p[%c]", (0, "expr", PRECEDENCE["subscript"]), (1, "expr")),
-    "unpack": ("%C%,", (1, maxint, ", ")),
-    # This nonterminal we create on the fly in semantic routines
-    "unpack_w_parens": ("(%C%,)", (1, maxint, ", ")),
     # This nonterminal we create on the fly in semantic routines
     "attribute_w_parens": ("(%c).%[1]{pattr}", (0, "expr")),
-    # This nonterminal we create on the fly in semantic routines
-    "store_w_parens": ("(%c).%[1]{pattr}", (0, "expr")),
-    "unpack_list": ("[%C]", (1, maxint, ", ")),
-    "build_tuple2": ("%P", (0, -1, ", ", 100)),
-    "list_iter": ("%c", 0),
-    "list_for": (" for %c in %c%c", 2, 0, 3),
-    "list_if": (" if %p%c", (0, "expr", 27), 2),
-    "list_if_not": (" if not %p%c", (0, "expr", PRECEDENCE["unary_not"]), 2),
-    "lc_body": ("",),  # ignore when recursing
-    "comp_iter": ("%c", 0),
-    "comp_if": (" if %c%c", 0, 2),
-    "comp_if_not": (" if not %p%c", (0, "expr", PRECEDENCE["unary_not"]), 2),
-    "comp_body": ("",),  # ignore when recusing
 
-    "set_comp_body": ("%c", 0),
-    "gen_comp_body": ("%c", 0),
-    "dict_comp_body": ("%c: %c", 1, 0),
-    "assign": ("%|%c = %p\n", -1, (0, 200)),
     # The 2nd parameter should have a = suffix.
     # There is a rule with a 4th parameter "store"
     # which we don't use here.
     "aug_assign1": ("%|%c %c %c\n", 0, 2, 1),
     "aug_assign2": ("%|%c.%[2]{pattr} %c %c\n", 0, -3, -4),
-    "designList": ("%c = %c", 0, -1),
-    "and": ("%c and %c", 0, 2),
-    "ret_and": ("%c and %c", 0, 2),
-    "and2": ("%c", 3),
-    "or": ("%p or %p", (0, PRECEDENCE["or"]), (1, PRECEDENCE["or"])),
-    "ret_or": ("%c or %c", 0, 2),
-    "if_exp": ("%p if %c else %c", (2, "expr", 27), 0, 4),
-    "if_exp_lambda": ("%p if %c else %c", (2, "expr", 27), (0, "expr"), 4),
-    "if_exp_true": ("%p if 1 else %c", (0, "expr", 27), 2),
-    "if_exp_ret": ("%p if %p else %p", (2, 27), (0, 27), (-1, 27)),
-    "if_exp_not": (
-        "%p if not %p else %p",
-        (2, 27),
-        (0, "expr", PRECEDENCE["unary_not"]),
-        (4, 27),
-    ),
-    "if_exp_not_lambda": ("%p if not %c else %c", (2, "expr", 27), 0, 4),
-    "compare_single": ('%p %[-1]{pattr.replace("-", " ")} %p', (0, 19), (1, 19)),
-    "compare_chained": ("%p %p", (0, 29), (1, 30)),
-    "compared_chained_middle": ('%[3]{pattr.replace("-", " ")} %p %p', (0, 19), (-2, 19)),
-    "compare_chained_right": ('%[1]{pattr.replace("-", " ")} %p', (0, 19)),
-    #   "classdef": 	(), # handled by n_classdef()
-    # A custom rule in n_function def distinguishes whether to call this or
-    # function_def_async
 
-    "function_def": ("\n\n%|def %c\n", -2),  # -2 to handle closures
-    "function_def_deco": ("\n\n%c", 0),
-    "mkfuncdeco": ("%|@%c\n%c", 0, 1),
-    # A custom rule in n_function def distinguishes whether to call this or
-    # function_def_async
-    "mkfuncdeco0": ("%|def %c\n", 0),
-    "classdefdeco": ("\n\n%c", 0),
-    "classdefdeco1": ("%|@%c\n%c", 0, 1),
-    "kwarg": ("%[0]{pattr}=%c", 1),  # Change when Python 2 does LOAD_STR
-    "kwargs": ("%D", (0, maxint, ", ")),
-    "kwargs1": ("%D", (0, maxint, ", ")),
-    "assert_expr_or": ("%c or %c", 0, 2),
-    "assert_expr_and": ("%c and %c", 0, 2),
-    "print_items_stmt": ("%|print %c%c,\n", 0, 2),  # Python 2 only
-    "print_items_nl_stmt": ("%|print %c%c\n", 0, 2),
-    "print_item": (", %c", 0),
-    "print_nl": ("%|print\n",),
-    "print_to": ("%|print >> %c, %c,\n", 0, 1),
-    "print_to_nl": ("%|print >> %c, %c\n", 0, 1),
-    "print_nl_to": ("%|print >> %c\n", 0),
-    "print_to_items": ("%C", (0, 2, ", ")),
-    # This is only generated by transform
-    # it is a string at the beginning of a function that is *not* a docstring
-    # 3.7 test_fstring.py tests for this kind of crap.
-    # For compatibility with older Python, we'll use "%" instead of
-    # a format string.
-    "string_at_beginning": ('%|"%%s" %% %c\n', 0),
+    # bin_op (formerly "binary_expr") is the Python AST BinOp
+    "bin_op": ("%c %c %c", 0, (-1, "binary_operator"), (1, "expr")),
+
+    "break": ("%|break\n",),
+    "build_tuple2": (
+        "%P",
+        (0, -1, ", ", NO_PARENTHESIS_EVER)
+    ),
+
     "call_stmt":	   ( "%|%p\n",
                              # When a call statement contains only a named_expr (:=)
                              # the named_expr should have parenthesis around it.
                              (0, PRECEDENCE["named_expr"]-1)),
-    "break": ("%|break\n",),
+
+    # "classdef": (), # handled by n_classdef()
+    # A custom rule in n_function def distinguishes whether to call this or
+    # function_def_async
+
+    "classdefdeco": ("\n\n%c", 0),
+    "classdefdeco1": ("%|@%c\n%c", 0, 1),
+
+    "comp_body": ("",),  # ignore when recusing
+    "comp_if": (" if %c%c", 0, 2),
+    "comp_if_not": (" if not %p%c", (0, "expr", PRECEDENCE["unary_not"]), 2),
+    "comp_iter": ("%c", 0),
+
+    "compare_single": ('%p %[-1]{pattr.replace("-", " ")} %p', (0, 19), (1, 19)),
+    "compare_chained": ("%p %p", (0, 29), (1, 30)),
+    "compared_chained_middle": ('%[3]{pattr.replace("-", " ")} %p %p', (0, 19), (-2, 19)),
+    "compare_chained_right": ('%[1]{pattr.replace("-", " ")} %p', (0, 19)),
+
     "continue": ("%|continue\n",),
 
+    "delete_subscript": (
+        "%|del %p[%c]\n",
+        (0, "expr", PRECEDENCE["subscript"]),
+        (1, "expr"),
+    ),
+    "designList": ("%c = %c", 0, -1),
+    "dict_comp_body": ("%c: %c", 1, 0),
+
+    "elifelifstmt": ("%|elif %c:\n%+%c%-%c", 0, 1, 3),
+    "elifelsestmt": ("%|elif %c:\n%+%c%-%|else:\n%+%c%-", 0, 1, 3),
+    "elifelsestmtr": ("%|elif %c:\n%+%c%-%|else:\n%+%c%-\n\n", 0, 1, 2),
+    "elifelsestmtr2": (
+        "%|elif %c:\n%+%c%-%|else:\n%+%c%-\n\n",
+        0,
+        1,
+        3,
+    ),  # has COME_FROM
+    "elifstmt": ("%|elif %c:\n%+%c%-", 0, 1),
+
     "except": ("%|except:\n%+%c%-", 3),
     "except_cond1": ("%|except %c:\n", 1),
     "except_cond2": ("%|except %c as %c:\n", (1, "expr"), (5, "store")),
     "except_suite": ("%+%c%-%C", 0, (1, maxint, "")),
+
     # In Python 3.6+, this is more complicated in the presence of "returns"
     "except_suite_finalize": ("%+%c%-%C", 1, (3, maxint, "")),
 
@@ -391,7 +331,7 @@ TABLE_DIRECT = {
         # When a statement contains only a named_expr (:=)
         # the named_expr should have parenthesis around it.
         (0, "expr", PRECEDENCE["named_expr"] - 1)
-        ),
+    ),
 
     # Note: Python 3.8+ changes this
     "for": ("%|for %c in %c:\n%+%c%-\n\n", (3, "store"), (1, "expr"), (4, "for_block")),
@@ -418,24 +358,24 @@ TABLE_DIRECT = {
         -2,
     ),
 
-    "raise_stmt0": ("%|raise\n",),
-    "raise_stmt1": ("%|raise %c\n", 0),
-    "raise_stmt3": ("%|raise %c, %c, %c\n", 0, 1, 2),
-    #    "yield":	        ( "yield %c", 0),
+    "function_def": ("\n\n%|def %c\n", -2),  # -2 to handle closures
+    "function_def_deco": ("\n\n%c", 0),
 
-    # Note: we have a custom rule, which calls when we don't
-    # have "return None"
-    "return":	( "%|return %c\n", 0),
+    "gen_comp_body": ("%c", 0),
+    "get_iter": ("iter(%c)", (0, "expr"),),
 
-    "return_if_stmt": ("return %c\n", 0),
-    "ifstmt": (
-        "%|if %c:\n%+%c%-",
-        0,  # "testexpr" or "testexpr_then"
-        1,  # "_ifstmts_jump" or "return_stmts"
+    "if_exp": ("%p if %c else %c", (2, "expr", 27), 0, 4),
+    "if_exp_lambda": ("%p if %c else %c", (2, "expr", 27), (0, "expr"), 4),
+    "if_exp_true": ("%p if 1 else %c", (0, "expr", 27), 2),
+    "if_exp_ret": ("%p if %p else %p", (2, 27), (0, 27), (-1, 27)),
+    "if_exp_not": (
+        "%p if not %p else %p",
+        (2, 27),
+        (0, "expr", PRECEDENCE["unary_not"]),
+        (4, 27),
     ),
-    "iflaststmt": ("%|if %c:\n%+%c%-", 0, 1),
-    "iflaststmtl": ("%|if %c:\n%+%c%-", 0, 1),
-    "testtrue": ("not %p", (0, PRECEDENCE["unary_not"])),
+    "if_exp_not_lambda": ("%p if not %c else %c", (2, "expr", 27), 0, 4),
+
     # Generally the args here are 0: (some sort of) "testexpr",
     #                             1: (some sort of) "cstmts_opt",
     #                             2 or 3: "else_suite"
@@ -445,20 +385,21 @@ TABLE_DIRECT = {
     "ifelsestmt": ("%|if %c:\n%+%c%-%|else:\n%+%c%-", 0, 1, 3),
     "ifelsestmtc": ("%|if %c:\n%+%c%-%|else:\n%+%c%-", 0, 1, 3),
     "ifelsestmtl": ("%|if %c:\n%+%c%-%|else:\n%+%c%-", 0, 1, 3),
-    #  These are created only via transformation
+
+    #  This is created only via transformation.
     "ifelifstmt": ("%|if %c:\n%+%c%-%c", 0, 1, 3),  # "testexpr" or "testexpr_then"
-    "elifelifstmt": ("%|elif %c:\n%+%c%-%c", 0, 1, 3),
-    "elifstmt": ("%|elif %c:\n%+%c%-", 0, 1),
-    "elifelsestmt": ("%|elif %c:\n%+%c%-%|else:\n%+%c%-", 0, 1, 3),
+
     "ifelsestmtr": ("%|if %c:\n%+%c%-%|else:\n%+%c%-", 0, 1, 2),
     "ifelsestmtr2": ("%|if %c:\n%+%c%-%|else:\n%+%c%-\n\n", 0, 1, 3),  # has COME_FROM
-    "elifelsestmtr": ("%|elif %c:\n%+%c%-%|else:\n%+%c%-\n\n", 0, 1, 2),
-    "elifelsestmtr2": (
-        "%|elif %c:\n%+%c%-%|else:\n%+%c%-\n\n",
-        0,
-        1,
-        3,
-    ),  # has COME_FROM
+    "iflaststmt": ("%|if %c:\n%+%c%-", 0, 1),
+
+    "iflaststmtl": ("%|if %c:\n%+%c%-", 0, 1),
+
+    "ifstmt": (
+        "%|if %c:\n%+%c%-",
+        0,  # "testexpr" or "testexpr_then"
+        1,  # "_ifstmts_jump" or "return_stmts"
+    ),
 
     "import": ("%|import %c\n", 2),
     "importlist": ("%C", (0, maxint, ", ")),
@@ -476,8 +417,131 @@ TABLE_DIRECT = {
 
     "kv": ("%c: %c", 3, 1),
     "kv2": ("%c: %c", 1, 2),
+
+    "kwarg": ("%[0]{pattr}=%c", 1),  # Change when Python 2 does LOAD_STR
+    "kwargs": ("%D", (0, maxint, ", ")),
+    "kwargs1": ("%D", (0, maxint, ", ")),
+
+    "lc_body": ("",),  # ignore when recursing
+    "list_iter": ("%c", 0),
+    "list_for": (" for %c in %c%c", 2, 0, 3),
+    "list_if": (" if %p%c", (0, "expr", 27), 2),
+    "list_if_not": (" if not %p%c", (0, "expr", PRECEDENCE["unary_not"]), 2),
+
+    "mkfuncdeco": ("%|@%c\n%c", (0, "expr"), 1),
+    # A custom rule in n_function def distinguishes whether to call this or
+    # function_def_async
+    "mkfuncdeco0": ("%|def %c\n", (0, ("mkfunc", "mkfunc_annotate"))),
+
+    # In cases where we desire an explicit new line.
+    # After docstrings which are followed by a "def" is
+    # one situations where Python formatting desires two newlines,
+    # and this is added, as a transformation rule.
+    "newline": ("\n"),
+
+    "or": ("%p or %p", (0, PRECEDENCE["or"]), (1, PRECEDENCE["or"])),
+
     "pass": ("%|pass\n",),
 
+    "print_item": (", %c", 0),
+    "print_items_nl_stmt": ("%|print %c%c\n", 0, 2),
+    "print_items_stmt": ("%|print %c%c,\n", 0, 2),  # Python 2 only
+    "print_nl": ("%|print\n",),
+    "print_nl_to": ("%|print >> %c\n", 0),
+    "print_to": ("%|print >> %c, %c,\n", 0, 1),
+    "print_to_items": ("%C", (0, 2, ", ")),
+    "print_to_nl": ("%|print >> %c, %c\n", 0, 1),
+
+    "raise_stmt0": ("%|raise\n",),
+    "raise_stmt1": ("%|raise %c\n", 0),
+    "raise_stmt3": ("%|raise %c, %c, %c\n", 0, 1, 2),
+
+    "ret_and": ("%c and %c", 0, 2),
+    "ret_or": ("%c or %c", 0, 2),
+
+    # Note: we have a custom rule, which calls when we don't
+    # have "return None"
+    "return":	( "%|return %c\n", 0),
+
+    "set_comp_body": ("%c", 0),
+
+    "set_iter":        ( "%c", 0 ),
+
+    "return_if_stmt": ("return %c\n", 0),
+    "slice0": (
+        "%c[:]",
+        (0, "expr"),
+        ),
+    "slice1": (
+        "%c[%p:]",
+        (0, "expr"),
+        (1, NO_PARENTHESIS_EVER)
+        ),
+
+    "slice2": ( "%c[:%p]",
+        (0, "expr"),
+        (1, NO_PARENTHESIS_EVER)
+        ),
+
+    "slice3": (
+        "%c[%p:%p]",
+        (0, "expr"),
+        (1, NO_PARENTHESIS_EVER),
+        (2, NO_PARENTHESIS_EVER)
+        ),
+
+    "store_subscript": (
+        "%p[%c]",
+        (0, "expr", PRECEDENCE["subscript"]), (1, "expr")
+    ),
+
+    # This nonterminal we create on the fly in semantic routines
+    "store_w_parens": (
+        "(%c).%[1]{pattr}",
+        (0, "expr")
+    ),
+
+    # This is only generated by transform
+    # it is a string at the beginning of a function that is *not* a docstring
+    # 3.7 test_fstring.py tests for this kind of crap.
+    # For compatibility with older Python, we'll use "%" instead of
+    # a format string.
+    "string_at_beginning": ('%|"%%s" %% %c\n', 0),
+
+    "subscript": (
+        "%p[%p]",
+        (0, "expr", PRECEDENCE["subscript"]),
+        (1, "expr", NO_PARENTHESIS_EVER)
+    ),
+
+    "subscript2": (
+        "%p[%p]",
+        (0, "expr", PRECEDENCE["subscript"]),
+        (1, "expr", NO_PARENTHESIS_EVER)
+    ),
+
+    "testtrue": ("not %p", (0, PRECEDENCE["unary_not"])),
+
+    # Note: this is generated generated by grammar rules but in this phase.
+    "tf_try_except": ("%c%-%c%+", 1, 3),
+    "tf_tryelsestmt": ("%c%-%c%|else:\n%+%c", 1, 3, 4),
+
+    "try_except": ("%|try:\n%+%c%-%c\n\n", 1, 3),
+    "tryelsestmt": ("%|try:\n%+%c%-%c%|else:\n%+%c%-\n\n", 1, 3, 4),
+    "tryelsestmtc": ("%|try:\n%+%c%-%c%|else:\n%+%c%-", 1, 3, 4),
+    "tryelsestmtl": ("%|try:\n%+%c%-%c%|else:\n%+%c%-", 1, 3, 4),
+    "tryfinallystmt": ("%|try:\n%+%c%-%|finally:\n%+%c%-\n\n", 1, 5),
+
+    # unary_op (formerly "unary_expr") is the Python AST UnaryOp
+    "unary_op": ("%c%c", (1, "unary_operator"), (0, "expr")),
+    "unary_not": ("not %c", (0, "expr")),
+    "unary_convert": ("`%c`", (0, "expr"),),
+
+    "unpack": ("%C%,", (1, maxint, ", ")),
+    "unpack_list": ("[%C]", (1, maxint, ", ")),
+    # This nonterminal we create on the fly in semantic routines
+    "unpack_w_parens": ("(%C%,)", (1, maxint, ", ")),
+
     "whileTruestmt": ("%|while True:\n%+%c%-\n\n", 1),
     "whilestmt": ("%|while %c:\n%+%c%-\n\n", 1, 2),
     "while1stmt": ("%|while 1:\n%+%c%-\n\n", 1),
@@ -495,15 +559,10 @@ TABLE_DIRECT = {
         (3, ("suite_stmts_opt", "suite_stmts")),
     ),
 
-    "try_except": ("%|try:\n%+%c%-%c\n\n", 1, 3),
-    "tryelsestmt": ("%|try:\n%+%c%-%c%|else:\n%+%c%-\n\n", 1, 3, 4),
-    "tryelsestmtc": ("%|try:\n%+%c%-%c%|else:\n%+%c%-", 1, 3, 4),
-    "tryelsestmtl": ("%|try:\n%+%c%-%c%|else:\n%+%c%-", 1, 3, 4),
-    # Note: this is generated generated by grammar rules but in this phase.
-    "tf_try_except": ("%c%-%c%+", 1, 3),
-    "tf_tryelsestmt": ("%c%-%c%|else:\n%+%c", 1, 3, 4),
-    "tryfinallystmt": ("%|try:\n%+%c%-%|finally:\n%+%c%-\n\n", 1, 5),
+    #    "yield":	        ( "yield %c", 0),
+
 }
+# fmt: on
 
 
 MAP_DIRECT = (TABLE_DIRECT,)
@@ -516,7 +575,7 @@ MAP = {
     "store": MAP_R,
 }
 
-ASSIGN_TUPLE_PARAM = lambda param_name: SyntaxTree(
+ASSIGN_TUPLE_PARAM = lambda param_name: SyntaxTree(  # noqa
     "expr", [Token("LOAD_FAST", pattr=param_name)]
 )
 

uncompyle6/semantics/customize.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2018-2019, 2021-2022 by Rocky Bernstein
+#  Copyright (c) 2018-2019, 2021-2022 2024 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -17,15 +17,15 @@
 """
 
 from uncompyle6.parsers.treenode import SyntaxTree
+from uncompyle6.scanners.tok import Token
 from uncompyle6.semantics.consts import (
     INDENT_PER_LEVEL,
     NO_PARENTHESIS_EVER,
     PRECEDENCE,
-    TABLE_R,
     TABLE_DIRECT,
+    TABLE_R,
 )
 from uncompyle6.semantics.helper import flatten_list
-from uncompyle6.scanners.tok import Token
 
 
 def customize_for_version(self, is_pypy, version):
@@ -34,7 +34,7 @@ def customize_for_version(self, is_pypy, version):
         # PyPy changes
         #######################
         # fmt: off
-        TABLE_DIRECT.update({
+        self.TABLE_DIRECT.update({
 
             "assert":           ("%|assert %c\n", 0),
             # This can happen as a result of an if transformation
@@ -114,7 +114,7 @@ def customize_for_version(self, is_pypy, version):
         ########################
         # Without PyPy
         #######################
-        TABLE_DIRECT.update(
+        self.TABLE_DIRECT.update(
             {
                 # "assert" and "assert_expr" are added via transform rules.
                 "assert": ("%|assert %c\n", 0),
@@ -133,23 +133,23 @@ def customize_for_version(self, is_pypy, version):
         )
     if version >= (3, 0):
         if version >= (3, 2):
-            TABLE_DIRECT.update(
+            self.TABLE_DIRECT.update(
                 {"del_deref_stmt": ("%|del %c\n", 0), "DELETE_DEREF": ("%{pattr}", 0)}
             )
         from uncompyle6.semantics.customize3 import customize_for_version3
 
         customize_for_version3(self, version)
     else:  # < 3.0
-        TABLE_DIRECT.update(
+        self.TABLE_DIRECT.update(
             {"except_cond3": ("%|except %c, %c:\n", (1, "expr"), (-2, "store"))}
         )
         if version <= (2, 6):
-            TABLE_DIRECT["testtrue_then"] = TABLE_DIRECT["testtrue"]
+            self.TABLE_DIRECT["testtrue_then"] = self.TABLE_DIRECT["testtrue"]
 
         if (2, 4) <= version <= (2, 6):
-            TABLE_DIRECT.update({"comp_for": (" for %c in %c", 3, 1)})
+            self.TABLE_DIRECT.update({"comp_for": (" for %c in %c", 3, 1)})
         else:
-            TABLE_DIRECT.update({"comp_for": (" for %c in %c%c", 2, 0, 3)})
+            self.TABLE_DIRECT.update({"comp_for": (" for %c in %c%c", 2, 0, 3)})
 
         if version >= (2, 5):
             from uncompyle6.semantics.customize25 import customize_for_version25
@@ -197,7 +197,7 @@ def customize_for_version(self, is_pypy, version):
                     )
                 ],
             )
-            TABLE_DIRECT.update(
+            self.TABLE_DIRECT.update(
                 {
                     "importmultiple": ("%|import %c%c\n", 2, 3),
                     "import_cont": (", %c", 2),
@@ -247,9 +247,9 @@ def customize_for_version(self, is_pypy, version):
                 self.n_call = n_call
 
             else:  # 1.0 <= version <= 2.3:
-                TABLE_DIRECT.update({"if1_stmt": ("%|if 1\n%+%c%-", 5)})
+                self.TABLE_DIRECT.update({"if1_stmt": ("%|if 1\n%+%c%-", 5)})
                 if version <= (2, 1):
-                    TABLE_DIRECT.update(
+                    self.TABLE_DIRECT.update(
                         {
                             "importmultiple": ("%c", 2),
                             # FIXME: not quite right. We have indiividual imports
@@ -263,7 +263,7 @@ def customize_for_version(self, is_pypy, version):
 
         # < 3.0 continues
 
-        TABLE_R.update(
+        self.TABLE_R.update(
             {
                 "STORE_SLICE+0": ("%c[:]", 0),
                 "STORE_SLICE+1": ("%c[%p:]", 0, (1, -1)),
@@ -275,7 +275,7 @@ def customize_for_version(self, is_pypy, version):
                 "DELETE_SLICE+3": ("%|del %c[%c:%c]\n", 0, 1, 2),
             }
         )
-        TABLE_DIRECT.update({"raise_stmt2": ("%|raise %c, %c\n", 0, 1)})
+        self.TABLE_DIRECT.update({"raise_stmt2": ("%|raise %c, %c\n", 0, 1)})
 
         # exec as a built-in statement is only in Python 2.x
         def n_exec_stmt(node):

uncompyle6/semantics/customize14.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2022 by Rocky Bernstein
+#  Copyright (c) 2022, 2024 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -15,16 +15,13 @@
 """Isolate Python 1.4- version-specific semantic actions here.
 """
 
-from uncompyle6.semantics.consts import TABLE_DIRECT
 
 #######################
 # Python 1.4- Changes #
 #######################
-def customize_for_version14(self, version):
-    TABLE_DIRECT.update(
+def customize_for_version14(self, version: tuple):
+    self.TABLE_DIRECT.update(
         {
-            "print_expr_stmt": (
-                ("%|print %c\n", 0)
-            ),
+            "print_expr_stmt": (("%|print %c\n", 0)),
         }
     )

uncompyle6/semantics/customize25.py

@@ -25,7 +25,7 @@ def customize_for_version25(self, version):
     ########################
     # Import style for 2.5+
     ########################
-    TABLE_DIRECT.update(
+    self.TABLE_DIRECT.update(
         {
             "importmultiple": ("%|import %c%c\n", 2, 3),
             "import_cont": (", %c", 2),
@@ -33,6 +33,7 @@ def customize_for_version25(self, version):
             # Note: It is safe to put the variables after "as" in parenthesis,
             # and sometimes it is needed.
             "with": ("%|with %c:\n%+%c%-", 0, 3),
+            "and_then": ("%c and %c", (0, "expr"), (4, "expr")),
         }
     )
 

uncompyle6/semantics/customize26_27.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019 2021 by Rocky Bernstein
+#  Copyright (c) 2019 2021, 2024 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -17,8 +17,8 @@
 
 from uncompyle6.semantics.consts import TABLE_DIRECT
 
-def customize_for_version26_27(self, version):
 
+def customize_for_version26_27(self, version: tuple):
     ########################################
     # Python 2.6+
     #    except <condition> as <var>
@@ -29,16 +29,20 @@ def customize_for_version26_27(self, version):
     # matches how we parse this in bytecode
     ########################################
     if version > (2, 6):
-        TABLE_DIRECT.update({
-            "except_cond2":	( "%|except %c as %c:\n", 1, 5 ),
-            # When a generator is a single parameter of a function,
-            # it doesn't need the surrounding parenethesis.
-            "call_generator": ('%c%P', 0, (1, -1, ', ', 100)),
-        })
+        self.TABLE_DIRECT.update(
+            {
+                "except_cond2": ("%|except %c as %c:\n", 1, 5),
+                # When a generator is a single parameter of a function,
+                # it doesn't need the surrounding parenethesis.
+                "call_generator": ("%c%P", 0, (1, -1, ", ", 100)),
+            }
+        )
     else:
-        TABLE_DIRECT.update({
-            'testtrue_then': ( 'not %p', (0, 22) ),
-        })
+        self.TABLE_DIRECT.update(
+            {
+                "testtrue_then": ("not %p", (0, 22)),
+            }
+        )
 
     # FIXME: this should be a transformation
     def n_call(node):
@@ -47,22 +51,24 @@ def customize_for_version26_27(self, version):
         for i in mapping[1:]:
             key = key[i]
             pass
-        if key.kind == 'CALL_FUNCTION_1':
+        if key.kind == "CALL_FUNCTION_1":
             # A function with one argument. If this is a generator,
             # no parenthesis is needed.
             args_node = node[-2]
-            if args_node == 'expr':
+            if args_node == "expr":
                 n = args_node[0]
-                if n == 'generator_exp':
-                    node.kind = 'call_generator'
+                if n == "generator_exp":
+                    node.kind = "call_generator"
                     pass
                 pass
 
         self.default(node)
+
     self.n_call = n_call
 
     def n_import_from(node):
         if node[0].pattr > 0:
             node[2].pattr = ("." * node[0].pattr) + node[2].pattr
         self.default(node)
+
     self.n_import_from = n_import_from

uncompyle6/semantics/customize3.py

@@ -29,8 +29,8 @@ from uncompyle6.semantics.make_function3 import make_function3_annotate
 from uncompyle6.util import get_code_name
 
 
-def customize_for_version3(self, version):
-    TABLE_DIRECT.update(
+def customize_for_version3(self, version: tuple):
+    self.TABLE_DIRECT.update(
         {
             "comp_for": (" for %c in %c", (2, "store"), (0, "expr")),
             "if_exp_not": (
@@ -183,7 +183,7 @@ def customize_for_version3(self, version):
         # the iteration variable. These rules we can ignore
         # since we pick up the iteration variable some other way and
         # we definitely don't include in the source  _[dd].
-        TABLE_DIRECT.update(
+        self.TABLE_DIRECT.update(
             {
                 "ifstmt30": (
                     "%|if %c:\n%+%c%-",
@@ -335,7 +335,7 @@ def customize_for_version3(self, version):
 
     self.n_mkfunc_annotate = n_mkfunc_annotate
 
-    TABLE_DIRECT.update(
+    self.TABLE_DIRECT.update(
         {
             "tryelsestmtl3": (
                 "%|try:\n%+%c%-%c%|else:\n%+%c%-",
@@ -350,7 +350,7 @@ def customize_for_version3(self, version):
         #######################
         # Python 3.4+ Changes #
         #######################
-        TABLE_DIRECT.update(
+        self.TABLE_DIRECT.update(
             {
                 "LOAD_CLASSDEREF": ("%{pattr}",),
                 "yield_from": ("yield from %c", (0, "expr")),

uncompyle6/semantics/customize35.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019-2020, 2022 by Rocky Bernstein
+#  Copyright (c) 2019-2020, 2022, 2024 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -16,21 +16,17 @@
 """
 
 from xdis import co_flags_is_async, iscode
-from uncompyle6.semantics.consts import (
-    INDENT_PER_LEVEL,
-    PRECEDENCE,
-    TABLE_DIRECT,
-)
 
+from uncompyle6.semantics.consts import INDENT_PER_LEVEL, PRECEDENCE, TABLE_DIRECT
 from uncompyle6.semantics.helper import flatten_list, gen_function_parens_adjust
 
 
 #######################
 # Python 3.5+ Changes #
 #######################
-def customize_for_version35(self, version):
+def customize_for_version35(self, version: tuple):
     # fmt: off
-    TABLE_DIRECT.update(
+    self.TABLE_DIRECT.update(
         {
             # nested await expressions like:
             #   return await (await bar())
@@ -197,7 +193,11 @@ def customize_for_version35(self, version):
                 self.template_engine(template, args_node)
             else:
                 if len(node) - nargs > 3:
-                    template = ("*%c, %P)", nargs + 1, (nargs + kwargs + 1, -1, ", ", 100))
+                    template = (
+                        "*%c, %P)",
+                        nargs + 1,
+                        (nargs + kwargs + 1, -1, ", ", 100),
+                    )
                 else:
                     template = ("*%c)", nargs + 1)
                 self.template_engine(template, node)

uncompyle6/semantics/customize36.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019-2023 by Rocky Bernstein
+#  Copyright (c) 2019-2024 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -15,16 +15,17 @@
 """Isolate Python 3.6 version-specific semantic actions here.
 """
 
-from xdis import iscode
 from spark_parser.ast import GenericASTTraversalPruningException
+from xdis import iscode
+
 from uncompyle6.scanners.tok import Token
-from uncompyle6.semantics.helper import flatten_list, escape_string, strip_quotes
 from uncompyle6.semantics.consts import (
     INDENT_PER_LEVEL,
     PRECEDENCE,
     TABLE_DIRECT,
     TABLE_R,
 )
+from uncompyle6.semantics.helper import escape_string, flatten_list, strip_quotes
 from uncompyle6.util import get_code_name
 
 
@@ -37,8 +38,7 @@ def escape_format(s):
 #######################
 
 
-def customize_for_version36(self, version):
-
+def customize_for_version36(self, version: tuple):
     # fmt: off
     PRECEDENCE["call_kw"]          =   0
     PRECEDENCE["call_kw36"]        =   1
@@ -50,7 +50,7 @@ def customize_for_version36(self, version):
     PRECEDENCE["dict_pack"]        =   0  # **{ ... }
     PRECEDENCE["formatted_value1"] = 100
 
-    TABLE_DIRECT.update(
+    self.TABLE_DIRECT.update(
         {
             "ann_assign_init_value": (
                 "%|%c = %p\n",
@@ -96,7 +96,7 @@ def customize_for_version36(self, version):
         }
     )
 
-    TABLE_R.update(
+    self.TABLE_R.update(
         {
             "CALL_FUNCTION_EX":    ("%c(*%P)", 0, (1, 2, ", ", 100)),
             # Not quite right
@@ -276,7 +276,7 @@ def customize_for_version36(self, version):
         if value == "":
             fmt = "%c(%p)"
         else:
-            fmt = "%%c(%s, %%p)" % value
+            fmt = "%c" + ("(%s, " % value).replace("%", "%%") + "%p)"
 
         self.template_engine(
             (fmt, (0, "expr"), (2, "build_map_unpack_with_call", 100)), node
@@ -295,7 +295,7 @@ def customize_for_version36(self, version):
         if value == "":
             fmt = "%c(%p)"
         else:
-            fmt = "%%c(%s, %%p)" % value
+            fmt = "%c" + ("(%s, " % value).replace("%", "%%") + "%p)"
 
         self.template_engine(
             (fmt, (0, "expr"), (2, "build_map_unpack_with_call", 100)), node
@@ -707,6 +707,7 @@ def customize_for_version36(self, version):
             self.comprehension_walk_newer(node, iter_index=3, code_index=0)
         self.write("]")
         self.prune()
+
     self.n_list_comp_async = n_list_comp_async
 
     # def kwargs_only_36(node):

uncompyle6/semantics/customize37.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019-2023 by Rocky Bernstein
+#  Copyright (c) 2019-2024 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -17,14 +17,15 @@
 
 import re
 
-from uncompyle6.semantics.consts import INDENT_PER_LEVEL, PRECEDENCE, TABLE_DIRECT
+from uncompyle6.semantics.consts import INDENT_PER_LEVEL, PRECEDENCE
 from uncompyle6.semantics.helper import flatten_list
 
+# FIXME get from a newer xdis
 FSTRING_CONVERSION_MAP = {1: "!s", 2: "!r", 3: "!a", "X": ":X"}
 
 
 #######################
-def customize_for_version37(self, version):
+def customize_for_version37(self, version: tuple):
     ########################
     # Python 3.7+ changes
     #######################
@@ -46,7 +47,7 @@ def customize_for_version37(self, version):
     PRECEDENCE["dict_unpack"]      =   0  # **{...}
 
     # fmt: on
-    TABLE_DIRECT.update(
+    self.TABLE_DIRECT.update(
         {
             "and_not": ("%c and not %c", (0, "expr"), (2, "expr")),
             "ann_assign": (

uncompyle6/semantics/customize38.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2019-2020, 2022 by Rocky Bernstein
+#  Copyright (c) 2019-2020, 2022, 2024 by Rocky Bernstein
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -24,13 +24,12 @@ from uncompyle6.semantics.customize37 import FSTRING_CONVERSION_MAP
 from uncompyle6.semantics.helper import escape_string, strip_quotes
 
 
-def customize_for_version38(self, version):
+def customize_for_version38(self, version: tuple):
     # FIXME: pytest doesn't add proper keys in testing. Reinstate after we have fixed pytest.
     # for lhs in 'for forelsestmt forelselaststmt '
     #             'forelselaststmtc tryfinally38'.split():
     #     del TABLE_DIRECT[lhs]
-
-    TABLE_DIRECT.update(
+    self.TABLE_DIRECT.update(
         {
             "async_for_stmt38": (
                 "%|async for %c in %c:\n%+%c%-%-\n\n",
@@ -144,7 +143,7 @@ def customize_for_version38(self, version):
             "whilestmt38": (
                 "%|while %c:\n%+%c%-\n\n",
                 (1, ("bool_op", "testexpr", "testexprc")),
-                (2, ("l_stmts", "l_stmts_opt", "pass")),
+                (2, ("_stmts", "l_stmts", "l_stmts_opt", "pass")),
             ),
             "whileTruestmt38": (
                 "%|while True:\n%+%c%-\n\n",