Get ready for release 3.6.0

With grammar reduction tests. Same as tryelsestmt. Lots of stdlib tests work
now. (More remain though.)

NEWS.md

@@ -1,12 +1,32 @@
-3.5.1 2019-10-29 JNC
+3.6.0: 2019-12-10 gecko gecko
+=============================
+
+The main focus in this release was more accurate decompilation especially
+for 3.7 and 3.8. However there are some improvments to Python 2.x as well,
+including one of the long-standing problems of detecting the difference between
+`try ... ` and `try else ...`.
+
+With this release we now rebase Python 3.7 on off of a 3.7 base; This
+is also as it is (now) in decompyle3.  This facilitates removing some of the
+cruft in control-flow detection in the 2.7 uncompyle2 base.
+
+Alas, decompilation speed for 3.7 on is greatly increased. Hopefull
+this is temporary (cough, cough) until we can do a static control flow
+pass.
+
+Finally, runing in 3.9-dev is tolerated. We can disassemble, but no parse tables yet.
+
+
+3.5.1 2019-11-17 JNC
 ====================
 
 - Pypy 3.3, 3.5, 3.6, and 3.6.9 support
+- bump xdis version to handle newer Python releases, e.g. 2.7.17, 3.5.8, and 3.5.9
 - Improve 3.0 decompilation
     - no parse errors on stlib bytecode. However accurate translation in
 	  control-flow and and/or detection needs work
 - Remove extraneous iter() in "for" of list comprehension  Fixes #272
-- "for" block without a POP_BLOCK and confusing JUMP_BACK for CONTINUE. Fixes #293
+- "for" block without a `POP_BLOCK `and confusing `JUMP_BACK` for `CONTINUE`. Fixes #293
 - Fix unmarshal incompletness detected in Pypy 3.6
 - Miscellaneous bugs fixed
 
@@ -14,19 +34,19 @@
 =================================
 
 - Fix fragment bugs
-   * missing RETURN_LAST introduced when adding transformation layer
+   * missing `RETURN_LAST` introduced when adding transformation layer
    * more parent entries on tokens
-- Preliminary support for decompiling Python 1.0, 1.1. 1.2 and 1.6
-   * Newer xdis version needed
+- Preliminary support for decompiling Python 1.0, 1.1, 1.2, and 1.6
+   * Newer _xdis_ version needed
 
 3.4.1 2019-10-02
 ================
 
 - Correct assert{,2} transforms Fixes #289
 - Fragment parsing fixes:
-     * Wasn't handling 3-arg %p
-   	 * fielding error in code_deparse()
-- Use newer xdis to better track Python 3.8.0
+     * Wasn't handling 3-arg `%p`
+   	 * fielding error in `code_deparse()`
+- Use newer _xdis_ to better track Python 3.8.0
 
 
 3.4.0 2019-08-24 Totoro

README.rst

@@ -15,7 +15,7 @@ Introduction
 *uncompyle6* translates Python bytecode back into equivalent Python
 source code. It accepts bytecodes from Python version 1.0 to version
 3.8, spanning over 24 years of Python releases. We include Dropbox's
-Python 2.5 bytecode and some PyPy bytecode.
+Python 2.5 bytecode and some PyPy bytecodes.
 
 Why this?
 ---------
@@ -46,14 +46,15 @@ not exist and there is just bytecode. Again, my debuggers make use of
 this.
 
 There were (and still are) a number of decompyle, uncompyle,
-uncompyle2, uncompyle3 forks around. Almost all of them come basically
-from the same code base, and (almost?) all of them are no longer
-actively maintained. One was really good at decompiling Python 1.5-2.3
-or so, another really good at Python 2.7, but that only. Another
-handles Python 3.2 only; another patched that and handled only 3.3.
-You get the idea. This code pulls all of these forks together and
-*moves forward*. There is some serious refactoring and cleanup in this
-code base over those old forks.
+uncompyle2, uncompyle3 forks around. Many of them come basically from
+the same code base, and (almost?) all of them are no longer actively
+maintained. One was really good at decompiling Python 1.5-2.3, another
+really good at Python 2.7, but that only. Another handles Python 3.2
+only; another patched that and handled only 3.3.  You get the
+idea. This code pulls all of these forks together and *moves
+forward*. There is some serious refactoring and cleanup in this code
+base over those old forks. Even more experimental refactoring is going
+on in decompile3_.
 
 This demonstrably does the best in decompiling Python across all
 Python versions. And even when there is another project that only
@@ -75,11 +76,11 @@ fixed in the other decompilers.
 Requirements
 ------------
 
-The code here can be run on Python versions 2.6 or later, PyPy 3-2.4,
-or PyPy-5.0.1.  Python versions 2.4-2.7 are supported in the
-python-2.4 branch.  The bytecode files it can read have been tested on
-Python bytecodes from versions 1.4, 2.1-2.7, and 3.0-3.8 and the
-above-mentioned PyPy versions.
+The code here can be run on Python versions 2.6 or later, PyPy 3-2.4
+and later.  Python versions 2.4-2.7 are supported in the python-2.4
+branch.  The bytecode files it can read have been tested on Python
+bytecodes from versions 1.4, 2.1-2.7, and 3.0-3.8 and later PyPy
+versions.
 
 Installation
 ------------
@@ -186,15 +187,21 @@ they had been rare.  Perhaps to compensate for the additional
 added. So in sum handling control flow by ad hoc means as is currently
 done is worse.
 
-Between Python 3.5, 3.6 and 3.7 there have been major changes to the
-:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.
+Between Python 3.5, 3.6, 3.7 there have been major changes to the
+:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.  Python
+
+Python 3.8 removes :code:`SETUP_LOOP`, :code:`SETUP_EXCEPT`,
+:code:`BREAK_LOOP`, and :code:`CONTINUE_LOOP`, instructions which may
+make control-flow detection harder, lacking the more sophisticated
+control-flow analysis that is planned. We'll see.
 
 Currently not all Python magic numbers are supported. Specifically in
 some versions of Python, notably Python 3.6, the magic number has
 changes several times within a version.
 
-**We support only released versions, not candidate versions.** Note however
-that the magic of a released version is usually the same as the *last* candidate version prior to release.
+**We support only released versions, not candidate versions.** Note
+however that the magic of a released version is usually the same as
+the *last* candidate version prior to release.
 
 There are also customized Python interpreters, notably Dropbox,
 which use their own magic and encrypt bytcode. With the exception of
@@ -216,7 +223,7 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/zrax/pycdc : purports to support all versions of Python. It is written in C++ and is most accurate for Python versions around 2.7 and 3.3 when the code was more actively developed. Accuracy for more recent versions of Python 3 and early versions of Python are especially lacking. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
+* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back ehre.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
 * https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details
@@ -225,6 +232,7 @@ See Also
 * https://github.com/rocky/python-xdis : Cross Python version disassembler
 * https://github.com/rocky/python-xasm : Cross Python version assembler
 * https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
+* https://github.com/zrax/pycdc : The README for this C++ code syas it aims to support all versions of Python. It is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
 
 
 .. _trepan: https://pypi.python.org/pypi/trepan2g
@@ -233,6 +241,7 @@ See Also
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake
 .. _pycdc: https://github.com/zrax/pycdc
+.. _decompile3: https://github.com/rocky/python-decompile3
 .. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
 .. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
 		 :target: https://travis-ci.org/rocky/python-uncompyle6

__pkginfo__.py

@@ -58,7 +58,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ["spark-parser >= 1.8.9, < 1.9.0",
-                      "xdis >= 4.1.3, < 4.2.0"]
+                      "xdis >= 4.2.0, < 4.3.0"]
 
 license            = "GPL3"
 mailing_list       = "python-debugger@googlegroups.com"

pytest/test_grammar.py

@@ -20,8 +20,12 @@ def test_grammar():
     (lhs, rhs, tokens, right_recursive, dup_rhs) = p.check_sets()
 
     # We have custom rules that create the below
-    expect_lhs = set(["pos_arg", "attribute"])
+    expect_lhs = set(["pos_arg"])
+
     if PYTHON_VERSION < 3.8:
+        if PYTHON_VERSION < 3.7:
+            expect_lhs.add("attribute")
+
         expect_lhs.add("get_iter")
     else:
         expect_lhs.add("async_with_as_stmt")
@@ -31,7 +35,7 @@ def test_grammar():
 
     expect_right_recursive = set([("designList", ("store", "DUP_TOP", "designList"))])
 
-    if PYTHON_VERSION < 3.7:
+    if PYTHON_VERSION <= 3.7:
         unused_rhs.add("call")
 
     if PYTHON_VERSION > 2.6:
@@ -50,9 +54,11 @@ def test_grammar():
             )
         )
         if PYTHON_VERSION >= 3.0:
-            expect_lhs.add("annotate_arg")
-            expect_lhs.add("annotate_tuple")
-            unused_rhs.add("mkfunc_annotate")
+            if PYTHON_VERSION < 3.7:
+                expect_lhs.add("annotate_arg")
+                expect_lhs.add("annotate_tuple")
+                unused_rhs.add("mkfunc_annotate")
+
             unused_rhs.add("dict_comp")
             unused_rhs.add("classdefdeco1")
             unused_rhs.add("tryelsestmtl")

setup.py

@@ -4,8 +4,8 @@ import sys
 """Setup script for the 'uncompyle6' distribution."""
 
 SYS_VERSION = sys.version_info[0:2]
-if not ((2, 6) <= SYS_VERSION  <= (3, 8)):
-    mess = "Python Release 2.6 .. 3.8 are supported in this code branch."
+if not ((2, 6) <= SYS_VERSION  <= (3, 9)):
+    mess = "Python Release 2.6 .. 3.9 are supported in this code branch."
     if ((2, 4) <= SYS_VERSION <= (2, 7)):
         mess += ("\nFor your Python, version %s, use the python-2.4 code/branch." %
                  sys.version[0:3])

test/simple_source/bug26/03_complex_and.py

@@ -14,4 +14,4 @@ for y in (1, 2, 10):
         expected = 3
     result.append(expected)
 
-assert result == [10, 2, 3]
+assert result == [3, 2, 3]

test/simple_source/bug37/03_ifelse_chained_for.py

@@ -0,0 +1,24 @@
+# From decompyle3/semantics/customize3.py
+# The bug is handling "for" loop inside the
+# chained compare ifelse
+def n_classdef3(a, b, c, l):
+    r = 1
+    if 3.0 <= a <= 3.2:
+        for n in l:
+            if b:
+                break
+            elif c:
+                r = 2
+            pass
+        pass
+    else:
+        r = 3
+        pass
+    return r
+
+assert n_classdef3(10, True, True, []) == 3
+assert n_classdef3(0, False, True, []) == 3
+assert n_classdef3(3.1, True, True, []) == 1
+assert n_classdef3(3.1, True, False, [1]) == 1
+assert n_classdef3(3.1, True, True, [2]) == 1
+assert n_classdef3(3.1, False, True, [3]) == 2

test/simple_source/bug38/01_named_expr.py

@@ -0,0 +1,5 @@
+(x, y) = "foo", 0
+if x := __name__:
+    y = 1
+assert x == "__main__", "Walrus operator changes value"
+assert y == 1, "Walrus operator branch taken"

test/simple_source/bug38/04_for_no_jump_back.py

@@ -0,0 +1,17 @@
+# from mult_by_const/instruction.py
+# Bug in 3.8 was handling no JUMP_BACK in "for" loop. It is
+# in the "if" instead
+def instruction_sequence_value(instrs, a, b):
+    for instr in instrs:
+        if a:
+            a = 6
+        elif b:
+            return 0
+        pass
+
+    return a
+
+assert instruction_sequence_value([], True, True) == 1
+assert instruction_sequence_value([1], True, True) == 6
+assert instruction_sequence_value([1], False, True) == 0
+assert instruction_sequence_value([1], False, False) == False

test/simple_source/expression/02_true_divide.py

@@ -1,4 +1,4 @@
 # Tests BINARY_TRUE_DIVIDE and INPLACE_TRUE_DIVIDE
-from __future__ import division  # needed on 2.6 and 2.7
+from __future__ import division  # needed on 2.2 .. 2.7
 x = len(__file__) / 1
 x /= 1

test/simple_source/expression/14_mixed_expressions.py

@@ -0,0 +1,53 @@
+# Covers a large number of operators
+#
+# This code is RUNNABLE!
+
+import sys
+PYTHON_VERSION = sys.version_info[0] + (sys.version_info[1] / 10.0)
+
+assert PYTHON_VERSION >= 3.7
+
+# some floats (from 01_float.py)
+
+x = 1e300
+assert 0.0 == x * 0
+assert x * 1e300 == float("inf")
+assert str(float("inf") * 0.0) == "nan"
+assert str(float("-inf") * 0.0) == "nan"
+assert -1e300 * 1e300 == float("-inf")
+
+# Complex (adapted from 02_complex.py)
+y = 5j
+assert y ** 2 == -25
+y **= 3
+assert y == (-0-125j)
+
+
+# Tests BINARY_TRUE_DIVIDE and INPLACE_TRUE_DIVIDE (from 02_try_divide.py)
+x = 2
+assert 4 / x == 2
+
+x = 5
+assert x / 2 == 2.5
+x = 3
+x /= 2
+assert x == 1.5
+
+x = 2
+assert 4 // x == 2
+x = 7
+x //= 2
+assert x == 3
+
+x = 3
+assert x % 2 == 1
+x %= 2
+assert x == 1
+
+assert x << 2 == 4
+x <<= 3
+assert x == 8
+
+assert x >> 1 == 4
+x >>= 1
+assert x == 4

test/simple_source/expression/15_mixed_expressions.py

@@ -0,0 +1,55 @@
+# Covers a large number of operators
+#
+# This code is RUNNABLE!
+from __future__ import division  # needed on 2.2 .. 2.7
+
+import sys
+PYTHON_VERSION = sys.version_info[0] + (sys.version_info[1] / 10.0)
+
+# some floats (from 01_float.py)
+
+x = 1e300
+assert 0.0 == x * 0
+assert x * 1e300 == float("inf")
+if PYTHON_VERSION > 2.41:
+    assert str(float("inf") * 0.0) == "nan"
+else:
+    assert str(float("inf") * 0.0) == "-nan"
+assert -1e300 * 1e300 == float("-inf")
+
+# Complex (adapted from 02_complex.py)
+y = 5j
+assert y ** 2 == -25
+y **= 3
+assert y == (-0-125j)
+
+
+# Tests BINARY_TRUE_DIVIDE and INPLACE_TRUE_DIVIDE (from 02_try_divide.py)
+x = 2
+assert 4 / x == 2
+
+if PYTHON_VERSION >= 2.19:
+    x = 5
+    assert x / 2 == 2.5
+    x = 3
+    x /= 2
+    assert x == 1.5
+
+x = 2
+assert 4 // x == 2
+x = 7
+x //= 2
+assert x == 3
+
+x = 3
+assert x % 2 == 1
+x %= 2
+assert x == 1
+
+assert x << 2 == 4
+x <<= 3
+assert x == 8
+
+assert x >> 1 == 4
+x >>= 1
+assert x == 4

test/stdlib/runtests.sh

@@ -34,10 +34,17 @@ case $PYVERSION in
 	SKIP_TESTS=(
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_grp.py]=1      # Long test - might work Control flow?
-	    [test_pwd.py]=1 # Long test - might work? Control flow?
 	    [test_pep247.py]=1 # Long test - might work? Control flow?
+	    [test_pwd.py]=1 # Long test - might work? Control flow?
+	    [test_pyclbr.py]=1 # Investigate
+	    [test_pyexpat.py]=1 # Investigate
 	    [test_queue.py]=1 # Control flow?
-	    # [test_threading.py]=1 # Long test - works
+	    [test_re.py]=1 # try confused with try-else again
+	    [test_socketserver.py]=1 # -- test takes too long to run: 40 seconds
+	    [test_threading.py]=1 # Line numbers are expected to be different
+	    [test_thread.py]=1 # test takes too long to run: 36 seconds
+	    [test_trace.py]=1 # Long test - works
+	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
 	)
 	;;
     2.5)
@@ -47,53 +54,35 @@ case $PYVERSION in
 	    [test_grammar.py]=1    # Too many stmts. Handle large stmts
 	    [test_grp.py]=1        # Long test - might work Control flow?
 	    [test_pdb.py]=1        # Line-number specific
+	    [test_pep247.py]=1     # "assert xxx or .." not detected properly in check_hash_module()
+	    [test_pep352.py]=1     # Investigate
 	    [test_pwd.py]=1 # Long test - might work? Control flow?
+	    [test_pyclbr.py]=1 # Investigate
 	    [test_queue.py]=1 # Control flow?
-	    [test_re.py]=1 # Probably Control flow?
+	    [test_re.py]=1 # Possibly try confused with try-else again
+	    [test_struct.py]=1 # "if and" confused for if .. assert and
+	    [test_sys.py]=1 # try confused with try-else again; in test_current_frames()
+	    [test_tarfile.py]=1  # try confused with try-else again; top-level import
+	    [test_threading.py]=1 # Line numbers are expected to be different
+	    [test_thread.py]=1 # test takes too long to run: 36 seconds
 	    [test_trace.py]=1  # Line numbers are expected to be different
 	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
 	)
 	;;
     2.6)
 	SKIP_TESTS=(
-	    [test_aepack.py]=1
-	    [test_aifc.py]=1
-	    [test_array.py]=1
-	    [test_audioop.py]=1
-	    [test_base64.py]=1
-	    [test_bigmem.py]=1
-	    [test_binascii.py]=1
-	    [test_builtin.py]=1
-	    [test_bytes.py]=1
-	    [test_class.py]=1
-	    [test_codeccallbacks.py]=1
-	    [test_codecencodings_cn.py]=1
-	    [test_codecencodings_hk.py]=1
-	    [test_codecencodings_jp.py]=1
-	    [test_codecencodings_kr.py]=1
-	    [test_codecencodings_tw.py]=1
-	    [test_codecencodings_cn.py]=1
-	    [test_codecmaps_hk.py]=1
-	    [test_codecmaps_jp.py]=1
-	    [test_codecmaps_kr.py]=1
-	    [test_codecmaps_tw.py]=1
-	    [test_codecs.py]=1
+	    [test_aepack.py]=1 # Fails on its own
+	    [test_codeccallbacks.py]=1  # Fails on its own
 	    [test_compile.py]=1  # Intermittent - sometimes works and sometimes doesn't
-	    [test_cookielib.py]=1
-	    [test_copy.py]=1
-	    [test_decimal.py]=1
-	    [test_descr.py]=1    # Problem in pickle.py?
 	    [test_exceptions.py]=1
-	    [test_extcall.py]=1
-	    [test_float.py]=1
-	    [test_future4.py]=1
-	    [test_generators.py]=1
+	    [test_generators.py]=1 # Investigate
 	    [test_grp.py]=1      # Long test - might work Control flow?
-	    [test_opcodes.py]=1
+	    [test_pep352.py]=1     # Investigate
+	    [test_pprint.py]=1
+	    [test_pyclbr.py]=1 # Investigate
 	    [test_pwd.py]=1 # Long test - might work? Control flow?
-	    [test_re.py]=1 # Probably Control flow?
-	    [test_queue.py]=1 # Control flow?
 	    [test_trace.py]=1  # Line numbers are expected to be different
+	    [test_urllib2net.py]=1 # Fails on its own. May need interactive input
 	    [test_zipfile64.py]=1  # Skip Long test
 	    [test_zlib.py]=1  # Takes too long to run (more than 3 minutes 39 seconds)
 	    # .pyenv/versions/2.6.9/lib/python2.6/lib2to3/refactor.pyc
@@ -105,8 +94,6 @@ case $PYVERSION in
 	    # .pyenv/versions/2.6.9/lib/python2.6/tabnanny.pyc
 	    # .pyenv/versions/2.6.9/lib/python2.6/tarfile.pyc
 
-	    # Not getting set by bach below?
-	    [test_pprint.py]=1
 
 	)
 	if (( batch )) ; then
@@ -133,41 +120,36 @@ case $PYVERSION in
 
 	    [test_capi.py]=1
 	    [test_curses.py]=1  # Possibly fails on its own but not detected
+	    [test test_cmd_line.py]=1 # Takes too long, maybe hangs, or looking for interactive input?
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_doctest.py]=1 # Fails on its own
 	    [test_exceptions.py]=1
 	    [test_format.py]=1  # control flow. uncompyle2 does not have problems here
 	    [test_generators.py]=1  # control flow. uncompyle2 has problem here too
 	    [test_grammar.py]=1     # Too many stmts. Handle large stmts
+	    [test_grp.py]=1     # test takes to long, works interactively though
+	    [test_hashlib.py]=1 # Investigate
 	    [test_io.py]=1 # Test takes too long to run
 	    [test_ioctl.py]=1 # Test takes too long to run
-	    [test_itertools.py]=1 # Fix erroneous reduction to "conditional_true".
-	                          # See test/simple_source/bug27+/05_not_unconditional.py
 	    [test_long.py]=1
 	    [test_long_future.py]=1
 	    [test_math.py]=1
 	    [test_memoryio.py]=1 # FIX
-	    [test_multiprocessing.py]=1 # On uncompyle2, taks 24 secs
-	    [test_pep352.py]=1  # ?
-	    [test_posix.py]=1   # Bug in try-else detection inside test_initgroups()
-	                        # Deal with when we have better flow-control detection
+	    [test_modulefinder.py]=1 # FIX
+	    [test_multiprocessing.py]=1 # On uncompyle2, takes 24 secs
 	    [test_pwd.py]=1     # Takes too long
 	    [test_pty.py]=1
-	    [test_queue.py]=1   # Control flow?
-	    [test_re.py]=1      # Probably Control flow?
 	    [test_runpy.py]=1   # Long and fails on its own
 	    [test_select.py]=1  # Runs okay but takes 11 seconds
 	    [test_socket.py]=1  # Runs ok but takes 22 seconds
 	    [test_subprocess.py]=1 # Runs ok but takes 22 seconds
 	    [test_sys_setprofile.py]=1
 	    [test_sys_settrace.py]=1 # Line numbers are expected to be different
-	    [test_strtod.py]=1 # FIX
 	    [test_traceback.py]=1 # Line numbers change - duh.
-	    [test_types.py]=1     # try/else confusions
 	    [test_unicode.py]=1  # Too long to run 11 seconds
 	    [test_xpickle.py]=1 # Runs ok but takes 72 seconds
 	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
-	    [test_zipimport.py]=1  # We can't distinguish try from try/else yet
+	    [test_zipimport.py]=1  # FIXME: improper try from try/else ?
         )
 	if (( batch )) ; then
 	    # Fails in crontab environment?

test/test_pyenvlib.py

@@ -80,6 +80,9 @@ for vers in TEST_VERSIONS:
     else:
         if vers == "native":
             short_vers = os.path.basename(sys.path[-1])
+            from xdis import PYTHON_VERSION
+            if PYTHON_VERSION > 3.0:
+                PYC = "*.cpython-%d.pyc" % int(PYTHON_VERSION * 10)
             test_options[vers] = (sys.path[-1], PYC, short_vers)
         else:
             short_vers = vers[:3]

uncompyle6/parsers/parse2.py

@@ -638,6 +638,9 @@ class Python2Parser(PythonParser):
             pass
 
         self.check_reduce["raise_stmt1"] = "tokens"
+        self.check_reduce["assert_expr_and"] = "AST"
+        self.check_reduce["tryelsestmt"] = "AST"
+        self.check_reduce["tryelsestmtl"] = "AST"
         self.check_reduce["aug_assign2"] = "AST"
         self.check_reduce["or"] = "AST"
         # self.check_reduce['_stmts'] = 'AST'
@@ -660,6 +663,10 @@ class Python2Parser(PythonParser):
             and ast[0][0] in ("and", "or")
         ):
             return True
+        elif lhs == "assert_expr_and":
+            jmp_false = ast[1]
+            jump_target = jmp_false[0].attr
+            return jump_target > tokens[last].off2int()
         elif lhs in ("raise_stmt1",):
             # We will assume 'LOAD_ASSERT' will be handled by an assert grammar rule
             return tokens[first] == "LOAD_ASSERT" and (last >= len(tokens))
@@ -669,6 +676,33 @@ class Python2Parser(PythonParser):
         elif lhs in ("delete_subscript", "del_expr"):
             op = ast[0][0]
             return op.kind in ("and", "or")
+        elif lhs in ("tryelsestmt", "tryelsestmtl"):
+            # Check the end of the except handler that there isn't a jump from
+            # inside the except handler to the end. If that happens
+            # then this is a "try" with no "else".
+            except_handler = ast[3]
+            if except_handler == "except_handler":
+
+                come_from = except_handler[-1]
+                # We only care about the *first* come_from because that is the
+                # the innermost one. So if the "tryelse" is invalid (should be a "try")
+                # ti will be invalid here.
+                if come_from == "COME_FROM":
+                    first_come_from = except_handler[-1]
+                else:
+                    assert come_from == "come_froms"
+                    first_come_from = come_from[0]
+                leading_jump = except_handler[0]
+
+                # We really don't care that this is a jump per-se. But
+                # we could also check that this jumps to the end of the except if
+                # desired.
+                if isinstance(leading_jump, SyntaxTree):
+                    except_handler_first_offset = leading_jump.first_child().off2int()
+                else:
+                    except_handler_first_offset = leading_jump.off2int()
+                return first_come_from.attr > except_handler_first_offset
+
 
         return False
 

uncompyle6/parsers/parse25.py

@@ -25,11 +25,17 @@ class Python25Parser(Python26Parser):
         setupwithas ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0
                         setup_finally
         # opcode SETUP_WITH
-        setupwith ::= DUP_TOP LOAD_ATTR STORE_NAME LOAD_ATTR CALL_FUNCTION_0 POP_TOP
-        withstmt ::= expr setupwith SETUP_FINALLY suite_stmts_opt
-                     POP_BLOCK LOAD_CONST COME_FROM with_cleanup
+        setupwith ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 POP_TOP
+        withstmt  ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+                      POP_BLOCK LOAD_CONST COME_FROM with_cleanup
+
+        # Semantic actions want store to be at index 2
+        withasstmt ::= expr setupwithas store suite_stmts_opt
+                       POP_BLOCK LOAD_CONST COME_FROM with_cleanup
+
 
         store ::= STORE_NAME
+        store ::= STORE_FAST
 
         # tryelsetmtl doesn't need COME_FROM since the jump might not
         # be the the join point at the end of the "try" but instead back to the

uncompyle6/parsers/parse27.py

@@ -214,13 +214,17 @@ class Python27Parser(Python2Parser):
 
 
         super(Python27Parser, self).customize_grammar_rules(tokens, customize)
-        self.check_reduce['and'] = 'AST'
-        # self.check_reduce['or'] = 'AST'
-        self.check_reduce['raise_stmt1'] = 'AST'
-        self.check_reduce['list_if_not'] = 'AST'
-        self.check_reduce['list_if'] = 'AST'
-        self.check_reduce['if_expr_true'] = 'tokens'
-        self.check_reduce['whilestmt'] = 'tokens'
+        self.check_reduce["and"] = "AST"
+        self.check_reduce["conditional"] = "AST"
+        # self.check_reduce["or"] = "AST"
+        self.check_reduce["raise_stmt1"] = "AST"
+        self.check_reduce["iflaststmtl"] = "AST"
+        self.check_reduce["list_if_not"] = "AST"
+        self.check_reduce["list_if"] = "AST"
+        self.check_reduce["comp_if"] = "AST"
+        self.check_reduce["if_expr_true"] = "tokens"
+        self.check_reduce["whilestmt"] = "tokens"
+
         return
 
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
@@ -231,54 +235,99 @@ class Python27Parser(Python2Parser):
         if invalid:
             return invalid
 
-        if rule == ('and', ('expr', 'jmp_false', 'expr', '\\e_come_from_opt')):
-            # If the instruction after the instructions formin "and"  is an "YIELD_VALUE"
+        if rule == ("and", ("expr", "jmp_false", "expr", "\\e_come_from_opt")):
+            # If the instruction after the instructions forming the "and"  is an "YIELD_VALUE"
             # then this is probably an "if" inside a comprehension.
-            if tokens[last] == 'YIELD_VALUE':
+            if tokens[last] == "YIELD_VALUE":
                 # Note: We might also consider testing last+1 being "POP_TOP"
                 return True
 
+            # Test that jump_false jump somewhere beyond the end of the "and"
+            # it might not be exactly the end of the "and" because this and can
+            # be a part of a larger condition. Oddly in 2.7 there doesn't seem to be
+            # an optimization where the "and" jump_false is back to a loop.
+            jmp_false = ast[1]
+            if jmp_false[0] == "POP_JUMP_IF_FALSE":
+                while (first < last and isinstance(tokens[last].offset, str)):
+                    last -= 1
+                if jmp_false[0].attr < tokens[last].offset:
+                    return True
+
             # Test that jmp_false jumps to the end of "and"
             # or that it jumps to the same place as the end of "and"
             jmp_false = ast[1][0]
             jmp_target = jmp_false.offset + jmp_false.attr + 3
             return not (jmp_target == tokens[last].offset or
                         tokens[last].pattr == jmp_false.pattr)
-        elif rule[0] == ('raise_stmt1'):
-            return ast[0] == 'expr' and ast[0][0] == 'or'
-        elif rule[0] in ('assert', 'assert2'):
+        elif rule == ("comp_if", ("expr", "jmp_false", "comp_iter")):
+            jmp_false = ast[1]
+            if jmp_false[0] == "POP_JUMP_IF_FALSE":
+                return tokens[first].offset < jmp_false[0].attr < tokens[last].offset
+            pass
+        elif (rule[0], rule[1][0:5]) == (
+                "conditional",
+                ("expr", "jmp_false", "expr", "JUMP_ABSOLUTE", "expr")):
+            jmp_false = ast[1]
+            if jmp_false[0] == "POP_JUMP_IF_FALSE":
+                else_instr = ast[4].first_child()
+                if jmp_false[0].attr != else_instr.offset:
+                    return True
+                end_offset = ast[3].attr
+                return end_offset < tokens[last].offset
+            pass
+        elif rule[0] == ("raise_stmt1"):
+            return ast[0] == "expr" and ast[0][0] == "or"
+        elif rule[0] in ("assert", "assert2"):
             jump_inst = ast[1][0]
             jump_target = jump_inst.attr
             return not (last >= len(tokens)
                         or jump_target == tokens[last].offset
                         or jump_target == next_offset(ast[-1].op, ast[-1].opc, ast[-1].offset))
-        elif rule == ('list_if_not', ('expr', 'jmp_true', 'list_iter')):
+        elif rule == ("iflaststmtl", ("testexpr", "c_stmts")):
+            testexpr = ast[0]
+            if testexpr[0] == "testfalse":
+                testfalse = testexpr[0]
+                if testfalse[1] == "jmp_false":
+                    jmp_false = testfalse[1]
+                    if last == len(tokens):
+                        last -= 1
+                    while (isinstance(tokens[first].offset, str) and first < last):
+                        first += 1
+                    if first == last:
+                        return True
+                    while (first < last and isinstance(tokens[last].offset, str)):
+                        last -= 1
+                    return tokens[first].offset < jmp_false[0].attr < tokens[last].offset
+                    pass
+                pass
+            pass
+        elif rule == ("list_if_not", ("expr", "jmp_true", "list_iter")):
             jump_inst = ast[1][0]
             jump_offset = jump_inst.attr
             return jump_offset > jump_inst.offset and jump_offset < tokens[last].offset
-        elif rule == ('list_if', ('expr', 'jmp_false', 'list_iter')):
+        elif rule == ("list_if", ("expr", "jmp_false", "list_iter")):
             jump_inst = ast[1][0]
             jump_offset = jump_inst.attr
             return jump_offset > jump_inst.offset and jump_offset < tokens[last].offset
-        elif rule == ('or', ('expr', 'jmp_true', 'expr', '\\e_come_from_opt')):
-            # Test that jmp_true doesn't jump inside the middle the "or"
+        elif rule == ("or", ("expr", "jmp_true", "expr", "\\e_come_from_opt")):
+            # Test that jmp_true doesn"t jump inside the middle the "or"
             # or that it jumps to the same place as the end of "and"
             jmp_true = ast[1][0]
             jmp_target = jmp_true.offset + jmp_true.attr + 3
             return not (jmp_target == tokens[last].offset or
                         tokens[last].pattr == jmp_true.pattr)
 
-        elif (rule[0] == 'whilestmt' and
+        elif (rule[0] == "whilestmt" and
               rule[1][0:-2] ==
-                      ('SETUP_LOOP', 'testexpr', 'l_stmts_opt',
-                       'JUMP_BACK', 'JUMP_BACK')):
+                      ("SETUP_LOOP", "testexpr", "l_stmts_opt",
+                       "JUMP_BACK", "JUMP_BACK")):
             # Make sure that the jump backs all go to the same place
             i = last-1
-            while (tokens[i] != 'JUMP_BACK'):
+            while (tokens[i] != "JUMP_BACK"):
                 i -= 1
             return tokens[i].attr != tokens[i-1].attr
-        elif rule[0] == 'if_expr_true':
-            return (first) > 0 and tokens[first-1] == 'POP_JUMP_IF_FALSE'
+        elif rule[0] == "if_expr_true":
+            return (first) > 0 and tokens[first-1] == "POP_JUMP_IF_FALSE"
 
         return False
 
@@ -286,7 +335,7 @@ class Python27Parser(Python2Parser):
 class Python27ParserSingle(Python27Parser, PythonParserSingle):
     pass
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     # Check grammar
     p = Python27Parser()
     p.check_grammar()
@@ -302,9 +351,9 @@ if __name__ == '__main__':
             """.split()))
         remain_tokens = set(tokens) - opcode_set
         import re
-        remain_tokens = set([re.sub(r'_\d+$', '', t)
+        remain_tokens = set([re.sub(r"_\d+$", "", t)
                              for t in remain_tokens])
-        remain_tokens = set([re.sub('_CONT$', '', t)
+        remain_tokens = set([re.sub("_CONT$", "", t)
                              for t in remain_tokens])
         remain_tokens = set(remain_tokens) - opcode_set
         print(remain_tokens)

uncompyle6/parsers/parse30.py

@@ -213,6 +213,21 @@ class Python30Parser(Python31Parser):
         whilestmt      ::= SETUP_LOOP testexpr returns POP_TOP POP_BLOCK COME_FROM_LOOP
         withasstmt     ::= expr SETUP_WITH store suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
         withstmt       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
+
+        # lc_body ::= LOAD_FAST expr LIST_APPEND
+        # lc_body ::= LOAD_NAME expr LIST_APPEND
+        # lc_body ::= expr LIST_APPEND
+        # list_comp ::= BUILD_LIST_0 list_iter
+        # list_for ::= expr FOR_ITER store list_iter jb_or_c
+        # list_if ::= expr jmp_false list_iter
+        # list_if ::= expr jmp_false_then list_iter
+        # list_if_not ::= expr jmp_true list_iter
+        # list_iter ::= list_if JUMP_BACK
+        # list_iter ::= list_if JUMP_BACK _come_froms POP_TOP
+        # list_iter ::= list_if_not
+        # load_closure ::= BUILD_TUPLE_0
+        # load_genexpr ::= BUILD_TUPLE_1 LOAD_GENEXPR LOAD_STR
+
         ##########################################################################################
 
         iflaststmtl        ::= testexpr c_stmts_opt JUMP_BACK COME_FROM_LOOP

uncompyle6/parsers/parse31.py

@@ -36,6 +36,10 @@ class Python31Parser(Python32Parser):
         self.remove_rules("""
         # DUP_TOP_TWO is DUP_TOPX in 3.1 and earlier
         subscript2 ::= expr expr DUP_TOP_TWO BINARY_SUBSCR
+
+        # The were found using grammar coverage
+        list_if     ::= expr jmp_false list_iter COME_FROM
+        list_if_not ::= expr jmp_true list_iter COME_FROM
         """)
 
     def customize_grammar_rules(self, tokens, customize):

uncompyle6/parsers/parse38.py

@@ -22,6 +22,12 @@ from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.parsers.parse37 import Python37Parser
 
 class Python38Parser(Python37Parser):
+    def p_38walrus(self, args):
+        """
+        # named_expr is also known as the "walrus op" :=
+        expr              ::= named_expr
+        named_expr        ::= expr DUP_TOP store
+        """
 
     def p_38misc(self, args):
         """
@@ -38,6 +44,12 @@ class Python38Parser(Python37Parser):
         stmt               ::= whilestmt38
         stmt               ::= whileTruestmt38
         stmt               ::= call
+        stmt               ::= ifstmtl
+
+        break ::= POP_BLOCK BREAK_LOOP
+        break ::= POP_BLOCK POP_TOP BREAK_LOOP
+        break ::= POP_TOP BREAK_LOOP
+        break ::= POP_EXCEPT BREAK_LOOP
 
         # FIXME: this should be restricted to being inside a try block
         stmt               ::= except_ret38
@@ -89,27 +101,39 @@ class Python38Parser(Python37Parser):
 
         return             ::= ret_expr ROT_TWO POP_TOP RETURN_VALUE
 
+        # 3.8 can push a looping JUMP_BACK into into a JUMP_ from a statement that jumps to it
+        lastl_stmt         ::= ifpoplaststmtl
+        ifpoplaststmtl     ::= testexpr POP_TOP c_stmts_opt JUMP_BACK
+        ifelsestmtl        ::= testexpr c_stmts_opt jb_cfs else_suitel JUMP_BACK come_froms
+
+        _ifstmts_jumpl     ::= c_stmts JUMP_BACK
+        _ifstmts_jumpl     ::= _ifstmts_jump
+        ifstmtl            ::= testexpr _ifstmts_jumpl
+
         for38              ::= expr get_iter store for_block JUMP_BACK
-        for38              ::= expr for_iter store for_block JUMP_BACK
-        for38              ::= expr for_iter store for_block JUMP_BACK POP_BLOCK
-        for38              ::= expr for_iter store for_block
+        for38              ::= expr get_for_iter store for_block JUMP_BACK
+        for38              ::= expr get_for_iter store for_block JUMP_BACK POP_BLOCK
+        for38              ::= expr get_for_iter store for_block
 
-        forelsestmt38      ::= expr for_iter store for_block POP_BLOCK else_suite
-        forelselaststmt38  ::= expr for_iter store for_block POP_BLOCK else_suitec
-        forelselaststmtl38 ::= expr for_iter store for_block POP_BLOCK else_suitel
+        forelsestmt38      ::= expr get_for_iter store for_block POP_BLOCK else_suite
+        forelselaststmt38  ::= expr get_for_iter store for_block POP_BLOCK else_suitec
+        forelselaststmtl38 ::= expr get_for_iter store for_block POP_BLOCK else_suitel
 
-        whilestmt38        ::= testexpr l_stmts_opt COME_FROM JUMP_BACK POP_BLOCK
-        whilestmt38        ::= testexpr l_stmts_opt JUMP_BACK POP_BLOCK
-        whilestmt38        ::= testexpr returns               POP_BLOCK
-        whilestmt38        ::= testexpr l_stmts     JUMP_BACK
+        whilestmt38        ::= _come_froms testexpr l_stmts_opt COME_FROM JUMP_BACK POP_BLOCK
+        whilestmt38        ::= _come_froms testexpr l_stmts_opt JUMP_BACK POP_BLOCK
+        whilestmt38        ::= _come_froms testexpr l_stmts_opt JUMP_BACK come_froms
+        whilestmt38        ::= _come_froms testexpr returns               POP_BLOCK
+        whilestmt38        ::= _come_froms testexpr l_stmts     JUMP_BACK
+        whilestmt38        ::= _come_froms testexpr l_stmts     come_froms
 
         # while1elsestmt   ::=          l_stmts     JUMP_BACK
-        whileTruestmt      ::= l_stmts              JUMP_BACK POP_BLOCK
-        while1stmt         ::= l_stmts COME_FROM_LOOP
-        while1stmt         ::= l_stmts COME_FROM JUMP_BACK COME_FROM_LOOP
-        whileTruestmt38    ::= l_stmts JUMP_BACK
+        whileTruestmt      ::= _come_froms l_stmts              JUMP_BACK POP_BLOCK
+        while1stmt         ::= _come_froms l_stmts COME_FROM_LOOP
+        while1stmt         ::= _come_froms l_stmts COME_FROM JUMP_BACK COME_FROM_LOOP
+        whileTruestmt38    ::= _come_froms l_stmts JUMP_BACK
+        whileTruestmt38    ::= _come_froms l_stmts JUMP_BACK COME_FROM_EXCEPT_CLAUSE
 
-        for_block          ::= l_stmts_opt _come_from_loops JUMP_BACK
+        for_block          ::= _come_froms l_stmts_opt _come_from_loops JUMP_BACK
 
         except_cond1       ::= DUP_TOP expr COMPARE_OP jmp_false
                                POP_TOP POP_TOP POP_TOP
@@ -134,7 +158,8 @@ class Python38Parser(Python37Parser):
         except_ret38a      ::= COME_FROM_FINALLY POP_TOP POP_TOP POP_TOP
                                expr ROT_FOUR
                                POP_EXCEPT RETURN_VALUE END_FINALLY
-        except_handler38   ::= JUMP_FORWARD COME_FROM_FINALLY
+
+        except_handler38   ::= _jump COME_FROM_FINALLY
                                except_stmts END_FINALLY opt_come_from_except
         except_handler38a  ::= COME_FROM_FINALLY POP_TOP POP_TOP POP_TOP
                                POP_EXCEPT POP_TOP stmts END_FINALLY
@@ -160,13 +185,15 @@ class Python38Parser(Python37Parser):
         self.customized = {}
 
     def remove_rules_38(self):
-        self.remove_rules("""
+        self.remove_rules(
+            """
            stmt               ::= async_for_stmt37
            stmt               ::= for
            stmt               ::= forelsestmt
            stmt               ::= try_except36
+           stmt               ::= async_forelse_stmt
 
-           async_for_stmt     ::= SETUP_LOOP expr
+           async_for_stmt     ::= setup_loop expr
                                   GET_AITER
                                   SETUP_EXCEPT GET_ANEXT LOAD_CONST
                                   YIELD_FROM
@@ -178,7 +205,7 @@ class Python38Parser(Python37Parser):
                                   COME_FROM
                                   POP_TOP POP_TOP POP_TOP POP_EXCEPT POP_TOP POP_BLOCK
                                   COME_FROM_LOOP
-           async_for_stmt37   ::= SETUP_LOOP expr
+           async_for_stmt37   ::= setup_loop expr
                                   GET_AITER
                                   SETUP_EXCEPT GET_ANEXT
                                   LOAD_CONST YIELD_FROM
@@ -190,7 +217,7 @@ class Python38Parser(Python37Parser):
                                   POP_TOP POP_BLOCK
                                   COME_FROM_LOOP
 
-          async_forelse_stmt ::= SETUP_LOOP expr
+          async_forelse_stmt ::= setup_loop expr
                                  GET_AITER
                                  SETUP_EXCEPT GET_ANEXT LOAD_CONST
                                  YIELD_FROM
@@ -203,13 +230,13 @@ class Python38Parser(Python37Parser):
                                  POP_TOP POP_TOP POP_TOP POP_EXCEPT POP_TOP POP_BLOCK
                                  else_suite COME_FROM_LOOP
 
-           for                ::= SETUP_LOOP expr for_iter store for_block POP_BLOCK
-           for                ::= SETUP_LOOP expr for_iter store for_block POP_BLOCK NOP
+           for                ::= setup_loop expr get_for_iter store for_block POP_BLOCK
+           for                ::= setup_loop expr get_for_iter store for_block POP_BLOCK NOP
 
            for_block          ::= l_stmts_opt COME_FROM_LOOP JUMP_BACK
-           forelsestmt        ::= SETUP_LOOP expr for_iter store for_block POP_BLOCK else_suite
-           forelselaststmt    ::= SETUP_LOOP expr for_iter store for_block POP_BLOCK else_suitec
-           forelselaststmtl   ::= SETUP_LOOP expr for_iter store for_block POP_BLOCK else_suitel
+           forelsestmt        ::= setup_loop expr get_for_iter store for_block POP_BLOCK else_suite
+           forelselaststmt    ::= setup_loop expr get_for_iter store for_block POP_BLOCK else_suitec
+           forelselaststmtl   ::= setup_loop expr get_for_iter store for_block POP_BLOCK else_suitel
 
            tryelsestmtl3      ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
                                   except_handler COME_FROM else_suitel
@@ -223,15 +250,16 @@ class Python38Parser(Python37Parser):
                                   COME_FROM_FINALLY suite_stmts_opt END_FINALLY
            tryfinally_return_stmt ::= SETUP_FINALLY suite_stmts_opt POP_BLOCK
                                       LOAD_CONST COME_FROM_FINALLY
-
-
-        """)
+        """
+        )
 
     def customize_grammar_rules(self, tokens, customize):
         super(Python37Parser, self).customize_grammar_rules(tokens, customize)
         self.remove_rules_38()
-        self.check_reduce['ifstmt'] = 'tokens'
-        self.check_reduce['whileTruestmt38'] = 'tokens'
+        self.check_reduce["ifstmt"] = "tokens"
+        self.check_reduce["whileTruestmt38"] = "tokens"
+        self.check_reduce["whilestmt38"] = "tokens"
+        self.check_reduce["ifstmtl"] = "tokens"
 
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
         invalid = super(Python38Parser,
@@ -240,34 +268,47 @@ class Python38Parser(Python37Parser):
         self.remove_rules_38()
         if invalid:
             return invalid
-        if rule[0] == 'ifstmt':
+        lhs = rule[0]
+        if lhs == "ifstmt":
             # Make sure jumps don't extend beyond the end of the if statement.
             l = last
             if l == len(tokens):
                 l -= 1
             if isinstance(tokens[l].offset, str):
-                last_offset = int(tokens[l].offset.split('_')[0], 10)
+                last_offset = int(tokens[l].offset.split("_")[0], 10)
             else:
                 last_offset = tokens[l].offset
             for i in range(first, l):
                 t = tokens[i]
-                if t.kind == 'POP_JUMP_IF_FALSE':
+                if t.kind == "POP_JUMP_IF_FALSE":
                     if t.attr > last_offset:
                         return True
                     pass
                 pass
             pass
-        elif rule[0] == 'whileTruestmt38':
-            t = tokens[last-1]
-            if t.kind == 'JUMP_BACK':
-                return t.attr != tokens[first].offset
+        elif lhs == "ifstmtl":
+            if last == len(tokens):
+                last -= 1
+            if (tokens[last].attr and isinstance(tokens[last].attr, int)):
+                return tokens[first].offset < tokens[last].attr
+            pass
+        elif lhs in ("whileTruestmt38", "whilestmt38"):
+            jb_index = last - 1
+            while jb_index > 0 and tokens[jb_index].kind.startswith("COME_FROM"):
+                jb_index -= 1
+            t = tokens[jb_index]
+            if t.kind != "JUMP_BACK":
+                return True
+            return t.attr != tokens[first].off2int()
             pass
 
         return False
 
+
 class Python38ParserSingle(Python38Parser, PythonParserSingle):
     pass
 
+
 if __name__ == "__main__":
     # Check grammar
     # FIXME: DRY this with other parseXX.py routines
@@ -284,9 +325,11 @@ if __name__ == "__main__":
         opcode_set = set(s.opc.opname).union(
             set(
                 """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
-                LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_CLASSNAME
-                LAMBDA_MARKER RETURN_LAST
-            """.split()))
+               LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_CLASSNAME
+               LAMBDA_MARKER RETURN_LAST
+            """.split()
+            )
+        )
         remain_tokens = set(tokens) - opcode_set
         import re
 

uncompyle6/scanner.py

@@ -60,6 +60,7 @@ PYTHON_VERSIONS = frozenset(
         3.6,
         3.7,
         3.8,
+        3.9,
     )
 )
 

uncompyle6/scanners/scanner3.py

@@ -62,7 +62,8 @@ class Scanner3(Scanner):
         # Create opcode classification sets
         # Note: super initilization above initializes self.opc
 
-        # Ops that start SETUP_ ... We will COME_FROM with these names
+        # For ops that start SETUP_ ... we will add COME_FROM with these names
+        # at the their targets.
         # Some blocks and END_ statements. And they can start
         # a new statement
         if self.version < 3.8:

uncompyle6/scanners/scanner37.py

@@ -22,24 +22,50 @@ This sets up opcodes Python's 3.7 and calls a generalized
 scanner routine for Python 3.
 """
 
-from uncompyle6.scanners.scanner36 import Scanner36
-from uncompyle6.scanners.scanner3 import Scanner3
+from uncompyle6.scanners.scanner37base import Scanner37Base
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_37 as opc
+
+# bytecode verification, verify(), uses JUMP_OPS from here
 JUMP_OPs = opc.JUMP_OPS
 
-class Scanner37(Scanner36):
 
+class Scanner37(Scanner37Base):
     def __init__(self, show_asm=None):
-        Scanner3.__init__(self, 3.7, show_asm)
+        Scanner37Base.__init__(self, 3.7, show_asm)
         return
+
     pass
 
+    def ingest(self, co, classname=None, code_objects={}, show_asm=None):
+        tokens, customize = Scanner37Base.ingest(self, co, classname, code_objects, show_asm)
+        for t in tokens:
+            # The lowest bit of flags indicates whether the
+            # var-keyword argument is placed at the top of the stack
+            if t.op == self.opc.CALL_FUNCTION_EX and t.attr & 1:
+                t.kind = "CALL_FUNCTION_EX_KW"
+                pass
+            elif t.op == self.opc.BUILD_STRING:
+                t.kind = "BUILD_STRING_%s" % t.attr
+            elif t.op == self.opc.CALL_FUNCTION_KW:
+                t.kind = "CALL_FUNCTION_KW_%s" % t.attr
+            elif t.op == self.opc.FORMAT_VALUE:
+                if t.attr & 0x4:
+                    t.kind = "FORMAT_VALUE_ATTR"
+                    pass
+            elif t.op == self.opc.BUILD_MAP_UNPACK_WITH_CALL:
+                t.kind = "BUILD_MAP_UNPACK_WITH_CALL_%d" % t.attr
+            elif t.op == self.opc.BUILD_TUPLE_UNPACK_WITH_CALL:
+                t.kind = "BUILD_TUPLE_UNPACK_WITH_CALL_%d" % t.attr
+            pass
+        return tokens, customize
+
 if __name__ == "__main__":
     from uncompyle6 import PYTHON_VERSION
     if PYTHON_VERSION == 3.7:
         import inspect
+
         co = inspect.currentframe().f_code
         tokens, customize = Scanner37().ingest(co)
         for t in tokens:

uncompyle6/scanners/scanner37base.py

@@ -0,0 +1,953 @@
+#  Copyright (c) 2015-2019 by Rocky Bernstein
+#  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
+#  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""
+Python 37 bytecode scanner/deparser base.
+
+Also we *modify* the instruction sequence to assist deparsing code.
+For example:
+ -  we add "COME_FROM" instructions to help in figuring out
+    conditional branching and looping.
+ -  LOAD_CONSTs are classified further into the type of thing
+    they load:
+      lambda's, genexpr's, {dict,set,list} comprehension's,
+ -  PARAMETER counts appended  {CALL,MAKE}_FUNCTION, BUILD_{TUPLE,SET,SLICE}
+
+Finally we save token information.
+"""
+
+from xdis.code import iscode
+from xdis.bytecode import instruction_size, _get_const_info
+
+from uncompyle6.scanner import Token
+import xdis
+
+# Get all the opcodes into globals
+import xdis.opcodes.opcode_37 as op3
+
+from uncompyle6.scanner import Scanner
+
+import sys
+
+globals().update(op3.opmap)
+
+
+class Scanner37Base(Scanner):
+    def __init__(self, version, show_asm=None, is_pypy=False):
+        super(Scanner37Base, self).__init__(version, show_asm, is_pypy)
+
+        # Create opcode classification sets
+        # Note: super initilization above initializes self.opc
+
+        # Ops that start SETUP_ ... We will COME_FROM with these names
+        # Some blocks and END_ statements. And they can start
+        # a new statement
+        if self.version < 3.8:
+            setup_ops = [
+                self.opc.SETUP_LOOP,
+                self.opc.SETUP_EXCEPT,
+                self.opc.SETUP_FINALLY,
+            ]
+            self.setup_ops_no_loop = frozenset(setup_ops) - frozenset(
+                [self.opc.SETUP_LOOP]
+            )
+        else:
+            setup_ops = [self.opc.SETUP_FINALLY]
+            self.setup_ops_no_loop = frozenset(setup_ops)
+
+            # Add back these opcodes which help us detect "break" and
+            # "continue" statements via parsing.
+            self.opc.BREAK_LOOP = 80
+            self.opc.CONTINUE_LOOP = 119
+            pass
+
+        setup_ops.append(self.opc.SETUP_WITH)
+        self.setup_ops = frozenset(setup_ops)
+
+        self.pop_jump_tf = frozenset([self.opc.PJIF, self.opc.PJIT])
+        self.not_continue_follow = ("END_FINALLY", "POP_BLOCK")
+
+        # Opcodes that can start a statement.
+        statement_opcodes = [
+            self.opc.POP_BLOCK,
+            self.opc.STORE_FAST,
+            self.opc.DELETE_FAST,
+            self.opc.STORE_DEREF,
+            self.opc.STORE_GLOBAL,
+            self.opc.DELETE_GLOBAL,
+            self.opc.STORE_NAME,
+            self.opc.DELETE_NAME,
+            self.opc.STORE_ATTR,
+            self.opc.DELETE_ATTR,
+            self.opc.STORE_SUBSCR,
+            self.opc.POP_TOP,
+            self.opc.DELETE_SUBSCR,
+            self.opc.END_FINALLY,
+            self.opc.RETURN_VALUE,
+            self.opc.RAISE_VARARGS,
+            self.opc.PRINT_EXPR,
+            self.opc.JUMP_ABSOLUTE,
+            # These are phony for 3.8+
+            self.opc.BREAK_LOOP,
+            self.opc.CONTINUE_LOOP,
+        ]
+
+        self.statement_opcodes = frozenset(statement_opcodes) | self.setup_ops_no_loop
+
+        # Opcodes that can start a "store" non-terminal.
+        # FIXME: JUMP_ABSOLUTE is weird. What's up with that?
+        self.designator_ops = frozenset(
+            [
+                self.opc.STORE_FAST,
+                self.opc.STORE_NAME,
+                self.opc.STORE_GLOBAL,
+                self.opc.STORE_DEREF,
+                self.opc.STORE_ATTR,
+                self.opc.STORE_SUBSCR,
+                self.opc.UNPACK_SEQUENCE,
+                self.opc.JUMP_ABSOLUTE,
+                self.opc.UNPACK_EX,
+            ]
+        )
+
+        self.jump_if_pop = frozenset(
+            [self.opc.JUMP_IF_FALSE_OR_POP, self.opc.JUMP_IF_TRUE_OR_POP]
+        )
+
+        self.pop_jump_if_pop = frozenset(
+            [
+                self.opc.JUMP_IF_FALSE_OR_POP,
+                self.opc.JUMP_IF_TRUE_OR_POP,
+                self.opc.POP_JUMP_IF_TRUE,
+                self.opc.POP_JUMP_IF_FALSE,
+            ]
+        )
+        # Not really a set, but still clasification-like
+        self.statement_opcode_sequences = [
+            (self.opc.POP_JUMP_IF_FALSE, self.opc.JUMP_FORWARD),
+            (self.opc.POP_JUMP_IF_FALSE, self.opc.JUMP_ABSOLUTE),
+            (self.opc.POP_JUMP_IF_TRUE, self.opc.JUMP_FORWARD),
+            (self.opc.POP_JUMP_IF_TRUE, self.opc.JUMP_ABSOLUTE),
+        ]
+
+        # FIXME: remove this and use instead info from xdis.
+        # Opcodes that take a variable number of arguments
+        # (expr's)
+        varargs_ops = set(
+            [
+                self.opc.BUILD_LIST,
+                self.opc.BUILD_TUPLE,
+                self.opc.BUILD_SET,
+                self.opc.BUILD_SLICE,
+                self.opc.BUILD_MAP,
+                self.opc.UNPACK_SEQUENCE,
+                self.opc.RAISE_VARARGS,
+            ]
+        )
+
+        varargs_ops.add(self.opc.CALL_METHOD)
+        varargs_ops |= set(
+            [
+                self.opc.BUILD_SET_UNPACK,
+                self.opc.BUILD_MAP_UNPACK,  # we will handle this later
+                self.opc.BUILD_LIST_UNPACK,
+                self.opc.BUILD_TUPLE_UNPACK,
+            ]
+        )
+        varargs_ops.add(self.opc.BUILD_CONST_KEY_MAP)
+        # Below is in bit order, "default = bit 0, closure = bit 3
+        self.MAKE_FUNCTION_FLAGS = tuple(
+            """
+            default keyword-only annotation closure""".split()
+        )
+
+        self.varargs_ops = frozenset(varargs_ops)
+        # FIXME: remove the above in favor of:
+        # self.varargs_ops = frozenset(self.opc.hasvargs)
+        return
+
+    def ingest(self, co, classname=None, code_objects={}, show_asm=None):
+        """
+        Pick out tokens from an uncompyle6 code object, and transform them,
+        returning a list of uncompyle6 Token's.
+
+        The transformations are made to assist the deparsing grammar.
+        Specificially:
+           -  various types of LOAD_CONST's are categorized in terms of what they load
+           -  COME_FROM instructions are added to assist parsing control structures
+           -  MAKE_FUNCTION and FUNCTION_CALLS append the number of positional arguments
+           -  some EXTENDED_ARGS instructions are removed
+
+        Also, when we encounter certain tokens, we add them to a set which will cause custom
+        grammar rules. Specifically, variable arg tokens like MAKE_FUNCTION or BUILD_LIST
+        cause specific rules for the specific number of arguments they take.
+        """
+
+        def tokens_append(j, token):
+            tokens.append(token)
+            self.offset2tok_index[token.offset] = j
+            j += 1
+            assert j == len(tokens)
+            return j
+
+        if not show_asm:
+            show_asm = self.show_asm
+
+        bytecode = self.build_instructions(co)
+
+        # show_asm = 'both'
+        if show_asm in ("both", "before"):
+            for instr in bytecode.get_instructions(co):
+                print(instr.disassemble())
+
+        # "customize" is in the process of going away here
+        customize = {}
+
+        if self.is_pypy:
+            customize["PyPy"] = 0
+
+        # Scan for assertions. Later we will
+        # turn 'LOAD_GLOBAL' to 'LOAD_ASSERT'.
+        # 'LOAD_ASSERT' is used in assert statements.
+        self.load_asserts = set()
+
+        # list of tokens/instructions
+        tokens = []
+        self.offset2tok_index = {}
+
+        n = len(self.insts)
+        for i, inst in enumerate(self.insts):
+
+            # We need to detect the difference between:
+            #   raise AssertionError
+            #  and
+            #   assert ...
+            # If we have a JUMP_FORWARD after the
+            # RAISE_VARARGS then we have a "raise" statement
+            # else we have an "assert" statement.
+            assert_can_follow = inst.opname == "POP_JUMP_IF_TRUE" and i + 1 < n
+            if assert_can_follow:
+                next_inst = self.insts[i + 1]
+                if (
+                    next_inst.opname == "LOAD_GLOBAL"
+                    and next_inst.argval == "AssertionError"
+                    and inst.argval
+                ):
+                    raise_idx = self.offset2inst_index[self.prev_op[inst.argval]]
+                    raise_inst = self.insts[raise_idx]
+                    if raise_inst.opname.startswith("RAISE_VARARGS"):
+                        self.load_asserts.add(next_inst.offset)
+                    pass
+                pass
+
+        # Get jump targets
+        # Format: {target offset: [jump offsets]}
+        jump_targets = self.find_jump_targets(show_asm)
+        # print("XXX2", jump_targets)
+
+        last_op_was_break = False
+
+        j = 0
+        for i, inst in enumerate(self.insts):
+
+            argval = inst.argval
+            op = inst.opcode
+
+            if inst.opname == "EXTENDED_ARG":
+                # FIXME: The EXTENDED_ARG is used to signal annotation
+                # parameters
+                if i + 1 < n and self.insts[i + 1].opcode != self.opc.MAKE_FUNCTION:
+                    continue
+
+            if inst.offset in jump_targets:
+                jump_idx = 0
+                # We want to process COME_FROMs to the same offset to be in *descending*
+                # offset order so we have the larger range or biggest instruction interval
+                # last. (I think they are sorted in increasing order, but for safety
+                # we sort them). That way, specific COME_FROM tags will match up
+                # properly. For example, a "loop" with an "if" nested in it should have the
+                # "loop" tag last so the grammar rule matches that properly.
+                for jump_offset in sorted(jump_targets[inst.offset], reverse=True):
+                    come_from_name = "COME_FROM"
+                    opname = self.opname_for_offset(jump_offset)
+                    if opname == "EXTENDED_ARG":
+                        k = xdis.next_offset(op, self.opc, jump_offset)
+                        opname = self.opname_for_offset(k)
+
+                    if opname.startswith("SETUP_"):
+                        come_from_type = opname[len("SETUP_") :]
+                        come_from_name = "COME_FROM_%s" % come_from_type
+                        pass
+                    elif inst.offset in self.except_targets:
+                        come_from_name = "COME_FROM_EXCEPT_CLAUSE"
+                    j = tokens_append(
+                        j,
+                        Token(
+                            come_from_name,
+                            jump_offset,
+                            repr(jump_offset),
+                            offset="%s_%s" % (inst.offset, jump_idx),
+                            has_arg=True,
+                            opc=self.opc,
+                        ),
+                    )
+                    jump_idx += 1
+                    pass
+                pass
+            elif inst.offset in self.else_start:
+                end_offset = self.else_start[inst.offset]
+                j = tokens_append(
+                    j,
+                    Token(
+                        "ELSE",
+                        None,
+                        repr(end_offset),
+                        offset="%s" % (inst.offset),
+                        has_arg=True,
+                        opc=self.opc,
+                    ),
+                )
+
+                pass
+
+            pattr = inst.argrepr
+            opname = inst.opname
+
+            if op in self.opc.CONST_OPS:
+                const = argval
+                if iscode(const):
+                    if const.co_name == "<lambda>":
+                        assert opname == "LOAD_CONST"
+                        opname = "LOAD_LAMBDA"
+                    elif const.co_name == "<genexpr>":
+                        opname = "LOAD_GENEXPR"
+                    elif const.co_name == "<dictcomp>":
+                        opname = "LOAD_DICTCOMP"
+                    elif const.co_name == "<setcomp>":
+                        opname = "LOAD_SETCOMP"
+                    elif const.co_name == "<listcomp>":
+                        opname = "LOAD_LISTCOMP"
+                    else:
+                        opname = "LOAD_CODE"
+                    # verify() uses 'pattr' for comparison, since 'attr'
+                    # now holds Code(const) and thus can not be used
+                    # for comparison (todo: think about changing this)
+                    # pattr = 'code_object @ 0x%x %s->%s' %\
+                    # (id(const), const.co_filename, const.co_name)
+                    pattr = "<code_object " + const.co_name + ">"
+                elif isinstance(const, str):
+                    opname = "LOAD_STR"
+                else:
+                    if isinstance(inst.arg, int) and inst.arg < len(co.co_consts):
+                        argval, _ = _get_const_info(inst.arg, co.co_consts)
+                    # Why don't we use _ above for "pattr" rather than "const"?
+                    # This *is* a little hoaky, but we have to coordinate with
+                    # other parts like n_LOAD_CONST in pysource.py for example.
+                    pattr = const
+                    pass
+            elif opname in ("MAKE_FUNCTION", "MAKE_CLOSURE"):
+                flags = argval
+                opname = "MAKE_FUNCTION_%d" % (flags)
+                attr = []
+                for flag in self.MAKE_FUNCTION_FLAGS:
+                    bit = flags & 1
+                    attr.append(bit)
+                    flags >>= 1
+                attr = attr[:4]  # remove last value: attr[5] == False
+                j = tokens_append(
+                    j,
+                    Token(
+                        opname=opname,
+                        attr=attr,
+                        pattr=pattr,
+                        offset=inst.offset,
+                        linestart=inst.starts_line,
+                        op=op,
+                        has_arg=inst.has_arg,
+                        opc=self.opc,
+                    ),
+                )
+                continue
+            elif op in self.varargs_ops:
+                pos_args = argval
+                if self.is_pypy and not pos_args and opname == "BUILD_MAP":
+                    opname = "BUILD_MAP_n"
+                else:
+                    opname = "%s_%d" % (opname, pos_args)
+
+            elif self.is_pypy and opname == "JUMP_IF_NOT_DEBUG":
+                # The value in the dict is in special cases in semantic actions, such
+                # as JUMP_IF_NOT_DEBUG. The value is not used in these cases, so we put
+                # in arbitrary value 0.
+                customize[opname] = 0
+            elif opname == "UNPACK_EX":
+                # FIXME: try with scanner and parser by
+                # changing argval
+                before_args = argval & 0xFF
+                after_args = (argval >> 8) & 0xFF
+                pattr = "%d before vararg, %d after" % (before_args, after_args)
+                argval = (before_args, after_args)
+                opname = "%s_%d+%d" % (opname, before_args, after_args)
+
+            elif op == self.opc.JUMP_ABSOLUTE:
+                # Further classify JUMP_ABSOLUTE into backward jumps
+                # which are used in loops, and "CONTINUE" jumps which
+                # may appear in a "continue" statement.  The loop-type
+                # and continue-type jumps will help us classify loop
+                # boundaries The continue-type jumps help us get
+                # "continue" statements with would otherwise be turned
+                # into a "pass" statement because JUMPs are sometimes
+                # ignored in rules as just boundary overhead. In
+                # comprehensions we might sometimes classify JUMP_BACK
+                # as CONTINUE, but that's okay since we add a grammar
+                # rule for that.
+                pattr = argval
+                target = self.get_target(inst.offset)
+                if target <= inst.offset:
+                    next_opname = self.insts[i + 1].opname
+
+                    # 'Continue's include jumps to loops that are not
+                    # and the end of a block which follow with POP_BLOCK and COME_FROM_LOOP.
+                    # If the JUMP_ABSOLUTE is to a FOR_ITER and it is followed by another JUMP_FORWARD
+                    # then we'll take it as a "continue".
+                    is_continue = (
+                        self.insts[self.offset2inst_index[target]].opname == "FOR_ITER"
+                        and self.insts[i + 1].opname == "JUMP_FORWARD"
+                    )
+
+                    if self.version < 3.8 and (
+                        is_continue
+                        or (
+                            inst.offset in self.stmts
+                            and (
+                                inst.starts_line
+                                and next_opname not in self.not_continue_follow
+                            )
+                        )
+                    ):
+                        opname = "CONTINUE"
+                    else:
+                        opname = "JUMP_BACK"
+                        # FIXME: this is a hack to catch stuff like:
+                        #   if x: continue
+                        # the "continue" is not on a new line.
+                        # There are other situations where we don't catch
+                        # CONTINUE as well.
+                        if tokens[-1].kind == "JUMP_BACK" and tokens[-1].attr <= argval:
+                            if tokens[-2].kind == "BREAK_LOOP":
+                                del tokens[-1]
+                            else:
+                                # intern is used because we are changing the *previous* token
+                                tokens[-1].kind = sys.intern("CONTINUE")
+                    if last_op_was_break and opname == "CONTINUE":
+                        last_op_was_break = False
+                        continue
+
+            elif inst.offset in self.load_asserts:
+                opname = "LOAD_ASSERT"
+
+            last_op_was_break = opname == "BREAK_LOOP"
+            j = tokens_append(
+                j,
+                Token(
+                    opname=opname,
+                    attr=argval,
+                    pattr=pattr,
+                    offset=inst.offset,
+                    linestart=inst.starts_line,
+                    op=op,
+                    has_arg=inst.has_arg,
+                    opc=self.opc,
+                ),
+            )
+            pass
+
+        if show_asm in ("both", "after"):
+            for t in tokens:
+                print(t.format(line_prefix="L."))
+            print()
+        return tokens, customize
+
+    def find_jump_targets(self, debug):
+        """
+        Detect all offsets in a byte code which are jump targets
+        where we might insert a COME_FROM instruction.
+
+        Return the list of offsets.
+
+        Return the list of offsets. An instruction can be jumped
+        to in from multiple instructions.
+        """
+        code = self.code
+        n = len(code)
+        self.structs = [{"type": "root", "start": 0, "end": n - 1}]
+
+        # All loop entry points
+        self.loops = []
+
+        # Map fixed jumps to their real destination
+        self.fixed_jumps = {}
+        self.except_targets = {}
+        self.ignore_if = set()
+        self.build_statement_indices()
+        self.else_start = {}
+
+        # Containers filled by detect_control_flow()
+        self.not_continue = set()
+        self.return_end_ifs = set()
+        self.setup_loop_targets = {}  # target given setup_loop offset
+        self.setup_loops = {}  # setup_loop offset given target
+
+        targets = {}
+        for i, inst in enumerate(self.insts):
+            offset = inst.offset
+            op = inst.opcode
+
+            # Determine structures and fix jumps in Python versions
+            # since 2.3
+            self.detect_control_flow(offset, targets, i)
+
+            if inst.has_arg:
+                label = self.fixed_jumps.get(offset)
+                oparg = inst.arg
+                if self.code[offset] == self.opc.EXTENDED_ARG:
+                    j = xdis.next_offset(op, self.opc, offset)
+                    next_offset = xdis.next_offset(op, self.opc, j)
+                else:
+                    next_offset = xdis.next_offset(op, self.opc, offset)
+
+                if label is None:
+                    if op in self.opc.hasjrel and op != self.opc.FOR_ITER:
+                        label = next_offset + oparg
+                    elif op in self.opc.hasjabs:
+                        if op in self.jump_if_pop:
+                            if oparg > offset:
+                                label = oparg
+
+                if label is not None and label != -1:
+                    targets[label] = targets.get(label, []) + [offset]
+            elif op == self.opc.END_FINALLY and offset in self.fixed_jumps:
+                label = self.fixed_jumps[offset]
+                targets[label] = targets.get(label, []) + [offset]
+                pass
+
+            pass  # for loop
+
+        # DEBUG:
+        if debug in ("both", "after"):
+            import pprint as pp
+
+            pp.pprint(self.structs)
+
+        return targets
+
+    def build_statement_indices(self):
+        code = self.code
+        start = 0
+        end = codelen = len(code)
+
+        # Compose preliminary list of indices with statements,
+        # using plain statement opcodes
+        prelim = self.inst_matches(start, end, self.statement_opcodes)
+
+        # Initialize final container with statements with
+        # preliminary data
+        stmts = self.stmts = set(prelim)
+
+        # Same for opcode sequences
+        pass_stmts = set()
+        for sequence in self.statement_opcode_sequences:
+            for i in self.op_range(start, end - (len(sequence) + 1)):
+                match = True
+                for elem in sequence:
+                    if elem != code[i]:
+                        match = False
+                        break
+                    i += instruction_size(code[i], self.opc)
+
+                if match is True:
+                    i = self.prev_op[i]
+                    stmts.add(i)
+                    pass_stmts.add(i)
+
+        # Initialize statement list with the full data we've gathered so far
+        if pass_stmts:
+            stmt_offset_list = list(stmts)
+            stmt_offset_list.sort()
+        else:
+            stmt_offset_list = prelim
+        # 'List-map' which contains offset of start of
+        # next statement, when op offset is passed as index
+        self.next_stmt = slist = []
+        last_stmt_offset = -1
+        i = 0
+        # Go through all statement offsets
+        for stmt_offset in stmt_offset_list:
+            # Process absolute jumps, but do not remove 'pass' statements
+            # from the set
+            if (
+                code[stmt_offset] == self.opc.JUMP_ABSOLUTE
+                and stmt_offset not in pass_stmts
+            ):
+                # If absolute jump occurs in forward direction or it takes off from the
+                # same line as previous statement, this is not a statement
+                # FIXME: 0 isn't always correct
+                target = self.get_target(stmt_offset)
+                if (
+                    target > stmt_offset
+                    or self.lines[last_stmt_offset].l_no == self.lines[stmt_offset].l_no
+                ):
+                    stmts.remove(stmt_offset)
+                    continue
+                # Rewing ops till we encounter non-JUMP_ABSOLUTE one
+                j = self.prev_op[stmt_offset]
+                while code[j] == self.opc.JUMP_ABSOLUTE:
+                    j = self.prev_op[j]
+                # If we got here, then it's list comprehension which
+                # is not a statement too
+                if code[j] == self.opc.LIST_APPEND:
+                    stmts.remove(stmt_offset)
+                    continue
+            # Exclude ROT_TWO + POP_TOP
+            elif (
+                code[stmt_offset] == self.opc.POP_TOP
+                and code[self.prev_op[stmt_offset]] == self.opc.ROT_TWO
+            ):
+                stmts.remove(stmt_offset)
+                continue
+            # Exclude FOR_ITER + designators
+            elif code[stmt_offset] in self.designator_ops:
+                j = self.prev_op[stmt_offset]
+                while code[j] in self.designator_ops:
+                    j = self.prev_op[j]
+                if code[j] == self.opc.FOR_ITER:
+                    stmts.remove(stmt_offset)
+                    continue
+            # Add to list another list with offset of current statement,
+            # equal to length of previous statement
+            slist += [stmt_offset] * (stmt_offset - i)
+            last_stmt_offset = stmt_offset
+            i = stmt_offset
+        # Finish filling the list for last statement
+        slist += [codelen] * (codelen - len(slist))
+
+    def detect_control_flow(self, offset, targets, inst_index):
+        """
+        Detect type of block structures and their boundaries to fix optimized jumps
+        in python2.3+
+        """
+
+        code = self.code
+        inst = self.insts[inst_index]
+        op = inst.opcode
+
+        # Detect parent structure
+        parent = self.structs[0]
+        start = parent["start"]
+        end = parent["end"]
+
+        # Pick inner-most parent for our offset
+        for struct in self.structs:
+            current_start = struct["start"]
+            current_end = struct["end"]
+            if (current_start <= offset < current_end) and (
+                current_start >= start and current_end <= end
+            ):
+                start = current_start
+                end = current_end
+                parent = struct
+
+        if self.version < 3.8 and op == self.opc.SETUP_LOOP:
+            # We categorize loop types: 'for', 'while', 'while 1' with
+            # possibly suffixes '-loop' and '-else'
+            # Try to find the jump_back instruction of the loop.
+            # It could be a return instruction.
+
+            start += inst.inst_size
+            target = self.get_target(offset)
+            end = self.restrict_to_parent(target, parent)
+            self.setup_loops[target] = offset
+
+            if target != end:
+                self.fixed_jumps[offset] = end
+
+            (line_no, next_line_byte) = self.lines[offset]
+            jump_back = self.last_instr(
+                start, end, self.opc.JUMP_ABSOLUTE, next_line_byte, False
+            )
+
+            if jump_back:
+                jump_forward_offset = xdis.next_offset(
+                    code[jump_back], self.opc, jump_back
+                )
+            else:
+                jump_forward_offset = None
+
+            return_val_offset1 = self.prev[self.prev[end]]
+
+            if (
+                jump_back
+                and jump_back != self.prev_op[end]
+                and self.is_jump_forward(jump_forward_offset)
+            ):
+                if code[self.prev_op[end]] == self.opc.RETURN_VALUE or (
+                    code[self.prev_op[end]] == self.opc.POP_BLOCK
+                    and code[return_val_offset1] == self.opc.RETURN_VALUE
+                ):
+                    jump_back = None
+            if not jump_back:
+                # loop suite ends in return
+                jump_back = self.last_instr(start, end, self.opc.RETURN_VALUE)
+                if not jump_back:
+                    return
+
+                jb_inst = self.get_inst(jump_back)
+                jump_back = self.next_offset(jb_inst.opcode, jump_back)
+
+                if_offset = None
+                if code[self.prev_op[next_line_byte]] not in self.pop_jump_tf:
+                    if_offset = self.prev[next_line_byte]
+                if if_offset:
+                    loop_type = "while"
+                    self.ignore_if.add(if_offset)
+                else:
+                    loop_type = "for"
+                target = next_line_byte
+                end = xdis.next_offset(code[jump_back], self.opc, jump_back)
+            else:
+                if self.get_target(jump_back) >= next_line_byte:
+                    jump_back = self.last_instr(
+                        start, end, self.opc.JUMP_ABSOLUTE, start, False
+                    )
+
+                jb_inst = self.get_inst(jump_back)
+
+                jb_next_offset = self.next_offset(jb_inst.opcode, jump_back)
+                if end > jb_next_offset and self.is_jump_forward(end):
+                    if self.is_jump_forward(jb_next_offset):
+                        if self.get_target(jb_next_offset) == self.get_target(end):
+                            self.fixed_jumps[offset] = jb_next_offset
+                            end = jb_next_offset
+                elif target < offset:
+                    self.fixed_jumps[offset] = jb_next_offset
+                    end = jb_next_offset
+
+                target = self.get_target(jump_back)
+
+                if code[target] in (self.opc.FOR_ITER, self.opc.GET_ITER):
+                    loop_type = "for"
+                else:
+                    loop_type = "while"
+                    test = self.prev_op[next_line_byte]
+
+                    if test == offset:
+                        loop_type = "while 1"
+                    elif self.code[test] in self.opc.JUMP_OPs:
+                        self.ignore_if.add(test)
+                        test_target = self.get_target(test)
+                        if test_target > (jump_back + 3):
+                            jump_back = test_target
+                self.not_continue.add(jump_back)
+            self.loops.append(target)
+            self.structs.append(
+                {"type": loop_type + "-loop", "start": target, "end": jump_back}
+            )
+            after_jump_offset = xdis.next_offset(code[jump_back], self.opc, jump_back)
+            if after_jump_offset != end:
+                self.structs.append(
+                    {
+                        "type": loop_type + "-else",
+                        "start": after_jump_offset,
+                        "end": end,
+                    }
+                )
+        elif op in self.pop_jump_tf:
+            target = inst.argval
+            prev_op = self.prev_op
+
+            # FIXME: hack upon hack, test_pysource.py fails with this
+            # Until the grammar is corrected we do this fiction...
+            pretarget = self.get_inst(prev_op[target])
+            if (
+                pretarget.opcode in self.pop_jump_if_pop
+                and (target > offset)
+                and pretarget.offset != offset
+            ):
+
+                if pretarget.argval != target:
+                    # FIXME: this is not accurate The commented out below
+                    # is what it should be. However grammar rules right now
+                    # assume the incorrect offsets.
+                    # self.fixed_jumps[offset] = target
+                    self.fixed_jumps[offset] = pretarget.offset
+                    return
+
+            self.fixed_jumps[offset] = target
+
+        elif self.version < 3.8 and op == self.opc.SETUP_EXCEPT:
+            target = self.get_target(offset)
+            end = self.restrict_to_parent(target, parent)
+            self.fixed_jumps[offset] = end
+        elif op == self.opc.POP_EXCEPT:
+            next_offset = xdis.next_offset(op, self.opc, offset)
+            target = self.get_target(next_offset)
+            if target > next_offset:
+                next_op = code[next_offset]
+                if (
+                    self.opc.JUMP_ABSOLUTE == next_op
+                    and self.opc.END_FINALLY
+                    != code[xdis.next_offset(next_op, self.opc, next_offset)]
+                ):
+                    self.fixed_jumps[next_offset] = target
+                    self.except_targets[target] = next_offset
+
+        elif op == self.opc.SETUP_FINALLY:
+            target = self.get_target(offset)
+            end = self.restrict_to_parent(target, parent)
+            self.fixed_jumps[offset] = end
+        elif op in self.jump_if_pop:
+            target = self.get_target(offset)
+            if target > offset:
+                unop_target = self.last_instr(
+                    offset, target, self.opc.JUMP_FORWARD, target
+                )
+                if unop_target and code[unop_target + 3] != self.opc.ROT_TWO:
+                    self.fixed_jumps[offset] = unop_target
+                else:
+                    self.fixed_jumps[offset] = self.restrict_to_parent(target, parent)
+                    pass
+                pass
+        else:
+            # 3.5+ has Jump optimization which too often causes RETURN_VALUE to get
+            # misclassified as RETURN_END_IF. Handle that here.
+            # In RETURN_VALUE, JUMP_ABSOLUTE, RETURN_VALUE is never RETURN_END_IF
+            if op == self.opc.RETURN_VALUE:
+                next_offset = xdis.next_offset(op, self.opc, offset)
+                if next_offset < len(code) and (
+                    code[next_offset] == self.opc.JUMP_ABSOLUTE
+                    and offset in self.return_end_ifs
+                ):
+                    self.return_end_ifs.remove(offset)
+                    pass
+                pass
+            elif op == self.opc.JUMP_FORWARD:
+                # If we have:
+                #   JUMP_FORWARD x, [non-jump, insns], RETURN_VALUE, x:
+                # then RETURN_VALUE is not RETURN_END_IF
+                rtarget = self.get_target(offset)
+                rtarget_prev = self.prev[rtarget]
+                if (
+                    code[rtarget_prev] == self.opc.RETURN_VALUE
+                    and rtarget_prev in self.return_end_ifs
+                ):
+                    i = rtarget_prev
+                    while i != offset:
+                        if code[i] in [op3.JUMP_FORWARD, op3.JUMP_ABSOLUTE]:
+                            return
+                        i = self.prev[i]
+                    self.return_end_ifs.remove(rtarget_prev)
+                pass
+        return
+
+    def is_jump_back(self, offset, extended_arg):
+        """
+        Return True if the code at offset is some sort of jump back.
+        That is, it is ether "JUMP_FORWARD" or an absolute jump that
+        goes forward.
+        """
+        if self.code[offset] != self.opc.JUMP_ABSOLUTE:
+            return False
+        return offset > self.get_target(offset, extended_arg)
+
+    def next_except_jump(self, start):
+        """
+        Return the next jump that was generated by an except SomeException:
+        construct in a try...except...else clause or None if not found.
+        """
+
+        if self.code[start] == self.opc.DUP_TOP:
+            except_match = self.first_instr(
+                start, len(self.code), self.opc.POP_JUMP_IF_FALSE
+            )
+            if except_match:
+                jmp = self.prev_op[self.get_target(except_match)]
+                self.ignore_if.add(except_match)
+                self.not_continue.add(jmp)
+                return jmp
+
+        count_END_FINALLY = 0
+        count_SETUP_ = 0
+        for i in self.op_range(start, len(self.code)):
+            op = self.code[i]
+            if op == self.opc.END_FINALLY:
+                if count_END_FINALLY == count_SETUP_:
+                    assert self.code[self.prev_op[i]] in frozenset(
+                        [
+                            self.opc.JUMP_ABSOLUTE,
+                            self.opc.JUMP_FORWARD,
+                            self.opc.RETURN_VALUE,
+                        ]
+                    )
+                    self.not_continue.add(self.prev_op[i])
+                    return self.prev_op[i]
+                count_END_FINALLY += 1
+            elif op in self.setup_opts_no_loop:
+                count_SETUP_ += 1
+
+    def rem_or(self, start, end, instr, target=None, include_beyond_target=False):
+        """
+        Find offsets of all requested <instr> between <start> and <end>,
+        optionally <target>ing specified offset, and return list found
+        <instr> offsets which are not within any POP_JUMP_IF_TRUE jumps.
+        """
+        assert start >= 0 and end <= len(self.code) and start <= end
+
+        # Find all offsets of requested instructions
+        instr_offsets = self.inst_matches(
+            start, end, instr, target, include_beyond_target
+        )
+        # Get all POP_JUMP_IF_TRUE (or) offsets
+        jump_true_op = self.opc.POP_JUMP_IF_TRUE
+        pjit_offsets = self.inst_matches(start, end, jump_true_op)
+        filtered = []
+        for pjit_offset in pjit_offsets:
+            pjit_tgt = self.get_target(pjit_offset) - 3
+            for instr_offset in instr_offsets:
+                if instr_offset <= pjit_offset or instr_offset >= pjit_tgt:
+                    filtered.append(instr_offset)
+            instr_offsets = filtered
+            filtered = []
+        return instr_offsets
+
+
+if __name__ == "__main__":
+    from uncompyle6 import PYTHON_VERSION
+
+    if PYTHON_VERSION >= 3.7:
+        import inspect
+
+        co = inspect.currentframe().f_code
+        from uncompyle6 import PYTHON_VERSION
+
+        tokens, customize = Scanner37Base(PYTHON_VERSION).ingest(co)
+        for t in tokens:
+            print(t)
+    else:
+        print(
+            "Need to be Python 3.7 or greater to demo; I am version {PYTHON_VERSION}."
+            % PYTHON_VERSION
+        )
+    pass

uncompyle6/scanners/scanner38.py

@@ -12,34 +12,73 @@
 #
 #  You should have received a copy of the GNU General Public License
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-"""
-Python 3.8 bytecode decompiler scanner
+"""Python 3.8 bytecode decompiler scanner
 
-Does some additional massaging of xdis-disassembled instructions to
-make things easier for decompilation.
+Does some token massaging of xdis-disassembled instructions to make
+things easier for decompilation.
 
 This sets up opcodes Python's 3.8 and calls a generalized
-scanner routine for Python 3.
+scanner routine for Python 3.7 and up.
 """
 
 from uncompyle6.scanners.scanner37 import Scanner37
-from uncompyle6.scanners.scanner3 import Scanner3
+from uncompyle6.scanners.scanner37base import Scanner37Base
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_38 as opc
+
+# bytecode verification, verify(), uses JUMP_OPS from here
 JUMP_OPs = opc.JUMP_OPS
 
-class Scanner38(Scanner37):
 
+class Scanner38(Scanner37):
     def __init__(self, show_asm=None):
-        Scanner3.__init__(self, 3.8, show_asm)
+        Scanner37Base.__init__(self, 3.8, show_asm)
         return
+
     pass
 
+    def ingest(self, co, classname=None, code_objects={}, show_asm=None):
+        tokens, customize = super(Scanner38, self).ingest(
+            co, classname, code_objects, show_asm
+        )
+        for i, token in enumerate(tokens):
+            opname = token.kind
+            if opname in ("JUMP_FORWARD", "JUMP_ABSOLUTE"):
+                # Turn JUMPs into BREAK_LOOP
+                jump_target = token.attr
+
+                if opname == "JUMP_ABSOLUTE" and token.offset >= jump_target:
+                    # Not a forward jump, so continue
+                    # FIXME: Do we need "continue" detection?
+                    continue
+                if i + 1 < len(tokens) and tokens[i + 1] == "JUMP_BACK":
+                    # Sometimes the jump back is *after* the break...
+                    jump_back_index = i + 1
+                else:
+                    # and sometimes it is *before* where we jumped to.
+                    jump_back_index = self.offset2tok_index[jump_target] - 1
+                    while tokens[jump_back_index].kind.startswith("COME_FROM_"):
+                        jump_back_index -= 1
+                        pass
+                    pass
+                jump_back_token = tokens[jump_back_index]
+                if (
+                    jump_back_token == "JUMP_BACK"
+                    and jump_back_token.attr < token.offset
+                ):
+                    token.kind = "BREAK_LOOP"
+                pass
+            pass
+        return tokens, customize
+
+
 if __name__ == "__main__":
     from uncompyle6 import PYTHON_VERSION
+
     if PYTHON_VERSION == 3.8:
         import inspect
+
         co = inspect.currentframe().f_code
         tokens, customize = Scanner38().ingest(co)
         for t in tokens:

uncompyle6/scanners/scanner39.py

@@ -0,0 +1,55 @@
+#  Copyright (c) 2019 by Rocky Bernstein
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""Python 3.9 bytecode decompiler scanner.
+
+Does some token massaging of xdis-disassembled instructions to make
+things easier for decompilation.
+
+This sets up opcodes Python's 3.9 and calls a generalized
+scanner routine for Python 3.7 and up.
+"""
+
+from uncompyle6.scanners.scanner38 import Scanner38
+from uncompyle6.scanners.scanner37base import Scanner37Base
+
+# bytecode verification, verify(), uses JUMP_OPs from here
+from xdis.opcodes import opcode_38 as opc
+
+# bytecode verification, verify(), uses JUMP_OPS from here
+JUMP_OPs = opc.JUMP_OPS
+
+
+class Scanner39(Scanner38):
+    def __init__(self, show_asm=None):
+        Scanner37Base.__init__(self, 3.9, show_asm)
+        return
+
+    pass
+
+
+if __name__ == "__main__":
+    from uncompyle6 import PYTHON_VERSION
+
+    if PYTHON_VERSION == 3.9:
+        import inspect
+
+        co = inspect.currentframe().f_code
+        tokens, customize = Scanner39().ingest(co)
+        for t in tokens:
+            print(t.format())
+        pass
+    else:
+        print("Need to be Python 3.9 to demo; I am %s." %
+              PYTHON_VERSION)

uncompyle6/scanners/tok.py

@@ -159,5 +159,12 @@ class Token:
     def __getitem__(self, i):
         raise IndexError
 
+    def off2int(self):
+        if isinstance(self.offset, int):
+            return self.offset
+        else:
+            assert isinstance(self.offset, str)
+            return(int(self.offset.split("_")[0]))
+
 
 NoneToken = Token("LOAD_CONST", offset=-1, attr=None, pattr=None)

uncompyle6/semantics/consts.py

@@ -344,8 +344,12 @@ TABLE_DIRECT = {
 #    'return':	        ( '%|return %c\n', 0),
     'return_if_stmt':	( 'return %c\n', 0),
 
-    'ifstmt':		    ( '%|if %c:\n%+%c%-', 0, 1 ),
-    'iflaststmt':		( '%|if %c:\n%+%c%-', 0, 1 ),
+    'ifstmt':		( '%|if %c:\n%+%c%-',
+                            0, # "testexpr" or "testexpr_then"
+                            1, # "_ifstmts_jump" or "return_stmts"
+                        ),
+
+    'iflaststmt':	( '%|if %c:\n%+%c%-', 0, 1 ),
     'iflaststmtl':		( '%|if %c:\n%+%c%-', 0, 1 ),
     'testtrue':         ( 'not %p',
                           (0, PRECEDENCE['unary_not']) ),
@@ -359,18 +363,18 @@ TABLE_DIRECT = {
     'ifelsestmt':	( '%|if %c:\n%+%c%-%|else:\n%+%c%-', 0, 1, 3 ),
     'ifelsestmtc':	( '%|if %c:\n%+%c%-%|else:\n%+%c%-', 0, 1, 3 ),
     'ifelsestmtl':	( '%|if %c:\n%+%c%-%|else:\n%+%c%-', 0, 1, 3 ),
-    'ifelsestmtr':	( '%|if %c:\n%+%c%-%|else:\n%+%c%-', 0, 1, 2 ),
-    'ifelsestmtr2':	( '%|if %c:\n%+%c%-%|else:\n%+%c%-\n\n', 0, 1, 3 ), # has COME_FROM in position 2
 
-
-    # "elif" forms are not generated by the parser but are created through tree
-    # transformations. See "n_ifelsestmt".
-    'ifelifstmt':	( '%|if %c:\n%+%c%-%c', 0, 1, 3 ),
+    #  These are created only via transformation
+    'ifelifstmt':	( '%|if %c:\n%+%c%-%c',
+                          0, # "testexpr" or "testexpr_then"
+                          1, 3 ),
     'elifelifstmt':	( '%|elif %c:\n%+%c%-%c', 0, 1, 3 ),
     'elifstmt':		( '%|elif %c:\n%+%c%-', 0, 1 ),
     'elifelsestmt':	( '%|elif %c:\n%+%c%-%|else:\n%+%c%-', 0, 1, 3 ),
+    'ifelsestmtr':	( '%|if %c:\n%+%c%-%|else:\n%+%c%-', 0, 1, 2 ),
+    'ifelsestmtr2':	( '%|if %c:\n%+%c%-%|else:\n%+%c%-\n\n', 0, 1, 3 ), # has COME_FROM
     'elifelsestmtr':	( '%|elif %c:\n%+%c%-%|else:\n%+%c%-\n\n', 0, 1, 2 ),
-    'elifelsestmtr2':	( '%|elif %c:\n%+%c%-%|else:\n%+%c%-\n\n', 0, 1, 3 ), # has COME_FROM in position 2
+    'elifelsestmtr2':	( '%|elif %c:\n%+%c%-%|else:\n%+%c%-\n\n', 0, 1, 3 ), # has COME_FROM
 
     'whileTruestmt':	( '%|while True:\n%+%c%-\n\n', 1 ),
     'whilestmt':	    ( '%|while %c:\n%+%c%-\n\n', 1, 2 ),
@@ -414,10 +418,11 @@ TABLE_DIRECT = {
                           (1, 'expr'), (5, 'store') ),
     'except_suite':     ( '%+%c%-%C', 0, (1, maxint, '') ),
 
-    # In Python 3.6, this is more complicated in the presence of "returns"
+    # In Python 3.6+, this is more complicated in the presence of "returns"
     'except_suite_finalize':     ( '%+%c%-%C', 1, (3, maxint, '') ),
 
     'pass':	            ( '%|pass\n', ),
+    'STORE_FAST':	    ( '%{pattr}', ),
     'kv':		    ( '%c: %c', 3, 1 ),
     'kv2':		    ( '%c: %c', 1, 2 ),
     'import':               ( '%|import %c\n', 2),

uncompyle6/semantics/customize.py

@@ -71,6 +71,9 @@ def customize_for_version(self, is_pypy, version):
         TABLE_DIRECT.update(
             {"except_cond3": ("%|except %c, %c:\n", (1, "expr"), (-2, "store"))}
         )
+        if version <= 2.6:
+            TABLE_DIRECT["testtrue_then"] = TABLE_DIRECT["testtrue"]
+
         if 2.4 <= version <= 2.6:
             TABLE_DIRECT.update({"comp_for": (" for %c in %c", 3, 1)})
         else:
@@ -134,7 +137,6 @@ def customize_for_version(self, is_pypy, version):
                 }
             )
             if version == 2.4:
-
                 def n_iftrue_stmt24(node):
                     self.template_engine(("%c", 0), node)
                     self.default(node)

uncompyle6/semantics/customize3.py

@@ -37,6 +37,7 @@ def customize_for_version3(self, version):
                 (0, "expr"),
                 (4, "expr"),
             ),
+            "except_cond2": ("%|except %c as %c:\n", (1, "expr"), (5, "store")),
             "function_def_annotate": ("\n\n%|def %c%c\n", -1, 0),
             # When a generator is a single parameter of a function,
             # it doesn't need the surrounding parenethesis.
@@ -331,7 +332,8 @@ def customize_for_version3(self, version):
                 (1, "suite_stmts_opt"),
                 (3, "except_handler"),
                 (5, "else_suitel"),
-            )
+            ),
+            "LOAD_CLASSDEREF": ("%{pattr}",),
         }
     )
     if version >= 3.4:

uncompyle6/semantics/customize38.py

@@ -12,7 +12,7 @@
 #
 #  You should have received a copy of the GNU General Public License
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
-"""Isolate Python 3.6 version-specific semantic actions here.
+"""Isolate Python 3.8 version-specific semantic actions here.
 """
 
 ########################
@@ -80,10 +80,19 @@ def customize_for_version38(self, version):
             (0, 'expr'),
             (3, 'for_block'), -2 ),
 
+        'ifpoplaststmtl': ( '%|if %c:\n%+%c%-',
+                            (0, "testexpr"),
+                            (2, "c_stmts" ) ),
+
+        'ifstmtl':	  ( '%|if %c:\n%+%c%-',
+                            (0, "testexpr"),
+                            (1, "_ifstmts_jumpl") ),
+
         'whilestmt38': ( '%|while %c:\n%+%c%-\n\n',
-                         (0, 'testexpr'), (1, 'l_stmts') ),
+                         (1, 'testexpr'),
+                         2 ), # "l_stmts" or "pass"
         'whileTruestmt38': ( '%|while True:\n%+%c%-\n\n',
-                         (0, 'l_stmts') ),
+                             1 ), # "l_stmts" or "pass"
         'try_elsestmtl38': (
             '%|try:\n%+%c%-%c%|else:\n%+%c%-',
             (1, 'suite_stmts_opt'),
@@ -98,4 +107,7 @@ def customize_for_version38(self, version):
         'tryfinally38': (
             '%|try:\n%+%c%-%|finally:\n%+%c%-\n\n',
                    (3, 'returns'), 6 ),
+        "named_expr": ( # AKA "walrus operator"
+            "%c := %c", (2, "store"), (0, "expr")
+            )
     })

uncompyle6/semantics/make_function.py

@@ -848,10 +848,18 @@ def make_function3(self, node, is_lambda, nested=1, code_node=None):
 
             free_tup = ann_dict = kw_dict = default_tup = None
             fn_bits = node[-1].attr
-            index = -4  # Skip over:
+
+            # Skip over:
             #  MAKE_FUNCTION,
+            #  optional docstring
             #  LOAD_CONST qualified name,
             #  LOAD_CONST code object
+            index = -4  # Skip over:
+            if node[-2] == "docstring":
+                index = -5
+            else:
+                index = -4
+
             if fn_bits[-1]:
                 free_tup = node[index]
                 index -= 1

uncompyle6/semantics/pysource.py

@@ -1130,11 +1130,15 @@ class SourceWalker(GenericASTTraversal, object):
 
     def n_generator_exp(self, node):
         self.write("(")
+        iter_index = 3
         if self.version > 3.2:
             code_index = -6
+            if self.version > 3.6:
+                # Python 3.7+ adds optional "come_froms" at node[0]
+                iter_index = 4
         else:
             code_index = -5
-        self.comprehension_walk(node, iter_index=3, code_index=code_index)
+        self.comprehension_walk(node, iter_index=iter_index, code_index=code_index)
         self.write(")")
         self.prune()
 
@@ -1633,6 +1637,57 @@ class SourceWalker(GenericASTTraversal, object):
 
         self.write(")")
 
+    def kv_map(self, kv_node, sep, line_number, indent):
+        first_time = True
+        for kv in kv_node:
+            assert kv in ('kv', 'kv2', 'kv3')
+
+            # kv ::= DUP_TOP expr ROT_TWO expr STORE_SUBSCR
+            # kv2 ::= DUP_TOP expr expr ROT_THREE STORE_SUBSCR
+            # kv3 ::= expr expr STORE_MAP
+
+            # FIXME: DRY this and the above
+            if kv == 'kv':
+                self.write(sep)
+                name = self.traverse(kv[-2], indent='')
+                if first_time:
+                    line_number = self.indent_if_source_nl(line_number, indent)
+                    first_time = False
+                    pass
+                line_number = self.line_number
+                self.write(name, ': ')
+                value = self.traverse(kv[1], indent=self.indent+(len(name)+2)*' ')
+            elif kv == 'kv2':
+                self.write(sep)
+                name = self.traverse(kv[1], indent='')
+                if first_time:
+                    line_number = self.indent_if_source_nl(line_number, indent)
+                    first_time = False
+                    pass
+                line_number = self.line_number
+                self.write(name, ': ')
+                value = self.traverse(kv[-3], indent=self.indent+(len(name)+2)*' ')
+            elif kv == 'kv3':
+                self.write(sep)
+                name = self.traverse(kv[-2], indent='')
+                if first_time:
+                    line_number = self.indent_if_source_nl(line_number, indent)
+                    first_time = False
+                    pass
+                line_number = self.line_number
+                self.write(name, ': ')
+                line_number = self.line_number
+                value = self.traverse(kv[0], indent=self.indent+(len(name)+2)*' ')
+                pass
+            self.write(value)
+            sep = ", "
+            if line_number != self.line_number:
+                sep += "\n" + self.indent + "  "
+                line_number = self.line_number
+                pass
+            pass
+
+
     def n_dict(self, node):
         """
         prettyprint a dict
@@ -1753,14 +1808,15 @@ class SourceWalker(GenericASTTraversal, object):
             pass
         else:
             # Python 2 style kvlist. Find beginning of kvlist.
+            indent = self.indent + "  "
+            line_number = self.line_number
             if node[0].kind.startswith("BUILD_MAP"):
                 if len(node) > 1 and node[1].kind in ("kvlist", "kvlist_n"):
                     kv_node = node[1]
                 else:
                     kv_node = node[1:]
+                self.kv_map(kv_node, sep, line_number, indent)
             else:
-                indent = self.indent + "  "
-                line_number = self.line_number
                 sep = ''
                 opname = node[-1].kind
                 if self.is_pypy and self.version >= 3.5:
@@ -1798,54 +1854,7 @@ class SourceWalker(GenericASTTraversal, object):
                         pass
                 elif opname.startswith('kvlist'):
                     kv_node = node[-1]
-                    first_time = True
-                    for kv in kv_node:
-                        assert kv in ('kv', 'kv2', 'kv3')
-
-                        # kv ::= DUP_TOP expr ROT_TWO expr STORE_SUBSCR
-                        # kv2 ::= DUP_TOP expr expr ROT_THREE STORE_SUBSCR
-                        # kv3 ::= expr expr STORE_MAP
-
-                        # FIXME: DRY this and the above
-                        if kv == 'kv':
-                            self.write(sep)
-                            name = self.traverse(kv[-2], indent='')
-                            if first_time:
-                                line_number = self.indent_if_source_nl(line_number, indent)
-                                first_time = False
-                                pass
-                            line_number = self.line_number
-                            self.write(name, ': ')
-                            value = self.traverse(kv[1], indent=self.indent+(len(name)+2)*' ')
-                        elif kv == 'kv2':
-                            self.write(sep)
-                            name = self.traverse(kv[1], indent='')
-                            if first_time:
-                                line_number = self.indent_if_source_nl(line_number, indent)
-                                first_time = False
-                                pass
-                            line_number = self.line_number
-                            self.write(name, ': ')
-                            value = self.traverse(kv[-3], indent=self.indent+(len(name)+2)*' ')
-                        elif kv == 'kv3':
-                            self.write(sep)
-                            name = self.traverse(kv[-2], indent='')
-                            if first_time:
-                                line_number = self.indent_if_source_nl(line_number, indent)
-                                first_time = False
-                                pass
-                            line_number = self.line_number
-                            self.write(name, ': ')
-                            line_number = self.line_number
-                            value = self.traverse(kv[0], indent=self.indent+(len(name)+2)*' ')
-                            pass
-                        self.write(value)
-                        sep = ", "
-                        if line_number != self.line_number:
-                            sep += "\n" + self.indent + "  "
-                            line_number = self.line_number
-                            pass
-                        pass
+                    self.kv_map(node[-1], sep, line_number, indent)
 
                 pass
         if sep.startswith(",\n"):
@@ -2010,8 +2019,13 @@ class SourceWalker(GenericASTTraversal, object):
         self.default(node)
 
     def n_except_cond2(self, node):
-        if node[-2][0] == "unpack":
-            node[-2][0].kind = "unpack_w_parens"
+        if node[-1] == "come_from_opt":
+            unpack_node = -3
+        else:
+            unpack_node = -2
+
+        if node[unpack_node][0] == "unpack":
+            node[unpack_node][0].kind = "unpack_w_parens"
         self.default(node)
 
     def template_engine(self, entry, startnode):

uncompyle6/semantics/transform.py

@@ -181,6 +181,7 @@ class TreeTransform(GenericASTTraversal, object):
 
         n = else_suite[0]
         old_stmts = None
+        else_suite_index = 1
 
         if len(n) == 1 == len(n[0]) and n[0] == "stmt":
             n = n[0][0]
@@ -192,9 +193,12 @@ class TreeTransform(GenericASTTraversal, object):
                 "iflaststmtl",
                 "ifelsestmtl",
                 "ifelsestmtc",
+                "ifpoplaststmtl",
             ):
-                # This seems needed for Python 2.5-2.7
                 n = n[0]
+                if n.kind == "ifpoplaststmtl":
+                    old_stmts = n[2]
+                    else_suite_index = 2
                 pass
             pass
         elif len(n) > 1 and 1 == len(n[0]) and n[0] == "stmt" and n[1].kind == "stmt":
@@ -206,7 +210,7 @@ class TreeTransform(GenericASTTraversal, object):
         else:
             return node
 
-        if n.kind in ("ifstmt", "iflaststmt", "iflaststmtl"):
+        if n.kind in ("ifstmt", "iflaststmt", "iflaststmtl", "ifpoplaststmtl"):
             node.kind = "ifelifstmt"
             n.kind = "elifstmt"
         elif n.kind in ("ifelsestmtr",):
@@ -223,17 +227,24 @@ class TreeTransform(GenericASTTraversal, object):
             if old_stmts:
                 if n.kind == "elifstmt":
                     trailing_else = SyntaxTree("stmts", old_stmts[1:])
-                    # We use elifelsestmtr because it has 3 nodes
-                    elifelse_stmt = SyntaxTree(
-                        "elifelsestmtr", [n[0], n[1], trailing_else]
-                    )
-                    node[3] = elifelse_stmt
+                    if len(trailing_else):
+                        # We use elifelsestmtr because it has 3 nodes
+                        elifelse_stmt = SyntaxTree(
+                            "elifelsestmtr", [n[0], n[else_suite_index], trailing_else]
+                        )
+                        node[3] = elifelse_stmt
+                    else:
+                        elif_stmt = SyntaxTree(
+                            "elifstmt", [n[0], n[else_suite_index]]
+                        )
+                        node[3] = elif_stmt
+
+                    node.transformed_by = "n_ifelsestmt"
                     pass
                 else:
                     # Other cases for n.kind may happen here
                     pass
                 pass
-            node.transformed_by = "n_ifelsestmt"
             return node
 
     n_ifelsestmtc = n_ifelsestmtl = n_ifelsestmt

uncompyle6/version.py

@@ -12,4 +12,4 @@
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
 # This file is suitable for sourcing inside bash as
 # well as importing into Python
-VERSION="3.5.1"  # noqa
+VERSION="3.6.0"  # noqa