Merge branch 'master' into python-2.4

HISTORY.md

@@ -44,7 +44,8 @@ it appears that Hartmut did most of the work to get this code to
 accept the full Python language. He added precedence to the table
 specifiers, support for multiple versions of Python, the
 pretty-printing of docstrings, lists, and hashes. He also wrote test and verification routines of
-deparsed bytecode, and used this in an extensive set of tests that he also wrote. He could verify against the entire Python library.
+deparsed bytecode, and used this in an extensive set of tests that he also wrote. He says he could verify against the 
+entire Python library. However I have subsequently found small and relatively obscure bugs in the decompilation code. 
 
 decompyle2.2 was packaged for Debian (sarge) by
 [Ben Burton around 2002](https://packages.qa.debian.org/d/decompyle.html). As
@@ -65,10 +66,12 @@ code to handle first Python 2.3 and then 2.4 bytecodes. Because of
 jump optimization introduced in the CPython bytecode compiler at that
 time, various JUMP instructions were classifed as going backwards, and
 COME FROM instructions were reintroduced.  See
-RELEASE-2.4-CHANGELOG.txt for more details here. There wasn't a public
+[RELEASE-2.4-CHANGELOG.txt](https://github.com/rocky/python-uncompyle6/blob/master/DECOMPYLE-2.4-CHANGELOG.txt) 
+for more details here. There wasn't a public
 release of RELEASE-2.4 and bytecodes other than Python 2.4 weren't
 supported. Dan says the Python 2.3 version could verify the entire
-python library.
+Python library. But given subsequent bugs found like simply
+recognizing complex-number constants in bytecode, decompilation wasn't perfect.
 
 Next we get to ["uncompyle" and
 PyPI](https://pypi.python.org/pypi/uncompyle/1.1) and the era of
@@ -95,17 +98,17 @@ so. Then hamled made a few commits earler on, while Eike Siewertsen
 made a few commits later on. But mostly wibiti, and Guenther
 Starnberger got the code to where uncompyle2 was around 2012.
 
-In uncompyle2 decompilation of python bytecode 2.5 & 2.6 is done by
+In `uncompyle`, decompilation of python bytecode 2.5 & 2.6 is done by
 transforming the byte code into a a pseudo 2.7 python bytecode and is
 based on code from Eloi Vanderbeken.
 
-This project, uncompyle6, abandons that approach for various
+This project, `uncompyle6`, abandons that approach for various
 reasons. However the main reason is that we need offsets in fragment
 deparsing to be exactly the same, and the transformation process can
-remove instructions.  Adding instructions with psuedo_offsets is
+remove instructions.  _Adding_ instructions with psuedo offsets is
 however okay.
 
-Uncompyle6, however owes its existence to the fork of uncompyle2 by
+`Uncompyle6` however owes its existence to the fork of `uncompyle2` by
 Myst herie (Mysterie) whose first commit picks up at
 2012. I chose this since it seemed to have been at that time the most
 actively, if briefly, worked on. Also starting around 2012 is Dark
@@ -115,9 +118,12 @@ I started working on this late 2015, mostly to add fragment support.
 In that, I decided to make this runnable on Python 3.2+ and Python 2.6+
 while, handling Python bytecodes from Python versions 2.5+ and
 3.2+. In doing so, it has been expedient to separate this into three
-projects: bytecode loading and disassembly (xdis), parsing and tree
-building (spark_parser), and grammar and semantic actions for
-decompiling (uncompyle6).
+projects:
+
+* bytecode loading and disassembly ([xdis](https://pypi.python.org/pypi/xdis)),
+* parsing and tree building ([spark_parser](https://pypi.python.org/pypi/spark_parser)),
+* this project - grammar and semantic actions for decompiling
+  ([uncompyle6](https://pypi.python.org/pypi/spark_parser)).
 
 
 Over the many years, code styles and Python features have
@@ -142,16 +148,19 @@ if the grammar is LR or left recursive.
 
 Another approach that doesn't use grammars is to do something like
 simulate execution symbolically and build expression trees off of
-stack results. The two important projects that work this way are
-[unpyc3](https://code.google.com/p/unpyc3/) and most especially
-[pycdc](https://github.com/zrax/pycdc) The latter project is largely
-by Michael Hansen and Darryl Pogue. If they supported getting
-source-code fragments and I could call it from Python, I'd probably
-ditch this and use that. From what I've seen, the code runs blindingly
-fast and spans all versions of Python.
+stack results. Control flow in that apprproach still needs to be
+handled somewhat ad hoc.  The two important projects that work this
+way are [unpyc3](https://code.google.com/p/unpyc3/) and most
+especially [pycdc](https://github.com/zrax/pycdc) The latter project
+is largely by Michael Hansen and Darryl Pogue. If they supported
+getting source-code fragments, did a better job in supporting Python
+more fully, and had a way I could call it from Python, I'd probably
+would have ditched this and used that. The code runs blindingly fast
+and spans all versions of Python, although more recently Python 3
+support has been lagging.
 
 Tests for the project have been, or are being, culled from all of the
 projects mentioned.
 
-NB. If you find mistakes, want corrections, or want your name added (or removed),
-please contact me.
+NB. If you find mistakes, want corrections, or want your name added
+(or removed), please contact me.

HOW-TO-REPORT-A-BUG.md

@@ -0,0 +1,63 @@
+# How to report a Bug
+
+## The difficulty of the problem
+
+There is no Python decompiler yet, that I know about that will
+decompyle everything. This one probably does the
+best job of *any* Python decompiler. But it is a constant work in progress: Python keeps changing, and so does its code generation.
+
+I have found bugs in *every* Python decompiler I have tried. Even
+those where authors/maintainers claim that they have used it on 
+the entire Python standard library. And I don't mean that
+the program doesn't come out with the same Python source instructions,
+but that the program is *semantically* not equivalent.
+
+So it is likely you'll find a mistranslation in decompiling.
+
+## What to send (minimum requirements)
+
+The basic requirement is pretty simple:
+
+* Python bytecode
+* Source text
+
+## What to send (additional helpful information)
+
+Some kind folks also give the invocation they used and the output
+which usually includes an error message produced. This is helpful. I
+can figure out what OS you are running this on and what version of
+*uncomplye6* was used. Therefore, if you don't provide the input
+command and the output from that, please give:
+
+* _uncompile6_ version used
+* OS that you used this on
+* Python interpreter version used
+
+
+### But I don't *have* the source code!
+
+Sure, I get it. No problem. There is Python assembly code on parse
+errors, so simply by hand decompile that. To get a full disassembly, use pydisasm from the [xdis](https://pypi.python.org/pypi/xdis) package. Opcodes are described in the documentation for the [dis](https://docs.python.org/3.6/library/dis.html) module.
+
+### But I don't *have* the source code and am incapable of figuring how how to do a hand disassembly!
+
+Well, you could learn. No one is born into this world knowing how to disassemble Python bytecode. And as Richard Feynman once said, "What one fool can learn, so can another."
+
+## Narrowing the problem
+
+I don't need the entire source code base for which one file or module
+can't be decompiled. I just need that one file or module only. If
+there are several files, file a bug report for each file.
+
+Python modules can get quite large, and usually decompilation problems
+occur in a single function or maybe the main-line code but not any of
+the functions or classes. So please chop down the source code by
+removing those parts that do to decompile properly.
+
+By doing this, you'll probably have a better sense of what exactly is
+the problem. Perhaps you can find the boundary of what decompiles, and
+what doesn't. That is useful. Or maybe the same file will decompile
+properly on a neighboring version of Python. That is helpful too.
+
+In sum, the more you can isolate or narrow the problem, the more
+likley the problem will be fixed and fixed sooner.

MANIFEST.in

@@ -1,6 +1,7 @@
 include README.rst
 include ChangeLog
 include HISTORY.md
+include HOW-TO-REPORT-A-BUG.md
 include LICENSE
 include Makefile
 include requirements.txt

README.rst

@@ -21,7 +21,8 @@ There were a number of decompyle, uncompile, uncompyle2, uncompyle3
 forks around. All of them came basically from the same code base, and
 almost all of them no were no longer actively maintained. Only one
 handled Python 3, and even there, only 3.2 or 3.3 depending on which
-code is used. This code pulls these together and moves forward. It
+code is used. This code pulls these together and moves forward. This
+project has the most complete support for Python 3.3 and above. It
 also addresses a number of open issues in the previous forks.
 
 What makes this different from other CPython bytecode decompilers?: its
@@ -166,7 +167,7 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++
+* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for later Python 3 versions is a bit lacking though.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique what is used here.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Include some fixes like supporting function annotations
 * The HISTORY_ file.

test/simple_source/bug22/03_if1.py

@@ -0,0 +1,7 @@
+# From https://github.com/ToontownInfinite /src/otp/avatar/LocalAvatar.py#L364
+if 1:
+    def jumpLandAnimFix(self, jumpTime):
+        return 5
+
+    def jumpLand(self):
+        return 6

test/simple_source/bug31/02_ifelse_comprehension.py

@@ -0,0 +1,12 @@
+# Python 2.7 sqlalchemy-1.013/sql/crud.py
+def _extend_values_for_multiparams(compiler, stmt, c):
+    c(
+        [
+            (
+                (compiler() if compiler()
+                    else compiler())
+                if c in stmt else compiler(),
+            )
+        ]
+        for i in enumerate(stmt)
+    )

test/simple_source/bug32/01_try_except_raise.py

@@ -0,0 +1,9 @@
+# From 3.2 _abcoll.py
+def pop(self):
+    it = iter(self)
+    try:
+        value = next(it)
+    except StopIteration:
+        raise KeyError
+    self.discard(value)
+    return value

test/simple_source/bug32/03_if.py

@@ -0,0 +1,11 @@
+# From 3.2 shlex.py
+def _samefile(os, src, dst):
+    if hasattr(os.path, 'samefile'):
+        try:
+            return os.path.samefile(src, dst)
+        except OSError:
+            return False
+
+    # All other platforms: check for same pathname.
+    return (os.path.normcase(os.path.abspath(src)) ==
+            os.path.normcase(os.path.abspath(dst)))

test/simple_source/bug33/01_delete_deref.py

@@ -0,0 +1,4 @@
+def a():
+    del y
+    def b():
+        return y

test/simple_source/bug33/02_pos_args.py

@@ -1,3 +1,6 @@
 # From Python 3.3.6 hmac.py
 # Problem was getting wrong placement of positional args
 digest_cons = lambda d=b'': 5
+
+# Handle single kwarg
+lambda *, d=0: None

test/simple_source/bug35/01_map_unpack.py

@@ -0,0 +1,9 @@
+# Python 3.5+ PEP 448 - Additional Unpacking Generalizations for dictionaries
+{**{}}
+{**{'a': 1, 'b': 2}}
+## {**{'x': 1}, **{'y': 2}}
+# {'c': 1, {'d': 2}, **{'e': 3}}
+[*[]]
+{**{0:0 for a in b}}
+## {**{}, **{}}
+## {**{}, **{}, **{}}

test/simple_source/bug35/02_async_for.py

@@ -0,0 +1,3 @@
+async def a(b, c):
+    async for b in c:
+        pass

test/simple_source/bug35/03_async_await.py

@@ -24,3 +24,9 @@ async def awith_test():
 async def awith_as_test():
     async with 1 as i:
         print(i)
+
+async def f(z):
+    await z
+
+async def g(z):
+    return await z

test/simple_source/bug35/03_while-if-break.py

@@ -5,3 +5,10 @@ def display_date(loop):
         if loop.time():
             break
         x = 5
+
+    # Another loop to test 3.5 ifelsestmtl grammar rule
+    while loop:
+        if x:
+            True
+        else:
+            True

test/simple_source/bug36/01_call_function.py

@@ -0,0 +1,5 @@
+# Python 3.6's changes for calling functions.
+# See https://github.com/rocky/python-uncompyle6/issues/58
+# CALL_FUNCTION_EX takes 2 to 3 arguments on the stack: the function, the tuple of positional arguments,
+# and optionally the dict of keyword arguments if bit 0 of oparg is 1.
+a(*[])

test/simple_source/bug36/01_extended_arg.py

@@ -0,0 +1,2 @@
+if __file__:
+    0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0

uncompyle6/parsers/parse3.py

@@ -147,23 +147,25 @@ class Python3Parser(PythonParser):
         iflaststmtl ::= testexpr c_stmts_opt JUMP_BACK COME_FROM_LOOP
 
         # These are used to keep AST indices the same
-        jf_else    ::= JUMP_FORWARD ELSE
-        ja_else    ::= JUMP_ABSOLUTE ELSE
+        jump_forward_else  ::= JUMP_FORWARD ELSE
+        jump_absolute_else ::= JUMP_ABSOLUTE ELSE
 
         # Note: in if/else kinds of statements, we err on the side
         # of missing "else" clauses. Therefore we include grammar
         # rules with and without ELSE.
 
         ifelsestmt ::= testexpr c_stmts_opt JUMP_FORWARD else_suite opt_come_from_except
-        ifelsestmt ::= testexpr c_stmts_opt jf_else else_suite _come_from
+        ifelsestmt ::= testexpr c_stmts_opt jump_forward_else else_suite _come_from
 
         ifelsestmtc ::= testexpr c_stmts_opt JUMP_ABSOLUTE else_suitec
-        ifelsestmtc ::= testexpr c_stmts_opt ja_else else_suitec
+        ifelsestmtc ::= testexpr c_stmts_opt jump_absolute_else else_suitec
 
         ifelsestmtr ::= testexpr return_if_stmts return_stmts
 
         ifelsestmtl ::= testexpr c_stmts_opt JUMP_BACK else_suitel
-        ifelsestmtl ::= testexpr c_stmts_opt COME_FROM JUMP_BACK else_suitel
+        ifelsestmtl ::= testexpr c_stmts_opt cf_jump_back else_suitel
+
+        cf_jump_back ::= COME_FROM JUMP_BACK
 
         # FIXME: this feels like a hack. Is it just 1 or two
         # COME_FROMs?  the parsed tree for this and even with just the
@@ -404,8 +406,12 @@ class Python3Parser(PythonParser):
 
     def p_expr3(self, args):
         """
-        conditional    ::= expr jmp_false expr jf_else expr COME_FROM
-        conditionalnot ::= expr jmp_true  expr jf_else expr COME_FROM
+        conditional    ::= expr jmp_false expr jump_forward_else expr COME_FROM
+        conditionalnot ::= expr jmp_true  expr jump_forward_else expr COME_FROM
+
+        # a JUMP_FORWARD to another JUMP_FORWARD can get turned into
+        # a JUMP_ABSOLUTE with no COME_FROM
+        conditional    ::= expr jmp_false expr jump_absolute_else expr
 
         expr ::= LOAD_CLASSNAME
 
@@ -660,10 +666,20 @@ class Python3Parser(PythonParser):
                     rule = "mapexpr ::=  BUILD_MAP_n kvlist_n"
                 elif self.version >= 3.5:
                     if opname != 'BUILD_MAP_WITH_CALL':
-                        rule = kvlist_n + ' ::= ' + 'expr ' * (token.attr*2)
-                        self.add_unique_rule(rule, opname, token.attr, customize)
-                        rule = "mapexpr ::=  %s %s" % (kvlist_n, opname)
-
+                        if opname == 'BUILD_MAP_UNPACK':
+                            rule = kvlist_n + ' ::= ' + 'expr ' * (token.attr*2)
+                            self.add_unique_rule(rule, opname, token.attr, customize)
+                            rule = 'dict ::= ' + 'expr ' * (token.attr*2)
+                            self.add_unique_rule(rule, opname, token.attr, customize)
+                            rule = 'mapexpr ::= ' + 'dict ' * token.attr
+                            self.add_unique_rule(rule, opname, token.attr, customize)
+                            rule = ('unmap_dict ::= ' +
+                                    ('mapexpr ' * token.attr) +
+                                    ' BUILD_MAP_UNPACK')
+                        else:
+                            rule = kvlist_n + ' ::= ' + 'expr ' * (token.attr*2)
+                            self.add_unique_rule(rule, opname, token.attr, customize)
+                            rule = "mapexpr ::=  %s %s" % (kvlist_n, opname)
                 else:
                     rule = kvlist_n + ' ::= ' + 'expr expr STORE_MAP ' * token.attr
                     self.add_unique_rule(rule, opname, token.attr, customize)
@@ -690,7 +706,10 @@ class Python3Parser(PythonParser):
                 rule_pat = ("genexpr ::= %sload_genexpr %%s%s expr "
                             "GET_ITER CALL_FUNCTION_1" % ('pos_arg '* args_pos, opname))
                 self.add_make_function_rule(rule_pat, opname, token.attr, customize)
-                rule_pat = ('mklambda ::= %sLOAD_LAMBDA %%s%s' % ('pos_arg '* args_pos, opname))
+                rule_pat = ('mklambda ::= %s%sLOAD_LAMBDA %%s%s' %
+                            (('pos_arg '* args_pos),
+                            ('kwarg '* args_kw),
+                            opname))
                 self.add_make_function_rule(rule_pat, opname, token.attr, customize)
                 rule_pat  = ("listcomp ::= %sLOAD_LISTCOMP %%s%s expr "
                              "GET_ITER CALL_FUNCTION_1" % ('expr ' * args_pos, opname))

uncompyle6/parsers/parse32.py

@@ -18,10 +18,17 @@ class Python32Parser(Python3Parser):
         whileTruestmt     ::= SETUP_LOOP return_stmts
                               COME_FROM_LOOP
 
+        try_middle ::= JUMP_FORWARD COME_FROM_EXCEPT except_stmts
+                       END_FINALLY
+
         # Python 3.2+ has more loop optimization that removes
         # JUMP_FORWARD in some cases, and hence we also don't
         # see COME_FROM
         _ifstmts_jump ::= c_stmts_opt
+        _ifstmts_jump ::= c_stmts_opt JUMP_FORWARD _come_from
+
+        stmt           ::= del_deref_stmt
+        del_deref_stmt ::= DELETE_DEREF
         """
     pass
 

uncompyle6/parsers/parse33.py

@@ -19,7 +19,6 @@ class Python33Parser(Python32Parser):
 
         iflaststmt    ::= testexpr c_stmts_opt33
         c_stmts_opt33 ::= JUMP_BACK JUMP_ABSOLUTE c_stmts_opt
-        _ifstmts_jump ::= c_stmts_opt JUMP_FORWARD _come_from
 
         # Python 3.5+ has jump optimization to remove the redundant
         # jump_excepts. But in 3.3 we need them added

uncompyle6/parsers/parse34.py

@@ -15,6 +15,8 @@ class Python34Parser(Python33Parser):
 
     def p_misc34(self, args):
         """
+        expr ::= LOAD_ASSERT
+
         # Python 3.4+ optimizes the trailing two JUMPS away
 
         # Is this 3.4 only?

uncompyle6/parsers/parse35.py

@@ -26,8 +26,20 @@ class Python35Parser(Python34Parser):
                            POP_BLOCK else_suite COME_FROM_LOOP
 
         # Python 3.5+ Await statement
-        stmt ::= await_stmt
-        await_stmt ::= call_function GET_AWAITABLE LOAD_CONST YIELD_FROM POP_TOP
+        expr       ::= await_expr
+        await_expr ::= expr GET_AWAITABLE LOAD_CONST YIELD_FROM
+
+        stmt       ::= await_stmt
+        await_stmt ::= await_expr POP_TOP
+
+        expr       ::= unmap_dict
+        expr       ::= unmapexpr
+
+        unmap_dict ::= dictcomp BUILD_MAP_UNPACK
+
+        unmap_dict ::= kv_lists BUILD_MAP_UNPACK
+        kv_lists   ::= kv_list kv_lists
+        kv_lists   ::= kv_list
 
         # Python 3.5+ has WITH_CLEANUP_START/FINISH
 
@@ -67,20 +79,31 @@ class Python35Parser(Python34Parser):
                                GET_AWAITABLE LOAD_CONST YIELD_FROM
                                WITH_CLEANUP_FINISH END_FINALLY
 
-
         stmt               ::= async_for_stmt
         async_for_stmt     ::= SETUP_LOOP expr
                                GET_AITER
                                LOAD_CONST YIELD_FROM SETUP_EXCEPT GET_ANEXT LOAD_CONST
                                YIELD_FROM
                                designator
-                               POP_BLOCK JUMP_FORWARD COME_FROM_EXCEPT DUP_TOP
+                               POP_BLOCK jump_except COME_FROM_EXCEPT DUP_TOP
                                LOAD_GLOBAL COMPARE_OP POP_JUMP_IF_FALSE
                                POP_TOP POP_TOP POP_TOP POP_EXCEPT POP_BLOCK
                                JUMP_ABSOLUTE END_FINALLY COME_FROM
                                for_block POP_BLOCK JUMP_ABSOLUTE
                                opt_come_from_loop
 
+        async_for_stmt     ::= SETUP_LOOP expr
+                               GET_AITER
+                               LOAD_CONST YIELD_FROM SETUP_EXCEPT GET_ANEXT LOAD_CONST
+                               YIELD_FROM
+                               designator
+                               POP_BLOCK jump_except COME_FROM_EXCEPT DUP_TOP
+                               LOAD_GLOBAL COMPARE_OP POP_JUMP_IF_FALSE
+                               POP_TOP POP_TOP POP_TOP POP_EXCEPT POP_BLOCK
+                               JUMP_ABSOLUTE END_FINALLY JUMP_BACK
+                               passstmt POP_BLOCK JUMP_ABSOLUTE
+                               opt_come_from_loop
+
         stmt               ::= async_forelse_stmt
         async_forelse_stmt ::= SETUP_LOOP expr
                                GET_AITER
@@ -114,7 +137,6 @@ class Python35Parser(Python34Parser):
         # differently than 3.3, 3.4
 
         yield_from ::= expr GET_YIELD_FROM_ITER LOAD_CONST YIELD_FROM
-
         """
 
     def add_custom_rules(self, tokens, customize):

uncompyle6/parsers/parse36.py

@@ -15,6 +15,8 @@ class Python36Parser(Python35Parser):
 
     def p_36misc(self, args):
         """
+        expr ::= LOAD_NAME EXTENDED_ARG
+
         fstring_multi ::= fstring_expr_or_strs BUILD_STRING
         fstring_expr_or_strs ::= fstring_expr_or_str+
 
@@ -23,12 +25,16 @@ class Python36Parser(Python35Parser):
 
         withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST
                      WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+
+        call_function ::= expr expr CALL_FUNCTION_EX
+        call_function ::= expr expr expr CALL_FUNCTION_EX_KW
         """
 
     def add_custom_rules(self, tokens, customize):
         super(Python36Parser, self).add_custom_rules(tokens, customize)
         for i, token in enumerate(tokens):
             opname = token.type
+
             if opname == 'FORMAT_VALUE':
                 rules_str = """
                     expr ::= fstring_single

uncompyle6/scanners/scanner3.py

@@ -231,6 +231,8 @@ class Scanner3(Scanner):
                         pass
                     elif inst.offset in self.except_targets:
                         come_from_name = 'COME_FROM_EXCEPT_CLAUSE'
+                        if self.version <= 3.2:
+                            continue
                         pass
                     tokens.append(Token(come_from_name,
                                         None, repr(jump_offset),
@@ -900,7 +902,7 @@ class Scanner3(Scanner):
                                 return
                             pass
                     pass
-                if code[pre_rtarget] == self.opc.RETURN_VALUE:
+                if code[pre_rtarget] == self.opc.RETURN_VALUE and self.version < 3.5:
                     self.return_end_ifs.add(pre_rtarget)
                 else:
                     self.fixed_jumps[offset] = rtarget

uncompyle6/scanners/scanner36.py

@@ -1,8 +1,8 @@
 #  Copyright (c) 2016 by Rocky Bernstein
 """
-Python 3.5 bytecode scanner/deparser
+Python 3.6 bytecode scanner/deparser
 
-This sets up opcodes Python's 3.5 and calls a generalized
+This sets up opcodes Python's 3.6 and calls a generalized
 scanner routine for Python 3.
 """
 
@@ -17,6 +17,18 @@ class Scanner36(Scanner3):
     def __init__(self, show_asm=None):
         Scanner3.__init__(self, 3.6, show_asm)
         return
+
+    def ingest(self, co, classname=None, code_objects={}, show_asm=None):
+        tokens, customize = Scanner3.ingest(self, co, classname, code_objects, show_asm)
+        for t in tokens:
+            # The lowest bit of flags indicates whether the
+            # var-keyword argument is placed at the top of the stack
+            if t.op == self.opc.CALL_FUNCTION_EX and t.attr & 1:
+                t.type = 'CALL_FUNCTION_EX_KW'
+                pass
+            pass
+        return tokens, customize
+
     pass
 
 if __name__ == "__main__":

uncompyle6/semantics/make_function.py

@@ -576,7 +576,7 @@ def make_function3(self, node, isLambda, nested=1, codeNode=None):
             for n in node:
                 if n == 'pos_arg':
                     continue
-                elif self.version >= 3.4 and n.type != 'kwargs':
+                elif self.version >= 3.4 and not (n.type in ('kwargs', 'kwarg')):
                     continue
                 else:
                     self.preorder(n)

uncompyle6/semantics/pysource.py

@@ -202,11 +202,17 @@ class SourceWalker(GenericASTTraversal, object):
                 'raise_stmt2':	( '%|raise %c, %c\n', 0, 1),
             })
         else:
-            # Gotta love Python for its futzing around with syntax like this
             TABLE_DIRECT.update({
-                'raise_stmt2':	( '%|raise %c from %c\n', 0, 1),
+                # Gotta love Python for its futzing around with syntax like this
+                'raise_stmt2':	 ( '%|raise %c from %c\n', 0, 1),
             })
 
+            if version >= 3.2:
+                TABLE_DIRECT.update({
+                'del_deref_stmt': ( '%|del %c\n', 0),
+                'DELETE_DEREF': ( '%{pattr}', 0 ),
+                })
+
         if version < 2.0:
             TABLE_DIRECT.update({
                 'importlist':	( '%C', (0, maxint, ', ') ),
@@ -327,7 +333,8 @@ class SourceWalker(GenericASTTraversal, object):
                 #######################
                 if version >= 3.5:
                     TABLE_DIRECT.update({
-                        'await_stmt':	       ( '%|await %c', 0),
+                        'await_expr':	       ( 'await %c', 0),
+                        'await_stmt':	       ( '%|%c', 0),
                         'async_for_stmt':      (
                             '%|async for %c in %c:\n%+%c%-\n\n', 9, 1, 25 ),
                         'async_forelse_stmt':  (
@@ -336,6 +343,8 @@ class SourceWalker(GenericASTTraversal, object):
                             '%|async with %c:\n%+%c%-', 0, 7),
                         'async_with_as_stmt':  (
                             '%|async with %c as %c:\n%+%c%-', 0, 6, 7),
+                        'unmap_dict':	       ( '{**%C}', (0, -1, ', **') ),
+                        'unmapexpr':	       ( '{**%c}', 0),
 
                     })
                     def n_async_call_function(node):
@@ -382,7 +391,11 @@ class SourceWalker(GenericASTTraversal, object):
                         'fstring_multi':  ( "f'%c'", 0),
                         'func_args36':    ( "%c(**", 0),
                     })
-
+                    TABLE_R.update({
+                        'CALL_FUNCTION_EX': ('%c(*%P)', 0, (1, 2, ', ', 100)),
+                        # Not quite right
+                        'CALL_FUNCTION_EX_KW': ('%c(**%C', 0, (2,3, ',')),
+                        })
                     FSTRING_CONVERSION_MAP = {1: '!s', 2: '!r', 3: '!a'}
 
                     def f_conversion(node):
@@ -1439,7 +1452,7 @@ class SourceWalker(GenericASTTraversal, object):
                     i += 2
                     pass
                 pass
-            elif node[1].type.startswith('kvlist'):
+            elif len(node) > 1 and node[1].type.startswith('kvlist'):
                 # Python 3.0..3.4 style key/value list in mapexpr
                 kv_node = node[1]
                 l = list(kv_node)
@@ -1553,22 +1566,22 @@ class SourceWalker(GenericASTTraversal, object):
         # will assume that if the text ends in *.
         last_was_star = self.f.getvalue().endswith('*')
 
+        if lastnodetype.startswith('BUILD_LIST'):
+            self.write('['); endchar = ']'
+        elif lastnodetype.startswith('BUILD_TUPLE'):
+            self.write('('); endchar = ')'
+        elif lastnodetype.startswith('BUILD_SET'):
+            self.write('{'); endchar = '}'
+        elif lastnodetype.startswith('BUILD_MAP_UNPACK'):
+            self.write('{*'); endchar = '}'
+        elif lastnodetype.startswith('ROT_TWO'):
+            self.write('('); endchar = ')'
+        else:
+            raise 'Internal Error: n_build_list expects list, tuple, set, or unpack'
         have_star = False
         if lastnodetype.endswith('UNPACK'):
             # FIXME: need to handle range of BUILD_LIST_UNPACK
             have_star = True
-            endchar = ''
-        else:
-            if lastnodetype.startswith('BUILD_LIST'):
-                self.write('['); endchar = ']'
-            elif lastnodetype.startswith('BUILD_TUPLE'):
-                self.write('('); endchar = ')'
-            elif lastnodetype.startswith('BUILD_SET'):
-                self.write('{'); endchar = '}'
-            elif lastnodetype.startswith('ROT_TWO'):
-                self.write('('); endchar = ')'
-            else:
-                raise 'Internal Error: n_build_list expects list or tuple'
 
         flat_elems = []
         for elem in node:
@@ -1717,6 +1730,8 @@ class SourceWalker(GenericASTTraversal, object):
                     remaining -= 1
                     if remaining > 0:
                         self.write(sep)
+                        pass
+                    pass
                 arg += 1
             elif typ == 'D':
                 low, high, sep = entry[arg]

uncompyle6/verify.py

@@ -319,6 +319,9 @@ def cmp_code_objects(version, is_pypy, code_obj1, code_obj2,
                     elif tokens1[i1].type == 'LOAD_GLOBAL' and tokens2[i2].type == 'LOAD_NAME' \
                          and tokens1[i1].pattr == tokens2[i2].pattr:
                         pass
+                    elif tokens1[i1].type == 'LOAD_ASSERT' and tokens2[i2].type == 'LOAD_NAME' \
+                         and tokens1[i1].pattr == tokens2[i2].pattr:
+                        pass
                     elif (tokens1[i1].type == 'RETURN_VALUE' and
                           tokens2[i2].type == 'RETURN_END_IF'):
                         pass