3.x funciton and annotation bug fixes

next_offset, op_size, has_argument

ChangeLog

@@ -1,6 +1,137 @@
+2017-06-25  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/version.py: Get ready for release 2.11.1
+
+2017-06-24  rocky <rb@dustyfeet.com>
+
+	* __pkginfo__.py, uncompyle6/scanner.py,
+	uncompyle6/scanners/scanner2.py, uncompyle6/scanners/scanner3.py,
+	uncompyle6/scanners/scanner30.py, uncompyle6/semantics/pysource.py: 
+	Use xdis' instruction offset calculation fns..  next_offset, op_size, has_argument
+
+2017-06-19  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: Python 2 sometimes need
+	str->uncode in writing?
+
+2017-06-19  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: Allow deparsed out to be str as
+	well as unicode
+
+2017-06-18  rocky <rb@dustyfeet.com>
+
+	* ChangeLog, NEWS, uncompyle6/version.py: Get ready for release
+	2.11.0
+
+2017-06-13  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py: Adjust nodeInfo if it is a
+	Token
+
+2017-06-13  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py: Add nonterminal node in
+	extractInfo
+
+2017-06-10  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py,
+	uncompyle6/semantics/make_function.py: Fragment tag more expressions Revise make_function3 comment wrt args and kwargs
+
+2017-06-10  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py: Fragment tag array subscripts
+
+2017-06-10  R. Bernstein <rocky@users.noreply.github.com>
+
+	* README.rst: Create README.rst
+
+2017-06-10  R. Bernstein <rocky@users.noreply.github.com>
+
+	* README.rst: Create README.rst
+
+2017-06-10  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py: Set YIELD_VALUE offset in a
+	<yield> expr
+
+2017-06-10  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/make_function.py: Python 3.2 MAKE_FUNCTION
+	again..  Was handling bug32/01_named_and_kwargs.py wrong again
+
+2017-06-09  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #119 from rocky/scan-longconstant Simplify access to L65536 ...
+
+2017-06-09  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/make_function.py: Attempt to document the
+	MAKE_FUNCTION/MAKE_LAMBDA mess...  in Python 3.0+
+
+2017-06-08  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/make_function.py: Correct make_function3 for
+	Pytohn 3.2
+
+2017-06-08  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: Disable "continue" removal in
+	pysource.py "continue" could be the only statement and then removing it might
+	lead to a dangling "else".
+
+2017-06-07  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py: Mark "pass" offsets.  Start routine to find previous node.
+
+2017-06-06  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py, uncompyle6/semantics/fragments.py: 
+	Remove hacky fragments try fixup...  hacky call_function code is also not needed or will be reinstated
+	properly. Better grammar structure for Python 3.6 call_function.
+
+2017-06-05  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py, uncompyle6/parsers/parse36.py,
+	uncompyle6/scanners/scanner36.py: BUILD_{MAP,TUPLE}_UNPACK &
+	CALL_FUNCTION_EX_KW...  Bang on these in 3.6. Not totally succesfull right now.  In fact a
+	regression on one of the test cases
+
+2017-06-05  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py: Important fragments bug fix...  start, finish that had been adjusted wasn't getting reflected in
+	final returned deparsed.offsets dictionary. Redo keeping API
+	compatibility, i.e we still use namedtuple NodeInfo.
+
+2017-06-04  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py, uncompyle6/semantics/pysource.py: 
+	Python 3.5 *args with kwargs handling.  3.5 is a snowflake here. Thank you, Python.  Fully fixes Issue 95.  3.6 is broken on this source, but for a *different* reason. Sigh.
+
 2017-06-03  rocky <rb@dustyfeet.com>
 
-	* uncompyle6/version.py: Get ready for release 2.10.1
+	* README.rst, __pkginfo__.py,
+	test/simple_source/bug35/04_CALL_FUNCTION_VAR_KW.py,
+	uncompyle6/semantics/fragments.py: Small changes.  fragment tag EXEC_STMT
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* .travis.yml: Streamline .travis.yml a little bit
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* __pkginfo__.py: We need six
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* README.rst, circle.yml, requirements-dev.txt: Go over
+	administrivia
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* ChangeLog, NEWS, uncompyle6/version.py: Get ready for release
+	2.10.1
 
 2017-06-03  rocky <rb@dustyfeet.com>
 

NEWS

@@ -1,3 +1,19 @@
+uncompyle6 2.11.1 2016-06-22
+
+- Python 3.x annotatoin and function signature fixes
+- Bump xdis version
+- Small pysource bug fixes
+
+uncompyle6 2.11.0 2016-06-18 Fleetwood
+- Major improvements in fragment tracking
+  * Add nonterminal node in extractInfo
+  * tag more offsets in expressions
+  * tag array subscripts
+  * set YIELD value offset in a <yield> expr
+  * fix a long-standing bug in not adjusting final AST when melding other deparse ASTs
+- Fixes yet again for make_function node handling; document what's up here
+- Fix bug in snowflake Python 3.5 *args kwargs
+
 uncompyle6 2.10.1 2016-06-3 Marylin Frankel
 
 - fix some fragments parsing bugs

README.rst

@@ -56,7 +56,7 @@ This uses setup.py, so it follows the standard Python routine:
 
 ::
 
-    pip install -e setup.py
+    pip install -e .
     pip install -r requirements-dev.txt
     python setup.py install # may need sudo
     # or if you have pyenv:
@@ -171,7 +171,7 @@ See Also
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Include some fixes like supporting function annotations
 * The HISTORY_ file.
-
+* `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md>`_
 .. |downloads| image:: https://img.shields.io/pypi/dd/uncompyle6.svg
 .. _trepan: https://pypi.python.org/pypi/trepan
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md

__pkginfo__.py

@@ -33,14 +33,14 @@ classifiers =  ['Development Status :: 5 - Production/Stable',
 # The rest in alphabetic order
 author             = "Rocky Bernstein, Hartmut Goebel, John Aycock, and others"
 author_email       = "rb@dustyfeet.com"
-entry_points={
+entry_points       = {
     'console_scripts': [
         'uncompyle6=uncompyle6.bin.uncompile:main_bin',
         'pydisassemble=uncompyle6.bin.pydisassemble:main',
     ]}
 ftp_url            = None
 install_requires   = ['spark-parser >= 1.6.1, < 1.7.0',
-                      'xdis >= 3.3.1, < 3.4.0', 'six']
+                      'xdis >= 3.4.0, < 3.5.0', 'six']
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'

test/simple_source/bug31/04_def_annotate.py

@@ -9,7 +9,7 @@ def open(file, mode = "r", buffering = None,
          newline = None, closefd = True) -> "IOBase":
     return text
 
-def foo(x: 'an argument that defaults to 5' = 5):
+def foo1(x: 'an argument that defaults to 5' = 5):
     print(x)
 
 def div(a: dict(type=float, help='the dividend'),

test/simple_source/bug35/04_CALL_FUNCTION_VAR_KW.py

@@ -1,4 +1,5 @@
 # sql/schema.py
+# Note that kwargs comes before "positional" args
 def tometadata(self, metadata, schema, Table, args, name=None):
     table = Table(
         name, metadata, schema=schema,

uncompyle6/parsers/parse3.py

@@ -496,8 +496,8 @@ class Python3Parser(PythonParser):
         token.type = self.call_fn_name(token)
         uniq_param = args_kw + args_pos
         if self.version == 3.5 and opname.startswith('CALL_FUNCTION_VAR'):
-            # Python 3.5 changes the stack position of where * args, the
-            # first LOAD_FAST, below are located.
+            # Python 3.5 changes the stack position of *args. KW args come
+            # after *args.
             # Python 3.6+ replaces CALL_FUNCTION_VAR_KW with CALL_FUNCTION_EX
             if opname.endswith('KW'):
                 kw = 'expr '
@@ -507,7 +507,13 @@ class Python3Parser(PythonParser):
                     ('pos_arg ' * args_pos) +
                     ('kwarg ' * args_kw) + kw + token.type)
             self.add_unique_rule(rule, token.type, uniq_param, customize)
-
+        if self.version >= 3.6 and opname == 'CALL_FUNCTION_EX_KW':
+            rule = ('call_function36 ::= '
+                    'expr build_tuple_unpack_with_call build_map_unpack_with_call '
+                    'CALL_FUNCTION_EX_KW_1')
+            self.add_unique_rule(rule, token.type, uniq_param, customize)
+            rule = 'call_function ::= call_function36'
+        else:
             rule = ('call_function ::= expr ' +
                     ('pos_arg ' * args_pos) +
                     ('kwarg ' * args_kw) +
@@ -610,9 +616,9 @@ class Python3Parser(PythonParser):
                     assign2_pypy ::= expr expr designator designator
                 """, nop_func)
                 continue
-            elif opname in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
-                            'CALL_FUNCTION_VAR_KW') \
-                 or opname.startswith('CALL_FUNCTION_KW'):
+            elif (opname in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
+                             'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_EX_KW')
+                  or opname.startswith('CALL_FUNCTION_KW')):
                 self.custom_classfunc_rule(opname, token, customize)
             elif opname == 'LOAD_DICTCOMP':
                 rule_pat = ("dictcomp ::= LOAD_DICTCOMP %sMAKE_FUNCTION_0 expr "
@@ -633,6 +639,18 @@ class Python3Parser(PythonParser):
                 self.add_unique_rule(rule, opname, token.attr, customize)
                 rule = 'expr ::= build_list_unpack'
                 self.add_unique_rule(rule, opname, token.attr, customize)
+            elif opname.startswith('BUILD_TUPLE_UNPACK_WITH_CALL'):
+                v = token.attr
+                rule = ('build_tuple_unpack_with_call ::= ' + 'expr1024 ' * int(v//1024) +
+                        'expr32 ' * int((v//32) % 32) +
+                        'expr ' * (v % 32) + opname)
+                self.add_unique_rule(rule, opname, token.attr, customize)
+            elif opname.startswith('BUILD_MAP_UNPACK_WITH_CALL'):
+                v = token.attr
+                rule = ('build_map_unpack_with_call ::= ' + 'expr1024 ' * int(v//1024) +
+                        'expr32 ' * int((v//32) % 32) +
+                        'expr ' * (v % 32) + opname)
+                self.add_unique_rule(rule, opname, token.attr, customize)
             elif opname_base in ('BUILD_LIST', 'BUILD_TUPLE', 'BUILD_SET'):
                 v = token.attr
                 rule = ('build_list ::= ' + 'expr1024 ' * int(v//1024) +
@@ -642,7 +660,10 @@ class Python3Parser(PythonParser):
                 if opname_base == 'BUILD_TUPLE':
                     rule = ('load_closure ::= %s%s' % (('LOAD_CLOSURE ' * v), opname))
                     self.add_unique_rule(rule, opname, token.attr, customize)
-
+                    rule = ('build_tuple ::= ' + 'expr1024 ' * int(v//1024) +
+                        'expr32 ' * int((v//32) % 32) +
+                        'expr ' * (v % 32) + opname)
+                    self.add_unique_rule(rule, opname, token.attr, customize)
             elif opname == 'LOOKUP_METHOD':
                 # A PyPy speciality - DRY with parse2
                 self.add_unique_rule("load_attr ::= expr LOOKUP_METHOD",

uncompyle6/parsers/parse36.py

@@ -25,12 +25,13 @@ class Python36Parser(Python35Parser):
 
         func_args36   ::= expr BUILD_TUPLE_0
         call_function ::= func_args36 unmapexpr CALL_FUNCTION_EX
+        call_function ::= func_args36 build_map_unpack_with_call CALL_FUNCTION_EX_KW_1
 
         withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST
                      WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
         call_function ::= expr expr CALL_FUNCTION_EX
-        call_function ::= expr expr expr CALL_FUNCTION_EX_KW
+        call_function ::= expr expr expr CALL_FUNCTION_EX_KW_1
         """
 
     def add_custom_rules(self, tokens, customize):

uncompyle6/scanner.py

@@ -16,6 +16,7 @@ import sys
 
 from uncompyle6 import PYTHON3, IS_PYPY
 from uncompyle6.scanners.tok import Token
+from xdis.bytecode import op_size
 
 # The byte code versions we support
 PYTHON_VERSIONS = (1.5,
@@ -214,9 +215,6 @@ class Scanner(object):
                         result.append(offset)
         return result
 
-    def op_hasArgument(self, op):
-        return self.op_size(op) > 1
-
     def op_range(self, start, end):
         """
         Iterate through positions of opcodes, skipping
@@ -224,20 +222,7 @@ class Scanner(object):
         """
         while start < end:
             yield start
-            start += self.op_size(self.code[start])
-
-    def next_offset(self, op, offset):
-        return offset + self.op_size(op)
-
-    def op_size(self, op):
-        """
-        Return size of operator with its arguments
-        for given opcode <op>.
-        """
-        if op < self.opc.HAVE_ARGUMENT:
-            return 2 if self.version >= 3.6 else 1
-        else:
-            return 2 if self.version >= 3.6 else 3
+            start += op_size(self.code[start], self.opc)
 
     def remove_mid_line_ifs(self, ifs):
         """
@@ -269,9 +254,6 @@ class Scanner(object):
         self.Token = tokenClass
         return self.Token
 
-def op_has_argument(op, opc):
-    return op >= opc.HAVE_ARGUMENT
-
 def parse_fn_counts(argc):
     return ((argc & 0xFF), (argc >> 8) & 0xFF, (argc >> 16) & 0x7FFF)
 

uncompyle6/scanners/scanner2.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2015, 2016 by Rocky Bernstein
+# Copyright (c) 2015-2017 by Rocky Bernstein
 # Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 # Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 """
@@ -25,14 +25,15 @@ from __future__ import print_function
 from collections import namedtuple
 from array import array
 
-from uncompyle6.scanner import op_has_argument
+from uncompyle6.scanner import L65536
 from xdis.code import iscode
+from xdis.bytecode import op_has_argument, op_size
 
-import uncompyle6.scanner as scan
+from uncompyle6.scanner import Scanner
 
-class Scanner2(scan.Scanner):
+class Scanner2(Scanner):
     def __init__(self, version, show_asm=None, is_pypy=False):
-        scan.Scanner.__init__(self, version, show_asm, is_pypy)
+        Scanner.__init__(self, version, show_asm, is_pypy)
         self.pop_jump_if = frozenset([self.opc.PJIF, self.opc.PJIT])
         self.jump_forward = frozenset([self.opc.JUMP_ABSOLUTE, self.opc.JUMP_FORWARD])
         # This is the 2.5+ default
@@ -186,7 +187,7 @@ class Scanner2(scan.Scanner):
                 oparg = self.get_argument(offset) + extended_arg
                 extended_arg = 0
                 if op == self.opc.EXTENDED_ARG:
-                    extended_arg = oparg * scan.L65536
+                    extended_arg = oparg * L65536
                     continue
                 if op in self.opc.hasconst:
                     const = co.co_consts[oparg]
@@ -327,7 +328,7 @@ class Scanner2(scan.Scanner):
         for i in self.op_range(0, n):
             op = self.code[i]
             self.prev.append(i)
-            if self.op_hasArgument(op):
+            if op_has_argument(op, self.opc):
                 self.prev.append(i)
                 self.prev.append(i)
                 pass
@@ -380,7 +381,7 @@ class Scanner2(scan.Scanner):
                     if elem != code[i]:
                         match = False
                         break
-                    i += self.op_size(code[i])
+                    i += op_size(code[i], self.opc)
 
                 if match:
                     i = self.prev[i]
@@ -617,7 +618,7 @@ class Scanner2(scan.Scanner):
                                        'start': jump_back+3,
                                        'end':   end})
         elif op == self.opc.SETUP_EXCEPT:
-            start  = offset + self.op_size(op)
+            start  = offset + op_size(op, self.opc)
             target = self.get_target(offset, op)
             end    = self.restrict_to_parent(target, parent)
             if target != end:
@@ -641,7 +642,7 @@ class Scanner2(scan.Scanner):
                         setup_except_nest -= 1
                 elif self.code[end_finally_offset] == self.opc.SETUP_EXCEPT:
                     setup_except_nest += 1
-                end_finally_offset += self.op_size(code[end_finally_offset])
+                end_finally_offset += op_size(code[end_finally_offset], self.opc)
                 pass
 
             # Add the except blocks
@@ -842,7 +843,7 @@ class Scanner2(scan.Scanner):
                     else:
                         # We still have the case in 2.7 that the next instruction
                         # is a jump to a SETUP_LOOP target.
-                        next_offset = target + self.op_size(self.code[target])
+                        next_offset = target + op_size(self.code[target], self.opc)
                         next_op = self.code[next_offset]
                         if self.op_name(next_op) == 'JUMP_FORWARD':
                             jump_target = self.get_target(next_offset, next_op)

uncompyle6/scanners/scanner26.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015, 2016 by Rocky Bernstein
+#  Copyright (c) 2015-2017 by Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 """
@@ -15,6 +15,7 @@ if PYTHON3:
     intern = sys.intern
 
 import uncompyle6.scanners.scanner2 as scan
+from uncompyle6.scanner import L65536
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_26
@@ -178,7 +179,7 @@ class Scanner26(scan.Scanner2):
                 oparg = self.get_argument(offset) + extended_arg
                 extended_arg = 0
                 if op == self.opc.EXTENDED_ARG:
-                    extended_arg = oparg * scan.L65536
+                    extended_arg = oparg * L65536
                     continue
                 if op in self.opc.hasconst:
                     const = co.co_consts[oparg]

uncompyle6/scanners/scanner3.py

@@ -25,10 +25,11 @@ from __future__ import print_function
 from collections import namedtuple
 from array import array
 
-from uncompyle6.scanner import Scanner, op_has_argument
+from uncompyle6.scanner import Scanner
 from xdis.code import iscode
-from xdis.bytecode import Bytecode
+from xdis.bytecode import Bytecode, op_has_argument, op_size
 from uncompyle6.scanner import Token, parse_fn_counts
+import xdis
 
 # Get all the opcodes into globals
 import xdis.opcodes.opcode_33 as op3
@@ -468,7 +469,7 @@ class Scanner3(Scanner):
         self.prev = self.prev_op = [0]
         for offset in self.op_range(0, codelen):
             op = code[offset]
-            for _ in range(self.op_size(op)):
+            for _ in range(op_size(op, self.opc)):
                 self.prev_op.append(offset)
 
     def find_jump_targets(self, debug):
@@ -518,7 +519,7 @@ class Scanner3(Scanner):
                     oparg = code[offset+1]
                 else:
                     oparg = code[offset+1] + code[offset+2] * 256
-                next_offset = self.next_offset(op, offset)
+                next_offset = xdis.next_offset(op, self.opc, offset)
 
                 if label is None:
                     if op in op3.hasjrel and op != self.opc.FOR_ITER:
@@ -564,7 +565,7 @@ class Scanner3(Scanner):
                     if elem != code[i]:
                         match = False
                         break
-                    i += self.op_size(code[i])
+                    i += op_size(code[i], self.opc)
 
                 if match is True:
                     i = self.prev_op[i]
@@ -757,7 +758,7 @@ class Scanner3(Scanner):
                                        'start': jump_back+3,
                                        'end':   end})
         elif op in self.pop_jump_tf:
-            start = offset + self.op_size(op)
+            start = offset + op_size(op, self.opc)
             target = self.get_target(offset)
             rtarget = self.restrict_to_parent(target, parent)
             prev_op = self.prev_op
@@ -932,9 +933,9 @@ class Scanner3(Scanner):
                     # not from SETUP_EXCEPT
                     next_op = rtarget
                     if code[next_op] == self.opc.POP_BLOCK:
-                        next_op += self.op_size(self.code[next_op])
+                        next_op += op_size(self.code[next_op], self.opc)
                     if code[next_op] == self.opc.JUMP_ABSOLUTE:
-                        next_op += self.op_size(self.code[next_op])
+                        next_op += op_size(self.code[next_op], self.opc)
                     if next_op in targets:
                         for try_op in targets[next_op]:
                             come_from_op = code[try_op]
@@ -957,12 +958,12 @@ class Scanner3(Scanner):
             end    = self.restrict_to_parent(target, parent)
             self.fixed_jumps[offset] = end
         elif op == self.opc.POP_EXCEPT:
-            next_offset = self.next_offset(op, offset)
+            next_offset = xdis.next_offset(op, self.opc, offset)
             target = self.get_target(next_offset)
             if target > next_offset:
                 next_op = code[next_offset]
                 if (self.opc.JUMP_ABSOLUTE == next_op and
-                    END_FINALLY != code[self.next_offset(next_op, next_offset)]):
+                    END_FINALLY != code[xdis.next_offset(next_op, self.opc, next_offset)]):
                     self.fixed_jumps[next_offset] = target
                     self.except_targets[target] = next_offset
 

uncompyle6/scanners/scanner30.py

@@ -10,6 +10,8 @@ from __future__ import print_function
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_30 as opc
+from xdis.bytecode import op_size
+
 JUMP_OPs = map(lambda op: opc.opname[op], opc.hasjrel + opc.hasjabs)
 
 JUMP_TF = frozenset([opc.JUMP_IF_FALSE, opc.JUMP_IF_TRUE])
@@ -133,7 +135,7 @@ class Scanner30(Scanner3):
                                        'start': jump_back+3,
                                        'end':   end})
         elif op in JUMP_TF:
-            start = offset + self.op_size(op)
+            start = offset + op_size(op, self.opc)
             target = self.get_target(offset)
             rtarget = self.restrict_to_parent(target, parent)
             prev_op = self.prev_op
@@ -305,9 +307,9 @@ class Scanner30(Scanner3):
                     # not from SETUP_EXCEPT
                     next_op = rtarget
                     if code[next_op] == self.opc.POP_BLOCK:
-                        next_op += self.op_size(self.code[next_op])
+                        next_op += op_size(self.code[next_op], self.opc)
                     if code[next_op] == self.opc.JUMP_ABSOLUTE:
-                        next_op += self.op_size(self.code[next_op])
+                        next_op += op_size(self.code[next_op], self.opc)
                     if next_op in targets:
                         for try_op in targets[next_op]:
                             come_from_op = code[try_op]

uncompyle6/scanners/scanner36.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016 by Rocky Bernstein
+#  Copyright (c) 2016-2017 by Rocky Bernstein
 """
 Python 3.6 bytecode scanner/deparser
 
@@ -28,8 +28,12 @@ class Scanner36(Scanner3):
             if t.op == self.opc.CALL_FUNCTION_EX and t.attr & 1:
                 t.type = 'CALL_FUNCTION_EX_KW'
                 pass
-            if t.op == self.opc.CALL_FUNCTION_KW:
+            elif t.op == self.opc.CALL_FUNCTION_KW:
                 t.type = 'CALL_FUNCTION_KW_{t.attr}'.format(**locals())
+            elif t.op == self.opc.BUILD_TUPLE_UNPACK_WITH_CALL:
+                t.type = 'BUILD_TUPLE_UNPACK_WITH_CALL_%d' % t.attr
+            elif t.op == self.opc.BUILD_MAP_UNPACK_WITH_CALL:
+                t.type = 'BUILD_MAP_UNPACK_WITH_CALL_%d' % t.attr
             pass
         return tokens, customize
 

uncompyle6/semantics/fragments.py

@@ -44,7 +44,7 @@ do it recursively which is where offsets are probably located.
 
 For example in:
   'importmultiple':   ( '%|import%b %c%c\n', 0, 2, 3 ),
-
+n
 The node position 0 will be associated with "import".
 
 """
@@ -77,11 +77,12 @@ from uncompyle6.semantics.consts import (
     )
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser.ast import GenericASTTraversalPruningException
 
 from collections import namedtuple
 NodeInfo = namedtuple("NodeInfo", "node start finish")
 ExtractInfo = namedtuple("ExtractInfo",
-                         "lineNo lineStartOffset markerLine selectedLine selectedText")
+                         "lineNo lineStartOffset markerLine selectedLine selectedText nonterminal")
 
 TABLE_DIRECT_FRAGMENT = {
     'break_stmt':	( '%|%rbreak\n', ),
@@ -159,8 +160,9 @@ class FragmentsWalker(pysource.SourceWalker, object):
     def set_pos_info(self, node, start, finish, name=None):
         if name is None: name = self.name
         if hasattr(node, 'offset'):
-            self.offsets[name, node.offset] = \
-              NodeInfo(node = node, start = start, finish = finish)
+            node.start = start
+            node.finish = finish
+            self.offsets[name, node.offset] = node
 
         if hasattr(node, 'parent'):
             assert node.parent != node
@@ -176,6 +178,34 @@ class FragmentsWalker(pysource.SourceWalker, object):
 
         return
 
+    def table_r_node(self, node):
+        """General pattern where the last node should should
+        get the text span attributes of the entire tree"""
+        start = len(self.f.getvalue())
+        try:
+            self.default(node)
+        except GenericASTTraversalPruningException:
+            final = len(self.f.getvalue())
+            self.set_pos_info(node, start, final)
+            self.set_pos_info(node[-1], start, final)
+            raise GenericASTTraversalPruningException
+
+    n_slice0 = n_slice1 = n_slice2 = n_slice3 = n_binary_subscr = table_r_node
+    n_augassign_1 = n_print_item = exec_stmt = print_to_item = del_stmt = table_r_node
+    n_classdefco1 = n_classdefco2 = except_cond1 = except_cond2 = table_r_node
+
+    def n_passtmt(self, node):
+        start = len(self.f.getvalue()) + len(self.indent)
+        self.set_pos_info(node, start, start+len("pass"))
+        self.default(node)
+
+    def n_trystmt(self, node):
+        start = len(self.f.getvalue()) + len(self.indent)
+        self.set_pos_info(node[0], start, start+len("try:"))
+        self.default(node)
+
+    n_tryelsestmt = n_tryelsestmtc = n_tryelsestmtl = n_tryfinallystmt = n_trystmt
+
     def n_return_stmt(self, node):
         start = len(self.f.getvalue()) + len(self.indent)
         if self.params['isLambda']:
@@ -229,6 +259,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
             self.write(' ')
             node[0].parent = node
             self.preorder(node[0])
+        self.set_pos_info(node[-1], start, len(self.f.getvalue()))
         self.set_pos_info(node, start, len(self.f.getvalue()))
         self.prune() # stop recursing
 
@@ -366,6 +397,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
                 self.write(sep); sep = ", "
                 self.preorder(subnode)
         self.set_pos_info(node, start, len(self.f.getvalue()))
+        self.set_pos_info(node[-1], start, len(self.f.getvalue()))
         self.println()
         self.prune() # stop recursing
 
@@ -1089,7 +1121,6 @@ class FragmentsWalker(pysource.SourceWalker, object):
     def traverse(self, node, indent=None, isLambda=False):
         '''Buulds up fragment which can be used inside a larger
         block of code'''
-
         self.param_stack.append(self.params)
         if indent is None: indent = self.indent
         p = self.pending_newlines
@@ -1184,16 +1215,38 @@ class FragmentsWalker(pysource.SourceWalker, object):
 
         if elided: selectedLine += ' ...'
 
+        if isinstance(nodeInfo, Token):
+            nodeInfo = nodeInfo.parent
+        else:
+            nodeInfo = nodeInfo
+
+        if isinstance(nodeInfo, AST):
+            nonterminal = nodeInfo[0]
+        else:
+            nonterminal = nodeInfo.node
+
         return ExtractInfo(lineNo = len(lines), lineStartOffset = lineStart,
                            markerLine = markerLine,
                            selectedLine = selectedLine,
-                           selectedText = selectedText)
+                           selectedText = selectedText,
+                           nonterminal = nonterminal)
 
     def extract_line_info(self, name, offset):
         if (name, offset) not in list(self.offsets.keys()):
             return None
         return self.extract_node_info(self.offsets[name, offset])
 
+    def prev_node(self, node):
+        prev = None
+        if not hasattr(node, 'parent'):
+            return prev
+        p = node.parent
+        for n in p:
+            if node == n:
+                return prev
+            prev = n
+        return prev
+
     def extract_parent_info(self, node):
         if not hasattr(node, 'parent'):
             return None, None
@@ -1557,20 +1610,9 @@ class FragmentsWalker(pysource.SourceWalker, object):
             self.set_pos_info(startnode, startnode_start, fin)
 
         # FIXME rocky: figure out how to get these casess to be table driven.
-        #
-        # 1. for loops. For loops have two positions that correspond to a single text
-        # location. In "for i in ..." there is the initialization "i" code as well
-        # as the iteration code with "i".  A "copy" spec like %X3,3 - copy parame
-        # 3 to param 2 would work
-        #
         # 2. subroutine calls. It the last op is the call and for purposes of printing
         # we don't need to print anything special there. However it encompases the
         # entire string of the node fn(...)
-        match = re.search(r'^try', startnode.type)
-        if match:
-            self.set_pos_info(node[0], startnode_start, startnode_start+len("try:"))
-            self.set_pos_info(node[2], node[3].finish, node[3].finish)
-        else:
         match = re.search(r'^call_function', startnode.type)
         if match:
             last_node = startnode[-1]
@@ -1657,6 +1699,13 @@ def deparse_code(version, co, out=StringIO(), showasm=False, showast=False,
     if deparsed.ERROR:
         raise deparsed.ERROR
 
+    # To keep the API consistent with previous releases, convert
+    # deparse.offset values into NodeInfo items
+    for tup, node in deparsed.offsets.items():
+        deparsed.offsets[tup] = NodeInfo(node = node, start = node.start,
+                                         finish = node.finish)
+
+    deparsed.scanner = scanner
     return deparsed
 
 from bisect import bisect_right
@@ -1691,6 +1740,7 @@ def deparse_code_around_offset(name, offset, version, co, out=StringIO(),
 
 if __name__ == '__main__':
 
+    from uncompyle6 import IS_PYPY
     def deparse_test(co, is_pypy=IS_PYPY):
         sys_version = sys.version_info.major + (sys.version_info.minor / 10.0)
         walk = deparse_code(sys_version, co, showasm=False, showast=False,

uncompyle6/semantics/make_function.py

@@ -163,6 +163,9 @@ def make_function3_annotate(self, node, isLambda, nested=1,
 
     i = len(paramnames) - len(defparams)
     suffix = ''
+
+    no_paramnames = len(paramnames[:i]) == 0
+
     for param in paramnames[:i]:
         self.write(suffix, param)
         suffix = ', '
@@ -182,6 +185,7 @@ def make_function3_annotate(self, node, isLambda, nested=1,
     suffix = ', ' if i > 0 else ''
     for n in node:
         if n == 'pos_arg':
+            no_paramnames = False
             self.write(suffix)
             param = paramnames[i]
             self.write(param)
@@ -190,6 +194,10 @@ def make_function3_annotate(self, node, isLambda, nested=1,
                 if isinstance(aa, tuple):
                     aa = aa[0]
                     self.write(': "%s"' % aa)
+                elif isinstance(aa, AST):
+                    self.write(': ')
+                    self.preorder(aa)
+
             self.write('=')
             i += 1
             self.preorder(n)
@@ -202,9 +210,9 @@ def make_function3_annotate(self, node, isLambda, nested=1,
 
     # self.println(indent, '#flags:\t', int(code.co_flags))
     if kw_args + annotate_argc > 0:
+        if no_paramnames:
             if not code_has_star_arg(code):
                 if argc > 0:
-
                     self.write(", *, ")
                 else:
                     self.write("*, ")
@@ -249,6 +257,7 @@ def make_function3_annotate(self, node, isLambda, nested=1,
                 pass
             pass
 
+
         if code_has_star_star_arg(code):
             if argc > 0:
                 self.write(', ')
@@ -259,7 +268,7 @@ def make_function3_annotate(self, node, isLambda, nested=1,
     else:
         self.write(')')
         if 'return' in annotate_tuple[0].attr:
-            if (line_number != self.line_number):
+            if (line_number != self.line_number) and not no_paramnames:
                 self.write("\n" + indent)
                 line_number = self.line_number
             self.write(' -> ')
@@ -428,10 +437,34 @@ def make_function2(self, node, isLambda, nested=1, codeNode=None):
 
 
 def make_function3(self, node, isLambda, nested=1, codeNode=None):
-    """Dump function definition, doc string, and function body."""
+    """Dump function definition, doc string, and function body in
+      Python version 3.0 and above
+    """
 
-    # FIXME: call make_function3 if we are self.version >= 3.0
-    # and then simplify the below.
+    # For Python 3.3, the evaluation stack in MAKE_FUNCTION is:
+
+    # * default argument objects in positional order
+    # * pairs of name and default argument, with the name just below
+    #   the object on the stack, for keyword-only parameters
+    # * parameter annotation objects
+    # * a tuple listing the parameter names for the annotations
+    #   (only if there are ony annotation objects)
+    # * the code associated with the function (at TOS1)
+    # * the qualified name of the function (at TOS)
+
+    # For Python 3.0 .. 3.2 the evaluation stack is:
+    # The function object is defined to have argc default parameters,
+    # which are found below TOS.
+    # * first come positional args in the order they are given in the source,
+    # * next come the keyword args in the order they given in the source,
+    # * finally is the code associated with the function (at TOS)
+    #
+    # Note: There is no qualified name at TOS
+
+    # MAKE_CLOSURE adds an additional closure slot
+
+    # Thank you, Python, for a such a well-thought out system that has
+    # changed 4 or so times.
 
     def build_param(ast, name, default):
         """build parameters:
@@ -451,21 +484,31 @@ def make_function3(self, node, isLambda, nested=1, codeNode=None):
     # MAKE_FUNCTION_... or MAKE_CLOSURE_...
     assert node[-1].type.startswith('MAKE_')
 
+
+    # Python 3.3+ adds a qualified name at TOS (-1)
+    # moving down the LOAD_LAMBDA instruction
+    if 3.0 <= self.version <= 3.2:
+        lambda_index = -2
+    elif 3.03 <= self.version:
+        lambda_index = -3
+    else:
+        lambda_index = None
+
     args_node = node[-1]
     if isinstance(args_node.attr, tuple):
-        if self.version <= 3.3 and len(node) > 2 and node[-3] != 'LOAD_LAMBDA':
-            # positional args are after kwargs
+        pos_args, kw_args, annotate_argc  = args_node.attr
+        if self.version <= 3.3 and len(node) > 2 and node[lambda_index] != 'LOAD_LAMBDA':
+            # args are after kwargs; kwargs are bundled as one node
             defparams = node[1:args_node.attr[0]+1]
         else:
-            # positional args are before kwargs
+            # args are before kwargs; kwags as bundled as one node
             defparams = node[:args_node.attr[0]]
-        pos_args, kw_args, annotate_argc  = args_node.attr
     else:
         if self.version < 3.6:
             defparams = node[:args_node.attr]
         else:
             default, kw, annotate, closure = args_node.attr
-            # FIXME: start here.
+            # FIXME: start here for Python 3.6 and above:
             defparams = []
             # if default:
             #     defparams = node[-(2 +  kw + annotate  + closure)]
@@ -475,12 +518,6 @@ def make_function3(self, node, isLambda, nested=1, codeNode=None):
         kw_args  = 0
         pass
 
-    if 3.0 <= self.version <= 3.2:
-        lambda_index = -2
-    elif 3.03 <= self.version:
-        lambda_index = -3
-    else:
-        lambda_index = None
 
     if lambda_index and isLambda and iscode(node[lambda_index].attr):
         assert node[lambda_index].type == 'LOAD_LAMBDA'
@@ -496,7 +533,7 @@ def make_function3(self, node, isLambda, nested=1, codeNode=None):
     paramnames = list(code.co_varnames[:argc])
 
     # defaults are for last n parameters, thus reverse
-    if not 3.0 <= self.version <= 3.2:
+    if not 3.0 <= self.version <= 3.1:
         paramnames.reverse(); defparams.reverse()
 
     try:
@@ -510,12 +547,12 @@ def make_function3(self, node, isLambda, nested=1, codeNode=None):
         return
 
     kw_pairs = args_node.attr[1] if self.version >= 3.0 else 0
-    indent = self.indent
 
     # build parameters
-    if self.version != 3.2:
-        params = [build_param(ast, name, default) for
-                  name, default in zip_longest(paramnames, defparams, fillvalue=None)]
+    params = [build_param(ast, name, d) for
+              name, d in zip_longest(paramnames, defparams, fillvalue=None)]
+
+    if not 3.0 <= self.version <= 3.1:
         params.reverse() # back to correct order
 
     if code_has_star_arg(code):
@@ -532,37 +569,6 @@ def make_function3(self, node, isLambda, nested=1, codeNode=None):
         self.write("(", ", ".join(params))
     # self.println(indent, '#flags:\t', int(code.co_flags))
 
-    else:
-        if isLambda:
-            self.write("lambda ")
-        else:
-            self.write("(")
-            pass
-
-        last_line = self.f.getvalue().split("\n")[-1]
-        l = len(last_line)
-        indent = ' ' * l
-        line_number = self.line_number
-
-        if code_has_star_arg(code):
-            self.write('*%s' % code.co_varnames[argc + kw_pairs])
-            argc += 1
-
-        i = len(paramnames) - len(defparams)
-        self.write(", ".join(paramnames[:i]))
-        suffix = ', ' if i > 0 else ''
-        for n in node:
-            if n == 'pos_arg':
-                self.write(suffix)
-                self.write(paramnames[i] + '=')
-                i += 1
-                self.preorder(n)
-                if (line_number != self.line_number):
-                    suffix = ",\n" + indent
-                    line_number = self.line_number
-                else:
-                    suffix = ', '
-
     if kw_args > 0:
         if not (4 & code.co_flags):
             if argc > 0:

uncompyle6/semantics/pysource.py

@@ -311,6 +311,13 @@ class SourceWalker(GenericASTTraversal, object):
                     if node[annotate_last] == 'annotate_tuple':
                         break
 
+                # FIXME: the real situation is that when derived from
+                # funcdef_annotate we the name has been filled in.
+                # But when derived from funcdefdeco it hasn't Would like a better
+                # way to distinquish.
+                if self.f.getvalue()[-4:] == 'def ':
+                    self.write(code.attr.co_name)
+
                 # FIXME: handle and pass full annotate args
                 make_function3_annotate(self, node, isLambda=False,
                                         codeNode=code, annotate_last=annotate_last)
@@ -358,9 +365,33 @@ class SourceWalker(GenericASTTraversal, object):
                         self.prec = p
                         self.prune()
                     self.n_async_call_function = n_async_call_function
-
                     self.n_build_list_unpack = self.n_build_list
 
+                    if version == 3.5:
+                        def n_call_function(node):
+                            mapping = self._get_mapping(node)
+                            table = mapping[0]
+                            key = node
+                            for i in mapping[1:]:
+                                key = key[i]
+                                pass
+                            if key.type.startswith('CALL_FUNCTION_VAR_KW'):
+                                # Python 3.5 changes the stack position of *args. kwargs come
+                                # after *args whereas in earlier Pythons, *args is at the end
+                                # which simpilfiies things from our perspective.
+                                # Python 3.6+ replaces CALL_FUNCTION_VAR_KW with CALL_FUNCTION_EX
+                                # We will just swap the order to make it look like earlier Python 3.
+                                entry = table[key.type]
+                                kwarg_pos = entry[2][1]
+                                args_pos = kwarg_pos - 1
+                                # Put last node[args_pos] after subsequent kwargs
+                                while node[kwarg_pos] == 'kwarg' and kwarg_pos < len(node):
+                                    # swap node[args_pos] with node[kwargs_pos]
+                                    node[kwarg_pos], node[args_pos] = node[args_pos], node[kwarg_pos]
+                                    args_pos = kwarg_pos
+                                    kwarg_pos += 1
+                            self.default(node)
+                        self.n_call_function = n_call_function
 
                     def n_funcdef(node):
                         if self.version == 3.6:
@@ -525,6 +556,8 @@ class SourceWalker(GenericASTTraversal, object):
 
         if self.pending_newlines:
             out = out[:-self.pending_newlines]
+        if isinstance(out, str) and not PYTHON3:
+            out = unicode(out, 'utf-8')
         self.f.write(out)
 
     def println(self, *data):
@@ -547,26 +580,6 @@ class SourceWalker(GenericASTTraversal, object):
                     node == AST('return_stmt',
                                 [AST('ret_expr', [NONE]), Token('RETURN_VALUE')]))
 
-    def n_continue_stmt(self, node):
-        if self.version >= 3.0 and node[0] == 'CONTINUE':
-            t = node[0]
-            if not t.linestart:
-                # Artificially-added "continue" statements derived from JUMP_ABSOLUTE
-                # don't have line numbers associated with them.
-                # If this is a CONTINUE is to the same target as a JUMP_ABSOLUTE following it,
-                # then the "continue" can be suppressed.
-                op, offset = t.op, t.offset
-                next_offset = self.scanner.next_offset(op, offset)
-                scanner = self.scanner
-                code = scanner.code
-                if next_offset < len(code):
-                    next_inst = code[next_offset]
-                    if (scanner.opc.opname[next_inst] == 'JUMP_ABSOLUTE'
-                        and t.pattr == code[next_offset+1]):
-                        # Suppress "continue"
-                        self.prune()
-        self.default(node)
-
     def n_return_stmt(self, node):
         if self.params['isLambda']:
             self.preorder(node[0])

uncompyle6/version.py

@@ -1,3 +1,3 @@
 # This file is suitable for sourcing inside bash as
 # well as importing into Python
-VERSION='2.10.1'
+VERSION='2.11.1'