More merge fixups from master

Revise make_function3 comment wrt args and kwargs

.travis.yml

@@ -6,7 +6,7 @@ python:
   - '2.7' # this is a cheat here because travis doesn't do 2.4-2.6
 
 install:
-- pip install -r requirements.txt
+- pip install -e .
 - pip install -r requirements-dev.txt
 
 script:

ChangeLog

@@ -1,6 +1,297 @@
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* pytest/validate.py: 2.4 doesn't do six
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* appveyor.yml: Nope it (appveyor) doesn't.
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* __pkginfo__.py, appveyor.yml: Administrivia See if appveyor will handle 2.5
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* : commit 7c299fbf3777c452d6a10075964961783f510699 Author: rocky
+	<rb@dustyfeet.com> Date:   Sat Jun 3 05:38:05 2017 -0400
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* __pkginfo__.py: We need six
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* README.rst, circle.yml, requirements-dev.txt: Go over
+	administrivia
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* ChangeLog, NEWS, uncompyle6/version.py: Get ready for release
+	2.10.1
+
+2017-06-03  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py,
+	uncompyle6/semantics/pysource.py: Fragment bugs fragment.py: * deparse_code_aorund_offset: was sometimes returning the wrong type * capture function name offset * lint imports pysource.py: use a clearer variable name
+
+2017-06-02  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py: Track changes in ifelstmtr..  in fragments from pysource
+
+2017-05-30  rocky <rb@dustyfeet.com>
+
+	* pytest/test_function_call.py: No decorators in Python < 2.6
+
+2017-05-30  rocky <rb@dustyfeet.com>
+
+	* : commit ad98fae3d4b0b83f65b15da8201e33c0ee6dab17 Author: rocky
+	<rb@dustyfeet.com> Date:   Tue May 30 01:26:52 2017 -0400
+
+2017-05-30  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py: Python 3.6 makefunction
+	handling for fragments
+
+2017-05-23  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: Fix up 3.6 unmapexpr
+
+2017-05-23  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: Fix up retreiving "async"
+	property on 3.6
+
+2017-05-23  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: Fix bug in a 3.6 class name.
+
+2017-05-23  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py,
+	uncompyle6/semantics/pysource.py: Add fuzzy offset deparse lookup
+
+2017-05-21  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner3.py: Correct EXTENDED_ARG handling on
+	Python 3.6...  where it can appear several times and xdis may handle it as well.
+	It possibly in other versions bug since EXTENDED_ARG is used so
+	rarely there because it has such a high value 1<<16, it's hard to
+	test and determine that.
+
+2017-05-20  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: Worse results. Revert some of
+	the last changes
+
+2017-05-20  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py, uncompyle6/semantics/pysource.py: 
+	More explicit about 3.5 UNMAP_PACK Have to reduce 3.5 bytecode testing for now, code is more solid.
+
+2017-05-19  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py, uncompyle6/parsers/parse36.py,
+	uncompyle6/scanners/scanner3.py: Simplify EXTENDED_ARG on 3.x We largely remove them and fold them itno the next op.
+	MAKE_FUNCTION though before 3.6 is an exception as that indicates an
+	annotated function
+
+2017-05-19  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner26.py: EXTENDED_ARG is implemented in
+	2.6
+
+2017-05-19  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/expression/06_huge_list.py,
+	uncompyle6/parsers/parse3.py, uncompyle6/semantics/pysource.py: Fix
+	EXTENDED_ARG for long lists, sets, maps
+
+2017-05-19  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner3.py: Another attempt at getting
+	get_target() correct
+
+2017-05-19  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py: Bug in pypy JUMP_IF_NOT_DEBUG
+	handling
+
+2017-05-19  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse36.py, uncompyle6/scanners/scanner3.py: 
+	EXTENDED_ARG handling...  get_target() wasn't taking into account EXTENDED_ARG before opcode.  This is mostly relevant in Python 3.6 where the max size before
+	needing EXTENDED_ARG has been reduced to 256, but theoretically
+	possible in earlier versions.
+
+2017-05-18  rocky <rb@dustyfeet.com>
+
+	* __pkginfo__.py: Enforce using xdis >=3.3.1 ..  to pick up bug fixes to 3.6 in xdis
+
+2017-05-17  rocky <rb@dustyfeet.com>
+
+	* __pkginfo__.py, uncompyle6/parsers/parse36.py,
+	uncompyle6/scanners/scanner3.py: Small changes....  * __pkginfo__.py: Need spark parser 1.6.1 for corrected
+	remove_rules() fn * parser36.py: remove replaced Python3 rules * scanner3.py: corrected comment. Thanks to moagstar here.  *
+
+2017-05-16  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse36.py: Fix broken CI on 3.6...  Another grammar rule replacing SETUP_LOOP with setup_loop
+
+2017-05-16  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse36.py: More EXTENDED_ARGS on 3.6
+
+2017-05-16  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse36.py: extend use of EXTENDED_ARGS in 3.6 switching to a wordcode seems to have made opcode fields smaller so
+	we need EXTENDED_ARG more?
+
+2017-05-16  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse36.py, uncompyle6/semantics/pysource.py: 
+	Allow LOAD_CONST EXTENDED_ARG
+
+2017-05-15  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py: Reinstate 3.6 listcomp rule
+
+2017-05-15  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py: Bang on 3.6 MAKE_FUNCTION some more
+
+2017-05-14  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: towards fixing a
+	3.5.CALL_FUNCTONI_VAR bug
+
+2017-05-14  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug35/04_CALL_FUNCTION_VAR_KW.py,
+	uncompyle6/parsers/parse3.py: Python 3.5 kw arg can be an expr Fixes Issue #95
+
+2017-05-14  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #117 from rocky/3.6-MAKE_FUNCTION 3.6 make function
+
+2017-05-13  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner3.py: MAKE_FUNCTION_FLAGS can be a
+	simpler tuple
+
+2017-05-13  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py: Grammar rules for Python 3.6
+	MAKE_FUNCTION
+
+2017-05-13  rocky <rb@dustyfeet.com>
+
+	* README.rst, uncompyle6/parsers/parse3.py,
+	uncompyle6/parsers/parse36.py, uncompyle6/semantics/pysource.py: 
+	Bang on 3.6 MAKE_FUNCTION a bit more parse3.py, parse36.py: adding return_closure rule tags what's going
+	on with this rule pysource.py: start changing semantic rules to support code changed
+	by new make_function semantics README.rst: typo
+
+2017-05-13  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner3.py: Typo
+
+2017-05-12  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse27.py: Bug in 2.7 decompiling ourself! Troublesome file was uncompyle6.semantics.pysource.engine()
+
+2017-05-11  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #113 from grkov90/patch-1 Fixed out_base bug
+
+2017-05-11  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py, uncompyle6/scanners/scanner3.py,
+	uncompyle6/semantics/make_function.py: WIP: start 3.6 MAKE_FUNCTION
+	handling
+
+2017-05-11  Daniel Bradburn <moagstar@gmail.com>
+
+	* : Merge pull request #116 from moagstar/function_call_keyword_only Added support for Python 3.6 CALL_FUNCTION_KW
+
+2017-05-10  Daniel Bradburn <moagstar@gmail.com>
+
+	* uncompyle6/semantics/pysource.py: Fixed bug in compiling double
+	star arg only function calls where the closing parenthesis would be
+	missed
+
+2017-05-10  Daniel Bradburn <moagstar@gmail.com>
+
+	* requirements-dev.txt: Adding requirement for pytest >= 3.0 to fix
+	strange INTERNALERROR in combination with hypothesis when using
+	pytest 2.6.4
+
+2017-05-10  Daniel Bradburn <moagstar@gmail.com>
+
+	* pytest/test_CALL_FUNCTION_KW.sh, pytest/test_function_call.py,
+	uncompyle6/parsers/parse3.py, uncompyle6/parsers/parse36.py,
+	uncompyle6/scanners/scanner36.py, uncompyle6/semantics/pysource.py: 
+	Added support for support for Python 3.6 CALL_FUNCTION_KW
+
+2017-05-08  rocky <rb@dustyfeet.com>
+
+	* appveyor.yml, test/test_pyenvlib.py,
+	uncompyle6/semantics/pysource.py: pysource guard and another
+	appveyor test
+
+2017-05-08  rocky <rb@dustyfeet.com>
+
+	* appveyor.yml: appveyor take 2
+
+2017-05-08  rocky <rb@dustyfeet.com>
+
+	* appveyor.yml, appveyor/install.ps1, appveyor/run_with_env.cmd: Try
+	appveyor
+
+2017-05-07  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: More guarded CONTINUE deletion
+
+2017-05-07  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanner.py, uncompyle6/scanners/scanner3.py,
+	uncompyle6/semantics/pysource.py: Reduce spurious "continue"
+	statements
+
+2017-05-07  rocky <rb@dustyfeet.com>
+
+	* test/Makefile: --weak-verify on 3.3 with inclusion of last commit Note that the result is sematically equivalent, so it is is correct.
+
+2017-05-07  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/looping/12_if_while_true_pass.py,
+	uncompyle6/scanners/scanner3.py: Python 3.x control-flow bug...  "pass" statement inside "while True"
+
+2017-05-07  rocky <rb@dustyfeet.com>
+
+	* HOW-TO-REPORT-A-BUG.md: Small typo
+
+2017-05-07  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner3.py: Fix improper COME_FROM_EXCEPT in
+	Python 3.3+
+
 2017-05-06  rocky <rb@dustyfeet.com>
 
-	* uncompyle6/version.py: Get ready for release 2.9.11
+	* uncompyle6/parsers/parse33.py: python 3.3 while True parsing bug
+
+2017-05-06  rocky <rb@dustyfeet.com>
+
+	* ChangeLog, NEWS, uncompyle6/version.py: Get ready for release
+	2.9.11
+
+2017-05-06  rocky <rb@dustyfeet.com>
+
+	* ChangeLog, NEWS, uncompyle6/version.py: Get ready for release
+	2.9.11
+
+2017-05-06  rocky <rb@dustyfeet.com>
+
+	* test/Makefile: fix PYTHON variable setting in test/Makefile
 
 2017-05-06  rocky <rb@dustyfeet.com>
 
@@ -22,6 +313,14 @@
 
 	* .travis.yml: Try CI testing on Python 3.6
 
+2017-05-03  Gregory <grkov90@gmail.com>
+
+	* uncompyle6/main.py: Some fix
+
+2017-05-03  Gregory <grkov90@gmail.com>
+
+	* uncompyle6/main.py: Fixed out_base bug Variable filename using in for tags uncompyle6 -o haven't worked argument -o haven't worked
+
 2017-05-02  rocky <rb@dustyfeet.com>
 
 	* test/simple_source/bug35/01_map_unpack.py, uncompyle6/parser.py,

HOW-TO-REPORT-A-BUG.md

@@ -7,7 +7,7 @@ decompyle everything. This one probably does the
 best job of *any* Python decompiler. But it is a constant work in progress: Python keeps changing, and so does its code generation.
 
 I have found bugs in *every* Python decompiler I have tried. Even
-those where authors/maintainers claim that they have used it on 
+those where authors/maintainers claim that they have used it on
 the entire Python standard library. And I don't mean that
 the program doesn't come out with the same Python source instructions,
 but that the program is *semantically* not equivalent.
@@ -29,7 +29,7 @@ can figure out what OS you are running this on and what version of
 *uncomplye6* was used. Therefore, if you don't provide the input
 command and the output from that, please give:
 
-* _uncompile6_ version used
+* _uncompyle6_ version used
 * OS that you used this on
 * Python interpreter version used
 
@@ -37,11 +37,16 @@ command and the output from that, please give:
 ### But I don't *have* the source code!
 
 Sure, I get it. No problem. There is Python assembly code on parse
-errors, so simply by hand decompile that. To get a full disassembly, use pydisasm from the [xdis](https://pypi.python.org/pypi/xdis) package. Opcodes are described in the documentation for the [dis](https://docs.python.org/3.6/library/dis.html) module.
+errors, so simply by hand decompile that. To get a full disassembly,
+use pydisasm from the [xdis](https://pypi.python.org/pypi/xdis)
+package. Opcodes are described in the documentation for
+the [dis](https://docs.python.org/3.6/library/dis.html) module.
 
 ### But I don't *have* the source code and am incapable of figuring how how to do a hand disassembly!
 
-Well, you could learn. No one is born into this world knowing how to disassemble Python bytecode. And as Richard Feynman once said, "What one fool can learn, so can another."
+Well, you could learn. No one is born into this world knowing how to
+disassemble Python bytecode. And as Richard Feynman once said, "What
+one fool can learn, so can another."
 
 ## Narrowing the problem
 

NEWS

@@ -1,3 +1,40 @@
+uncompyle6 2.11.1 2016-06-18 Fleetwood
+- Major improvements in fragment tracking
+  * Add nonterminal node in extractInfo
+  * tag more offsets in expressions
+  * tag array subscripts
+  * set YIELD value offset in a <yield> expr
+  * fix a long-standing bug in not adjusting final AST when melding other deparse ASTs
+- Fixes yet again for make_function node handling; document what's up here
+- Fix bug in snowflake Python 3.5 *args kwargs
+
+uncompyle6 2.10.1 2016-06-3 Marylin Frankel
+
+- fix some fragments parsing bugs
+  - was returning the wrong type sometimes in deparse_code_around_offset()
+  - capture function name in offsets
+  - track changes to ifelstrmtr node from pysource into fragments
+
+uncompyle6 2.10.0 2016-05-30 Elaine Gordon
+
+- Add fuzzy offset deparse lookup
+- 3.6 bugfixes
+   - fix EXTENDED_ARGS handling (and in 2.6 and others)
+   - semantic routine make_function fragments.py
+   - MAKE_FUNCTION handling
+   - CALL_FUNCTION_EX handling
+   - async property on defs
+   - support for CALL_FUNCTION_KW (moagstar)
+- 3.5+ UNMAP_PACK and BUILD_UNMAP_PACK handling
+- 3.5 FUNCTION_VAR bug
+- 3.x pass statement insdie while True
+- Improve 3.2 decompilation
+- Fixed -o argument processing (Gregrory)
+- Reduce scope of LOAD_ASSERT as expr to 3.4+
+- "await" statement fixes
+- 2.3, 2.4 "if 1 .." fixes
+- 3.x annotation fixes
+
 uncompyle6 2.9.11 2016-04-06
 
 - Better support for Python 3.5+ BUILD_MAP_UNPACK

README.rst

@@ -56,7 +56,7 @@ This uses setup.py, so it follows the standard Python routine:
 
 ::
 
-    pip install -r requirements.txt
+    pip install -e .
     pip install -r requirements-dev.txt
     python setup.py install # may need sudo
     # or if you have pyenv:
@@ -168,10 +168,10 @@ See Also
 --------
 
 * https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for later Python 3 versions is a bit lacking though.
-* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique what is used here.
+* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Include some fixes like supporting function annotations
 * The HISTORY_ file.
-
+* `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md>`_
 .. |downloads| image:: https://img.shields.io/pypi/dd/uncompyle6.svg
 .. _trepan: https://pypi.python.org/pypi/trepan
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md

__pkginfo__.py

@@ -33,14 +33,14 @@ classifiers =  ['Development Status :: 5 - Production/Stable',
 # The rest in alphabetic order
 author             = "Rocky Bernstein, Hartmut Goebel, John Aycock, and others"
 author_email       = "rb@dustyfeet.com"
-entry_points={
+entry_points       = {
     'console_scripts': [
         'uncompyle6=uncompyle6.bin.uncompile:main_bin',
         'pydisassemble=uncompyle6.bin.pydisassemble:main',
     ]}
 ftp_url            = None
-install_requires   = ['spark-parser >= 1.6.0, < 1.7.0',
-                      'xdis >= 3.3.0, < 3.4.0']
+install_requires   = ['spark-parser >= 1.6.1, < 1.7.0',
+                      'xdis >= 3.3.1, < 3.4.0']
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'

appveyor.yml

@@ -0,0 +1,78 @@
+environment:
+  global:
+    # SDK v7.0 MSVC Express 2008's SetEnv.cmd script will fail if the
+    # /E:ON and /V:ON options are not enabled in the batch script intepreter
+    # See: http://stackoverflow.com/a/13751649/163740
+    CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\appveyor\\run_with_env.cmd"
+
+  matrix:
+
+    # Pre-installed Python versions, which Appveyor may upgrade to
+    # a later point release.
+    # See: http://www.appveyor.com/docs/installed-software#python
+
+    # - PYTHON: "C:\\Python27"
+    #   PYTHON_VERSION: "2.7.x"
+    #   PYTHON_ARCH: "32"
+
+    - PYTHON: "C:\\Python27-x64"
+      PYTHON_VERSION: "2.7.x"
+      PYTHON_ARCH: "64"
+
+    # - PYTHON: "C:\\Python26"
+    #   PYTHON_VERSION: "2.6.x"
+    #   PYTHON_ARCH: "32"
+
+    # - PYTHON: "C:\\Python26-x64"
+    #   PYTHON_VERSION: "2.6.x"
+    #   PYTHON_ARCH: "64"
+
+install:
+  # We need wheel installed to build wheels
+  - "%PYTHON%\\python.exe -m pip install wheel"
+
+  # Install Python (from the official .msi of http://python.org) and pip when
+  # not already installed.
+  - ps: if (-not(Test-Path($env:PYTHON))) { & appveyor\install.ps1 }
+
+  # Prepend newly installed Python to the PATH of this build (this cannot be
+  # done from inside the powershell script as it would require to restart
+  # the parent CMD process).
+  - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
+  - "SET HOME=."
+
+  # Check that we have the expected version and architecture for Python
+  - "python --version"
+  - "python -c \"import struct; print(struct.calcsize('P') * 8)\""
+
+  # Upgrade to the latest version of pip to avoid it displaying warnings
+  # about it being out of date.
+  - "pip install --disable-pip-version-check --user --upgrade pip"
+
+  # Install the build dependencies of the project. If some dependencies contain
+  # compiled extensions and are not provided as pre-built wheel packages,
+  # pip will build them from source using the MSVC compiler matching the
+  # target Python version and architecture
+  - "%CMD_IN_ENV% pip install -r requirements.txt"
+
+build_script:
+  # Build the compiled extension
+  - "%CMD_IN_ENV% python setup.py build"
+
+test_script:
+  # Run the project tests
+  - "%CMD_IN_ENV% python test/test_pyenvlib.py --native --weak-verify"
+
+after_test:
+  # If tests are successful, create binary packages for the project.
+  - "%CMD_IN_ENV% python setup.py bdist_wininst"
+  - "%CMD_IN_ENV% python setup.py bdist_msi"
+  - ps: "ls dist"
+
+artifacts:
+  # Archive the generated packages in the ci.appveyor.com build report.
+  - path: dist\*
+
+#on_success:
+#  - TODO: upload the content of dist/*.whl to a public wheelhouse
+#

appveyor/install.ps1

@@ -0,0 +1,229 @@
+# Sample script to install Python and pip under Windows
+# Authors: Olivier Grisel, Jonathan Helmus, Kyle Kastner, and Alex Willmer
+# License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
+
+$MINICONDA_URL = "http://repo.continuum.io/miniconda/"
+$BASE_URL = "https://www.python.org/ftp/python/"
+$GET_PIP_URL = "https://bootstrap.pypa.io/get-pip.py"
+$GET_PIP_PATH = "C:\get-pip.py"
+
+$PYTHON_PRERELEASE_REGEX = @"
+(?x)
+(?<major>\d+)
+\.
+(?<minor>\d+)
+\.
+(?<micro>\d+)
+(?<prerelease>[a-z]{1,2}\d+)
+"@
+
+
+function Download ($filename, $url) {
+    $webclient = New-Object System.Net.WebClient
+
+    $basedir = $pwd.Path + "\"
+    $filepath = $basedir + $filename
+    if (Test-Path $filename) {
+        Write-Host "Reusing" $filepath
+        return $filepath
+    }
+
+    # Download and retry up to 3 times in case of network transient errors.
+    Write-Host "Downloading" $filename "from" $url
+    $retry_attempts = 2
+    for ($i = 0; $i -lt $retry_attempts; $i++) {
+        try {
+            $webclient.DownloadFile($url, $filepath)
+            break
+        }
+        Catch [Exception]{
+            Start-Sleep 1
+        }
+    }
+    if (Test-Path $filepath) {
+        Write-Host "File saved at" $filepath
+    } else {
+        # Retry once to get the error message if any at the last try
+        $webclient.DownloadFile($url, $filepath)
+    }
+    return $filepath
+}
+
+
+function ParsePythonVersion ($python_version) {
+    if ($python_version -match $PYTHON_PRERELEASE_REGEX) {
+        return ([int]$matches.major, [int]$matches.minor, [int]$matches.micro,
+                $matches.prerelease)
+    }
+    $version_obj = [version]$python_version
+    return ($version_obj.major, $version_obj.minor, $version_obj.build, "")
+}
+
+
+function DownloadPython ($python_version, $platform_suffix) {
+    $major, $minor, $micro, $prerelease = ParsePythonVersion $python_version
+
+    if (($major -le 2 -and $micro -eq 0) `
+        -or ($major -eq 3 -and $minor -le 2 -and $micro -eq 0) `
+        ) {
+        $dir = "$major.$minor"
+        $python_version = "$major.$minor$prerelease"
+    } else {
+        $dir = "$major.$minor.$micro"
+    }
+
+    if ($prerelease) {
+        if (($major -le 2) `
+            -or ($major -eq 3 -and $minor -eq 1) `
+            -or ($major -eq 3 -and $minor -eq 2) `
+            -or ($major -eq 3 -and $minor -eq 3) `
+            ) {
+            $dir = "$dir/prev"
+        }
+    }
+
+    if (($major -le 2) -or ($major -le 3 -and $minor -le 4)) {
+        $ext = "msi"
+        if ($platform_suffix) {
+            $platform_suffix = ".$platform_suffix"
+        }
+    } else {
+        $ext = "exe"
+        if ($platform_suffix) {
+            $platform_suffix = "-$platform_suffix"
+        }
+    }
+
+    $filename = "python-$python_version$platform_suffix.$ext"
+    $url = "$BASE_URL$dir/$filename"
+    $filepath = Download $filename $url
+    return $filepath
+}
+
+
+function InstallPython ($python_version, $architecture, $python_home) {
+    Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
+    if (Test-Path $python_home) {
+        Write-Host $python_home "already exists, skipping."
+        return $false
+    }
+    if ($architecture -eq "32") {
+        $platform_suffix = ""
+    } else {
+        $platform_suffix = "amd64"
+    }
+    $installer_path = DownloadPython $python_version $platform_suffix
+    $installer_ext = [System.IO.Path]::GetExtension($installer_path)
+    Write-Host "Installing $installer_path to $python_home"
+    $install_log = $python_home + ".log"
+    if ($installer_ext -eq '.msi') {
+        InstallPythonMSI $installer_path $python_home $install_log
+    } else {
+        InstallPythonEXE $installer_path $python_home $install_log
+    }
+    if (Test-Path $python_home) {
+        Write-Host "Python $python_version ($architecture) installation complete"
+    } else {
+        Write-Host "Failed to install Python in $python_home"
+        Get-Content -Path $install_log
+        Exit 1
+    }
+}
+
+
+function InstallPythonEXE ($exepath, $python_home, $install_log) {
+    $install_args = "/quiet InstallAllUsers=1 TargetDir=$python_home"
+    RunCommand $exepath $install_args
+}
+
+
+function InstallPythonMSI ($msipath, $python_home, $install_log) {
+    $install_args = "/qn /log $install_log /i $msipath TARGETDIR=$python_home"
+    $uninstall_args = "/qn /x $msipath"
+    RunCommand "msiexec.exe" $install_args
+    if (-not(Test-Path $python_home)) {
+        Write-Host "Python seems to be installed else-where, reinstalling."
+        RunCommand "msiexec.exe" $uninstall_args
+        RunCommand "msiexec.exe" $install_args
+    }
+}
+
+function RunCommand ($command, $command_args) {
+    Write-Host $command $command_args
+    Start-Process -FilePath $command -ArgumentList $command_args -Wait -Passthru
+}
+
+
+function InstallPip ($python_home) {
+    $pip_path = $python_home + "\Scripts\pip.exe"
+    $python_path = $python_home + "\python.exe"
+    if (-not(Test-Path $pip_path)) {
+        Write-Host "Installing pip..."
+        $webclient = New-Object System.Net.WebClient
+        $webclient.DownloadFile($GET_PIP_URL, $GET_PIP_PATH)
+        Write-Host "Executing:" $python_path $GET_PIP_PATH
+        & $python_path $GET_PIP_PATH
+    } else {
+        Write-Host "pip already installed."
+    }
+}
+
+
+function DownloadMiniconda ($python_version, $platform_suffix) {
+    if ($python_version -eq "3.4") {
+        $filename = "Miniconda3-3.5.5-Windows-" + $platform_suffix + ".exe"
+    } else {
+        $filename = "Miniconda-3.5.5-Windows-" + $platform_suffix + ".exe"
+    }
+    $url = $MINICONDA_URL + $filename
+    $filepath = Download $filename $url
+    return $filepath
+}
+
+
+function InstallMiniconda ($python_version, $architecture, $python_home) {
+    Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
+    if (Test-Path $python_home) {
+        Write-Host $python_home "already exists, skipping."
+        return $false
+    }
+    if ($architecture -eq "32") {
+        $platform_suffix = "x86"
+    } else {
+        $platform_suffix = "x86_64"
+    }
+    $filepath = DownloadMiniconda $python_version $platform_suffix
+    Write-Host "Installing" $filepath "to" $python_home
+    $install_log = $python_home + ".log"
+    $args = "/S /D=$python_home"
+    Write-Host $filepath $args
+    Start-Process -FilePath $filepath -ArgumentList $args -Wait -Passthru
+    if (Test-Path $python_home) {
+        Write-Host "Python $python_version ($architecture) installation complete"
+    } else {
+        Write-Host "Failed to install Python in $python_home"
+        Get-Content -Path $install_log
+        Exit 1
+    }
+}
+
+
+function InstallMinicondaPip ($python_home) {
+    $pip_path = $python_home + "\Scripts\pip.exe"
+    $conda_path = $python_home + "\Scripts\conda.exe"
+    if (-not(Test-Path $pip_path)) {
+        Write-Host "Installing pip..."
+        $args = "install --yes pip"
+        Write-Host $conda_path $args
+        Start-Process -FilePath "$conda_path" -ArgumentList $args -Wait -Passthru
+    } else {
+        Write-Host "pip already installed."
+    }
+}
+
+function main () {
+    InstallPython $env:PYTHON_VERSION $env:PYTHON_ARCH $env:PYTHON
+    InstallPip $env:PYTHON
+}
+
+main

appveyor/run_with_env.cmd

@@ -0,0 +1,87 @@
+:: To build extensions for 64 bit Python 3, we need to configure environment
+:: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of:
+:: MS Windows SDK for Windows 7 and .NET Framework 4 (SDK v7.1)
+::
+:: To build extensions for 64 bit Python 2, we need to configure environment
+:: variables to use the MSVC 2008 C++ compilers from GRMSDKX_EN_DVD.iso of:
+:: MS Windows SDK for Windows 7 and .NET Framework 3.5 (SDK v7.0)
+::
+:: 32 bit builds, and 64-bit builds for 3.5 and beyond, do not require specific
+:: environment configurations.
+::
+:: Note: this script needs to be run with the /E:ON and /V:ON flags for the
+:: cmd interpreter, at least for (SDK v7.0)
+::
+:: More details at:
+:: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
+:: http://stackoverflow.com/a/13751649/163740
+::
+:: Author: Olivier Grisel
+:: License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
+::
+:: Notes about batch files for Python people:
+::
+:: Quotes in values are literally part of the values:
+::      SET FOO="bar"
+:: FOO is now five characters long: " b a r "
+:: If you don't want quotes, don't include them on the right-hand side.
+::
+:: The CALL lines at the end of this file look redundant, but if you move them
+:: outside of the IF clauses, they do not run properly in the SET_SDK_64==Y
+:: case, I don't know why.
+@ECHO OFF
+SET COMMAND_TO_RUN=%*
+SET WIN_SDK_ROOT=C:\Program Files\Microsoft SDKs\Windows
+SET WIN_WDK=c:\Program Files (x86)\Windows Kits\10\Include\wdf
+
+:: Extract the major and minor versions, and allow for the minor version to be
+:: more than 9.  This requires the version number to have two dots in it.
+SET MAJOR_PYTHON_VERSION=%PYTHON_VERSION:~0,1%
+IF "%PYTHON_VERSION:~3,1%" == "." (
+    SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,1%
+) ELSE (
+    SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,2%
+)
+
+:: Based on the Python version, determine what SDK version to use, and whether
+:: to set the SDK for 64-bit.
+IF %MAJOR_PYTHON_VERSION% == 2 (
+    SET WINDOWS_SDK_VERSION="v7.0"
+    SET SET_SDK_64=Y
+) ELSE (
+    IF %MAJOR_PYTHON_VERSION% == 3 (
+        SET WINDOWS_SDK_VERSION="v7.1"
+        IF %MINOR_PYTHON_VERSION% LEQ 4 (
+            SET SET_SDK_64=Y
+        ) ELSE (
+            SET SET_SDK_64=N
+            IF EXIST "%WIN_WDK%" (
+                :: See: https://connect.microsoft.com/VisualStudio/feedback/details/1610302/
+                REN "%WIN_WDK%" 0wdf
+            )
+        )
+    ) ELSE (
+        ECHO Unsupported Python version: "%MAJOR_PYTHON_VERSION%"
+        EXIT 1
+    )
+)
+
+IF %PYTHON_ARCH% == 64 (
+    IF %SET_SDK_64% == Y (
+        ECHO Configuring Windows SDK %WINDOWS_SDK_VERSION% for Python %MAJOR_PYTHON_VERSION% on a 64 bit architecture
+        SET DISTUTILS_USE_SDK=1
+        SET MSSdk=1
+        "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION%
+        "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /x64 /release
+        ECHO Executing: %COMMAND_TO_RUN%
+        call %COMMAND_TO_RUN% || EXIT 1
+    ) ELSE (
+        ECHO Using default MSVC build environment for 64 bit architecture
+        ECHO Executing: %COMMAND_TO_RUN%
+        call %COMMAND_TO_RUN% || EXIT 1
+    )
+) ELSE (
+    ECHO Using default MSVC build environment for 32 bit architecture
+    ECHO Executing: %COMMAND_TO_RUN%
+    call %COMMAND_TO_RUN% || EXIT 1
+)

circle.yml

@@ -6,7 +6,7 @@ machine:
 
 dependencies:
     override:
-        - pip install -r requirements.txt
+        - pip install -e .
         - pip install -r requirements-dev.txt
 test:
     override:

pytest/test_CALL_FUNCTION_KW.sh

@@ -0,0 +1,6 @@
+source ../.venv.3.6/bin/activate
+py.test -k test_CALL_FUNCTION_KW
+source ../.venv.3.5/bin/activate
+py.test -k test_CALL_FUNCTION_KW
+source ../.venv.2.7/bin/activate
+py.test -k test_CALL_FUNCTION_KW

pytest/test_function_call.py

@@ -1,128 +0,0 @@
-# std
-import string
-# 3rd party
-from hypothesis import given, assume, strategies as st
-import pytest
-# uncompyle
-from validate import validate_uncompyle
-
-
-alpha = st.sampled_from(string.ascii_lowercase)
-numbers = st.sampled_from(string.digits)
-alphanum = st.sampled_from(string.ascii_lowercase + string.digits)
-expressions = st.sampled_from([x for x in string.ascii_lowercase + string.digits] + ['x+1'])
-
-
-@st.composite
-def function_calls(draw):
-    """
-    Strategy factory for generating function calls.
-
-    :param draw: Callable which draws examples from other strategies.
-
-    :return: The function call text.
-    """
-    list1 = st.lists(alpha, min_size=0, max_size=1)
-    list3 = st.lists(alpha, min_size=0, max_size=3)
-
-    positional_args = draw(list3)
-    named_args = [x + '=0' for x in draw(list3)]
-    star_args = ['*' + x for x in draw(list1)]
-    double_star_args = ['**' + x for x in draw(list1)]
-
-    arguments = positional_args + named_args + star_args + double_star_args
-    draw(st.randoms()).shuffle(arguments)
-    arguments = ','.join(arguments)
-
-    function_call = 'fn({arguments})'.format(arguments=arguments)
-    try:
-        # TODO: Figure out the exact rules for ordering of positional, named,
-        # star args, double star args and in which versions the various
-        # types of arguments are supported so we don't need to check that the
-        # expression compiles like this.
-        compile(function_call, '<string>', 'single')
-    except:
-        assume(False)
-    return function_call
-
-
-@pytest.mark.xfail()
-def test_CALL_FUNCTION():
-    validate_uncompyle("fn(w,m,f)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_CONST_KEY_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(w=0,m=0,**v)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(a=0,**g)")
-
-
-@pytest.mark.xfail()
-def test_CALL_FUNCTION_KW():
-    validate_uncompyle("fn(j=0)")
-
-
-@pytest.mark.xfail()
-def test_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*g,**j)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*z,u=0)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(**a)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(b,b,b=0,*a)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*c,v)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_CONST_KEY_MAP_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(i=0,y=0,*p)")
-
-
-@pytest.mark.skip(reason='skipping property based test until all individual tests are passing')
-@given(function_calls())
-def test_function_call(function_call):
-    validate_uncompyle(function_call)
-
-
-examples = set()
-generate_examples = False
-
-
-@pytest.mark.skipif(not generate_examples, reason='not generating examples')
-@given(function_calls())
-def test_generate_hypothesis(function_call):
-    examples.add(function_call)
-
-
-@pytest.mark.skipif(not generate_examples, reason='not generating examples')
-def test_generate_examples():
-    import dis
-    example_opcodes = {}
-    for example in examples:
-        opcodes = tuple(sorted(set(
-            instruction.opname
-            for instruction in dis.Bytecode(example)
-            if instruction.opname not in ('LOAD_CONST', 'LOAD_NAME', 'RETURN_VALUE')
-        )))
-        example_opcodes[opcodes] = example
-    for k, v in example_opcodes.items():
-        print('def test_' + '_'.join(k) + '():\n    validate_uncompyle("' + v + '")\n\n')
-    return

pytest/validate.py

@@ -6,8 +6,7 @@ import difflib
 import subprocess
 import tempfile
 import functools
-# compatability
-import six
+from StringIO import StringIO
 # uncompyle6 / xdis
 from uncompyle6 import PYTHON_VERSION, IS_PYPY, deparse_code
 # TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
@@ -123,7 +122,7 @@ def validate_uncompyle(text, mode='exec'):
     original_text = text
 
     deparsed = deparse_code(PYTHON_VERSION, original_code,
-                            compile_mode=mode, out=six.StringIO())
+                            compile_mode=mode, out=StringIO())
     uncompyled_text = deparsed.text
     uncompyled_code = compile(uncompyled_text, '<string>', 'exec')
 

requirements-dev.txt

@@ -1,4 +1,3 @@
-pytest
+pytest>=3.0.0
 flake8
 hypothesis
-six

test/Makefile

@@ -16,8 +16,7 @@ check-short:
 
 # Run all tests
 check:
-	@$(PYTHON) -V && PYTHON_VERSION=`$(PYTHON) -V 2>&1 | cut -d ' ' -f 2 | cut -d'.' -f1,2`; \
-	$(MAKE) check-$$PYTHON_VERSION
+	$(MAKE) check-$(PYTHON_VERSION)
 
 #: Run working tests from Python 2.6 or 2.7
 check-2.4 check-2.5 check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1  check-native-short
@@ -36,7 +35,7 @@ check-3.2: check-bytecode
 
 #: Run working tests from Python 3.3
 check-3.3: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.3 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.4
 check-3.4: check-bytecode check-3.4-ok check-2.7-ok

test/simple_source/bug35/04_CALL_FUNCTION_VAR_KW.py

@@ -0,0 +1,8 @@
+# sql/schema.py
+# Note that kwargs comes before "positional" args
+def tometadata(self, metadata, schema, Table, args, name=None):
+    table = Table(
+        name, metadata, schema=schema,
+        *args, **self.kwargs
+    )
+    return table

test/simple_source/looping/12_if_while_true_pass.py

@@ -0,0 +1,14 @@
+# Python 3.3 pyclbr.py
+# Note that Python 3 adds a lot of unecessary "continues"
+# and puts that in for "pass"
+def _readmodule(g, token, path):
+    for tokentype in g:
+        if g:
+            while True:
+                if token:
+                    token = 1
+                elif token:
+                    pass
+                elif tokentype:
+                    token = 7
+            token = 10

test/test_pyenvlib.py

@@ -20,7 +20,7 @@ Step 2: Run the test:
 """
 
 from uncompyle6 import main, PYTHON3
-import os, time, shutil
+import os, time, shutil, sys
 from fnmatch import fnmatch
 
 #----- configure this for your needs
@@ -31,7 +31,7 @@ TEST_VERSIONS=('2.3.7', '2.4.6', '2.5.6', '2.6.9',
                '2.7.10', '2.7.11', '2.7.12', '2.7.13',
                '3.0.1', '3.1.5', '3.2.6',
                '3.3.5', '3.3.6',
-               '3.4.2', '3.5.1', '3.6.0')
+               '3.4.2', '3.5.1', '3.6.0', 'native')
 
 target_base = '/tmp/py-dis/'
 lib_prefix = os.path.join(os.environ['HOME'], '.pyenv/versions')
@@ -52,6 +52,11 @@ for vers in TEST_VERSIONS:
         short_vers = vers[0:-2]
         test_options[vers] = (os.path.join(lib_prefix, vers, 'lib_pypy'),
                               PYC, 'python-lib'+short_vers)
+    if vers == 'native':
+        short_vers = os.path.basename(sys.path[-1])
+        test_options[vers] = (sys.path[-1],
+                              PYC, short_vers)
+
     else:
         short_vers = vers[:3]
         test_options[vers] = (os.path.join(lib_prefix, vers, 'lib', 'python'+short_vers),

uncompyle6/main.py

@@ -110,6 +110,7 @@ def main(in_base, out_base, files, codes, outfile=None,
         return open(outfile, 'w')
 
     tot_files = okay_files = failed_files = verify_failed_files = 0
+    current_outfile = outfile
 
     for filename in files:
         infile = os.path.join(in_base, filename)
@@ -141,11 +142,11 @@ def main(in_base, out_base, files, codes, outfile=None,
                 os.dup2(tee.stdin.fileno(), sys.stderr.fileno())
         else:
             if filename.endswith('.pyc'):
-                outfile = os.path.join(out_base, filename[0:-1])
+                current_outfile = os.path.join(out_base, filename[0:-1])
             else:
-                outfile = os.path.join(out_base, filename) + '_dis'
-            outstream = _get_outstream(outfile)
-        # print(outfile, file=sys.stderr)
+                current_outfile = os.path.join(out_base, filename) + '_dis'
+            outstream = _get_outstream(current_outfile)
+        # print(current_outfile, file=sys.stderr)
 
         # Try to uncompile the input file
         try:
@@ -164,16 +165,16 @@ def main(in_base, out_base, files, codes, outfile=None,
             raise
         # except:
         #     failed_files += 1
-        #     if outfile:
+        #     if current_outfile:
         #         outstream.close()
-        #         os.rename(outfile, outfile + '_failed')
+        #         os.rename(current_outfile, current_outfile + '_failed')
         #     else:
         #         sys.stderr.write("\n# %s" % sys.exc_info()[1])
         #         sys.stderr.write("\n# Can't uncompile %s\n" % infile)
         else: # uncompile successful
-            if outfile:
+            if current_outfile:
                 if do_linemaps:
-                    mapping = line_number_mapping(infile, outfile)
+                    mapping = line_number_mapping(infile, current_outfile)
                     outstream.write("\n\n## Line number correspondences\n")
                     import pprint
                     s = pprint.pformat(mapping, indent=2, width=80)
@@ -184,8 +185,8 @@ def main(in_base, out_base, files, codes, outfile=None,
                 if do_verify:
                     weak_verify = do_verify == 'weak'
                     try:
-                        msg = verify.compare_code_with_srcfile(infile, outfile, weak_verify=weak_verify)
-                        if not outfile:
+                        msg = verify.compare_code_with_srcfile(infile, current_outfile, weak_verify=weak_verify)
+                        if not current_outfile:
                             if not msg:
                                 print '\n# okay decompiling %s' % infile
                                 okay_files += 1
@@ -194,7 +195,7 @@ def main(in_base, out_base, files, codes, outfile=None,
                     except verify.VerifyCmpError, e:
                         print(e)
                         verify_failed_files += 1
-                        os.rename(outfile, outfile + '_unverified')
+                        os.rename(current_outfile, current_outfile + '_unverified')
                         sys.stderr.write("### Error Verifying %s\n" % filename)
                         sys.stderr.write(str(e) + "\n")
                         if not outfile:
@@ -212,15 +213,15 @@ def main(in_base, out_base, files, codes, outfile=None,
                 pass
             else:
                 okay_files += 1
-                if not outfile:
+                if not current_outfile:
                     mess = '\n# okay decompiling'
                     # mem_usage = __memUsage()
-                    print mess, infile
-        if outfile:
+                    print(mess, infile)
+        if current_outfile:
             sys.stdout.write("%s\r" %
                              status_msg(do_verify, tot_files, okay_files, failed_files, verify_failed_files))
             sys.stdout.flush()
-    if outfile:
+    if current_outfile:
         sys.stdout.write("\n")
         sys.stdout.flush()
     return (tot_files, okay_files, failed_files, verify_failed_files)

uncompyle6/parsers/parse27.py

@@ -52,6 +52,8 @@ class Python27Parser(Python2Parser):
         come_froms ::= come_froms COME_FROM
         come_froms ::= COME_FROM
 
+        iflaststmtl ::= testexpr c_stmts_opt
+
         _ifstmts_jump ::= c_stmts_opt JUMP_FORWARD come_froms
         bp_come_from    ::= POP_BLOCK COME_FROM
 

uncompyle6/parsers/parse3.py

@@ -332,7 +332,9 @@ class Python3Parser(PythonParser):
 
     def p_stmt3(self, args):
         """
-        stmt ::= LOAD_CLOSURE RETURN_VALUE RETURN_LAST
+        stmt ::= return_closure
+        return_closure ::= LOAD_CLOSURE RETURN_VALUE RETURN_LAST
+
         stmt ::= whileTruestmt
         ifelsestmt ::= testexpr c_stmts_opt JUMP_FORWARD else_suite _come_from
         """
@@ -479,6 +481,8 @@ class Python3Parser(PythonParser):
         # high byte number of positional parameters
         args_pos = token.attr & 0xff
         args_kw = (token.attr >> 8) & 0xff
+        args_kw = (token.attr >> 8) & 0xff
+        # args_ann = (token.attr >> 16) & 0x7FFF
 
         # Additional exprs for * and ** args:
         #  0 if neither
@@ -490,22 +494,28 @@ class Python3Parser(PythonParser):
         token.type = self.call_fn_name(token)
         uniq_param = args_kw + args_pos
         if self.version == 3.5 and opname.startswith('CALL_FUNCTION_VAR'):
-            # Python 3.5 changes the stack position of where * args, the
-            # first LOAD_FAST, below are located.
+            # Python 3.5 changes the stack position of *args. KW args come
+            # after *args.
             # Python 3.6+ replaces CALL_FUNCTION_VAR_KW with CALL_FUNCTION_EX
             if opname.endswith('KW'):
-                kw = 'LOAD_FAST '
+                kw = 'expr '
             else:
                 kw = ''
             rule = ('call_function ::= expr expr ' +
                     ('pos_arg ' * args_pos) +
                     ('kwarg ' * args_kw) + kw + token.type)
             self.add_unique_rule(rule, token.type, uniq_param, customize)
-
-        rule = ('call_function ::= expr ' +
-                ('pos_arg ' * args_pos) +
-                ('kwarg ' * args_kw) +
-                'expr ' * nak + token.type)
+        if self.version >= 3.6 and opname == 'CALL_FUNCTION_EX_KW':
+            rule = ('call_function36 ::= '
+                    'expr build_tuple_unpack_with_call build_map_unpack_with_call '
+                    'CALL_FUNCTION_EX_KW_1')
+            self.add_unique_rule(rule, token.type, uniq_param, customize)
+            rule = 'call_function ::= call_function36'
+        else:
+            rule = ('call_function ::= expr ' +
+                    ('pos_arg ' * args_pos) +
+                    ('kwarg ' * args_kw) +
+                    'expr ' * nak + token.type)
 
         self.add_unique_rule(rule, token.type, uniq_param, customize)
         if self.version >= 3.5:
@@ -607,8 +617,9 @@ class Python3Parser(PythonParser):
                     assign2_pypy ::= expr expr designator designator
                 """, nop_func)
                 continue
-            elif opname in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
-                            'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_KW'):
+            elif (opname in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
+                             'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_EX_KW')
+                  or opname.startswith('CALL_FUNCTION_KW')):
                 self.custom_classfunc_rule(opname, token, customize)
             elif opname == 'LOAD_DICTCOMP':
                 rule_pat = ("dictcomp ::= LOAD_DICTCOMP %sMAKE_FUNCTION_0 expr "
@@ -621,6 +632,26 @@ class Python3Parser(PythonParser):
                 self.add_make_function_rule(rule_pat, opname, token.attr, customize)
             elif opname == 'LOAD_BUILD_CLASS':
                 self.custom_build_class_rule(opname, i, token, tokens, customize)
+            elif opname.startswith('BUILD_LIST_UNPACK'):
+                v = token.attr
+                rule = ('build_list_unpack ::= ' + 'expr1024 ' * int(v//1024) +
+                        'expr32 ' * int((v//32) % 32) +
+                        'expr ' * (v % 32) + opname)
+                self.add_unique_rule(rule, opname, token.attr, customize)
+                rule = 'expr ::= build_list_unpack'
+                self.add_unique_rule(rule, opname, token.attr, customize)
+            elif opname.startswith('BUILD_TUPLE_UNPACK_WITH_CALL'):
+                v = token.attr
+                rule = ('build_tuple_unpack_with_call ::= ' + 'expr1024 ' * int(v//1024) +
+                        'expr32 ' * int((v//32) % 32) +
+                        'expr ' * (v % 32) + opname)
+                self.add_unique_rule(rule, opname, token.attr, customize)
+            elif opname.startswith('BUILD_MAP_UNPACK_WITH_CALL'):
+                v = token.attr
+                rule = ('build_map_unpack_with_call ::= ' + 'expr1024 ' * int(v//1024) +
+                        'expr32 ' * int((v//32) % 32) +
+                        'expr ' * (v % 32) + opname)
+                self.add_unique_rule(rule, opname, token.attr, customize)
             elif opname_base in ('BUILD_LIST', 'BUILD_TUPLE', 'BUILD_SET'):
                 v = token.attr
                 rule = ('build_list ::= ' + 'expr1024 ' * int(v//1024) +
@@ -630,13 +661,17 @@ class Python3Parser(PythonParser):
                 if opname_base == 'BUILD_TUPLE':
                     rule = ('load_closure ::= %s%s' % (('LOAD_CLOSURE ' * v), opname))
                     self.add_unique_rule(rule, opname, token.attr, customize)
-
+                    rule = ('build_tuple ::= ' + 'expr1024 ' * int(v//1024) +
+                        'expr32 ' * int((v//32) % 32) +
+                        'expr ' * (v % 32) + opname)
+                    self.add_unique_rule(rule, opname, token.attr, customize)
             elif opname == 'LOOKUP_METHOD':
                 # A PyPy speciality - DRY with parse2
                 self.add_unique_rule("load_attr ::= expr LOOKUP_METHOD",
                                      opname, token.attr, customize)
                 continue
             elif opname == 'JUMP_IF_NOT_DEBUG':
+                v = token.attr
                 self.add_unique_rule(
                     "stmt ::= assert_pypy", opname, v, customize)
                 self.add_unique_rule(
@@ -675,7 +710,7 @@ class Python3Parser(PythonParser):
                             self.add_unique_rule(rule, opname, token.attr, customize)
                             rule = ('unmap_dict ::= ' +
                                     ('mapexpr ' * token.attr) +
-                                    ' BUILD_MAP_UNPACK')
+                                    'BUILD_MAP_UNPACK')
                         else:
                             rule = kvlist_n + ' ::= ' + 'expr ' * (token.attr*2)
                             self.add_unique_rule(rule, opname, token.attr, customize)
@@ -701,7 +736,30 @@ class Python3Parser(PythonParser):
                 rule = 'unpack_list ::= ' + opname + ' designator' * token.attr
             elif opname_base.startswith('MAKE_FUNCTION'):
                 # DRY with MAKE_CLOSURE
-                args_pos, args_kw, annotate_args  = token.attr
+                if self.version >= 3.6:
+                    # The semantics of MAKE_FUNCTION in 3.6 are totally different from
+                    # before.
+                    args_pos, args_kw, annotate_args, closure  = token.attr
+                    stack_count = args_pos + args_kw + annotate_args
+                    rule = ('mkfunc ::= %s%s%s%s' %
+                                ('expr ' * stack_count,
+                                 'load_closure ' * closure,
+                                 'LOAD_CONST ' * 2,
+                                 opname))
+                    self.add_unique_rule(rule, opname, token.attr, customize)
+                    rule_pat = ('mklambda ::= %s%sLOAD_LAMBDA %%s%s' %
+                                (('pos_arg '* args_pos),
+                                 ('kwarg '* args_kw),
+                                 opname))
+                    self.add_make_function_rule(rule_pat, opname, token.attr, customize)
+                    rule_pat  = ("listcomp ::= %sLOAD_LISTCOMP %%s%s expr "
+                                 "GET_ITER CALL_FUNCTION_1" % ('expr ' * args_pos, opname))
+                    self.add_make_function_rule(rule_pat, opname, token.attr, customize)
+                    continue
+                if self.version < 3.6:
+                    args_pos, args_kw, annotate_args  = token.attr
+                else:
+                    args_pos, args_kw, annotate_args, closure  = token.attr
 
                 rule_pat = ("genexpr ::= %sload_genexpr %%s%s expr "
                             "GET_ITER CALL_FUNCTION_1" % ('pos_arg '* args_pos, opname))
@@ -730,7 +788,18 @@ class Python3Parser(PythonParser):
                             ('pos_arg ' * args_pos, opname))
                 self.add_unique_rule(rule, opname, token.attr, customize)
                 if opname.startswith('MAKE_FUNCTION_A'):
+                    if self.version >= 3.6:
+                        rule = ('mkfunc_annotate ::= %s%sannotate_tuple LOAD_CONST LOAD_CONST %s' %
+                                (('pos_arg ' * (args_pos)),
+                                 ('call_function ' * (annotate_args-1)), opname))
+                        self.add_unique_rule(rule, opname, token.attr, customize)
+                        rule = ('mkfunc_annotate ::= %s%sannotate_tuple LOAD_CONST LOAD_CONST %s' %
+                                (('pos_arg ' * (args_pos)),
+                                 ('annotate_arg ' * (annotate_args-1)), opname))
                     if self.version >= 3.3:
+                        # Normally we remove EXTENDED_ARG from the opcodes, but in the case of
+                        # annotated functions can use the EXTENDED_ARG tuple to signal we have an annotated function.
+                        # Yes this is a little hacky
                         rule = ('mkfunc_annotate ::= %s%sannotate_tuple LOAD_CONST LOAD_CONST EXTENDED_ARG %s' %
                                 (('pos_arg ' * (args_pos)),
                                  ('call_function ' * (annotate_args-1)), opname))
@@ -739,6 +808,7 @@ class Python3Parser(PythonParser):
                                 (('pos_arg ' * (args_pos)),
                                  ('annotate_arg ' * (annotate_args-1)), opname))
                     else:
+                        # See above comment about use of EXTENDED_ARG
                         rule = ('mkfunc_annotate ::= %s%sannotate_tuple LOAD_CONST EXTENDED_ARG %s' %
                                 (('pos_arg ' * (args_pos)),
                                  ('annotate_arg ' * (annotate_args-1)), opname))

uncompyle6/parsers/parse33.py

@@ -20,6 +20,9 @@ class Python33Parser(Python32Parser):
         iflaststmt    ::= testexpr c_stmts_opt33
         c_stmts_opt33 ::= JUMP_BACK JUMP_ABSOLUTE c_stmts_opt
 
+        whileTruestmt ::= SETUP_LOOP l_stmts JUMP_ABSOLUTE
+                          JUMP_BACK COME_FROM_LOOP
+
         # Python 3.5+ has jump optimization to remove the redundant
         # jump_excepts. But in 3.3 we need them added
 

uncompyle6/parsers/parse36.py

@@ -13,21 +13,24 @@ class Python36Parser(Python35Parser):
         super(Python36Parser, self).__init__(debug_parser)
         self.customized = {}
 
+
     def p_36misc(self, args):
         """
-        expr ::= LOAD_NAME EXTENDED_ARG
+        # 3.6 redoes how return_closure works
+        return_closure ::= LOAD_CLOSURE DUP_TOP STORE_NAME RETURN_VALUE RETURN_LAST
 
         fstring_multi ::= fstring_expr_or_strs BUILD_STRING
         fstring_expr_or_strs ::= fstring_expr_or_str+
 
         func_args36   ::= expr BUILD_TUPLE_0
         call_function ::= func_args36 unmapexpr CALL_FUNCTION_EX
+        call_function ::= func_args36 build_map_unpack_with_call CALL_FUNCTION_EX_KW_1
 
         withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST
                      WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
 
         call_function ::= expr expr CALL_FUNCTION_EX
-        call_function ::= expr expr expr CALL_FUNCTION_EX_KW
+        call_function ::= expr expr expr CALL_FUNCTION_EX_KW_1
         """
 
     def add_custom_rules(self, tokens, customize):
@@ -57,6 +60,17 @@ class Python36Parser(Python35Parser):
                 """ % (fstring_expr_or_str_n, fstring_expr_or_str_n, "fstring_expr_or_str " * v)
                 self.add_unique_doc_rules(rules_str, customize)
 
+    def custom_classfunc_rule(self, opname, token, customize):
+
+        if opname.startswith('CALL_FUNCTION_KW'):
+            values = 'expr ' * token.attr
+            rule = 'call_function ::= expr kwargs_only_36 {token.type}'.format(**locals())
+            self.add_unique_rule(rule, token.type, token.attr, customize)
+            rule = 'kwargs_only_36 ::= {values} LOAD_CONST'.format(**locals())
+            self.add_unique_rule(rule, token.type, token.attr, customize)
+        else:
+            super(Python36Parser, self).custom_classfunc_rule(opname, token, customize)
+
 
 class Python36ParserSingle(Python36Parser, PythonParserSingle):
     pass

uncompyle6/scanner.py

@@ -224,6 +224,9 @@ class Scanner(object):
             yield start
             start += self.op_size(self.code[start])
 
+    def next_offset(self, op, offset):
+        return offset + self.op_size(op)
+
     def op_size(self, op):
         """
         Return size of operator with its arguments

uncompyle6/scanners/scanner2.py

@@ -23,20 +23,20 @@ Finally we save token information.
 from uncompyle6 import PYTHON_VERSION
 
 if PYTHON_VERSION < 2.6:
-    from xdis.namedtuple25 import namedtuple
+    from xdis.namedtuple24 import namedtuple
 else:
     from collections import namedtuple
 
 from array import array
 
-from uncompyle6.scanner import op_has_argument
+from uncompyle6.scanner import op_has_argument, L65536
 from xdis.code import iscode
 
-import uncompyle6.scanner as scan
+from uncompyle6.scanner import Scanner
 
-class Scanner2(scan.Scanner):
+class Scanner2(Scanner):
     def __init__(self, version, show_asm=None, is_pypy=False):
-        scan.Scanner.__init__(self, version, show_asm, is_pypy)
+        Scanner.__init__(self, version, show_asm, is_pypy)
         self.pop_jump_if = frozenset([self.opc.PJIF, self.opc.PJIT])
         self.jump_forward = frozenset([self.opc.JUMP_ABSOLUTE, self.opc.JUMP_FORWARD])
         # This is the 2.5+ default
@@ -192,7 +192,7 @@ class Scanner2(scan.Scanner):
                 oparg = self.get_argument(offset) + extended_arg
                 extended_arg = 0
                 if op == self.opc.EXTENDED_ARG:
-                    extended_arg = oparg * scan.L65536
+                    extended_arg = oparg * L65536
                     continue
                 if op in self.opc.hasconst:
                     const = co.co_consts[oparg]

uncompyle6/scanners/scanner26.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015, 2016 by Rocky Bernstein
+#  Copyright (c) 2015-2017 by Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 """
@@ -15,6 +15,7 @@ if PYTHON3:
     intern = sys.intern
 
 import uncompyle6.scanners.scanner2 as scan
+from uncompyle6.scanner import L65536
 
 # bytecode verification, verify(), uses JUMP_OPs from here
 from xdis.opcodes import opcode_26
@@ -180,8 +181,7 @@ class Scanner26(scan.Scanner2):
                 oparg = self.get_argument(offset) + extended_arg
                 extended_arg = 0
                 if op == self.opc.EXTENDED_ARG:
-                    raise NotImplementedError
-                    extended_arg = oparg * scan.L65536
+                    extended_arg = oparg * L65536
                     continue
                 if op in self.opc.hasconst:
                     const = co.co_consts[oparg]

uncompyle6/scanners/scanner3.py

@@ -23,7 +23,7 @@ Finally we save token information.
 from uncompyle6 import PYTHON_VERSION
 
 if PYTHON_VERSION < 2.6:
-    from xdis.namedtuple25 import namedtuple
+    from xdis.namedtuple24 import namedtuple
 else:
     from collections import namedtuple
 
@@ -134,11 +134,21 @@ class Scanner3(Scanner):
             varargs_ops.add(self.opc.CALL_METHOD)
         if self.version >= 3.6:
             varargs_ops.add(self.opc.BUILD_CONST_KEY_MAP)
+            # Below is in bit order, "default = bit 0, closure = bit 3
+            self.MAKE_FUNCTION_FLAGS = tuple("""
+             default keyword-only annotation closure""".split())
 
         self.varargs_ops = frozenset(varargs_ops)
         # FIXME: remove the above in favor of:
         # self.varargs_ops = frozenset(self.opc.hasvargs)
 
+    def extended_arg_val(self, val):
+        if self.version < 3.6:
+            return val * (1<<16)
+        else:
+            return val * (1<<8)
+
+
     def ingest(self, co, classname=None, code_objects={}, show_asm=None):
         """
         Pick out tokens from an uncompyle6 code object, and transform them,
@@ -211,9 +221,29 @@ class Scanner3(Scanner):
         jump_targets = self.find_jump_targets(show_asm)
         last_op_was_break = False
 
-        for inst in bytecode:
+        extended_arg = 0
+        for i, inst in enumerate(bytecode):
 
             argval = inst.argval
+            op     = inst.opcode
+            has_arg = op_has_argument(op, self.opc)
+            if has_arg:
+                if op == self.opc.EXTENDED_ARG:
+                    extended_arg += self.extended_arg_val(argval)
+
+                    # Normally we remove EXTENDED_ARG from the
+                    # opcodes, but in the case of annotated functions
+                    # can use the EXTENDED_ARG tuple to signal we have
+                    # an annotated function.
+                    if not bs[i+1].opname.startswith("MAKE_FUNCTION"):
+                        continue
+
+            if isinstance(argval, int) and extended_arg:
+                min_extended= self.extended_arg_val(1)
+                if argval < min_extended:
+                    argval += extended_arg
+            extended_arg = 0
+
             if inst.offset in jump_targets:
                 jump_idx = 0
                 # We want to process COME_FROMs to the same offset to be in *descending*
@@ -250,12 +280,11 @@ class Scanner3(Scanner):
 
                 pass
 
-            pattr =  inst.argrepr
+            pattr  = inst.argrepr
             opname = inst.opname
-            op = inst.opcode
 
             if opname in ['LOAD_CONST']:
-                const = inst.argval
+                const = argval
                 if iscode(const):
                     if const.co_name == '<lambda>':
                         opname = 'LOAD_LAMBDA'
@@ -277,20 +306,37 @@ class Scanner3(Scanner):
                     pattr = const
                     pass
             elif opname in ('MAKE_FUNCTION', 'MAKE_CLOSURE'):
-                pos_args, name_pair_args, annotate_args = parse_fn_counts(inst.argval)
-                if name_pair_args > 0:
-                    opname = '%s_N%d' % (opname, name_pair_args)
-                    pass
-                if annotate_args > 0:
-                    opname = '%s_A_%d' % (opname, annotate_args)
-                    pass
-                opname = '%s_%d' % (opname, pos_args)
-                pattr = ("%d positional, %d keyword pair, %d annotated" %
-                             (pos_args, name_pair_args, annotate_args))
+                if self.version >= 3.6:
+                    # 3.6+ doesn't have MAKE_CLOSURE, so opname == 'MAKE_FUNCTION'
+                    flags = argval
+                    opname = 'MAKE_FUNCTION_%d' % (flags)
+                    attr = []
+                    for flag in self.MAKE_FUNCTION_FLAGS:
+                        bit = flags & 1
+                        if bit:
+                            if pattr:
+                                pattr += ", " + flag
+                            else:
+                                pattr += flag
+                        attr.append(bit)
+                        flags >>= 1
+                    attr = attr[:4] # remove last value: attr[5] == False
+                else:
+                    pos_args, name_pair_args, annotate_args = parse_fn_counts(inst.argval)
+                    pattr = ("%d positional, %d keyword pair, %d annotated" %
+                                 (pos_args, name_pair_args, annotate_args))
+                    if name_pair_args > 0:
+                        opname = '%s_N%d' % (opname, name_pair_args)
+                        pass
+                    if annotate_args > 0:
+                        opname = '%s_A_%d' % (opname, annotate_args)
+                        pass
+                    opname = '%s_%d' % (opname, pos_args)
+                    attr = (pos_args, name_pair_args, annotate_args)
                 tokens.append(
                     Token(
                         type_ = opname,
-                        attr = (pos_args, name_pair_args, annotate_args),
+                        attr = attr,
                         pattr = pattr,
                         offset = inst.offset,
                         linestart = inst.starts_line,
@@ -301,7 +347,7 @@ class Scanner3(Scanner):
                 )
                 continue
             elif op in self.varargs_ops:
-                pos_args = inst.argval
+                pos_args = argval
                 if self.is_pypy and not pos_args and opname == 'BUILD_MAP':
                     opname = 'BUILD_MAP_n'
                 else:
@@ -313,9 +359,9 @@ class Scanner3(Scanner):
                 customize[opname] = 0
             elif opname == 'UNPACK_EX':
                 # FIXME: try with scanner and parser by
-                # changing inst.argval
-                before_args = inst.argval & 0xFF
-                after_args = (inst.argval >> 8) & 0xff
+                # changing argval
+                before_args = argval & 0xFF
+                after_args = (argval >> 8) & 0xff
                 pattr = "%d before vararg, %d after" % (before_args, after_args)
                 argval = (before_args, after_args)
                 opname = '%s_%d+%d' % (opname, before_args, after_args)
@@ -332,7 +378,7 @@ class Scanner3(Scanner):
                 # comprehensions we might sometimes classify JUMP_BACK
                 # as CONTINUE, but that's okay since we add a grammar
                 # rule for that.
-                pattr = inst.argval
+                pattr = argval
                 target = self.get_target(inst.offset)
                 if target <= inst.offset:
                     next_opname = self.opname[self.code[inst.offset+3]]
@@ -341,11 +387,7 @@ class Scanner3(Scanner):
                         (next_opname not in ('END_FINALLY', 'POP_BLOCK',
                                             # Python 3.0 only uses POP_TOP
                                             'POP_TOP'))):
-                        if (self.version >= 3.4 or
-                            (inst.offset not in self.not_continue) or
-                            (tokens[-1].type == 'RETURN_VALUE')):
-                            opname = 'CONTINUE'
-                            pass
+                        opname = 'CONTINUE'
                     else:
                         opname = 'JUMP_BACK'
                         # FIXME: this is a hack to catch stuff like:
@@ -482,7 +524,7 @@ class Scanner3(Scanner):
                     oparg = code[offset+1]
                 else:
                     oparg = code[offset+1] + code[offset+2] * 256
-                next_offset = offset + self.op_size(op)
+                next_offset = self.next_offset(op, offset)
 
                 if label is None:
                     if op in op3.hasjrel and op != self.opc.FOR_ITER:
@@ -593,14 +635,18 @@ class Scanner3(Scanner):
         Get target offset for op located at given <offset>.
         """
         op = self.code[offset]
+        rel_offset = 0
         if self.version  >= 3.6:
             target = self.code[offset+1]
             if op in self.opc.hasjrel:
-                target += offset + 2
+                rel_offset = offset + 2
         else:
             target = self.code[offset+1] + self.code[offset+2] * 256
             if op in self.opc.hasjrel:
-                target += offset + 3
+                rel_offset = offset + 3
+                pass
+            pass
+        target += rel_offset
 
         return target
 
@@ -745,7 +791,7 @@ class Scanner3(Scanner):
             pre_rtarget = prev_op[rtarget]
 
             # Is it an "and" inside an "if" or "while" block
-            if op == self.opc.POP_JUMP_IF_FALSE:
+            if op == self.opc.POP_JUMP_IF_FALSE and self.version < 3.6:
 
                 # Search for another POP_JUMP_IF_FALSE targetting the same op,
                 # in current statement, starting from current offset, and filter
@@ -917,14 +963,14 @@ class Scanner3(Scanner):
             end    = self.restrict_to_parent(target, parent)
             self.fixed_jumps[offset] = end
         elif op == self.opc.POP_EXCEPT:
-            if self.version <= 3.5:
-                next_offset = offset+1
-            else:
-                next_offset = offset+2
+            next_offset = self.next_offset(op, offset)
             target = self.get_target(next_offset)
             if target > next_offset:
-                self.fixed_jumps[next_offset] = target
-                self.except_targets[target] = next_offset
+                next_op = code[next_offset]
+                if (self.opc.JUMP_ABSOLUTE == next_op and
+                    END_FINALLY != code[self.next_offset(next_op, next_offset)]):
+                    self.fixed_jumps[next_offset] = target
+                    self.except_targets[target] = next_offset
 
         elif op == self.opc.SETUP_FINALLY:
             target = self.get_target(offset)

uncompyle6/scanners/scanner36.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016 by Rocky Bernstein
+#  Copyright (c) 2016-2017 by Rocky Bernstein
 """
 Python 3.6 bytecode scanner/deparser
 
@@ -26,6 +26,12 @@ class Scanner36(Scanner3):
             if t.op == self.opc.CALL_FUNCTION_EX and t.attr & 1:
                 t.type = 'CALL_FUNCTION_EX_KW'
                 pass
+            elif t.op == self.opc.CALL_FUNCTION_KW:
+                t.type = 'CALL_FUNCTION_KW_{t.attr}'.format(**locals())
+            elif t.op == self.opc.BUILD_TUPLE_UNPACK_WITH_CALL:
+                t.type = 'BUILD_TUPLE_UNPACK_WITH_CALL_%d' % t.attr
+            elif t.op == self.opc.BUILD_MAP_UNPACK_WITH_CALL:
+                t.type = 'BUILD_MAP_UNPACK_WITH_CALL_%d' % t.attr
             pass
         return tokens, customize
 

uncompyle6/semantics/fragments.py

@@ -44,7 +44,7 @@ do it recursively which is where offsets are probably located.
 
 For example in:
   'importmultiple':   ( '%|import%b %c%c\n', 0, 2, 3 ),
-
+n
 The node position 0 will be associated with "import".
 
 """
@@ -53,44 +53,39 @@ The node position 0 will be associated with "import".
 
 import re, sys
 
-from uncompyle6 import PYTHON3, IS_PYPY, PYTHON_VERSION
 from xdis.code import iscode
 from uncompyle6.semantics import pysource
 from uncompyle6 import parser
 from uncompyle6.scanner import Token, Code, get_scanner
 from uncompyle6.semantics.check_ast import checker
-from uncompyle6.semantics.helper import print_docstring
 
 from uncompyle6.show import (
     maybe_show_asm,
     maybe_show_ast,
-    maybe_show_ast_param_default,
 )
 
 from uncompyle6.parsers.astnode import AST
 
 from uncompyle6.semantics.pysource import (
-    ParserError, find_globals, StringIO)
+    ParserError, StringIO)
 
 from uncompyle6.semantics.consts import (
     INDENT_PER_LEVEL, NONE, PRECEDENCE,
     TABLE_DIRECT, escape, minint, MAP
     )
 
-from uncompyle6.semantics.make_function import (
-    find_all_globals, find_none, code_has_star_arg, code_has_star_star_arg
-)
-
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser.ast import GenericASTTraversalPruningException
 
+from uncompyle6 import PYTHON_VERSION
 if PYTHON_VERSION < 2.6:
-    from xdis.namedtuple25 import namedtuple
+    from xdis.namedtuple24 import namedtuple
 else:
     from collections import namedtuple
 
 NodeInfo = namedtuple("NodeInfo", "node start finish")
 ExtractInfo = namedtuple("ExtractInfo",
-                         "lineNo lineStartOffset markerLine selectedLine selectedText")
+                         "lineNo lineStartOffset markerLine selectedLine selectedText nonterminal")
 
 TABLE_DIRECT_FRAGMENT = {
     'break_stmt':	( '%|%rbreak\n', ),
@@ -168,8 +163,9 @@ class FragmentsWalker(pysource.SourceWalker, object):
     def set_pos_info(self, node, start, finish, name=None):
         if name is None: name = self.name
         if hasattr(node, 'offset'):
-            self.offsets[name, node.offset] = \
-              NodeInfo(node = node, start = start, finish = finish)
+            node.start = start
+            node.finish = finish
+            self.offsets[name, node.offset] = node
 
         if hasattr(node, 'parent'):
             assert node.parent != node
@@ -185,6 +181,34 @@ class FragmentsWalker(pysource.SourceWalker, object):
 
         return
 
+    def table_r_node(self, node):
+        """General pattern where the last node should should
+        get the text span attributes of the entire tree"""
+        start = len(self.f.getvalue())
+        try:
+            self.default(node)
+        except GenericASTTraversalPruningException:
+            final = len(self.f.getvalue())
+            self.set_pos_info(node, start, final)
+            self.set_pos_info(node[-1], start, final)
+            raise GenericASTTraversalPruningException
+
+    n_slice0 = n_slice1 = n_slice2 = n_slice3 = n_binary_subscr = table_r_node
+    n_augassign_1 = n_print_item = exec_stmt = print_to_item = del_stmt = table_r_node
+    n_classdefco1 = n_classdefco2 = except_cond1 = except_cond2 = table_r_node
+
+    def n_passtmt(self, node):
+        start = len(self.f.getvalue()) + len(self.indent)
+        self.set_pos_info(node, start, start+len("pass"))
+        self.default(node)
+
+    def n_trystmt(self, node):
+        start = len(self.f.getvalue()) + len(self.indent)
+        self.set_pos_info(node[0], start, start+len("try:"))
+        self.default(node)
+
+    n_tryelsestmt = n_tryelsestmtc = n_tryelsestmtl = n_tryfinallystmt = n_trystmt
+
     def n_return_stmt(self, node):
         start = len(self.f.getvalue()) + len(self.indent)
         if self.params['isLambda']:
@@ -238,6 +262,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
             self.write(' ')
             node[0].parent = node
             self.preorder(node[0])
+        self.set_pos_info(node[-1], start, len(self.f.getvalue()))
         self.set_pos_info(node, start, len(self.f.getvalue()))
         self.prune() # stop recursing
 
@@ -375,15 +400,23 @@ class FragmentsWalker(pysource.SourceWalker, object):
                 self.write(sep); sep = ", "
                 self.preorder(subnode)
         self.set_pos_info(node, start, len(self.f.getvalue()))
+        self.set_pos_info(node[-1], start, len(self.f.getvalue()))
         self.println()
         self.prune() # stop recursing
 
     def n_ifelsestmtr(self, node):
-        if len(node[2]) != 2:
+        if node[2] == 'COME_FROM':
+            return_stmts_node = node[3]
+            node.type = 'ifelsestmtr2'
+        else:
+            return_stmts_node = node[2]
+        if len(return_stmts_node) != 2:
             self.default(node)
 
-        if not (node[2][0][0][0] == 'ifstmt' and node[2][0][0][0][1][0] == 'return_if_stmts') \
-                and not (node[2][0][-1][0] == 'ifstmt' and node[2][0][-1][0][1][0] == 'return_if_stmts'):
+        if (not (return_stmts_node[0][0][0] == 'ifstmt'
+                 and return_stmts_node[0][0][0][1][0] == 'return_if_stmts')
+            and not (return_stmts_node[0][-1][0] == 'ifstmt'
+                     and return_stmts_node[0][-1][0][1][0] == 'return_if_stmts')):
             self.default(node)
             return
 
@@ -404,7 +437,7 @@ class FragmentsWalker(pysource.SourceWalker, object):
 
         past_else = False
         prev_stmt_is_if_ret = True
-        for n in node[2][0]:
+        for n in return_stmts_node[0]:
             if (n[0] == 'ifstmt' and n[0][1][0] == 'return_if_stmts'):
                 if prev_stmt_is_if_ret:
                     n[0].type = 'elifstmt'
@@ -486,17 +519,20 @@ class FragmentsWalker(pysource.SourceWalker, object):
             # LOAD_CONST code object ..
             # LOAD_CONST        'x0'  if >= 3.3
             # MAKE_FUNCTION ..
-            code_index = -3
+            code_node = node[-3]
+        elif node[-2] == 'expr':
+            code_node = node[-2][0]
         else:
             # LOAD_CONST code object ..
             # MAKE_FUNCTION ..
-            code_index = -2
-        code = node[code_index]
-        func_name = code.attr.co_name
+            code_node = node[-2]
+        func_name = code_node.attr.co_name
         self.write(func_name)
+        self.set_pos_info(code_node, start, len(self.f.getvalue()))
 
         self.indentMore()
-        self.make_function(node, isLambda=False, code=code)
+        start = len(self.f.getvalue())
+        self.make_function(node, isLambda=False, codeNode=code_node)
 
         self.set_pos_info(node, start, len(self.f.getvalue()))
 
@@ -1101,7 +1137,6 @@ class FragmentsWalker(pysource.SourceWalker, object):
     def traverse(self, node, indent=None, isLambda=False):
         '''Buulds up fragment which can be used inside a larger
         block of code'''
-
         self.param_stack.append(self.params)
         if indent is None: indent = self.indent
         p = self.pending_newlines
@@ -1196,16 +1231,38 @@ class FragmentsWalker(pysource.SourceWalker, object):
 
         if elided: selectedLine += ' ...'
 
+        if isinstance(nodeInfo, Token):
+            nodeInfo = nodeInfo.parent
+        else:
+            nodeInfo = nodeInfo
+
+        if isinstance(nodeInfo, AST):
+            nonterminal = nodeInfo[0]
+        else:
+            nonterminal = nodeInfo.node
+
         return ExtractInfo(lineNo = len(lines), lineStartOffset = lineStart,
                            markerLine = markerLine,
                            selectedLine = selectedLine,
-                           selectedText = selectedText)
+                           selectedText = selectedText,
+                           nonterminal = nonterminal)
 
     def extract_line_info(self, name, offset):
         if (name, offset) not in list(self.offsets.keys()):
             return None
         return self.extract_node_info(self.offsets[name, offset])
 
+    def prev_node(self, node):
+        prev = None
+        if not hasattr(node, 'parent'):
+            return prev
+        p = node.parent
+        for n in p:
+            if node == n:
+                return prev
+            prev = n
+        return prev
+
     def extract_parent_info(self, node):
         if not hasattr(node, 'parent'):
             return None, None
@@ -1569,155 +1626,16 @@ class FragmentsWalker(pysource.SourceWalker, object):
             self.set_pos_info(startnode, startnode_start, fin)
 
         # FIXME rocky: figure out how to get these casess to be table driven.
-        #
-        # 1. for loops. For loops have two positions that correspond to a single text
-        # location. In "for i in ..." there is the initialization "i" code as well
-        # as the iteration code with "i".  A "copy" spec like %X3,3 - copy parame
-        # 3 to param 2 would work
-        #
         # 2. subroutine calls. It the last op is the call and for purposes of printing
         # we don't need to print anything special there. However it encompases the
         # entire string of the node fn(...)
-        match = re.search(r'^try', startnode.type)
+        match = re.search(r'^call_function', startnode.type)
         if match:
-            self.set_pos_info(node[0], startnode_start, startnode_start+len("try:"))
-            self.set_pos_info(node[2], node[3].finish, node[3].finish)
-        else:
-            match = re.search(r'^call_function', startnode.type)
-            if match:
-                last_node = startnode[-1]
-                # import traceback; traceback.print_stack()
-                self.set_pos_info(last_node, startnode_start, self.last_finish)
+            last_node = startnode[-1]
+            # import traceback; traceback.print_stack()
+            self.set_pos_info(last_node, startnode_start, self.last_finish)
         return
 
-    def make_function(self, node, isLambda, nested=1, code=None):
-        """Dump function defintion, doc string, and function body."""
-
-        def build_param(ast, name, default):
-            """build parameters:
-                - handle defaults
-                - handle format tuple parameters
-            """
-            if self.version < 3.0:
-                # if formal parameter is a tuple, the paramater name
-                # starts with a dot (eg. '.1', '.2')
-                if name.startswith('.'):
-                    # replace the name with the tuple-string
-                    name = self.get_tuple_parameter(ast, name)
-                    pass
-                pass
-
-            if default:
-                maybe_show_ast_param_default(self.showast, name, default)
-                result = '%s=' % name
-                old_last_finish = self.last_finish
-                self.last_finish = len(result)
-                value = self.traverse(default, indent='')
-                self.last_finish = old_last_finish
-                result += value
-                if result[-2:] == '= ':	# default was 'LOAD_CONST None'
-                    result += 'None'
-                return result
-            else:
-                return name
-
-        # MAKE_FUNCTION_... or MAKE_CLOSURE_...
-        assert node[-1].type.startswith('MAKE_')
-
-        args_node = node[-1]
-        if isinstance(args_node.attr, tuple):
-            if self.version <= 3.3 and len(node) > 2 and node[-3] != 'LOAD_LAMBDA':
-                # positional args are after kwargs
-                defparams = node[1:args_node.attr[0]+1]
-            else:
-                # positional args are before kwargs
-                defparams = node[:args_node.attr[0]]
-            pos_args, kw_args, annotate_argc  = args_node.attr
-        else:
-            defparams = node[:args_node.attr]
-            kw_args, annotate_argc  = (0, 0)
-            pass
-
-        if self.version > 3.0 and isLambda and iscode(node[-3].attr):
-            code = node[-3].attr
-        else:
-            code = code.attr
-
-        assert iscode(code)
-        code = Code(code, self.scanner, self.currentclass)
-
-        # add defaults values to parameter names
-        argc = code.co_argcount
-        paramnames = list(code.co_varnames[:argc])
-
-        # defaults are for last n parameters, thus reverse
-        paramnames.reverse(); defparams.reverse()
-
-        try:
-            ast = self.build_ast(code._tokens,
-                                 code._customize,
-                                 isLambda = isLambda,
-                                 noneInNames = ('None' in code.co_names))
-        except ParserError(p):
-            self.write(str(p))
-            self.ERROR = p
-            return
-
-        # build parameters
-
-        tup = [paramnames, defparams]
-        params = [build_param(ast, name, default) for
-              name, default in map(lambda *tup:tup, *tup)]
-        params.reverse() # back to correct order
-
-        if code_has_star_arg(code):
-            params.append('*%s' % code.co_varnames[argc])
-            argc += 1
-        if code_has_star_star_arg(code):
-            params.append('**%s' % code.co_varnames[argc])
-            argc += 1
-
-        # dump parameter list (with default values)
-        indent = self.indent
-        if isLambda:
-            self.write("lambda ", ", ".join(params))
-        else:
-            self.write("(", ", ".join(params))
-            # self.println(indent, '#flags:\t', int(code.co_flags))
-
-        if kw_args > 0:
-            if argc > 0:
-                self.write(", *, ")
-            else:
-                self.write("*, ")
-            for n in node:
-                if n == 'pos_arg':
-                    continue
-                self.preorder(n)
-                break
-            pass
-
-        if isLambda:
-            self.write(": ")
-        else:
-            self.println("):")
-
-        if len(code.co_consts)>0 and code.co_consts[0] is not None and not isLambda: # ugly
-            # docstring exists, dump it
-            print_docstring(self, indent, code.co_consts[0])
-
-        code._tokens = None # save memory
-        assert ast == 'stmts'
-
-        all_globals = find_all_globals(ast, set())
-        for g in ((all_globals & self.mod_globs) | find_globals(ast, set())):
-            self.println(self.indent, 'global ', g)
-        self.mod_globs -= all_globals
-        rn = ('None' in code.co_names) and not find_none(ast)
-        self.gen_source(ast, code.co_name, code._customize, isLambda=isLambda,
-                          returnNone=rn)
-        code._tokens = None; code._customize = None # save memory
-
     @classmethod
     def _get_mapping(cls, node):
         return MAP.get(node, MAP_DIRECT_FRAGMENT)
@@ -1797,10 +1715,48 @@ def deparse_code(version, co, out=StringIO(), showasm=False, showast=False,
     if deparsed.ERROR:
         raise deparsed.ERROR
 
+    # To keep the API consistent with previous releases, convert
+    # deparse.offset values into NodeInfo items
+    for tup, node in deparsed.offsets.items():
+        deparsed.offsets[tup] = NodeInfo(node = node, start = node.start,
+                                         finish = node.finish)
+
+    deparsed.scanner = scanner
     return deparsed
 
+from bisect import bisect_right
+def find_gt(a, x):
+    'Find leftmost value greater than x'
+    i = bisect_right(a, x)
+    if i != len(a):
+        return a[i]
+    raise ValueError
+
+def deparse_code_around_offset(name, offset, version, co, out=StringIO(),
+                               showasm=False, showast=False,
+                               showgrammar=False, is_pypy=False):
+    """
+    Like deparse_code(), but given  a function/module name and
+    offset, finds the node closest to offset. If offset is not an instruction boundary,
+    we raise an IndexError.
+    """
+    deparsed = deparse_code(version, co, out, showasm, showast, showgrammar, is_pypy)
+    if (name, offset) in deparsed.offsets.keys():
+        # This is the easy case
+        return deparsed
+
+    valid_offsets = [t for t in deparsed.offsets if isinstance(t[1], int)]
+    offset_list = sorted([t[1] for t in valid_offsets if t[0] == name])
+
+    # FIXME: should check for branching?
+    found_offset = find_gt(offset_list, offset)
+    deparsed.offsets[name, offset] = deparsed.offsets[name, found_offset]
+    return deparsed
+
+
 if __name__ == '__main__':
 
+    from uncompyle6 import IS_PYPY
     def deparse_test(co, is_pypy=IS_PYPY):
         sys_version = sys.version_info.major + (sys.version_info.minor / 10.0)
         walk = deparse_code(sys_version, co, showasm=False, showast=False,
@@ -1830,6 +1786,35 @@ if __name__ == '__main__':
             pass
         return
 
+    def deparse_test_around(offset, name, co, is_pypy=IS_PYPY):
+        sys_version = sys.version_info.major + (sys.version_info.minor / 10.0)
+        walk = deparse_code_around_offset(name, offset, sys_version, co, showasm=False, showast=False,
+                            showgrammar=False, is_pypy=IS_PYPY)
+        print("deparsed source")
+        print(walk.text, "\n")
+        print('------------------------')
+        for name, offset in sorted(walk.offsets.keys(),
+                                   key=lambda x: str(x[0])):
+            print("name %s, offset %s" % (name, offset))
+            nodeInfo = walk.offsets[name, offset]
+            node = nodeInfo.node
+            extractInfo = walk.extract_node_info(node)
+            print("code: %s" % node.type)
+            # print extractInfo
+            print(extractInfo.selectedText)
+            print(extractInfo.selectedLine)
+            print(extractInfo.markerLine)
+            extractInfo, p = walk.extract_parent_info(node)
+            if extractInfo:
+                print("Contained in...")
+                print(extractInfo.selectedLine)
+                print(extractInfo.markerLine)
+                print("code: %s" % p.type)
+                print('=' * 40)
+                pass
+            pass
+        return
+
     def get_code_for_fn(fn):
         return fn.__code__
 
@@ -1849,6 +1834,9 @@ if __name__ == '__main__':
 
     # check_args(['3', '5'])
     # deparse_test(get_code_for_fn(gcd))
-    deparse_test(get_code_for_fn(test))
+    # deparse_test(get_code_for_fn(test))
     # deparse_test(get_code_for_fn(FragmentsWalker.fixup_offsets))
+    # deparse_test(get_code_for_fn(FragmentsWalker.n_build_list))
+    print('=' * 30)
+    deparse_test_around(408, 'n_build_list', get_code_for_fn(FragmentsWalker.n_build_list))
     # deparse_test(inspect.currentframe().f_code)

uncompyle6/semantics/make_function.py

@@ -463,7 +463,17 @@ def make_function3(self, node, isLambda, nested=1, codeNode=None):
             defparams = node[:args_node.attr[0]]
         pos_args, kw_args, annotate_argc  = args_node.attr
     else:
-        defparams = node[:args_node.attr]
+        if self.version < 3.6:
+            defparams = node[:args_node.attr]
+        else:
+            default, kw, annotate, closure = args_node.attr
+            # FIXME: start here.
+            defparams = []
+            # if default:
+            #     defparams = node[-(2 +  kw + annotate  + closure)]
+            # else:
+            #     defparams = []
+
         kw_args  = 0
         pass
 

uncompyle6/semantics/pysource.py

@@ -344,7 +344,7 @@ class SourceWalker(GenericASTTraversal, object):
                         'async_with_as_stmt':  (
                             '%|async with %c as %c:\n%+%c%-', 0, 6, 7),
                         'unmap_dict':	       ( '{**%C}', (0, -1, ', **') ),
-                        'unmapexpr':	       ( '{**%c}', 0),
+                        # 'unmapexpr':	       ( '{**%c}', 0), # done by n_unmapexpr
 
                     })
                     def n_async_call_function(node):
@@ -356,11 +356,43 @@ class SourceWalker(GenericASTTraversal, object):
                         self.prec = p
                         self.prune()
                     self.n_async_call_function = n_async_call_function
+                    self.n_build_list_unpack = self.n_build_list
+
+                    if version == 3.5:
+                        def n_call_function(node):
+                            mapping = self._get_mapping(node)
+                            table = mapping[0]
+                            key = node
+                            for i in mapping[1:]:
+                                key = key[i]
+                                pass
+                            if key.type.startswith('CALL_FUNCTION_VAR_KW'):
+                                # Python 3.5 changes the stack position of *args. kwargs come
+                                # after *args whereas in earlier Pythons, *args is at the end
+                                # which simpilfiies things from our perspective.
+                                # Python 3.6+ replaces CALL_FUNCTION_VAR_KW with CALL_FUNCTION_EX
+                                # We will just swap the order to make it look like earlier Python 3.
+                                entry = table[key.type]
+                                kwarg_pos = entry[2][1]
+                                args_pos = kwarg_pos - 1
+                                # Put last node[args_pos] after subsequent kwargs
+                                while node[kwarg_pos] == 'kwarg' and kwarg_pos < len(node):
+                                    # swap node[args_pos] with node[kwargs_pos]
+                                    node[kwarg_pos], node[args_pos] = node[args_pos], node[kwarg_pos]
+                                    args_pos = kwarg_pos
+                                    kwarg_pos += 1
+                            self.default(node)
+                        self.n_call_function = n_call_function
 
                     def n_funcdef(node):
-                        code_node = node[0][1]
-                        if (code_node == 'LOAD_CONST' and iscode(code_node.attr)
-                            and code_node.attr.co_flags & COMPILER_FLAG_BIT['COROUTINE']):
+                        if self.version == 3.6:
+                            code_node = node[0][0]
+                        else:
+                            code_node = node[0][1]
+
+                        is_code = hasattr(code_node, 'attr') and iscode(code_node.attr)
+                        if (is_code and
+                            (code_node.attr.co_flags & COMPILER_FLAG_BIT['COROUTINE'])):
                             self.engine(('\n\n%|async def %c\n', -2), node)
                         else:
                             self.engine(('\n\n%|def %c\n', -2), node)
@@ -375,8 +407,6 @@ class SourceWalker(GenericASTTraversal, object):
                                 self.f.write(', **')
                                 pass
                             pass
-                        if version >= 3.6:
-                            self.f.write(')')
                         self.prune()
                         pass
                     self.n_unmapexpr = n_unmapexpr
@@ -390,11 +420,12 @@ class SourceWalker(GenericASTTraversal, object):
                         'fstring_single': ( "f'{%c%{conversion}}'", 0),
                         'fstring_multi':  ( "f'%c'", 0),
                         'func_args36':    ( "%c(**", 0),
+                        #'kwargs_only_36': ( "%c(**", 0),
                     })
                     TABLE_R.update({
                         'CALL_FUNCTION_EX': ('%c(*%P)', 0, (1, 2, ', ', 100)),
                         # Not quite right
-                        'CALL_FUNCTION_EX_KW': ('%c(**%C', 0, (2,3, ',')),
+                        'CALL_FUNCTION_EX_KW': ('%c(**%C)', 0, (2,3, ',')),
                         })
                     FSTRING_CONVERSION_MAP = {1: '!s', 2: '!r', 3: '!a'}
 
@@ -411,6 +442,27 @@ class SourceWalker(GenericASTTraversal, object):
                         self.default(node)
                     self.n_fstring_single = n_fstring_single
 
+                    def n_kwargs_only_36(node):
+                        keys = node[-1].attr
+                        num_kwargs = len(keys)
+                        values = node[:num_kwargs]
+                        for i, (key, value) in enumerate(zip(keys, values)):
+                            self.write(key + '=')
+                            self.preorder(value)
+                            if i < num_kwargs:
+                                self.write(',')
+                        self.prune()
+                        return
+                    self.n_kwargs_only_36 = n_kwargs_only_36
+
+                    def n_return_closure(node):
+                        # Nothing should be output here
+                        self.prune()
+                        return
+                    self.n_return_closure = n_return_closure
+                    pass # version > 3.6
+                pass # version > 3.4
+            pass # version > 3.0
         return
 
     f = property(lambda s: s.params['f'],
@@ -517,6 +569,33 @@ class SourceWalker(GenericASTTraversal, object):
                     node == AST('return_stmt',
                                 [AST('ret_expr', [NONE]), Token('RETURN_VALUE')]))
 
+    ## The below doesn't work because continue may be the only thing inside an 'else'. For example
+    # for ...
+    #   if ...
+    #   else:
+    #     continue
+    #
+    # def n_continue_stmt(self, node):
+    #     if self.version >= 3.0 and node[0] == 'CONTINUE':
+    #         t = node[0]
+    #         if not t.linestart:
+    #             # Artificially-added "continue" statements derived from JUMP_ABSOLUTE
+    #             # don't have line numbers associated with them.
+    #             # If this is a CONTINUE is to the same target as a JUMP_ABSOLUTE following it,
+    #             # then the "continue" can be suppressed.
+    #             op, offset = t.op, t.offset
+    #             next_offset = self.scanner.next_offset(op, offset)
+    #             scanner = self.scanner
+    #             code = scanner.code
+    #             if next_offset < len(code):
+    #                 next_inst = code[next_offset]
+    #                 if (scanner.opc.opname[next_inst] == 'JUMP_ABSOLUTE'
+    #                     and t.pattr == code[next_offset+1]):
+    #                     # Suppress "continue"
+    #                     import pdb; pdb.set_trace()
+    #                     self.prune()
+    #     self.default(node)
+
     def n_return_stmt(self, node):
         if self.params['isLambda']:
             self.preorder(node[0])
@@ -850,19 +929,19 @@ class SourceWalker(GenericASTTraversal, object):
             # LOAD_CONST code object ..
             # LOAD_CONST        'x0'  if >= 3.3
             # MAKE_FUNCTION ..
-            code = node[-3]
+            code_node = node[-3]
         elif node[-2] == 'expr':
-            code = node[-2][0]
+            code_node = node[-2][0]
         else:
             # LOAD_CONST code object ..
             # MAKE_FUNCTION ..
-            code = node[-2]
+            code_node = node[-2]
 
-        func_name = code.attr.co_name
+        func_name = code_node.attr.co_name
         self.write(func_name)
 
         self.indentMore()
-        self.make_function(node, isLambda=False, codeNode=code)
+        self.make_function(node, isLambda=False, codeNode=code_node)
 
         if len(self.param_stack) > 1:
             self.write('\n\n')
@@ -1269,11 +1348,18 @@ class SourceWalker(GenericASTTraversal, object):
 
         if self.version > 3.0:
             if node == 'classdefdeco2':
-                currentclass = node[1][2].pattr
+                if self.version >= 3.6:
+                    class_name = node[1][1].pattr
+                else:
+                    class_name = node[1][2].pattr
                 buildclass = node
             else:
-                currentclass = node[1][0].pattr
-                buildclass = node[0]
+                if self.version >= 3.6:
+                    class_name = node[0][1][0].attr.co_name
+                    buildclass = node[0]
+                else:
+                    class_name = node[1][0].pattr
+                    buildclass = node[0]
 
             assert 'mkfunc' == buildclass[1]
             mkfunc = buildclass[1]
@@ -1281,16 +1367,16 @@ class SourceWalker(GenericASTTraversal, object):
                 if 3.0 <= self.version <= 3.2:
                     for n in mkfunc:
                         if hasattr(n, 'attr') and iscode(n.attr):
-                            subclass = n.attr
+                            subclass_code = n.attr
                             break
                         elif n == 'expr':
-                            subclass = n[0].attr
+                            subclass_code = n[0].attr
                         pass
                     pass
                 else:
                     for n in mkfunc:
                         if hasattr(n, 'attr') and iscode(n.attr):
-                            subclass = n.attr
+                            subclass_code = n.attr
                             break
                         pass
                     pass
@@ -1305,10 +1391,10 @@ class SourceWalker(GenericASTTraversal, object):
                     # Python 3.3 classes with closures work like this.
                     # Note have to test before 3.2 case because
                     # index -2 also has an attr.
-                    subclass = load_closure[-3].attr
+                    subclass_code = load_closure[-3].attr
                 elif hasattr(load_closure[-2], 'attr'):
                     # Python 3.2 works like this
-                    subclass = load_closure[-2].attr
+                    subclass_code = load_closure[-2].attr
                 else:
                     raise 'Internal Error n_classdef: cannot find class body'
                 if hasattr(buildclass[3], '__len__'):
@@ -1317,8 +1403,11 @@ class SourceWalker(GenericASTTraversal, object):
                     subclass_info = buildclass[2]
                 else:
                     raise 'Internal Error n_classdef: cannot superclass name'
+            elif self.version >= 3.6 and node == 'classdefdeco2':
+                subclass_info = node
+                subclass_code = buildclass[1][0].attr
             else:
-                subclass = buildclass[1][0].attr
+                subclass_code = buildclass[1][0].attr
                 subclass_info = node[0]
         else:
             if node == 'classdefdeco2':
@@ -1327,11 +1416,11 @@ class SourceWalker(GenericASTTraversal, object):
                 buildclass = node[0]
             build_list = buildclass[1][0]
             if hasattr(buildclass[-3][0], 'attr'):
-                subclass = buildclass[-3][0].attr
-                currentclass = buildclass[0].pattr
+                subclass_code = buildclass[-3][0].attr
+                class_name = buildclass[0].pattr
             elif hasattr(node[0][0], 'pattr'):
-                subclass = buildclass[-3][1].attr
-                currentclass = node[0][0].pattr
+                subclass_code = buildclass[-3][1].attr
+                class_name = node[0][0].pattr
             else:
                 raise 'Internal Error n_classdef: cannot find class name'
 
@@ -1340,7 +1429,7 @@ class SourceWalker(GenericASTTraversal, object):
         else:
             self.write('\n\n')
 
-        self.currentclass = str(currentclass)
+        self.currentclass = str(class_name)
         self.write(self.indent, 'class ', self.currentclass)
 
         if self.version > 3.0:
@@ -1351,7 +1440,7 @@ class SourceWalker(GenericASTTraversal, object):
 
         # class body
         self.indentMore()
-        self.build_class(subclass)
+        self.build_class(subclass_code)
         self.indentLess()
 
         self.currentclass = cclass
@@ -1487,10 +1576,21 @@ class SourceWalker(GenericASTTraversal, object):
                 values = node[:-2]
                 # FIXME: Line numbers?
                 for key, value in zip(keys, values):
+                    self.write(sep)
                     self.write(repr(key))
+                    line_number = self.line_number
                     self.write(':')
                     self.write(self.traverse(value[0]))
-                    self.write(',')
+                    sep = ","
+                    if line_number != self.line_number:
+                        sep += "\n" + self.indent + INDENT_PER_LEVEL[:-1]
+                        line_number = self.line_number
+                    else:
+                        sep += " "
+                        pass
+                    pass
+                if sep.startswith(",\n"):
+                    self.write(sep[1:])
                 pass
             pass
         else:
@@ -1544,6 +1644,9 @@ class SourceWalker(GenericASTTraversal, object):
                 if line_number != self.line_number:
                     sep += "\n" + self.indent + "  "
                     line_number = self.line_number
+                    pass
+                pass
+            pass
         if sep.startswith(",\n"):
             self.write(sep[1:])
         self.write('}')
@@ -1566,6 +1669,13 @@ class SourceWalker(GenericASTTraversal, object):
         # will assume that if the text ends in *.
         last_was_star = self.f.getvalue().endswith('*')
 
+        if lastnodetype.endswith('UNPACK'):
+            # FIXME: need to handle range of BUILD_LIST_UNPACK
+            have_star = True
+            # endchar = ''
+        else:
+            have_star = False
+
         if lastnodetype.startswith('BUILD_LIST'):
             self.write('['); endchar = ']'
         elif lastnodetype.startswith('BUILD_TUPLE'):
@@ -1577,11 +1687,7 @@ class SourceWalker(GenericASTTraversal, object):
         elif lastnodetype.startswith('ROT_TWO'):
             self.write('('); endchar = ')'
         else:
-            raise 'Internal Error: n_build_list expects list, tuple, set, or unpack'
-        have_star = False
-        if lastnodetype.endswith('UNPACK'):
-            # FIXME: need to handle range of BUILD_LIST_UNPACK
-            have_star = True
+            raise TypeError('Internal Error: n_build_list expects list, tuple, set, or unpack')
 
         flat_elems = []
         for elem in node:
@@ -1599,7 +1705,7 @@ class SourceWalker(GenericASTTraversal, object):
         sep = ''
 
         for elem in flat_elems:
-            if elem == 'ROT_THREE':
+            if elem in ('ROT_THREE', 'EXTENDED_ARG'):
                 continue
             assert elem == 'expr'
             line_number = self.line_number
@@ -1688,12 +1794,8 @@ class SourceWalker(GenericASTTraversal, object):
 
             typ = m.group('type') or '{'
             node = startnode
-            try:
-                if m.group('child'):
-                    node = node[int(m.group('child'))]
-            except:
-                print(node.__dict__)
-                raise
+            if m.group('child'):
+                node = node[int(m.group('child'))]
 
             if   typ == '%':	self.write('%')
             elif typ == '+':
@@ -1768,7 +1870,6 @@ class SourceWalker(GenericASTTraversal, object):
                 try:
                     self.write(eval(expr, d, d))
                 except:
-                    print(node)
                     raise
             m = escape.search(fmt, i)
         self.write(fmt[i:])
@@ -1799,6 +1900,8 @@ class SourceWalker(GenericASTTraversal, object):
             if k.startswith('CALL_METHOD'):
                 # This happens in PyPy only
                 TABLE_R[k] = ('%c(%P)', 0, (1, -1, ', ', 100))
+            elif self.version >= 3.6 and k.startswith('CALL_FUNCTION_KW'):
+                TABLE_R[k] = ('%c(%P)', 0, (1, -1, ', ', 100))
             elif op == 'CALL_FUNCTION':
                 TABLE_R[k] = ('%c(%P)', 0, (1, -1, ', ', 100))
             elif op in ('CALL_FUNCTION_VAR',
@@ -1815,6 +1918,9 @@ class SourceWalker(GenericASTTraversal, object):
                     if self.version == 3.5:
                         if str == '%c(%C, ':
                             str = '%c(*%C, %c)'
+                        elif str == '%c(%C':
+                            str = '%c(*%C)'
+                            # p2 = (1, -1, 100)
                     else:
                         str += '*%c)'
                     entry = (str, 0, p2, -2)

uncompyle6/version.py

@@ -1,3 +1,3 @@
 # This file is suitable for sourcing inside bash as
 # well as importing into Python
-VERSION='2.9.11'
+VERSION='2.11.0'