Get ready for release 2.3.1

LOAD_FAST         '__locals__'
STORE_LOCALS      ''

Also have to adjust doc constants for this crap

astnode.py: minor format change

.gitignore

@@ -3,9 +3,12 @@
 /.cache
 /.eggs
 /.python-version
+/.tox
+/README
 /__pkginfo__.pyc
 /dist
 /how-to-make-a-release.txt
+/tmp
 /uncompyle6.egg-info
 __pycache__
 build

.travis.yml

@@ -9,6 +9,7 @@ python:
   - '3.5'
 
 install:
+- pip install -r requirements.txt
 - pip install -r requirements-dev.txt
 
 script:

ChangeLog

@@ -1,6 +1,216 @@
+2016-04-30  rocky <rocky@gnu.org>
+
+	* README.rst, __pkginfo__.py: Get ready for release 2.3.0
+
+2016-04-30  rocky <rocky@gnu.org>
+
+	* uncompyle6/parsers/astnode.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/semantics/pysource.py: Python 3.0..3.2 bug in
+	LOAD_FAST/STORE_LOCAL LOAD_FAST         '__locals__' STORE_LOCALS      '' Also have to adjust doc constants for this crap astnode.py: minor format change
+
+2016-04-30  rocky <rocky@gnu.org>
+
+	* test/Makefile, test/simple_source/def/06_classbug.py,
+	test/test_pythonlib.py: Test optimized Python code and Python 3.2
+
+2016-04-30  rocky <rocky@gnu.org>
+
+	* uncompyle6/parsers/parse3.py: Pevious commit grammar change is
+	Python 3.5 and up
+
+2016-04-30  rocky <rocky@gnu.org>
+
+	* uncompyle6/parsers/parse3.py: Python 3.5 if statments decompyle Sometimes it doesn't need JUMP_FORWARD _come_from _come_from For example: def handle2(module):     if module == 'foo':         try:             module = 1         except ImportError as exc:             module = exc     return module And: if __name__:     for i in (1, 2):         x = 3
+
+2016-04-28  rocky <rocky@gnu.org>
+
+	* requirements.txt, uncompyle6/parser.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/semantics/fragments.py, uncompyle6/semantics/pysource.py: 
+	spark -> spark_parser
+
+2016-04-28  rocky <rocky@gnu.org>
+
+	* uncompyle6/parsers/spark.py: Really remove spark - Use external
+	package instead
+
+2016-04-27  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #8 from rocky/external-spark External spark
+
+2016-04-27  rocky <rocky@gnu.org>
+
+	* .travis.yml, circle.yml: Note dependencies on spark
+
+2016-04-27  rocky <rocky@gnu.org>
+
+	* .gitignore, README.rst, requirements.txt, uncompyle6/parser.py,
+	uncompyle6/parsers/astnode.py, uncompyle6/parsers/parse2.py,
+	uncompyle6/parsers/parse3.py, uncompyle6/semantics/fragments.py,
+	uncompyle6/semantics/pysource.py: Use external spark now.
+
+2016-04-20  rocky <rocky@gnu.org>
+
+	* circle.yml: Back to 2.7.8
+
+2016-04-20  rocky <rocky@gnu.org>
+
+	* circle.yml: Try python 2.7.10
+
+2016-04-19  rocky <rocky@gnu.org>
+
+	* README.rst: Remove link to Mysterie uncompyle2 per request
+
+2016-04-19  rocky <rocky@gnu.org>
+
+	* ChangeLog, NEWS, __pkginfo__.py: Get ready for release 2.2.0
+
+2016-04-18  rocky <rocky@gnu.org>
+
+	* README.rst: Another typo
+
+2016-04-18  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #7 from rocky/single-compile Support single-mode compile
+
+2016-04-18  rocky <rocky@gnu.org>
+
+	* README.rst: Doc typo
+
+2016-04-18  Thomas Grainger <tom.grainger@procensus.com>
+
+	* __pkginfo__.py, setup.cfg, setup.py: declare Python3 support in
+	wheel and trove
+
+2016-04-18  rocky <rocky@gnu.org>
+
+	* uncompyle6/parser.py, uncompyle6/parsers/parse2.py,
+	uncompyle6/parsers/parse3.py: Start to DRY Python 2 and Python 3
+	grammar code Move common code to parser.py
+
+2016-04-18  rocky <rocky@gnu.org>
+
+	* pytest/test_single_compile.py, uncompyle6/parser.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py: Add
+	simgle-mode compilation
+
+2016-04-17  rocky <rocky@gnu.org>
+
+	* pytest/test_single_compile.py, uncompyle6/parser.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/semantics/pysource.py: Towards single compilation
+
+2016-04-10  rocky <rb@dustyfeet.com>
+
+	* : Back off if_else_ternary pending Fails on Python 3.4 investigation Python 3.5 works though
+
+2016-04-10  rocky <rb@dustyfeet.com>
+
+	* .gitignore: Test administrivia
+
+2016-04-07  rocky <rb@dustyfeet.com>
+
+	* HISTORY.md, README.rst,
+	test/simple_source/branching/10_if_else_ternary.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py: Handle
+	Ternary "or". Remove mention of uncompyle3 uncompyle3 removed per Mysterie's request [Fixes Issue #5]
+
+2016-03-11  rocky <rb@dustyfeet.com>
+
+	* test/Makefile: remove uncompyle3 and make test work again * uncompyle3 removed by request * make test on python 2.7 is failing on some python3 and python3.5   bytecodes. Remove for now.
+
+2016-02-23  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #3 from lelicopter/master Bug correction (parse cmd options)
+
+2016-01-07  rocky <rb@dustyfeet.com>
+
+	* test/Makefile, test/bytecompile-tests, test/test_pythonlib.py: Add
+	Python 3.5 tests that we can do.
+
+2016-01-05  R. Bernstein <rocky@users.noreply.github.com>
+
+	* HISTORY.md: Grammar fixes
+
+2016-01-02  rocky <rb@dustyfeet.com>
+
+	* ChangeLog, NEWS, __pkginfo__.py: Get ready for release 2.1.3
+
+2016-01-02  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/opcodes/opcode_23.py, uncompyle6/opcodes/opcode_25.py,
+	uncompyle6/opcodes/opcode_26.py, uncompyle6/opcodes/opcode_27.py,
+	uncompyle6/opcodes/opcode_32.py, uncompyle6/opcodes/opcode_33.py,
+	uncompyle6/opcodes/opcode_34.py, uncompyle6/opcodes/opcode_35.py,
+	uncompyle6/opcodes/opcode_3x.py, uncompyle6/parser.py,
+	uncompyle6/scanner.py, uncompyle6/scanners/scanner32.py,
+	uncompyle6/scanners/scanner33.py, uncompyle6/scanners/scanner34.py,
+	uncompyle6/scanners/scanner35.py: Start to DRY opcode code. Limited
+	support for decopyling Python 3.5
+
+2016-01-02  rocky <rb@dustyfeet.com>
+
+	* test/Makefile, test/ok_lib3.4/antigravity.py,
+	test/ok_lib3.4/bisect.py, test/test_pythonlib.py: Start 3.4 library
+	verify tests
+
+2016-01-02  rocky <rb@dustyfeet.com>
+
+	* README.rst: Regularize spelling of bytecode
+
+2016-01-02  rocky <rb@dustyfeet.com>
+
+	* Makefile, README.rst: Add download shield. Add check-rst target
+
+2016-01-02  R. Bernstein <rocky@users.noreply.github.com>
+
+	* README.rst: Update README.rst
+
+2016-01-02  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py: Track recent source class
+	semantic actions in fragment actions
+
+2016-01-02  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanner.py, uncompyle6/scanners/scanner25.py,
+	uncompyle6/scanners/scanner26.py, uncompyle6/scanners/scanner27.py: 
+	Make ScannerXX() initialization the same on Python 2.x and 3.x
+
+2016-01-02  rocky <rb@dustyfeet.com>
+
+	* test/Makefile,
+	test/simple_source/comprehension/05_list_comprehension.py,
+	test/simple_source/def/02_closure.py, test/test_pythonlib.py,
+	uncompyle6/main.py, uncompyle6/verify.py: Verify 3.4 bytecode.
+	verify API call bug fixed.
+
+2016-01-02  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/def/05_class.py, uncompyle6/load.py,
+	uncompyle6/main.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/semantics/pysource.py: Python 3 class deparsing.  stop
+	earlier in uncompyle6 on a syntax error.
+
+2016-01-01  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner3.py: Fix make_closure compilation from
+	2.x of 3.3 bytecode
+
+2016-01-01  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/def/02_closure.py,
+	uncompyle6/parsers/parse3.py, uncompyle6/semantics/pysource.py: Work
+	on MAKE_CLOSURE rules for Python 3.3
+
 2015-12-31  rocky <rb@dustyfeet.com>
 
-	* __pkginfo__.py: Get ready for release 2.1.2
+	* uncompyle6/semantics/fragments.py: track source deparsing
+	superclass bug fix
+
+2015-12-31  rocky <rb@dustyfeet.com>
+
+	* ChangeLog, NEWS, __pkginfo__.py: Get ready for release 2.1.2
 
 2015-12-31  rocky <rb@dustyfeet.com>
 

HISTORY.md

@@ -30,7 +30,7 @@ The last mention of a release of SPARK from John is around 2002.
 
 In the fall of 2000, Hartmut Goebel
 [took over maintaining the code](https://groups.google.com/forum/#!searchin/comp.lang.python/hartmut$20goebel/comp.lang.python/35s3mp4-nuY/UZALti6ujnQJ). The
-first subsequennt public release announcement that I can find is
+first subsequent public release announcement that I can find is
 ["decompyle - A byte-code-decompiler version 2.2 beta 1"](https://mail.python.org/pipermail/python-announce-list/2002-February/001272.html).
 
 From the CHANGES file found in
@@ -38,9 +38,8 @@ From the CHANGES file found in
 it appears that Hartmut did most of the work to get this code to
 accept the full Python language. He added precidence to the table
 specifiers, support for multiple versions of Python, the
-pretty-printing of docstrings, lists and hashes. He also wrote
-extensive tests and routines to the testing and verification of
-decompiled bytecode.
+pretty-printing of docstrings, lists, and hashes. He also wrote test and verification routines of
+deparsed bytecode, and used this in an extensive set of tests that he also wrote. He could verify against the entire Python library.
 
 decompyle2.2 was packaged for Debian (sarge) by
 [Ben Burton around 2002](https://packages.qa.debian.org/d/decompyle.html). As
@@ -91,10 +90,6 @@ Hartmut a decade an a half ago:
     NB. This is not a masterpiece of software, but became more like a hack.
     Probably a complete rewrite would be sensefull. hG/2000-12-27
 
-One of the attempts to modernize it and make it available for Python3
-is [the one by Anton Vorobyov (DarkFenX)](https://github.com/DarkFenX/uncompyle3). I've
-followed some of the ideas there in this project.
-
 Lastly, I should mention [unpyc](https://code.google.com/p/unpyc3/)
 and most especially [pycdc](https://github.com/zrax/pycdc), largely by
 Michael Hansen and Darryl Pogue. If they supported getting source-code

Makefile

@@ -60,6 +60,11 @@ sdist:
 #: Style check. Set env var LINT to pyflakes, flake, or flake8
 lint: flake8
 
+# Check StructuredText long description formatting
+check-rst:
+	$(PYTHON) setup.py --long-description | rst2html.py > python3-trepan.html
+
+#: Lint program
 flake8:
 	$(LINT) uncompyle6
 

NEWS

@@ -1,3 +1,30 @@
+uncompyle6 2.2.1 2016-04-30
+
+- Require spark_parser  >= 1.1.0
+
+uncompyle6 2.2.0 2016-04-30
+
+- Spark is no longer here but pulled separate package spark_parse
+- Python 3 parsing fixes
+- More tests
+
+uncompyle6 2.2.0 2016-04-02
+
+- Support single-mode (in addtion to exec-mode) compilation
+- Start to DRY Python 2 and Python 3 grammars
+- Fix bug in if else ternary  construct
+- Fix bug in uncomplye6 -d and -r options (via lelicopter)
+
+
+uncompyle6 2.1.3 2016-01-02
+
+- Limited support for decompiling Python 3.5
+- Improve Python 3 class deparsing
+- Handle MAKE_CLOSURE opcode
+- Start to DRY opcode code.
+- increase test coverage
+- fix misc small bugs and some improvements
+
 uncompyle6 2.1.2 2015-12-31
 
 - Fix cross-version Marshal loading

README.rst

@@ -3,28 +3,28 @@
 uncompyle6
 ==========
 
-A native Python Byte-code Disassembler, Decompiler, Fragment Decompiler
-and byte-code library
+A native Python bytecode Disassembler, Decompiler, Fragment Decompiler
+and bytecode library
 
 
 Introduction
 ------------
 
-*uncompyle6* translates Python byte-code back into equivalent Python
-source code. It accepts byte-codes from Python version 2.5 to 3.4 or
+*uncompyle6* translates Python bytecode back into equivalent Python
+source code. It accepts bytecodes from Python version 2.5 to 3.4 or
 so and has been tested on Python running versions 2.6, 2.7, 3.3,
 3.4 and 3.5.
 
 Why this?
 ---------
 
-What makes this different other CPython byte-code decompilers?  Its
+What makes this different from other CPython bytecode decompilers?  Its
 ability to deparse just fragments and give source-code information
 around a given bytecode offset.
 
-I using this to deparse fragments of code inside my trepan_
+I use this to deparse fragments of code inside my trepan_
 debuggers_. For that, I need to record text fragments for all
-byte-code offsets (of interest). This purpose although largely
+bytecode offsets (of interest). This purpose although largely
 compatible with the original intention is yet a little bit different.
 See this_ for more information.
 
@@ -37,7 +37,7 @@ information.
 Other parts of the library can be used inside Python for various
 bytecode-related tasks. For example you can read in bytecode,
 i.e. perform a version-independent `marshal.loads()`, and disassemble
-the bytecode using version of Python different from the one used to
+the bytecode using a version of Python different from the one used to
 compile the bytecode.
 
 
@@ -48,6 +48,8 @@ This uses setup.py, so it follows the standard Python routine:
 
 ::
 
+    pip install -r requirements.txt
+    pip install -r requirements-dev.txt
     python setup.py install # may need sudo
     # or if you have pyenv:
     python setup.py develop
@@ -77,7 +79,7 @@ Run
 ::
 
      ./bin/uncompyle6 -h
-     ./bin/pydisassemble -y
+     ./bin/pydisassemble -h
 
 for usage help
 
@@ -92,12 +94,11 @@ See Also
 --------
 
 * https://github.com/zrax/pycdc
-* https://github.com/Mysterie/uncompyle2
-* https://github.com/DarkFenX/uncompyle3
 * https://code.google.com/p/unpyc3/
 
 The HISTORY file.
 
+.. |downloads| image:: https://img.shields.io/pypi/dd/uncompyle6.svg
 .. _trepan: https://pypi.python.org/pypi/trepan
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake

__pkginfo__.py

@@ -9,13 +9,20 @@
 
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015, 2016 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
 classifiers =  ['Development Status :: 3 - Alpha',
                 'Intended Audience :: Developers',
                 'Operating System :: OS Independent',
                 'Programming Language :: Python',
+                'Programming Language :: Python :: 2',
+                'Programming Language :: Python :: 2.6',
+                'Programming Language :: Python :: 2.7',
+                'Programming Language :: Python :: 3',
+                'Programming Language :: Python :: 3.3',
+                'Programming Language :: Python :: 3.4',
+                'Programming Language :: Python :: 3.5',
                 'Topic :: Software Development :: Debuggers',
                 'Topic :: Software Development :: Libraries :: Python Modules',
                 ]
@@ -24,6 +31,9 @@ classifiers =  ['Development Status :: 3 - Alpha',
 author             = "Rocky Bernstein, Hartmut Goebel, John Aycock, and others"
 author_email       = "rb@dustyfeet.com"
 ftp_url            = None
+install_requires   = ['python-spark >= 1.1.0']
+license            = 'GPL'
+
 # license            = 'BSDish'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'
@@ -40,7 +50,7 @@ def get_srcdir():
     return os.path.realpath(filename)
 
 ns = {}
-version            = '2.1.2'
+version            = '2.3.1'
 web                = 'https://github.com/rocky/python-uncompyle6/'
 
 # tracebacks in zip files are funky and not debuggable

bin/uncompyle6

@@ -92,13 +92,13 @@ for opt, val in opts:
         options['showgrammar'] = True
     elif opt == '-o':
         outfile = val
-    elif opt == ('--timestamp', '-d'):
+    elif opt in ('--timestamp', '-d'):
         timestamp = True
     elif opt == '-c':
         codes.append(val)
     elif opt == '-p':
         numproc = int(val)
-    elif opt == ('--recurse', '-r'):
+    elif opt in ('--recurse', '-r'):
         recurse_dirs = True
     else:
         print(opt, file=sys.stderr)

circle.yml

@@ -6,7 +6,8 @@ machine:
 
 dependencies:
     override:
-        - pip install -r test-requirements.txt
+        - pip install -r requirements.txt
+        - pip install -r requirements-dev.txt
 test:
     override:
         - python ./setup.py develop && make check-2.7

pytest/test_single_compile.py

@@ -0,0 +1,21 @@
+import pytest
+from uncompyle6 import PYTHON_VERSION, PYTHON3, deparse_code
+
+def test_single_mode():
+    single_expressions = (
+        'i = 1',
+        'i and (j or k)',
+        'i += 1',
+        'i = j % 4',
+        'i = {}',
+        'i = []',
+        'while i < 1 or stop:\n    i\n',
+        'while i < 1 or stop:\n    print%s\n' % ('(i)' if PYTHON3 else ' i'),
+        'for i in range(10):\n    i\n',
+        'for i in range(10):\n    for j in range(10):\n        i + j\n',
+        'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
+    )
+
+    for expr in single_expressions:
+        code = compile(expr + '\n', '<string>', 'single')
+        assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'

requirements.txt

@@ -0,0 +1 @@
+spark_parser >= 1.1.0

setup.cfg

@@ -6,3 +6,6 @@ doc_files = README
 #		USAGE.txt
 #		doc/
 #		examples/
+
+[bdist_wheel]
+universal=1

setup.py

@@ -1,9 +1,7 @@
-#! python
+#!/usr/bin/env python
 
 """Setup script for the 'uncompyle6' distribution."""
 
-from distutils.core import setup, Extension
-
 # Get the package information used in setup().
 # from __pkginfo__ import \
 #     author,           author_email,       classifiers,                    \

test-requirements.txt

@@ -1 +0,0 @@
-pytest

test/Makefile

@@ -20,7 +20,7 @@ check:
 	$(MAKE) check-$$PYTHON_VERSION
 
 #: Run working tests from Python 2.6 or 2.7
-check-2.6 check-2.7: check-bytecode check-2.7-ok
+check-2.6 check-2.7: check-bytecode-sans-3.5 check-2.7-ok
 
 #: Run working tests from Python 3.3
 check-3.3: check-bytecode
@@ -30,7 +30,8 @@ check-3.3: check-bytecode
 check-3.5: check-bytecode
 
 #: Run working tests from Python 3.4
-check-3.4: check-bytecode check-2.7-ok
+check-3.4: check-bytecode check-3.4-ok check-2.7-ok
+	$(PYTHON) test_pythonlib.py --bytecode-3.4 --verify $(COMPILE)
 
 #: Check deparsing only, but from a different Python version
 check-disasm:
@@ -42,6 +43,11 @@ check-bytecode-2:
 
 #: Check deparsing bytecode only
 check-bytecode:
+	$(PYTHON) test_pythonlib.py --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
+        --bytecode-3.2 --bytecode-3.3 --bytecode-3.4 --bytecode-3.5
+
+#: Check deparsing bytecode only
+check-bytecode-sans-3.5:
 	$(PYTHON) test_pythonlib.py --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
         --bytecode-3.2 --bytecode-3.3 --bytecode-3.4
 
@@ -69,6 +75,10 @@ check-bytecode-3.3:
 check-bytecode-3.4:
 	$(PYTHON) test_pythonlib.py --bytecode-3.4
 
+#: Check deparsing Python 3.5
+check-bytecode-3.5:
+	$(PYTHON) test_pythonlib.py --bytecode-3.5
+
 #: short tests for bytecodes only for this version of Python
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --verify $(COMPILE)
@@ -77,6 +87,14 @@ check-native-short:
 check-2.7-ok:
 	$(PYTHON) test_pythonlib.py --ok-2.7 --verify $(COMPILE)
 
+#: Run longer Python 3.2's lib files known to be okay
+check-3.2-ok:
+	$(PYTHON) test_pythonlib.py --ok-3.2 --verify $(COMPILE)
+
+#: Run longer Python 3.4's lib files known to be okay
+check-3.4-ok:
+	$(PYTHON) test_pythonlib.py --ok-3.4 --verify $(COMPILE)
+
 clean: clean-py-dis clean-dis clean-unverified
 
 clean-dis:

test/bytecompile-tests

@@ -70,7 +70,8 @@ for root, dirs, basenames in os.walk('simple_source'):
             simple_source.append(os.path.join(root, basename)[0:-3])
     pass
 
-tests['2.6'] = tests['2.7'] = tests['3.2'] = tests['3.3'] = tests['3.4'] = simple_source
+tests['2.6'] = tests['2.7'] = tests['3.2'] = \
+  tests['3.3'] = tests['3.4'] = tests['3.5'] = simple_source
 
 total_tests = len(tests['2.7'])
 #tests['2.2'].sort(); print tests['2.2']

test/ok_lib3.4/antigravity.py

@@ -0,0 +1,17 @@
+
+import webbrowser
+import hashlib
+
+webbrowser.open("http://xkcd.com/353/")
+
+def geohash(latitude, longitude, datedow):
+    '''Compute geohash() using the Munroe algorithm.
+
+    >>> geohash(37.421542, -122.085589, b'2005-05-26-10458.68')
+    37.857713 -122.544543
+
+    '''
+    # http://xkcd.com/426/
+    h = hashlib.md5(datedow).hexdigest()
+    p, q = [('%f' % float.fromhex('0.' + x)) for x in (h[:16], h[16:32])]
+    print('%d%s %d%s' % (latitude, p[1:], longitude, q[1:]))

test/ok_lib3.4/bisect.py

@@ -0,0 +1,92 @@
+"""Bisection algorithms."""
+
+def insort_right(a, x, lo=0, hi=None):
+    """Insert item x in list a, and keep it sorted assuming a is sorted.
+
+    If x is already in a, insert it to the right of the rightmost x.
+
+    Optional args lo (default 0) and hi (default len(a)) bound the
+    slice of a to be searched.
+    """
+
+    if lo < 0:
+        raise ValueError('lo must be non-negative')
+    if hi is None:
+        hi = len(a)
+    while lo < hi:
+        mid = (lo+hi)//2
+        if x < a[mid]: hi = mid
+        else: lo = mid+1
+    a.insert(lo, x)
+
+insort = insort_right   # backward compatibility
+
+def bisect_right(a, x, lo=0, hi=None):
+    """Return the index where to insert item x in list a, assuming a is sorted.
+
+    The return value i is such that all e in a[:i] have e <= x, and all e in
+    a[i:] have e > x.  So if x already appears in the list, a.insert(x) will
+    insert just after the rightmost x already there.
+
+    Optional args lo (default 0) and hi (default len(a)) bound the
+    slice of a to be searched.
+    """
+
+    if lo < 0:
+        raise ValueError('lo must be non-negative')
+    if hi is None:
+        hi = len(a)
+    while lo < hi:
+        mid = (lo+hi)//2
+        if x < a[mid]: hi = mid
+        else: lo = mid+1
+    return lo
+
+bisect = bisect_right   # backward compatibility
+
+def insort_left(a, x, lo=0, hi=None):
+    """Insert item x in list a, and keep it sorted assuming a is sorted.
+
+    If x is already in a, insert it to the left of the leftmost x.
+
+    Optional args lo (default 0) and hi (default len(a)) bound the
+    slice of a to be searched.
+    """
+
+    if lo < 0:
+        raise ValueError('lo must be non-negative')
+    if hi is None:
+        hi = len(a)
+    while lo < hi:
+        mid = (lo+hi)//2
+        if a[mid] < x: lo = mid+1
+        else: hi = mid
+    a.insert(lo, x)
+
+
+def bisect_left(a, x, lo=0, hi=None):
+    """Return the index where to insert item x in list a, assuming a is sorted.
+
+    The return value i is such that all e in a[:i] have e < x, and all e in
+    a[i:] have e >= x.  So if x already appears in the list, a.insert(x) will
+    insert just before the leftmost x already there.
+
+    Optional args lo (default 0) and hi (default len(a)) bound the
+    slice of a to be searched.
+    """
+
+    if lo < 0:
+        raise ValueError('lo must be non-negative')
+    if hi is None:
+        hi = len(a)
+    while lo < hi:
+        mid = (lo+hi)//2
+        if a[mid] < x: lo = mid+1
+        else: hi = mid
+    return lo
+
+# Overwrite above definitions with a fast C implementation
+try:
+    from _bisect import *
+except ImportError:
+    pass

test/simple_source/branching/10_if_else_ternary.py

@@ -0,0 +1,11 @@
+# Tests:
+
+# ret_expr_or_cond ::= ret_expr
+# ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
+# ret_expr_or_cond ::= ret_cond
+# ret_or ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+
+# See https://github.com/rocky/python-uncompyle6/issues/5
+
+def minimize(x, y):
+    return x or (x if x < y else y)

test/simple_source/comprehension/05_list_comprehension.py

@@ -19,4 +19,4 @@
 #   list_for ::= expr _for designator list_iter JUMP_BACK
 #   list_iter ::= lc_body
 #   lc_body ::= expr LIST_APPEND
-[ i * j for i in range(4) for j in range(7) ]
+# [ i * j for i in range(4) for j in range(7) ]

test/simple_source/def/02_closure.py

@@ -0,0 +1,19 @@
+# Tests
+# Python3:
+#   funcdef ::= mkfunc designator
+#   designator ::= STORE_DEREF
+#   mkfunc ::= load_closure BUILD_TUPLE_1 LOAD_CONST LOAD_CONST MAKE_CLOSURE_0
+#   load_closure ::= LOAD_CLOSURE
+#
+# Python2:
+
+#   funcdef ::= mkfunc designator
+#   designator ::= STORE_DEREF
+#   mkfunc ::= load_closure LOAD_CONST MAKE_CLOSURE_0
+#   load_closure ::= LOAD_CLOSURE
+
+
+def bug():
+    def convert(node):
+        return node and convert(node.left)
+    return

test/simple_source/def/05_class.py

@@ -11,3 +11,7 @@
 import io
 class BZ2File(io.BufferedIOBase):
     pass
+
+
+class ABC(metaclass=BZ2File):
+    pass

test/simple_source/def/06_classbug.py

@@ -0,0 +1,13 @@
+# Python 3.2 Bug from abc.py
+
+# Make sure we handle:
+
+# LOAD_FAST         '__locals__'
+# STORE_LOCALS      ''
+
+class abstractclassmethod(object):
+    """A Python 3.2 STORE_LOCALS bug
+    """
+
+    def __init__(self, callable):
+        callable.__isabstractmethod__ = True

test/test_pythonlib.py

@@ -55,22 +55,31 @@ PYO = ('*.pyo', )
 PYOC = ('*.pyc', '*.pyo')
 
 test_options = {
-    # name:   (src_basedir, pattern, output_base_suffix, pythoin_version)
+    # name:   (src_basedir, pattern, output_base_suffix, python_version)
     'test':
         ('test', PYC, 'test'),
 
     'ok-2.6':
         (os.path.join(src_dir, 'ok_2.6'),
-         PYC, 'ok-2.6', 2.6),
+         PYOC, 'ok-2.6', 2.6),
 
     'ok-2.7':    (os.path.join(src_dir, 'ok_lib2.7'),
-                 PYC, 'ok-2.7', 2.7),
+                 PYOC, 'ok-2.7', 2.7),
+
+    'ok-3.2':    (os.path.join(src_dir, 'ok_lib3.2'),
+                 PYOC, 'ok-3.2', 3.5),
 
     'base-2.7':  (os.path.join(src_dir, 'base_tests', 'python2.7'),
-                  PYC, 'base_2.7', 2.7),
+                  PYOC, 'base_2.7', 2.7),
 }
 
-for  vers in (2.5, 2.6, 2.7, 3.2, 3.3, 3.4):
+for  vers in (2.7, 3.4, 3.5):
+    pythonlib = "ok_lib%s" % vers
+    key = "ok-%s" % vers
+    test_options[key] = (os.path.join(src_dir, pythonlib), PYOC, key, vers)
+    pass
+
+for  vers in (2.5, 2.6, 2.7, 3.2, 3.3, 3.4, 3.5):
     bytecode = "bytecode_%s" % vers
     key = "bytecode-%s" % vers
     test_options[key] =  (bytecode, PYC, bytecode, vers)
@@ -78,7 +87,7 @@ for  vers in (2.5, 2.6, 2.7, 3.2, 3.3, 3.4):
     pythonlib = "python%s" % vers
     if vers >= 3.0:
         pythonlib = os.path.join(pythonlib, '__pycache__')
-    test_options[key] =  (os.path.join(lib_prefix, pythonlib), PYC, pythonlib, vers)
+    test_options[key] =  (os.path.join(lib_prefix, pythonlib), PYOC, pythonlib, vers)
 
 #-----
 
@@ -93,7 +102,6 @@ def help():
 
   # decompile and verify known good python 2.7
   test_pythonlib.py --ok-2.7 --verify
-
 """)
     sys.exit(1)
 

uncompyle6/load.py

@@ -1,5 +1,5 @@
-# Copyright (c) 2000 by hartmut Goebel <h.goebel@crazy-compilers.com>
 # Copyright (c) 2015 by Rocky Bernstein
+# Copyright (c) 2000 by hartmut Goebel <h.goebel@crazy-compilers.com>
 from __future__ import print_function
 
 import imp, marshal, os, py_compile, sys, tempfile
@@ -26,7 +26,7 @@ def check_object_path(path):
         spath = path if PYTHON3 else path.decode('utf-8')
         path = tempfile.mkstemp(prefix=basename + '-',
                                 suffix='.pyc', text=False)[1]
-        py_compile.compile(spath, cfile=path)
+        py_compile.compile(spath, cfile=path, doraise=True)
 
     if not path.endswith(".pyc") and not path.endswith(".pyo"):
         raise ValueError("path %s must point to a .py or .pyc file\n" %

uncompyle6/main.py

@@ -1,5 +1,5 @@
 from __future__ import print_function
-import datetime, inspect, os, sys
+import datetime, os, sys
 
 from uncompyle6 import verify, PYTHON_VERSION
 from uncompyle6.code import iscode
@@ -57,9 +57,10 @@ def uncompyle_file(filename, outstream=None, showasm=False, showast=False,
                   timestamp, showgrammar, code_objects=code_objects)
     co = None
 
+# FIXME: combine into an options parameter
 def main(in_base, out_base, files, codes, outfile=None,
          showasm=False, showast=False, do_verify=False,
-         showgrammar=False):
+         showgrammar=False, raise_on_error=False):
     """
     in_base	base directory for input files
     out_base	base directory for output files (ignored when
@@ -108,7 +109,7 @@ def main(in_base, out_base, files, codes, outfile=None,
         try:
             uncompyle_file(infile, outstream, showasm, showast, showgrammar)
             tot_files += 1
-        except ValueError as e:
+        except (ValueError, SyntaxError) as e:
             sys.stderr.write("\n# %s" % e)
             failed_files += 1
         except KeyboardInterrupt:
@@ -143,6 +144,8 @@ def main(in_base, out_base, files, codes, outfile=None,
                         if not outfile:
                             print("### Error Verifiying %s" % filename,  file=sys.stderr)
                             print(e, file=sys.stderr)
+                            if raise_on_error:
+                                raise
                             pass
                         pass
                 pass

uncompyle6/opcodes/opcode_23.py

@@ -7,6 +7,8 @@ This is a superset of Python 2.3's opcode.py with some opcodes that simplify
 parsing and semantic interpretation.
 """
 
+# FIXME: DRY this along the lines of opcode_3x.
+
 cmp_op = ('<', '<=', '==', '!=', '>', '>=', 'in', 'not in', 'is',
         'is not', 'exception match', 'BAD')
 

uncompyle6/opcodes/opcode_25.py

@@ -7,6 +7,8 @@ This is a superset of Python 2.5's opcode.py with some opcodes that simplify
 parsing and semantic interpretation.
 """
 
+# FIXME: DRY this along the lines of opcode_3x.
+
 cmp_op = ('<', '<=', '==', '!=', '>', '>=', 'in', 'not in', 'is',
         'is not', 'exception match', 'BAD')
 

uncompyle6/opcodes/opcode_26.py

@@ -7,6 +7,8 @@ This is a superset of Python 3.4's opcode.py with some opcodes that simplify
 parsing and semantic interpretation.
 """
 
+# FIXME: DRY this along the lines of opcode_3x.
+
 cmp_op = ('<', '<=', '==', '!=', '>', '>=', 'in', 'not in', 'is',
         'is not', 'exception match', 'BAD')
 

uncompyle6/opcodes/opcode_27.py

@@ -7,6 +7,8 @@ This is a superset of Python 3.4's opcode.py with some opcodes that simplify
 parsing and semantic interpretation.
 """
 
+# FIXME: DRY this along the lines of opcode_3x.
+
 cmp_op = ('<', '<=', '==', '!=', '>', '>=', 'in', 'not in', 'is',
         'is not', 'exception match', 'BAD')
 

uncompyle6/opcodes/opcode_32.py

@@ -3,42 +3,27 @@ CPython 3.2 bytecode opcodes
 
 This is used in scanner (bytecode disassembly) and parser (Python grammar).
 
-This is a superset of Python 3.4's opcode.py with some opcodes that simplify
+This is a superset of Python 3.2's opcode.py with some opcodes that simplify
 parsing and semantic interpretation.
 """
 
-cmp_op = ('<', '<=', '==', '!=', '>', '>=', 'in', 'not in', 'is',
-        'is not', 'exception match', 'BAD')
+from copy import deepcopy
 
-hasconst = []
-hasname = []
-hasjrel = []
-hasjabs = []
-haslocal = []
-hascompare = []
-hasfree = []
-hasnargs = []
+import uncompyle6.opcodes.opcode_3x as opcode_3x
+from uncompyle6.opcodes.opcode_3x import fields2copy
+
+# FIXME: can we DRY this even more?
 
 opmap = {}
 opname = [''] * 256
-for op in range(256): opname[op] = '<%r>' % (op,)
-del op
+hasjrel = []
+hasjabs = []
 
-def def_op(name, op):
-    opname[op] = name
-    opmap[name] = op
+for object in fields2copy:
+    globals()[object] =  deepcopy(getattr(opcode_3x, object))
 
-def name_op(name, op):
-    def_op(name, op)
-    hasname.append(op)
-
-def jrel_op(name, op):
-    def_op(name, op)
-    hasjrel.append(op)
-
-def jabs_op(name, op):
-    def_op(name, op)
-    hasjabs.append(op)
+# There are no opcodes to add or change.
+# If there were, they'd be listed below.
 
 def updateGlobal():
     # JUMP_OPs are used in verification are set in the scanner
@@ -50,165 +35,14 @@ def updateGlobal():
     globals().update(dict([(k.replace('+', '_'), v) for (k, v) in opmap.items()]))
     globals().update({'JUMP_OPs': map(lambda op: opname[op], hasjrel + hasjabs)})
 
-# Instruction opcodes for compiled code
-# Blank lines correspond to available opcodes
-
-def_op('STOP_CODE', 0)
-def_op('POP_TOP', 1)
-def_op('ROT_TWO', 2)
-def_op('ROT_THREE', 3)
-def_op('DUP_TOP', 4)
-def_op('DUP_TOP_TWO', 5)
-
-def_op('NOP', 9)
-def_op('UNARY_POSITIVE', 10)
-def_op('UNARY_NEGATIVE', 11)
-def_op('UNARY_NOT', 12)
-
-def_op('UNARY_INVERT', 15)
-
-def_op('BINARY_POWER', 19)
-def_op('BINARY_MULTIPLY', 20)
-
-def_op('BINARY_MODULO', 22)
-def_op('BINARY_ADD', 23)
-def_op('BINARY_SUBTRACT', 24)
-def_op('BINARY_SUBSCR', 25)
-def_op('BINARY_FLOOR_DIVIDE', 26)
-def_op('BINARY_TRUE_DIVIDE', 27)
-def_op('INPLACE_FLOOR_DIVIDE', 28)
-def_op('INPLACE_TRUE_DIVIDE', 29)
-
-# Gone from Python 3 are Python2's
-# SLICE+0 .. SLICE+3
-# STORE_SLICE+0 .. STORE_SLICE+3
-# DELETE_SLICE+0 .. DELETE_SLICE+3
-
-def_op('STORE_MAP', 54)
-def_op('INPLACE_ADD', 55)
-def_op('INPLACE_SUBTRACT', 56)
-def_op('INPLACE_MULTIPLY', 57)
-
-def_op('INPLACE_MODULO', 59)
-def_op('STORE_SUBSCR', 60)
-def_op('DELETE_SUBSCR', 61)
-def_op('BINARY_LSHIFT', 62)
-def_op('BINARY_RSHIFT', 63)
-def_op('BINARY_AND', 64)
-def_op('BINARY_XOR', 65)
-def_op('BINARY_OR', 66)
-def_op('INPLACE_POWER', 67)
-def_op('GET_ITER', 68)
-def_op('STORE_LOCALS', 69)
-
-def_op('PRINT_EXPR', 70)
-def_op('LOAD_BUILD_CLASS', 71)
-
-# Python3 drops/changes:
-#  def_op('PRINT_ITEM', 71)
-#  def_op('PRINT_NEWLINE', 72)
-#  def_op('PRINT_ITEM_TO', 73)
-#  def_op('PRINT_NEWLINE_TO', 74)
-
-def_op('INPLACE_LSHIFT', 75)
-def_op('INPLACE_RSHIFT', 76)
-def_op('INPLACE_AND', 77)
-def_op('INPLACE_XOR', 78)
-def_op('INPLACE_OR', 79)
-def_op('BREAK_LOOP', 80)
-def_op('WITH_CLEANUP', 81)
-
-def_op('RETURN_VALUE', 83)
-def_op('IMPORT_STAR', 84)
-
-def_op('YIELD_VALUE', 86)
-def_op('POP_BLOCK', 87)
-def_op('END_FINALLY', 88)
-def_op('POP_EXCEPT', 89)
-
-HAVE_ARGUMENT = 90              # Opcodes from here have an argument:
-
-name_op('STORE_NAME', 90)       # Index in name list
-name_op('DELETE_NAME', 91)      # ""
-def_op('UNPACK_SEQUENCE', 92)   # Number of tuple items
-jrel_op('FOR_ITER', 93)
-def_op('UNPACK_EX', 94)
-name_op('STORE_ATTR', 95)       # Index in name list
-name_op('DELETE_ATTR', 96)      # ""
-name_op('STORE_GLOBAL', 97)     # ""
-name_op('DELETE_GLOBAL', 98)    # ""
-
-# Python 2's DUP_TOPX is gone
-
-def_op('LOAD_CONST', 100)       # Index in const list
-hasconst.append(100)
-name_op('LOAD_NAME', 101)       # Index in name list
-def_op('BUILD_TUPLE', 102)      # Number of tuple items
-def_op('BUILD_LIST', 103)       # Number of list items
-def_op('BUILD_SET', 104)        # Number of set items
-def_op('BUILD_MAP', 105)        # Number of dict entries (upto 255)
-name_op('LOAD_ATTR', 106)       # Index in name list
-def_op('COMPARE_OP', 107)       # Comparison operator
-hascompare.append(107)
-name_op('IMPORT_NAME', 108)     # Index in name list
-name_op('IMPORT_FROM', 109)     # Index in name list
-
-jrel_op('JUMP_FORWARD', 110)    # Number of bytes to skip
-jabs_op('JUMP_IF_FALSE_OR_POP', 111) # Target byte offset from beginning of code
-jabs_op('JUMP_IF_TRUE_OR_POP', 112)  # ""
-jabs_op('JUMP_ABSOLUTE', 113)        # ""
-jabs_op('POP_JUMP_IF_FALSE', 114)    # ""
-jabs_op('POP_JUMP_IF_TRUE', 115)     # ""
-
-name_op('LOAD_GLOBAL', 116)     # Index in name list
-
-jabs_op('CONTINUE_LOOP', 119)   # Target address
-jrel_op('SETUP_LOOP', 120)      # Distance to target address
-jrel_op('SETUP_EXCEPT', 121)    # ""
-jrel_op('SETUP_FINALLY', 122)   # ""
-
-def_op('LOAD_FAST', 124)        # Local variable number
-haslocal.append(124)
-def_op('STORE_FAST', 125)       # Local variable number
-haslocal.append(125)
-def_op('DELETE_FAST', 126)      # Local variable number
-haslocal.append(126)
-
-def_op('RAISE_VARARGS', 130)    # Number of raise arguments (1, 2, or 3)
-def_op('CALL_FUNCTION', 131)    # #args + (#kwargs << 8)
-hasnargs.append(131)
-def_op('MAKE_FUNCTION', 132)    # Number of args with default values
-def_op('BUILD_SLICE', 133)      # Number of items
-def_op('MAKE_CLOSURE', 134)
-def_op('LOAD_CLOSURE', 135)
-hasfree.append(135)
-def_op('LOAD_DEREF', 136)
-hasfree.append(136)
-def_op('STORE_DEREF', 137)
-hasfree.append(137)
-def_op('DELETE_DEREF', 138)
-hasfree.append(138)
-
-def_op('CALL_FUNCTION_VAR', 140)     # #args + (#kwargs << 8)
-hasnargs.append(140)
-def_op('CALL_FUNCTION_KW', 141)      # #args + (#kwargs << 8)
-hasnargs.append(141)
-def_op('CALL_FUNCTION_VAR_KW', 142)  # #args + (#kwargs << 8)
-hasnargs.append(142)
-
-jrel_op('SETUP_WITH', 143)
-
-def_op('LIST_APPEND', 145)
-def_op('SET_ADD', 146)
-def_op('MAP_ADD', 147)
-
-def_op('EXTENDED_ARG', 144)
-EXTENDED_ARG = 144
-
 updateGlobal()
-del def_op, name_op, jrel_op, jabs_op
 
 from uncompyle6 import PYTHON_VERSION
 if PYTHON_VERSION == 3.2:
     import dis
+    # for item in dis.opmap.items():
+    #     if item not in opmap.items():
+    #         print(item)
     assert all(item in opmap.items() for item in dis.opmap.items())
+
+# opcode_3x.dump_opcodes(opmap)

uncompyle6/opcodes/opcode_33.py

@@ -7,37 +7,31 @@ This is a superset of Python 3.3's opcode.py with some opcodes that simplify
 parsing and semantic interpretation.
 """
 
-cmp_op = ('<', '<=', '==', '!=', '>', '>=', 'in', 'not in', 'is',
-        'is not', 'exception match', 'BAD')
+from copy import deepcopy
 
-hasconst = []
-hasname = []
-hasjrel = []
-hasjabs = []
-haslocal = []
-hascompare = []
-hasfree = []
+import uncompyle6.opcodes.opcode_3x as opcode_3x
+from uncompyle6.opcodes.opcode_3x import fields2copy, rm_op
+
+# FIXME: can we DRY this even more?
 
 opmap = {}
 opname = [''] * 256
-for op in range(256): opname[op] = '<%r>' % (op,)
-del op
+hasconst = []
+hasjrel = []
+hasjabs = []
 
 def def_op(name, op):
     opname[op] = name
     opmap[name] = op
 
-def name_op(name, op):
-    def_op(name, op)
-    hasname.append(op)
+for object in fields2copy:
+    globals()[object] =  deepcopy(getattr(opcode_3x, object))
 
-def jrel_op(name, op):
-    def_op(name, op)
-    hasjrel.append(op)
+# Below are opcodes since Python 3.2
 
-def jabs_op(name, op):
-    def_op(name, op)
-    hasjabs.append(op)
+rm_op(opname, opmap, 'STOP_CODE', 0)
+
+def_op('YIELD_FROM', 72)
 
 def updateGlobal():
     # JUMP_OPs are used in verification are set in the scanner
@@ -49,156 +43,9 @@ def updateGlobal():
     globals().update(dict([(k.replace('+', '_'), v) for (k, v) in opmap.items()]))
     globals().update({'JUMP_OPs': map(lambda op: opname[op], hasjrel + hasjabs)})
 
-# Instruction opcodes for compiled code
-# Blank lines correspond to available opcodes
-
-def_op('STOP_CODE', 0)
-def_op('POP_TOP', 1)
-def_op('ROT_TWO', 2)
-def_op('ROT_THREE', 3)
-def_op('DUP_TOP', 4)
-def_op('DUP_TOP_TWO', 5)
-
-def_op('NOP', 9)
-def_op('UNARY_POSITIVE', 10)
-def_op('UNARY_NEGATIVE', 11)
-def_op('UNARY_NOT', 12)
-
-def_op('UNARY_INVERT', 15)
-
-def_op('BINARY_POWER', 19)
-def_op('BINARY_MULTIPLY', 20)
-
-def_op('BINARY_MODULO', 22)
-def_op('BINARY_ADD', 23)
-def_op('BINARY_SUBTRACT', 24)
-def_op('BINARY_SUBSCR', 25)
-def_op('BINARY_FLOOR_DIVIDE', 26)
-def_op('BINARY_TRUE_DIVIDE', 27)
-def_op('INPLACE_FLOOR_DIVIDE', 28)
-def_op('INPLACE_TRUE_DIVIDE', 29)
-
-# Gone from Python 3 are
-# Python 2's SLICE+0 .. SLICE+3
-
-def_op('STORE_MAP', 54)
-def_op('INPLACE_ADD', 55)
-def_op('INPLACE_SUBTRACT', 56)
-def_op('INPLACE_MULTIPLY', 57)
-
-def_op('INPLACE_MODULO', 59)
-def_op('STORE_SUBSCR', 60)
-def_op('DELETE_SUBSCR', 61)
-def_op('BINARY_LSHIFT', 62)
-def_op('BINARY_RSHIFT', 63)
-def_op('BINARY_AND', 64)
-def_op('BINARY_XOR', 65)
-def_op('BINARY_OR', 66)
-def_op('INPLACE_POWER', 67)
-def_op('GET_ITER', 68)
-def_op('STORE_LOCALS', 69)
-
-def_op('PRINT_EXPR', 70)
-def_op('LOAD_BUILD_CLASS', 71)
-
-# Python3 drops/changes:
-#  def_op('PRINT_ITEM', 71)
-#  def_op('PRINT_NEWLINE', 72)
-def_op('YIELD_FROM', 72)
-#  def_op('PRINT_ITEM_TO', 73)
-#  def_op('PRINT_NEWLINE_TO', 74)
-
-def_op('INPLACE_LSHIFT', 75)
-def_op('INPLACE_RSHIFT', 76)
-def_op('INPLACE_AND', 77)
-def_op('INPLACE_XOR', 78)
-def_op('INPLACE_OR', 79)
-def_op('BREAK_LOOP', 80)
-def_op('WITH_CLEANUP', 81)
-
-def_op('RETURN_VALUE', 83)
-def_op('IMPORT_STAR', 84)
-
-def_op('YIELD_VALUE', 86)
-def_op('POP_BLOCK', 87)
-def_op('END_FINALLY', 88)
-def_op('POP_EXCEPT', 89)
-
-HAVE_ARGUMENT = 90              # Opcodes from here have an argument:
-
-name_op('STORE_NAME', 90)       # Index in name list
-name_op('DELETE_NAME', 91)      # ""
-def_op('UNPACK_SEQUENCE', 92)   # Number of tuple items
-jrel_op('FOR_ITER', 93)
-def_op('UNPACK_EX', 94)
-name_op('STORE_ATTR', 95)       # Index in name list
-name_op('DELETE_ATTR', 96)      # ""
-name_op('STORE_GLOBAL', 97)     # ""
-name_op('DELETE_GLOBAL', 98)    # ""
-def_op('LOAD_CONST', 100)       # Index in const list
-hasconst.append(100)
-name_op('LOAD_NAME', 101)       # Index in name list
-def_op('BUILD_TUPLE', 102)      # Number of tuple items
-def_op('BUILD_LIST', 103)       # Number of list items
-def_op('BUILD_SET', 104)        # Number of set items
-def_op('BUILD_MAP', 105)        # Number of dict entries (upto 255)
-name_op('LOAD_ATTR', 106)       # Index in name list
-def_op('COMPARE_OP', 107)       # Comparison operator
-hascompare.append(107)
-name_op('IMPORT_NAME', 108)     # Index in name list
-name_op('IMPORT_FROM', 109)     # Index in name list
-
-jrel_op('JUMP_FORWARD', 110)    # Number of bytes to skip
-jabs_op('JUMP_IF_FALSE_OR_POP', 111) # Target byte offset from beginning of code
-jabs_op('JUMP_IF_TRUE_OR_POP', 112)  # ""
-jabs_op('JUMP_ABSOLUTE', 113)        # ""
-jabs_op('POP_JUMP_IF_FALSE', 114)    # ""
-jabs_op('POP_JUMP_IF_TRUE', 115)     # ""
-
-name_op('LOAD_GLOBAL', 116)     # Index in name list
-
-jabs_op('CONTINUE_LOOP', 119)   # Target address
-jrel_op('SETUP_LOOP', 120)      # Distance to target address
-jrel_op('SETUP_EXCEPT', 121)    # ""
-jrel_op('SETUP_FINALLY', 122)   # ""
-
-def_op('LOAD_FAST', 124)        # Local variable number
-haslocal.append(124)
-def_op('STORE_FAST', 125)       # Local variable number
-haslocal.append(125)
-def_op('DELETE_FAST', 126)      # Local variable number
-haslocal.append(126)
-
-def_op('RAISE_VARARGS', 130)    # Number of raise arguments (1, 2, or 3)
-def_op('CALL_FUNCTION', 131)    # #args + (#kwargs << 8)
-def_op('MAKE_FUNCTION', 132)    # Number of args with default values
-def_op('BUILD_SLICE', 133)      # Number of items
-def_op('MAKE_CLOSURE', 134)
-def_op('LOAD_CLOSURE', 135)
-hasfree.append(135)
-def_op('LOAD_DEREF', 136)
-hasfree.append(136)
-def_op('STORE_DEREF', 137)
-hasfree.append(137)
-def_op('DELETE_DEREF', 138)
-hasfree.append(138)
-
-def_op('CALL_FUNCTION_VAR', 140)     # #args + (#kwargs << 8)
-def_op('CALL_FUNCTION_KW', 141)      # #args + (#kwargs << 8)
-def_op('CALL_FUNCTION_VAR_KW', 142)  # #args + (#kwargs << 8)
-
-jrel_op('SETUP_WITH', 143)
-
-def_op('LIST_APPEND', 145)
-def_op('SET_ADD', 146)
-def_op('MAP_ADD', 147)
-
-def_op('EXTENDED_ARG', 144)
-EXTENDED_ARG = 144
-
 updateGlobal()
-del def_op, name_op, jrel_op, jabs_op
 
+# FIXME: turn into pytest test
 from uncompyle6 import PYTHON_VERSION
 if PYTHON_VERSION == 3.3:
     import dis
@@ -206,3 +53,5 @@ if PYTHON_VERSION == 3.3:
     #     if item not in opmap.items():
     #         print(item)
     assert all(item in opmap.items() for item in dis.opmap.items())
+
+# opcode3x.dump_opcodes(opmap)

uncompyle6/opcodes/opcode_34.py

@@ -1,44 +1,41 @@
 """
 CPython 3.4 bytecode opcodes
 
-This is used in scanner (bytecode disassembly) and parser (Python grammar).
+used in scanner (bytecode disassembly) and parser (Python grammar)
 
 This is a superset of Python 3.4's opcode.py with some opcodes that simplify
 parsing and semantic interpretation.
 """
 
-cmp_op = ('<', '<=', '==', '!=', '>', '>=', 'in', 'not in', 'is',
-        'is not', 'exception match', 'BAD')
+from copy import deepcopy
 
-hasconst = []
-hasname = []
-hasjrel = []
-hasjabs = []
-haslocal = []
-hascompare = []
-hasfree = []
-hasnargs = []
+import uncompyle6.opcodes.opcode_3x as opcode_3x
+from uncompyle6.opcodes.opcode_3x import fields2copy, hasfree, rm_op
+
+# FIXME: can we DRY this even more?
 
 opmap = {}
 opname = [''] * 256
-for op in range(256): opname[op] = '<%r>' % (op,)
-del op
+hasconst = []
+hasjrel = []
+hasjabs = []
 
 def def_op(name, op):
     opname[op] = name
     opmap[name] = op
 
-def name_op(name, op):
-    def_op(name, op)
-    hasname.append(op)
+for object in fields2copy:
+    globals()[object] =  deepcopy(getattr(opcode_3x, object))
 
-def jrel_op(name, op):
-    def_op(name, op)
-    hasjrel.append(op)
+# Below are opcodes changes since Python 3.2
 
-def jabs_op(name, op):
-    def_op(name, op)
-    hasjabs.append(op)
+rm_op(opname, opmap, 'STOP_CODE', 0)
+rm_op(opname, opmap, 'STORE_LOCALS', 69)
+
+# These are new since Python 3.3
+def_op('YIELD_FROM', 72)
+def_op('LOAD_CLASSDEREF', 148)
+hasfree.append(148)
 
 def updateGlobal():
     # JUMP_OPs are used in verification are set in the scanner
@@ -50,171 +47,15 @@ def updateGlobal():
     globals().update(dict([(k.replace('+', '_'), v) for (k, v) in opmap.items()]))
     globals().update({'JUMP_OPs': map(lambda op: opname[op], hasjrel + hasjabs)})
 
-# Instruction opcodes for compiled code
-# Blank lines correspond to available opcodes
-
-def_op('STOP_CODE', 0)
-def_op('POP_TOP', 1)
-def_op('ROT_TWO', 2)
-def_op('ROT_THREE', 3)
-def_op('DUP_TOP', 4)
-def_op('DUP_TOP_TWO', 5)
-
-def_op('NOP', 9)
-def_op('UNARY_POSITIVE', 10)
-def_op('UNARY_NEGATIVE', 11)
-def_op('UNARY_NOT', 12)
-
-def_op('UNARY_INVERT', 15)
-
-def_op('BINARY_POWER', 19)
-def_op('BINARY_MULTIPLY', 20)
-
-def_op('BINARY_MODULO', 22)
-def_op('BINARY_ADD', 23)
-def_op('BINARY_SUBTRACT', 24)
-def_op('BINARY_SUBSCR', 25)
-def_op('BINARY_FLOOR_DIVIDE', 26)
-def_op('BINARY_TRUE_DIVIDE', 27)
-def_op('INPLACE_FLOOR_DIVIDE', 28)
-def_op('INPLACE_TRUE_DIVIDE', 29)
-
-# Gone from Python 3 are Python2's
-# SLICE+0 .. SLICE+3
-# STORE_SLICE+0 .. STORE_SLICE+3
-# DELETE_SLICE+0 .. DELETE_SLICE+3
-
-def_op('STORE_MAP', 54)
-def_op('INPLACE_ADD', 55)
-def_op('INPLACE_SUBTRACT', 56)
-def_op('INPLACE_MULTIPLY', 57)
-
-def_op('INPLACE_MODULO', 59)
-def_op('STORE_SUBSCR', 60)
-def_op('DELETE_SUBSCR', 61)
-def_op('BINARY_LSHIFT', 62)
-def_op('BINARY_RSHIFT', 63)
-def_op('BINARY_AND', 64)
-def_op('BINARY_XOR', 65)
-def_op('BINARY_OR', 66)
-def_op('INPLACE_POWER', 67)
-def_op('GET_ITER', 68)
-
-# FIXME: no code generates this
-def_op('STORE_LOCALS', 69)
-
-def_op('PRINT_EXPR', 70)
-def_op('LOAD_BUILD_CLASS', 71)
-def_op('YIELD_FROM', 72)
-
-# Python3 drops/changes:
-#  def_op('PRINT_ITEM', 71)
-#  def_op('PRINT_NEWLINE', 72)
-#  def_op('PRINT_ITEM_TO', 73)
-#  def_op('PRINT_NEWLINE_TO', 74)
-
-def_op('INPLACE_LSHIFT', 75)
-def_op('INPLACE_RSHIFT', 76)
-def_op('INPLACE_AND', 77)
-def_op('INPLACE_XOR', 78)
-def_op('INPLACE_OR', 79)
-def_op('BREAK_LOOP', 80)
-def_op('WITH_CLEANUP', 81)
-
-def_op('RETURN_VALUE', 83)
-def_op('IMPORT_STAR', 84)
-
-def_op('YIELD_VALUE', 86)
-def_op('POP_BLOCK', 87)
-def_op('END_FINALLY', 88)
-def_op('POP_EXCEPT', 89)
-
-HAVE_ARGUMENT = 90              # Opcodes from here have an argument:
-
-name_op('STORE_NAME', 90)       # Index in name list
-name_op('DELETE_NAME', 91)      # ""
-def_op('UNPACK_SEQUENCE', 92)   # Number of tuple items
-jrel_op('FOR_ITER', 93)
-def_op('UNPACK_EX', 94)
-name_op('STORE_ATTR', 95)       # Index in name list
-name_op('DELETE_ATTR', 96)      # ""
-name_op('STORE_GLOBAL', 97)     # ""
-name_op('DELETE_GLOBAL', 98)    # ""
-
-# Python 2's DUP_TOPX is gone
-
-def_op('LOAD_CONST', 100)       # Index in const list
-hasconst.append(100)
-name_op('LOAD_NAME', 101)       # Index in name list
-def_op('BUILD_TUPLE', 102)      # Number of tuple items
-def_op('BUILD_LIST', 103)       # Number of list items
-def_op('BUILD_SET', 104)        # Number of set items
-def_op('BUILD_MAP', 105)        # Number of dict entries (upto 255)
-name_op('LOAD_ATTR', 106)       # Index in name list
-def_op('COMPARE_OP', 107)       # Comparison operator
-hascompare.append(107)
-name_op('IMPORT_NAME', 108)     # Index in name list
-name_op('IMPORT_FROM', 109)     # Index in name list
-
-jrel_op('JUMP_FORWARD', 110)    # Number of bytes to skip
-jabs_op('JUMP_IF_FALSE_OR_POP', 111) # Target byte offset from beginning of code
-jabs_op('JUMP_IF_TRUE_OR_POP', 112)  # ""
-jabs_op('JUMP_ABSOLUTE', 113)        # ""
-jabs_op('POP_JUMP_IF_FALSE', 114)    # ""
-jabs_op('POP_JUMP_IF_TRUE', 115)     # ""
-
-name_op('LOAD_GLOBAL', 116)     # Index in name list
-
-jabs_op('CONTINUE_LOOP', 119)   # Target address
-jrel_op('SETUP_LOOP', 120)      # Distance to target address
-jrel_op('SETUP_EXCEPT', 121)    # ""
-jrel_op('SETUP_FINALLY', 122)   # ""
-
-def_op('LOAD_FAST', 124)        # Local variable number
-haslocal.append(124)
-def_op('STORE_FAST', 125)       # Local variable number
-haslocal.append(125)
-def_op('DELETE_FAST', 126)      # Local variable number
-haslocal.append(126)
-
-def_op('RAISE_VARARGS', 130)    # Number of raise arguments (1, 2, or 3)
-def_op('CALL_FUNCTION', 131)    # #args + (#kwargs << 8)
-hasnargs.append(131)
-def_op('MAKE_FUNCTION', 132)    # Number of args with default values
-def_op('BUILD_SLICE', 133)      # Number of items
-def_op('MAKE_CLOSURE', 134)
-def_op('LOAD_CLOSURE', 135)
-hasfree.append(135)
-def_op('LOAD_DEREF', 136)
-hasfree.append(136)
-def_op('STORE_DEREF', 137)
-hasfree.append(137)
-def_op('DELETE_DEREF', 138)
-hasfree.append(138)
-
-def_op('CALL_FUNCTION_VAR', 140)     # #args + (#kwargs << 8)
-hasnargs.append(140)
-def_op('CALL_FUNCTION_KW', 141)      # #args + (#kwargs << 8)
-hasnargs.append(141)
-def_op('CALL_FUNCTION_VAR_KW', 142)  # #args + (#kwargs << 8)
-hasnargs.append(142)
-
-jrel_op('SETUP_WITH', 143)
-
-def_op('LIST_APPEND', 145)
-def_op('SET_ADD', 146)
-def_op('MAP_ADD', 147)
-
-def_op('LOAD_CLASSDEREF', 148)
-hasfree.append(148)
-
-def_op('EXTENDED_ARG', 144)
-EXTENDED_ARG = 144
-
 updateGlobal()
-del def_op, name_op, jrel_op, jabs_op
 
+# FIXME: turn into pytest test
 from uncompyle6 import PYTHON_VERSION
 if PYTHON_VERSION == 3.4:
     import dis
+    # for item in dis.opmap.items():
+    #     if item not in opmap.items():
+    #         print(item)
     assert all(item in opmap.items() for item in dis.opmap.items())
+
+# opcode_3x.dump_opcodes(opmap)

uncompyle6/opcodes/opcode_35.py

@@ -0,0 +1,81 @@
+# (C) Copyright 2016 by Rocky Bernstein
+"""
+CPython 3.5 bytecode opcodes
+
+used in scanner (bytecode disassembly) and parser (Python grammar)
+
+This is a superset of Python 3.5's opcode.py with some opcodes that simplify
+parsing and semantic interpretation.
+"""
+
+from copy import deepcopy
+
+import uncompyle6.opcodes.opcode_3x as opcode_3x
+from uncompyle6.opcodes.opcode_3x import fields2copy, hasfree, rm_op
+
+# FIXME: can we DRY this even more?
+
+opmap = {}
+opname = [''] * 256
+hasconst = []
+hasjrel = []
+hasjabs = []
+
+def def_op(name, op):
+    opname[op] = name
+    opmap[name] = op
+
+for object in fields2copy:
+    globals()[object] =  deepcopy(getattr(opcode_3x, object))
+
+# Below are opcodes changes since Python 3.2
+
+rm_op(opname, opmap, 'STOP_CODE', 0)
+rm_op(opname, opmap, 'STORE_LOCALS', 69)
+
+# These are new since Python 3.3
+def_op('YIELD_FROM', 72)
+def_op('LOAD_CLASSDEREF', 148)
+
+# These are new since Python 3.4
+def_op('BINARY_MATRIX_MULTIPLY', 16)
+def_op('INPLACE_MATRIX_MULTIPLY', 17)
+def_op('GET_AITER', 50)
+def_op('GET_ANEXT', 51)
+def_op('BEFORE_ASYNC_WITH', 52)
+def_op('GET_YIELD_FROM_ITER', 69)
+def_op('GET_AWAITABLE', 73)
+def_op('WITH_CLEANUP_START', 81)
+def_op('WITH_CLEANUP_FINISH', 82)
+def_op('BUILD_LIST_UNPACK', 149)
+def_op('BUILD_MAP_UNPACK', 150)
+def_op('BUILD_MAP_UNPACK_WITH_CALL', 151)
+def_op('BUILD_TUPLE_UNPACK', 152)
+def_op('BUILD_SET_UNPACK', 153)
+def_op('SETUP_ASYNC_WITH', 154)
+
+
+hasfree.append(148)
+
+def updateGlobal():
+    # JUMP_OPs are used in verification are set in the scanner
+    # and used in the parser grammar
+    globals().update({'PJIF': opmap['POP_JUMP_IF_FALSE']})
+    globals().update({'PJIT': opmap['POP_JUMP_IF_TRUE']})
+    globals().update({'JA': opmap['JUMP_ABSOLUTE']})
+    globals().update({'JF': opmap['JUMP_FORWARD']})
+    globals().update(dict([(k.replace('+', '_'), v) for (k, v) in opmap.items()]))
+    globals().update({'JUMP_OPs': map(lambda op: opname[op], hasjrel + hasjabs)})
+
+updateGlobal()
+
+# FIXME: turn into pytest test
+from uncompyle6 import PYTHON_VERSION
+if PYTHON_VERSION == 3.5:
+    import dis
+    for item in dis.opmap.items():
+        if item not in opmap.items():
+            print(item)
+    assert all(item in opmap.items() for item in dis.opmap.items())
+
+# opcode_3x.dump_opcodes(opmap)

uncompyle6/opcodes/opcode_3x.py

@@ -0,0 +1,210 @@
+"""
+CPython 3.2 bytecode opcodes to be used as a base for other opcodes including 3.2.
+If this file changes the other opcode files may have to a adjusted accordingly.
+"""
+
+cmp_op = ('<', '<=', '==', '!=', '>', '>=', 'in', 'not in', 'is',
+        'is not', 'exception match', 'BAD')
+
+hasconst = []
+hasname = []
+hasjrel = []
+hasjabs = []
+haslocal = []
+hascompare = []
+hasfree = []
+hasnargs = []
+
+opmap = {}
+opname = [''] * 256
+for op in range(256): opname[op] = '<%r>' % (op,)
+del op
+
+def def_op(name, op):
+    opname[op] = name
+    opmap[name] = op
+
+def name_op(name, op):
+    def_op(name, op)
+    hasname.append(op)
+
+def jrel_op(name, op):
+    def_op(name, op)
+    hasjrel.append(op)
+
+def jabs_op(name, op):
+    def_op(name, op)
+    hasjabs.append(op)
+
+# Instruction opcodes for compiled code
+# Blank lines correspond to available opcodes
+
+def_op('STOP_CODE', 0)
+def_op('POP_TOP', 1)
+def_op('ROT_TWO', 2)
+def_op('ROT_THREE', 3)
+def_op('DUP_TOP', 4)
+def_op('DUP_TOP_TWO', 5)
+
+def_op('NOP', 9)
+def_op('UNARY_POSITIVE', 10)
+def_op('UNARY_NEGATIVE', 11)
+def_op('UNARY_NOT', 12)
+
+def_op('UNARY_INVERT', 15)
+
+def_op('BINARY_POWER', 19)
+def_op('BINARY_MULTIPLY', 20)
+
+def_op('BINARY_MODULO', 22)
+def_op('BINARY_ADD', 23)
+def_op('BINARY_SUBTRACT', 24)
+def_op('BINARY_SUBSCR', 25)
+def_op('BINARY_FLOOR_DIVIDE', 26)
+def_op('BINARY_TRUE_DIVIDE', 27)
+def_op('INPLACE_FLOOR_DIVIDE', 28)
+def_op('INPLACE_TRUE_DIVIDE', 29)
+
+# Gone from Python 3 are Python2's
+# SLICE+0 .. SLICE+3
+# STORE_SLICE+0 .. STORE_SLICE+3
+# DELETE_SLICE+0 .. DELETE_SLICE+3
+
+def_op('STORE_MAP', 54)
+def_op('INPLACE_ADD', 55)
+def_op('INPLACE_SUBTRACT', 56)
+def_op('INPLACE_MULTIPLY', 57)
+
+def_op('INPLACE_MODULO', 59)
+def_op('STORE_SUBSCR', 60)
+def_op('DELETE_SUBSCR', 61)
+def_op('BINARY_LSHIFT', 62)
+def_op('BINARY_RSHIFT', 63)
+def_op('BINARY_AND', 64)
+def_op('BINARY_XOR', 65)
+def_op('BINARY_OR', 66)
+def_op('INPLACE_POWER', 67)
+def_op('GET_ITER', 68)
+def_op('STORE_LOCALS', 69)
+
+def_op('PRINT_EXPR', 70)
+def_op('LOAD_BUILD_CLASS', 71)
+
+# Python3 drops/changes:
+#  def_op('PRINT_ITEM', 71)
+#  def_op('PRINT_NEWLINE', 72)
+#  def_op('PRINT_ITEM_TO', 73)
+#  def_op('PRINT_NEWLINE_TO', 74)
+
+def_op('INPLACE_LSHIFT', 75)
+def_op('INPLACE_RSHIFT', 76)
+def_op('INPLACE_AND', 77)
+def_op('INPLACE_XOR', 78)
+def_op('INPLACE_OR', 79)
+def_op('BREAK_LOOP', 80)
+def_op('WITH_CLEANUP', 81)
+
+def_op('RETURN_VALUE', 83)
+def_op('IMPORT_STAR', 84)
+
+def_op('YIELD_VALUE', 86)
+def_op('POP_BLOCK', 87)
+def_op('END_FINALLY', 88)
+def_op('POP_EXCEPT', 89)
+
+HAVE_ARGUMENT = 90              # Opcodes from here have an argument:
+
+name_op('STORE_NAME', 90)       # Index in name list
+name_op('DELETE_NAME', 91)      # ""
+def_op('UNPACK_SEQUENCE', 92)   # Number of tuple items
+jrel_op('FOR_ITER', 93)
+def_op('UNPACK_EX', 94)
+name_op('STORE_ATTR', 95)       # Index in name list
+name_op('DELETE_ATTR', 96)      # ""
+name_op('STORE_GLOBAL', 97)     # ""
+name_op('DELETE_GLOBAL', 98)    # ""
+
+# Python 2's DUP_TOPX is gone
+
+def_op('LOAD_CONST', 100)       # Index in const list
+hasconst.append(100)
+name_op('LOAD_NAME', 101)       # Index in name list
+def_op('BUILD_TUPLE', 102)      # Number of tuple items
+def_op('BUILD_LIST', 103)       # Number of list items
+def_op('BUILD_SET', 104)        # Number of set items
+def_op('BUILD_MAP', 105)        # Number of dict entries (upto 255)
+name_op('LOAD_ATTR', 106)       # Index in name list
+def_op('COMPARE_OP', 107)       # Comparison operator
+hascompare.append(107)
+name_op('IMPORT_NAME', 108)     # Index in name list
+name_op('IMPORT_FROM', 109)     # Index in name list
+
+jrel_op('JUMP_FORWARD', 110)    # Number of bytes to skip
+jabs_op('JUMP_IF_FALSE_OR_POP', 111) # Target byte offset from beginning of code
+jabs_op('JUMP_IF_TRUE_OR_POP', 112)  # ""
+jabs_op('JUMP_ABSOLUTE', 113)        # ""
+jabs_op('POP_JUMP_IF_FALSE', 114)    # ""
+jabs_op('POP_JUMP_IF_TRUE', 115)     # ""
+
+name_op('LOAD_GLOBAL', 116)     # Index in name list
+
+jabs_op('CONTINUE_LOOP', 119)   # Target address
+jrel_op('SETUP_LOOP', 120)      # Distance to target address
+jrel_op('SETUP_EXCEPT', 121)    # ""
+jrel_op('SETUP_FINALLY', 122)   # ""
+
+def_op('LOAD_FAST', 124)        # Local variable number
+haslocal.append(124)
+def_op('STORE_FAST', 125)       # Local variable number
+haslocal.append(125)
+def_op('DELETE_FAST', 126)      # Local variable number
+haslocal.append(126)
+
+def_op('RAISE_VARARGS', 130)    # Number of raise arguments (1, 2, or 3)
+def_op('CALL_FUNCTION', 131)    # #args + (#kwargs << 8)
+hasnargs.append(131)
+def_op('MAKE_FUNCTION', 132)    # Number of args with default values
+def_op('BUILD_SLICE', 133)      # Number of items
+def_op('MAKE_CLOSURE', 134)
+def_op('LOAD_CLOSURE', 135)
+hasfree.append(135)
+def_op('LOAD_DEREF', 136)
+hasfree.append(136)
+def_op('STORE_DEREF', 137)
+hasfree.append(137)
+def_op('DELETE_DEREF', 138)
+hasfree.append(138)
+
+def_op('CALL_FUNCTION_VAR', 140)     # #args + (#kwargs << 8)
+hasnargs.append(140)
+def_op('CALL_FUNCTION_KW', 141)      # #args + (#kwargs << 8)
+hasnargs.append(141)
+def_op('CALL_FUNCTION_VAR_KW', 142)  # #args + (#kwargs << 8)
+hasnargs.append(142)
+
+jrel_op('SETUP_WITH', 143)
+
+def_op('LIST_APPEND', 145)
+def_op('SET_ADD', 146)
+def_op('MAP_ADD', 147)
+
+def_op('EXTENDED_ARG', 144)
+EXTENDED_ARG = 144
+
+fields2copy = """cmp_op hasconst hasname hasjrel hasjabs haslocal hascompare hasfree hasnargs
+opmap opname HAVE_ARGUMENT EXTENDED_ARG""".split()
+
+def rm_op(opname, opmap, name, op):
+    # opname is an array, so we need to keep the position in there.
+    opname[op] = ''
+
+    assert opmap[name] == op
+    del opmap[name]
+
+def dump_opcodes(opmap):
+    """Utility for dumping opcodes"""
+    op2name = {}
+    for k in opmap.keys():
+        op2name[opmap[k]] = k
+    for i in sorted(op2name.keys()):
+        print("%-3s %s" % (str(i), op2name[i]))

uncompyle6/parser.py

@@ -1,9 +1,9 @@
-#  Copyright (c) 1999 John Aycock
-#  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
+#  Copyright (c) 2015-2016 Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
-#  Copyright (c) 2015 Rocky Bernstein
+#  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
+#  Copyright (c) 1999 John Aycock
 """
-Common spark parser routines Python.
+Common uncompyle parser routines.
 """
 
 from __future__ import print_function
@@ -11,7 +11,7 @@ from __future__ import print_function
 import sys
 
 from uncompyle6.code import iscode
-from uncompyle6.parsers.spark import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 
 class ParserError(Exception):
     def __init__(self, token, offset):
@@ -69,6 +69,130 @@ class PythonParser(GenericASTBuilder):
         # print >> sys.stderr, 'resolve', str(list)
         return GenericASTBuilder.resolve(self, list)
 
+    ##############################################
+    ## Common Python 2 and Python 3 grammar rules
+    ##############################################
+    def p_start(self, args):
+        '''
+        # The start or goal symbol
+        stmts ::= stmts sstmt
+        stmts ::= sstmt
+        '''
+
+    def p_call_stmt(self, args):
+        '''
+        # eval-mode compilation.  Single-mode interactive compilation
+        # adds another rule.
+        call_stmt ::= expr POP_TOP
+        '''
+
+    def p_funcdef(self, args):
+        '''
+        stmt ::= funcdef
+        funcdef ::= mkfunc designator
+        stmt ::= funcdefdeco
+        funcdefdeco ::= mkfuncdeco designator
+        mkfuncdeco ::= expr mkfuncdeco CALL_FUNCTION_1
+        mkfuncdeco ::= expr mkfuncdeco0 CALL_FUNCTION_1
+        mkfuncdeco0 ::= mkfunc
+        load_closure ::= load_closure LOAD_CLOSURE
+        load_closure ::= LOAD_CLOSURE
+        '''
+
+    def p_genexpr(self, args):
+        '''
+        expr ::= genexpr
+
+        genexpr ::= LOAD_GENEXPR MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
+
+        stmt ::= genexpr_func
+
+        genexpr_func ::= LOAD_FAST FOR_ITER designator comp_iter JUMP_BACK
+        '''
+
+    def p_dictcomp(self, args):
+        '''
+        expr ::= dictcomp
+        dictcomp ::= LOAD_DICTCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
+        stmt ::= dictcomp_func
+
+        dictcomp_func ::= BUILD_MAP LOAD_FAST FOR_ITER designator
+                comp_iter JUMP_BACK RETURN_VALUE RETURN_LAST
+
+        '''
+
+    def p_augmented_assign(self, args):
+        '''
+        stmt ::= augassign1
+        stmt ::= augassign2
+        augassign1 ::= expr expr inplace_op designator
+        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SUBSCR
+        augassign1 ::= expr expr inplace_op ROT_TWO   STORE_SLICE+0
+        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+1
+        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+2
+        augassign1 ::= expr expr inplace_op ROT_FOUR  STORE_SLICE+3
+        augassign2 ::= expr DUP_TOP LOAD_ATTR expr
+                inplace_op ROT_TWO   STORE_ATTR
+
+        inplace_op ::= INPLACE_ADD
+        inplace_op ::= INPLACE_SUBTRACT
+        inplace_op ::= INPLACE_MULTIPLY
+        inplace_op ::= INPLACE_DIVIDE
+        inplace_op ::= INPLACE_TRUE_DIVIDE
+        inplace_op ::= INPLACE_FLOOR_DIVIDE
+        inplace_op ::= INPLACE_MODULO
+        inplace_op ::= INPLACE_POWER
+        inplace_op ::= INPLACE_LSHIFT
+        inplace_op ::= INPLACE_RSHIFT
+        inplace_op ::= INPLACE_AND
+        inplace_op ::= INPLACE_XOR
+        inplace_op ::= INPLACE_OR
+        '''
+
+    def p_assign(self, args):
+        '''
+        stmt ::= assign
+        assign ::= expr DUP_TOP designList
+        assign ::= expr designator
+
+        stmt ::= assign2
+        stmt ::= assign3
+        assign2 ::= expr expr ROT_TWO designator designator
+        assign3 ::= expr expr expr ROT_THREE ROT_TWO designator designator designator
+        '''
+
+    def p_import20(self, args):
+        '''
+        stmt ::= importstmt
+        stmt ::= importfrom
+        stmt ::= importstar
+        stmt ::= importmultiple
+
+        importlist2 ::= importlist2 import_as
+        importlist2 ::= import_as
+        import_as ::= IMPORT_NAME designator
+        import_as ::= IMPORT_NAME load_attrs designator
+        import_as ::= IMPORT_FROM designator
+
+        importstmt ::= LOAD_CONST LOAD_CONST import_as
+        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME IMPORT_STAR
+        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist2 POP_TOP
+        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT IMPORT_STAR
+        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT importlist2 POP_TOP
+        importmultiple ::= LOAD_CONST LOAD_CONST import_as imports_cont
+
+        imports_cont ::= imports_cont import_cont
+        imports_cont ::= import_cont
+        import_cont ::= LOAD_CONST LOAD_CONST import_as_cont
+        import_as_cont ::= IMPORT_NAME_CONT designator
+        import_as_cont ::= IMPORT_NAME_CONT load_attrs designator
+        import_as_cont ::= IMPORT_FROM designator
+
+        load_attrs ::= LOAD_ATTR
+        load_attrs ::= load_attrs LOAD_ATTR
+        '''
+
+
 def parse(p, tokens, customize):
     p.add_custom_rules(tokens, customize)
     ast = p.parse(tokens)
@@ -76,17 +200,27 @@ def parse(p, tokens, customize):
     return ast
 
 
-def get_python_parser(version, debug_parser):
+def get_python_parser(version, debug_parser, compile_mode='exec'):
     """
     Returns parser object for Python version 2 or 3
-    depending on the parameter passed.
+    depending on the parameter passed.  *compile_mode*
+    is either 'exec', 'eval', or 'single'. See
+    https://docs.python.org/3.6/library/functions.html#compile for an explanation
+    of the different modes.
     """
+
     if version < 3.0:
         import uncompyle6.parsers.parse2 as parse2
-        p = parse2.Python2Parser(debug_parser)
+        if compile_mode == 'exec':
+            p = parse2.Python2Parser(debug_parser)
+        else:
+            p = parse2.Python2ParserSingle(debug_parser)
     else:
         import uncompyle6.parsers.parse3 as parse3
-        p = parse3.Python3Parser(debug_parser)
+        if compile_mode == 'exec':
+            p = parse3.Python3Parser(debug_parser)
+        else:
+            p = parse3.Python3ParserSingle(debug_parser)
     p.version = version
     return p
 
@@ -105,7 +239,7 @@ def python_parser(version, co, out=sys.stdout, showasm=False,
 
 if __name__ == '__main__':
     def parse_test(co):
-        from uncompyl6e import PYTHON_VERSION
+        from uncompyle6 import PYTHON_VERSION
         ast = python_parser(PYTHON_VERSION, co, showasm=True)
         print(ast)
         return

uncompyle6/parsers/astnode.py

@@ -10,8 +10,8 @@ else:
 
 
 class AST(UserList):
-    def __init__(self, type, kids=[]):
-        self.type = intern(type)
+    def __init__(self, kind, kids=[]):
+        self.type = intern(kind)
         UserList.__init__(self, kids)
 
     def isNone(self):
@@ -28,7 +28,8 @@ class AST(UserList):
         else:
             return self.type == o
 
-    def __hash__(self):            return hash(self.type)
+    def __hash__(self):
+        return hash(self.type)
 
     def __repr__(self, indent=''):
         rv = str(self.type)

uncompyle6/parsers/parse2.py

@@ -19,27 +19,18 @@ from __future__ import print_function
 
 from uncompyle6.parser import PythonParser, nop_func
 from uncompyle6.parsers.astnode import AST
-from uncompyle6.parsers.spark import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6 import PYTHON3
 
 class Python2Parser(PythonParser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        GenericASTBuilder.__init__(self, AST, 'stmts', debug=debug_parser)
+        if PYTHON3:
+            super().__init__(AST, 'stmts', debug=debug_parser)
+        else:
+            super(Python2Parser, self).__init__(AST, 'stmts', debug=debug_parser)
         self.customized = {}
 
-    def p_funcdef(self, args):
-        '''
-        stmt ::= funcdef
-        funcdef ::= mkfunc designator
-        stmt ::= funcdefdeco
-        funcdefdeco ::= mkfuncdeco designator
-        mkfuncdeco ::= expr mkfuncdeco CALL_FUNCTION_1
-        mkfuncdeco ::= expr mkfuncdeco0 CALL_FUNCTION_1
-        mkfuncdeco0 ::= mkfunc
-        load_closure ::= load_closure LOAD_CLOSURE
-        load_closure ::= LOAD_CLOSURE
-        '''
-
     def p_list_comprehension(self, args):
         '''
         expr ::= list_compr
@@ -84,71 +75,11 @@ class Python2Parser(PythonParser):
 
         comp_if ::= expr jmp_false comp_iter
         comp_ifnot ::= expr jmp_true comp_iter
+
+        # This is different in python3 - shout it be?
         comp_for ::= expr _for designator comp_iter JUMP_BACK
         '''
 
-    def p_genexpr(self, args):
-        '''
-        expr ::= genexpr
-
-        genexpr ::= LOAD_GENEXPR MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-
-        stmt ::= genexpr_func
-
-        genexpr_func ::= LOAD_FAST FOR_ITER designator comp_iter JUMP_BACK
-        '''
-
-    def p_dictcomp(self, args):
-        '''
-        expr ::= dictcomp
-        dictcomp ::= LOAD_DICTCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-        stmt ::= dictcomp_func
-
-        dictcomp_func ::= BUILD_MAP LOAD_FAST FOR_ITER designator
-                comp_iter JUMP_BACK RETURN_VALUE RETURN_LAST
-
-        '''
-
-    def p_augmented_assign(self, args):
-        '''
-        stmt ::= augassign1
-        stmt ::= augassign2
-        augassign1 ::= expr expr inplace_op designator
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SUBSCR
-        augassign1 ::= expr expr inplace_op ROT_TWO   STORE_SLICE+0
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+1
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+2
-        augassign1 ::= expr expr inplace_op ROT_FOUR  STORE_SLICE+3
-        augassign2 ::= expr DUP_TOP LOAD_ATTR expr
-                inplace_op ROT_TWO   STORE_ATTR
-
-        inplace_op ::= INPLACE_ADD
-        inplace_op ::= INPLACE_SUBTRACT
-        inplace_op ::= INPLACE_MULTIPLY
-        inplace_op ::= INPLACE_DIVIDE
-        inplace_op ::= INPLACE_TRUE_DIVIDE
-        inplace_op ::= INPLACE_FLOOR_DIVIDE
-        inplace_op ::= INPLACE_MODULO
-        inplace_op ::= INPLACE_POWER
-        inplace_op ::= INPLACE_LSHIFT
-        inplace_op ::= INPLACE_RSHIFT
-        inplace_op ::= INPLACE_AND
-        inplace_op ::= INPLACE_XOR
-        inplace_op ::= INPLACE_OR
-        '''
-
-    def p_assign(self, args):
-        '''
-        stmt ::= assign
-        assign ::= expr DUP_TOP designList
-        assign ::= expr designator
-
-        stmt ::= assign2
-        stmt ::= assign3
-        assign2 ::= expr expr ROT_TWO designator designator
-        assign3 ::= expr expr expr ROT_THREE ROT_TWO designator designator designator
-        '''
-
     def p_print(self, args):
         '''
         stmt ::= print_items_stmt
@@ -178,41 +109,8 @@ class Python2Parser(PythonParser):
         print_to_item ::= DUP_TOP expr ROT_TWO PRINT_ITEM_TO
         '''
 
-    def p_import20(self, args):
-        '''
-        stmt ::= importstmt
-        stmt ::= importfrom
-        stmt ::= importstar
-        stmt ::= importmultiple
-
-        importlist2 ::= importlist2 import_as
-        importlist2 ::= import_as
-        import_as ::= IMPORT_NAME designator
-        import_as ::= IMPORT_NAME load_attrs designator
-        import_as ::= IMPORT_FROM designator
-
-        importstmt ::= LOAD_CONST LOAD_CONST import_as
-        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME IMPORT_STAR
-        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist2 POP_TOP
-        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT IMPORT_STAR
-        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT importlist2 POP_TOP
-        importmultiple ::= LOAD_CONST LOAD_CONST import_as imports_cont
-
-        imports_cont ::= imports_cont import_cont
-        imports_cont ::= import_cont
-        import_cont ::= LOAD_CONST LOAD_CONST import_as_cont
-        import_as_cont ::= IMPORT_NAME_CONT designator
-        import_as_cont ::= IMPORT_NAME_CONT load_attrs designator
-        import_as_cont ::= IMPORT_FROM designator
-
-        load_attrs ::= LOAD_ATTR
-        load_attrs ::= load_attrs LOAD_ATTR
-        '''
-
     def p_grammar(self, args):
         '''
-        stmts ::= stmts sstmt
-        stmts ::= sstmt
         sstmt ::= stmt
         sstmt ::= ifelsestmtr
         sstmt ::= return_stmt RETURN_LAST
@@ -581,8 +479,8 @@ class Python2Parser(PythonParser):
         _mklambda ::= load_closure mklambda
         _mklambda ::= mklambda
 
-        or   ::= expr jmp_true expr _come_from
         or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
+        or   ::= expr jmp_true expr _come_from
         and  ::= expr jmp_false expr _come_from
         and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
         and2 ::= _jump jmp_false COME_FROM expr COME_FROM
@@ -601,10 +499,10 @@ class Python2Parser(PythonParser):
         ret_expr_or_cond ::= ret_cond
         ret_expr_or_cond ::= ret_cond_not
 
-        ret_and  ::= expr jmp_false ret_expr_or_cond COME_FROM
-        ret_or   ::= expr jmp_true ret_expr_or_cond COME_FROM
-        ret_cond ::= expr jmp_false expr RETURN_END_IF ret_expr_or_cond
-        ret_cond_not ::= expr jmp_true expr RETURN_END_IF ret_expr_or_cond
+        ret_and  ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
+        ret_or   ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+        ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
+        ret_cond_not ::= expr POP_JUMP_IF_TRUE expr RETURN_END_IF ret_expr_or_cond
 
         stmt ::= return_lambda
         stmt ::= conditional_lambda
@@ -710,3 +608,13 @@ class Python2Parser(PythonParser):
             else:
                 raise Exception('unknown customize token %s' % k)
             self.addRule(rule, nop_func)
+
+class Python2ParserSingle(Python2Parser):
+    def p_call_stmt(self, args):
+        '''
+        # single-mode compilation. eval-mode interactive compilation
+        # drops the last rule.
+
+        call_stmt ::= expr POP_TOP
+        call_stmt ::= expr PRINT_EXPR
+        '''

uncompyle6/parsers/parse3.py

@@ -19,13 +19,17 @@ from __future__ import print_function
 
 from uncompyle6.parser import PythonParser, nop_func
 from uncompyle6.parsers.astnode import AST
-from uncompyle6.parsers.spark import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6 import PYTHON3
 
 class Python3Parser(PythonParser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
         self.added_rules = set()
-        GenericASTBuilder.__init__(self, AST, 'stmts', debug=debug_parser)
+        if PYTHON3:
+            super().__init__(AST, 'stmts', debug=debug_parser)
+        else:
+            super(Python3Parser, self).__init__(AST, 'stmts', debug=debug_parser)
         self.new_rules = set()
 
     def add_unique_rule(self, rule, opname, count, customize):
@@ -38,19 +42,6 @@ class Python3Parser(PythonParser):
             pass
         return
 
-    def p_funcdef(self, args):
-        '''
-        stmt ::= funcdef
-        funcdef ::= mkfunc designator
-        stmt ::= funcdefdeco
-        funcdefdeco ::= mkfuncdeco designator
-        mkfuncdeco ::= expr mkfuncdeco CALL_FUNCTION_1
-        mkfuncdeco ::= expr mkfuncdeco0 CALL_FUNCTION_1
-        mkfuncdeco0 ::= mkfunc
-        load_closure ::= load_closure LOAD_CLOSURE
-        load_closure ::= LOAD_CLOSURE
-        '''
-
     def p_list_comprehension(self, args):
         '''
         # Python3 scanner adds LOAD_LISTCOMP. Python3 does list comprehension like
@@ -101,135 +92,13 @@ class Python3Parser(PythonParser):
 
         comp_if ::= expr jmp_false comp_iter
         comp_ifnot ::= expr jmp_true comp_iter
+
+        # This is different in python2 - should it be?
         comp_for ::= expr _for designator comp_iter JUMP_ABSOLUTE
         '''
 
-    def p_genexpr(self, args):
-        '''
-        expr ::= genexpr
-
-        genexpr ::= LOAD_GENEXPR MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-
-        stmt ::= genexpr_func
-
-        genexpr_func ::= LOAD_FAST FOR_ITER designator comp_iter JUMP_BACK
-        '''
-
-    def p_dictcomp(self, args):
-        '''
-        expr ::= dictcomp
-        dictcomp ::= LOAD_DICTCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-        stmt ::= dictcomp_func
-
-        dictcomp_func ::= BUILD_MAP LOAD_FAST FOR_ITER designator
-                comp_iter JUMP_BACK RETURN_VALUE RETURN_LAST
-
-        '''
-
-    def p_augmented_assign(self, args):
-        '''
-        stmt ::= augassign1
-        stmt ::= augassign2
-        augassign1 ::= expr expr inplace_op designator
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SUBSCR
-        augassign1 ::= expr expr inplace_op ROT_TWO   STORE_SLICE+0
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+1
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+2
-        augassign1 ::= expr expr inplace_op ROT_FOUR  STORE_SLICE+3
-        augassign2 ::= expr DUP_TOP LOAD_ATTR expr
-                inplace_op ROT_TWO   STORE_ATTR
-
-        inplace_op ::= INPLACE_ADD
-        inplace_op ::= INPLACE_SUBTRACT
-        inplace_op ::= INPLACE_MULTIPLY
-        inplace_op ::= INPLACE_DIVIDE
-        inplace_op ::= INPLACE_TRUE_DIVIDE
-        inplace_op ::= INPLACE_FLOOR_DIVIDE
-        inplace_op ::= INPLACE_MODULO
-        inplace_op ::= INPLACE_POWER
-        inplace_op ::= INPLACE_LSHIFT
-        inplace_op ::= INPLACE_RSHIFT
-        inplace_op ::= INPLACE_AND
-        inplace_op ::= INPLACE_XOR
-        inplace_op ::= INPLACE_OR
-        '''
-
-    def p_assign(self, args):
-        '''
-        stmt ::= assign
-        assign ::= expr DUP_TOP designList
-        assign ::= expr designator
-
-        stmt ::= assign2
-        stmt ::= assign3
-        assign2 ::= expr expr ROT_TWO designator designator
-        assign3 ::= expr expr expr ROT_THREE ROT_TWO designator designator designator
-        '''
-
-    # Python3 doesn't have a built-in print.
-    # def p_print(self, args):
-    #     '''
-    #     stmt ::= print_items_stmt
-    #     stmt ::= print_nl
-    #     stmt ::= print_items_nl_stmt
-
-    #     print_items_stmt ::= expr PRINT_ITEM print_items_opt
-    #     print_items_nl_stmt ::= expr PRINT_ITEM print_items_opt PRINT_NEWLINE_CONT
-    #     print_items_opt ::= print_items
-    #     print_items_opt ::=
-    #     print_items ::= print_items print_item
-    #     print_items ::= print_item
-    #     print_item ::= expr PRINT_ITEM_CONT
-    #     print_nl ::= PRINT_NEWLINE
-    #     '''
-
-    # def p_print_to(self, args):
-    #     '''
-    #     stmt ::= print_to
-    #     stmt ::= print_to_nl
-    #     stmt ::= print_nl_to
-    #     print_to ::= expr print_to_items POP_TOP
-    #     print_to_nl ::= expr print_to_items PRINT_NEWLINE_TO
-    #     print_nl_to ::= expr PRINT_NEWLINE_TO
-    #     print_to_items ::= print_to_items print_to_item
-    #     print_to_items ::= print_to_item
-    #     print_to_item ::= DUP_TOP expr ROT_TWO PRINT_ITEM_TO
-    #     '''
-
-    def p_import20(self, args):
-        '''
-        stmt ::= importstmt
-        stmt ::= importfrom
-        stmt ::= importstar
-        stmt ::= importmultiple
-
-        importlist2 ::= importlist2 import_as
-        importlist2 ::= import_as
-        import_as ::= IMPORT_NAME designator
-        import_as ::= IMPORT_NAME load_attrs designator
-        import_as ::= IMPORT_FROM designator
-
-        importstmt ::= LOAD_CONST LOAD_CONST import_as
-        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME IMPORT_STAR
-        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist2 POP_TOP
-        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT IMPORT_STAR
-        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT importlist2 POP_TOP
-        importmultiple ::= LOAD_CONST LOAD_CONST import_as imports_cont
-
-        imports_cont ::= imports_cont import_cont
-        imports_cont ::= import_cont
-        import_cont ::= LOAD_CONST LOAD_CONST import_as_cont
-        import_as_cont ::= IMPORT_NAME_CONT designator
-        import_as_cont ::= IMPORT_NAME_CONT load_attrs designator
-        import_as_cont ::= IMPORT_FROM designator
-
-        load_attrs ::= LOAD_ATTR
-        load_attrs ::= load_attrs LOAD_ATTR
-        '''
-
     def p_grammar(self, args):
-        '''stmts ::= stmts sstmt
-        stmts ::= sstmt
+        '''
         sstmt ::= stmt
         sstmt ::= ifelsestmtr
         sstmt ::= return_stmt RETURN_LAST
@@ -285,6 +154,9 @@ class Python3Parser(PythonParser):
         designList ::= designator designator
         designList ::= designator DUP_TOP designList
 
+        # FIXME: Store local is only used in Python 3.2
+        designator ::= STORE_LOCALS
+
         designator ::= STORE_FAST
         designator ::= STORE_NAME
         designator ::= STORE_GLOBAL
@@ -301,7 +173,6 @@ class Python3Parser(PythonParser):
 
         stmt ::= classdef
         stmt ::= call_stmt
-        call_stmt ::= expr POP_TOP
 
         stmt ::= return_stmt
         return_stmt ::= ret_expr RETURN_VALUE
@@ -368,9 +239,9 @@ class Python3Parser(PythonParser):
 
         kwarg   ::= LOAD_CONST expr
 
-        classdef ::= buildclass designator
+        classdef ::= build_class designator
         # Python3 introduced LOAD_BUILD_CLASS
-        # the definition of buildclass is a custom rule
+        # the definition of build_class is a custom rule
 
         stmt ::= classdefdeco
         classdefdeco ::= classdefdeco1 designator
@@ -407,6 +278,9 @@ class Python3Parser(PythonParser):
         _ifstmts_jump ::= return_if_stmts
         _ifstmts_jump ::= c_stmts_opt JUMP_FORWARD _come_from _come_from
 
+        # FIXME: this optimization is only used in Python 3.5 and beyond
+        _ifstmts_jump ::= c_stmts_opt
+
         iflaststmt ::= testexpr c_stmts_opt JUMP_ABSOLUTE
 
         iflaststmtl ::= testexpr c_stmts_opt JUMP_BACK
@@ -547,6 +421,7 @@ class Python3Parser(PythonParser):
         expr ::= LOAD_GLOBAL
         expr ::= LOAD_DEREF
         expr ::= LOAD_LOCALS
+        expr ::= LOAD_CLASSNAME
         expr ::= load_attr
         expr ::= binary_expr
         expr ::= binary_expr_na
@@ -630,10 +505,10 @@ class Python3Parser(PythonParser):
         ret_expr_or_cond ::= ret_cond
         ret_expr_or_cond ::= ret_cond_not
 
-        ret_and  ::= expr jmp_false ret_expr_or_cond COME_FROM
-        ret_or   ::= expr jmp_true ret_expr_or_cond COME_FROM
-        ret_cond ::= expr jmp_false expr RETURN_END_IF ret_expr_or_cond
-        ret_cond_not ::= expr jmp_true expr RETURN_END_IF ret_expr_or_cond
+        ret_and  ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
+        ret_or   ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+        ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
+        ret_cond_not ::= expr POP_JUMP_IF_TRUE expr RETURN_END_IF ret_expr_or_cond
 
         stmt ::= return_lambda
         stmt ::= conditional_lambda
@@ -677,72 +552,61 @@ class Python3Parser(PythonParser):
         nullexprlist ::=
         '''
 
-    def custom_buildclass_rule(self, opname, i, token, tokens, customize):
-        """
-        Python >= 3.3:
-          buildclass ::= LOAD_BUILD_CLASS mkfunc
-                         LOAD_CLASSNAME expr CALL_FUNCTION_3
-          or
-          buildclass ::= LOAD_BUILD_CLASS mkfunc
-                         LOAD_CONST CALL_FUNCTION_3
-        Python < 3.3
-          buildclass ::= LOAD_BUILD_CLASS LOAD_CONST MAKE_FUNCTION_0 LOAD_CONST
-                         CALL_FUNCTION_n
+    @staticmethod
+    def call_fn_name(token):
+        """Customize CALL_FUNCTION to add the number of positional arguments"""
+        return 'CALL_FUNCTION_%i' % token.attr
 
+    def custom_build_class_rule(self, opname, i, token, tokens, customize):
         """
-
+        build_class ::= LOAD_BUILD_CLASS mkfunc
+                        LOAD_CLASSNAME {expr}^n CALL_FUNCTION_n+2
+                        LOAD_CONST CALL_FUNCTION_n
+        """
+        # FIXME: I bet this can be simplified
         # look for next MAKE_FUNCTION
         for i in range(i+1, len(tokens)):
             if tokens[i].type.startswith('MAKE_FUNCTION'):
                 break
             pass
-        assert i < len(tokens)
-        assert tokens[i+1].type == 'LOAD_CONST'
-        # find load names
-        have_loadname = False
-        for i in range(i+1, len(tokens)):
-            if tokens[i].type == 'LOAD_NAME':
-                if tokens[i+1].type != 'LOAD_ATTR':
-                    tokens[i].type = 'LOAD_CLASSNAME'
-                have_loadname = True
+        assert i < len(tokens), "build_class needs to find MAKE_FUNCTION"
+        assert tokens[i+1].type == 'LOAD_CONST', \
+          "build_class expecing CONST after MAKE_FUNCTION"
+        for i in range(i, len(tokens)):
+            if tokens[i].type == 'CALL_FUNCTION':
+                call_fn_tok = tokens[i]
                 break
-            if tokens[i].type in 'CALL_FUNCTION':
-                break
-            pass
-        assert i < len(tokens)
-        have_load_attr = False
-        if have_loadname:
-            j = 1
-            for i in range(i+1, len(tokens)):
-                if tokens[i].type in ['CALL_FUNCTION', 'LOAD_ATTR']:
-                    if tokens[i].type == 'LOAD_ATTR':
-                        have_load_attr = True
-                    break
-                assert tokens[i].type == 'LOAD_NAME', \
-                  'Expecting LOAD_NAME after CALL_FUNCTION'
-                tokens[i].type = 'LOAD_CLASSNAME'
-                j += 1
-                pass
-            load_names = 'LOAD_CLASSNAME ' * j
-        else:
-            j = 0
-            load_names = ''
+        assert call_fn_tok, "build_class custom rule needs to find CALL_FUNCTION"
+
         # customize CALL_FUNCTION
-        if self.version >= 3.3:
-            if not have_load_attr:
-                call_function = 'CALL_FUNCTION_%d' % (j + 2)
-                rule = ("buildclass ::= LOAD_BUILD_CLASS mkfunc LOAD_CONST "
-                        "%s%s" % (load_names, call_function))
-            else:
-                rule = ("buildclass ::= LOAD_BUILD_CLASS mkfunc LOAD_CONST expr "
-                        "CALL_FUNCTION_3")
-        else:
-            call_function = 'CALL_FUNCTION_%d' % (j + 1)
-            rule = ("buildclass ::= LOAD_BUILD_CLASS mkfunc %s%s" %
-                    (load_names, call_function))
+        call_function = self.call_fn_name(call_fn_tok)
+        args_pos = call_fn_tok.attr & 0xff
+        args_kw = (call_fn_tok.attr >> 8) & 0xff
+        rule = ("build_class ::= LOAD_BUILD_CLASS mkfunc %s"
+                "%s" % (('expr ' * (args_pos - 1) + ('kwarg ' * args_kw)),
+                        call_function))
         self.add_unique_rule(rule, opname, token.attr, customize)
         return
 
+    def custom_classfunc_rule(self, opname, token, customize):
+        """
+        call_function ::= expr {expr}^n CALL_FUNCTION_n
+        call_function ::= expr {expr}^n CALL_FUNCTION_VAR_n POP_TOP
+        call_function ::= expr {expr}^n CALL_FUNCTION_VAR_KW_n POP_TOP
+        call_function ::= expr {expr}^n CALL_FUNCTION_KW_n POP_TOP
+        """
+        # Low byte indicates number of positional paramters,
+        # high byte number of positional parameters
+        args_pos = token.attr & 0xff
+        args_kw = (token.attr >> 8) & 0xff
+        nak = ( len(opname)-len('CALL_FUNCTION') ) // 3
+        token.type = self.call_fn_name(token)
+        rule = ('call_function ::= expr '
+                + ('expr ' * args_pos)
+                + ('kwarg ' * args_kw)
+                + 'expr ' * nak + token.type)
+        self.add_unique_rule(rule, token.type, args_pos, customize)
+
     def add_custom_rules(self, tokens, customize):
         """
         Special handling for opcodes that take a variable number
@@ -755,7 +619,7 @@ class Python3Parser(PythonParser):
             listcomp ::= LOAD_LISTCOMP MAKE_FUNCTION_0 expr
                          GET_ITER CALL_FUNCTION_1
 
-            buildclass (see load_build_class)
+            build_class (see load_build_class)
 
             build_list  ::= {expr}^n BUILD_LIST_n
             build_list  ::= {expr}^n BUILD_TUPLE_n
@@ -766,10 +630,8 @@ class Python3Parser(PythonParser):
             mkfunc      ::= {expr}^n LOAD_CONST MAKE_FUNCTION_n
             mklambda    ::= {expr}^n LOAD_LAMBDA MAKE_FUNCTION_n
             mkfunc      ::= {expr}^n load_closure LOAD_CONST MAKE_FUNCTION_n
-            expr ::= expr {expr}^n CALL_FUNCTION_n
-            expr ::= expr {expr}^n CALL_FUNCTION_VAR_n POP_TOP
-            expr ::= expr {expr}^n CALL_FUNCTION_VAR_KW_n POP_TOP
-            expr ::= expr {expr}^n CALL_FUNCTION_KW_n POP_TOP
+
+            call_function (see custom_classfunc_rule)
         """
         # from trepan.api import debug
         # debug(start_opts={'startup-profile': True})
@@ -779,17 +641,7 @@ class Python3Parser(PythonParser):
 
             if opname in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
                           'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_KW'):
-                # Low byte indicates number of positional paramters,
-                # high byte number of positional parameters
-                args_pos = token.attr & 0xff
-                args_kw = (token.attr >> 8) & 0xff
-                nak = ( len(opname)-len('CALL_FUNCTION') ) // 3
-                token.type = 'CALL_FUNCTION_%i' % token.attr
-                rule = ('call_function ::= expr '
-                        + ('expr ' * args_pos)
-                        + ('kwarg ' * args_kw)
-                        + 'expr ' * nak + token.type)
-                self.add_unique_rule(rule, token.type, args_pos, customize)
+                self.custom_classfunc_rule(opname, token, customize)
             elif opname == 'LOAD_LISTCOMP':
                 if self.version >= 3.4:
                     rule = ("listcomp ::= LOAD_LISTCOMP LOAD_CONST MAKE_FUNCTION_0 expr "
@@ -799,7 +651,7 @@ class Python3Parser(PythonParser):
                             "GET_ITER CALL_FUNCTION_1")
                 self.add_unique_rule(rule, opname, token.attr, customize)
             elif opname == 'LOAD_BUILD_CLASS':
-                self.custom_buildclass_rule(opname, i, token, tokens, customize)
+                self.custom_build_class_rule(opname, i, token, tokens, customize)
             elif opname_base in ('BUILD_LIST', 'BUILD_TUPLE', 'BUILD_SET'):
                 rule = 'build_list ::= ' + 'expr ' * token.attr + opname
                 self.add_unique_rule(rule, opname, token.attr, customize)
@@ -817,4 +669,36 @@ class Python3Parser(PythonParser):
                     rule = 'mkfunc ::= %s LOAD_CONST %s' % ('expr ' * token.attr, opname)
                 self.add_unique_rule(rule, opname, token.attr, customize)
                 pass
+            elif opname.startswith('MAKE_CLOSURE'):
+                self.add_unique_rule('mklambda ::= %s load_closure LOAD_LAMBDA %s' %
+                                     ('expr ' * token.attr, opname), opname, token.attr,
+                                     customize)
+                self.add_unique_rule('genexpr ::= %s load_closure LOAD_GENEXPR %s '
+                                     'expr GET_ITER CALL_FUNCTION_1' %
+                                     ('expr ' * token.attr, opname),
+                                     opname, token.attr, customize)
+                self.add_unique_rule('setcomp ::= %s load_closure LOAD_SETCOMP %s expr '
+                                     'GET_ITER CALL_FUNCTION_1' %
+                                     ('expr ' * token.attr, opname),
+                                     opname, token.attr, customize)
+                self.add_unique_rule('dictcomp ::= %s load_closure LOAD_DICTCOMP %s '
+                                     'expr GET_ITER CALL_FUNCTION_1' %
+                                     ('expr '* token.attr, opname),
+                                     opname, token.attr, customize)
+                rule = ('mkfunc ::= %s load_closure BUILD_TUPLE_1 LOAD_CONST LOAD_CONST %s'
+                        % ('expr ' * token.attr, opname))
+                self.add_unique_rule(rule, opname, token.attr, customize)
+                rule = ('mkfunc ::= %s load_closure BUILD_TUPLE_1 LOAD_GENXPR LOAD_CONST %s'
+                        % ('expr ' * token.attr, opname))
+                self.add_unique_rule(rule, opname, token.attr, customize)
         return
+
+class Python3ParserSingle(Python3Parser):
+    def p_call_stmt(self, args):
+        '''
+        # single-mode compilation. Eval-mode interactive compilation
+        # drops the last rule.
+
+        call_stmt ::= expr POP_TOP
+        call_stmt ::= expr PRINT_EXPR
+        '''

uncompyle6/parsers/spark.py

@@ -1,737 +0,0 @@
-"""
-Copyright (c) 1998-2002 John Aycock
-Copyright (c) 2015 Rocky Bernstein
-
-  Permission is hereby granted, free of charge, to any person obtaining
-  a copy of this software and associated documentation files (the
-  "Software"), to deal in the Software without restriction, including
-  without limitation the rights to use, copy, modify, merge, publish,
-  distribute, sublicense, and/or sell copies of the Software, and to
-  permit persons to whom the Software is furnished to do so, subject to
-  the following conditions:
-
-  The above copyright notice and this permission notice shall be
-  included in all copies or substantial portions of the Software.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-  IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-  CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-  TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-  SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-"""
-from __future__ import print_function
-
-import os, re
-
-__version__ = 'SPARK-1.0 Python3 compatible'
-
-def _namelist(instance):
-    namelist, namedict, classlist = [], {}, [instance.__class__]
-    for c in classlist:
-        for b in c.__bases__:
-            classlist.append(b)
-        for name in list(c.__dict__.keys()):
-            if name not in namedict:
-                namelist.append(name)
-                namedict[name] = 1
-    return namelist
-
-class _State:
-    '''
-    Extracted from GenericParser and made global so that [un]picking works.
-    '''
-    def __init__(self, stateno, items):
-        self.T, self.complete, self.items = [], [], items
-        self.stateno = stateno
-
-# DEFAULT_DEBUG = {'rules': True, 'transition': True, 'reduce' : True}
-# DEFAULT_DEBUG = {'rules': False, 'transition': False, 'reduce' : True}
-DEFAULT_DEBUG = {'rules': False, 'transition': False, 'reduce': False}
-
-class GenericParser:
-    '''
-    An Earley parser, as per J. Earley, "An Efficient Context-Free
-    Parsing Algorithm", CACM 13(2), pp. 94-102.  Also J. C. Earley,
-    "An Efficient Context-Free Parsing Algorithm", Ph.D. thesis,
-    Carnegie-Mellon University, August 1968.  New formulation of
-    the parser according to J. Aycock, "Practical Earley Parsing
-    and the SPARK Toolkit", Ph.D. thesis, University of Victoria,
-    2001, and J. Aycock and R. N. Horspool, "Practical Earley
-    Parsing", unpublished paper, 2001.
-    '''
-
-    def __init__(self, start, debug=DEFAULT_DEBUG):
-        self.rules = {}
-        self.rule2func = {}
-        self.rule2name = {}
-        self.collectRules()
-        self.augment(start)
-        self.ruleschanged = True
-        self.debug = debug
-
-    _NULLABLE = '\e_'
-    _START = 'START'
-    _BOF = '|-'
-
-    #
-    #  When pickling, take the time to generate the full state machine;
-    #  some information is then extraneous, too.  Unfortunately we
-    #  can't save the rule2func map.
-    #
-    def __getstate__(self):
-        if self.ruleschanged:
-            #
-            #  XXX - duplicated from parse()
-            #
-            self.computeNull()
-            self.newrules = {}
-            self.new2old = {}
-            self.makeNewRules()
-            self.ruleschanged = False
-            self.edges, self.cores = {}, {}
-            self.states = { 0: self.makeState0() }
-            self.makeState(0, self._BOF)
-        #
-        #  XXX - should find a better way to do this..
-        #
-        changes = 1
-        while changes:
-            changes = 0
-            for k, v in list(self.edges.items()):
-                if v is None:
-                    state, sym = k
-                    if state in self.states:
-                        self.goto(state, sym)
-                        changes = 1
-        rv = self.__dict__.copy()
-        for s in list(self.states.values()):
-            del s.items
-        del rv['rule2func']
-        del rv['nullable']
-        del rv['cores']
-        return rv
-
-    def __setstate__(self, D):
-        self.rules = {}
-        self.rule2func = {}
-        self.rule2name = {}
-        self.collectRules()
-        start = D['rules'][self._START][0][1][1] # Blech.
-        self.augment(start)
-        D['rule2func'] = self.rule2func
-        D['makeSet'] = self.makeSet_fast
-        self.__dict__ = D
-
-    #
-    #  A hook for GenericASTBuilder and GenericASTMatcher.  Mess
-    #  thee not with this; nor shall thee toucheth the _preprocess
-    #  argument to addRule.
-    #
-    def preprocess(self, rule, func):
-        return rule, func
-
-    def addRule(self, doc, func, _preprocess=True):
-        """Add a grammar rules to self.rules, self.rule2func and self.rule2name"""
-        fn = func
-
-        # remove blanks lines and comment lines, e.g. lines starting with "#"
-        doc = os.linesep.join([s for s in doc.splitlines() if s and not re.match("^\s*#", s)])
-
-        rules = doc.split()
-
-        index = []
-        for i in range(len(rules)):
-            if rules[i] == '::=':
-                index.append(i-1)
-        index.append(len(rules))
-
-        for i in range(len(index)-1):
-            lhs = rules[index[i]]
-            rhs = rules[index[i]+2:index[i+1]]
-            rule = (lhs, tuple(rhs))
-
-            if _preprocess:
-                rule, fn = self.preprocess(rule, func)
-
-            if lhs in self.rules:
-                self.rules[lhs].append(rule)
-            else:
-                self.rules[lhs] = [ rule ]
-            self.rule2func[rule] = fn
-            self.rule2name[rule] = func.__name__[2:]
-        self.ruleschanged = True
-
-    def collectRules(self):
-        for name in _namelist(self):
-            if name[:2] == 'p_':
-                func = getattr(self, name)
-                doc = func.__doc__
-                self.addRule(doc, func)
-
-    def augment(self, start):
-        rule = '%s ::= %s %s' % (self._START, self._BOF, start)
-        self.addRule(rule, lambda args: args[1], 0)
-
-    def computeNull(self):
-        self.nullable = {}
-        tbd = []
-
-        for rulelist in list(self.rules.values()):
-            lhs = rulelist[0][0]
-            self.nullable[lhs] = 0
-            for rule in rulelist:
-                rhs = rule[1]
-                if len(rhs) == 0:
-                    self.nullable[lhs] = 1
-                    continue
-                #
-                #  We only need to consider rules which
-                #  consist entirely of nonterminal symbols.
-                #  This should be a savings on typical
-                #  grammars.
-                #
-                for sym in rhs:
-                    if sym not in self.rules:
-                        break
-                else:
-                    tbd.append(rule)
-        changes = 1
-        while changes:
-            changes = 0
-            for lhs, rhs in tbd:
-                if self.nullable[lhs]:
-                    continue
-                for sym in rhs:
-                    if not self.nullable[sym]:
-                        break
-                else:
-                    self.nullable[lhs] = 1
-                    changes = 1
-
-    def makeState0(self):
-        s0 = _State(0, [])
-        for rule in self.newrules[self._START]:
-            s0.items.append((rule, 0))
-        return s0
-
-    def finalState(self, tokens):
-        #
-        #  Yuck.
-        #
-        if len(self.newrules[self._START]) == 2 and len(tokens) == 0:
-            return 1
-        start = self.rules[self._START][0][1][1]
-        return self.goto(1, start)
-
-    def makeNewRules(self):
-        worklist = []
-        for rulelist in list(self.rules.values()):
-            for rule in rulelist:
-                worklist.append((rule, 0, 1, rule))
-
-        for rule, i, candidate, oldrule in worklist:
-            lhs, rhs = rule
-            n = len(rhs)
-            while i < n:
-                sym = rhs[i]
-                if sym not in self.rules or \
-                    not self.nullable[sym]:
-                        candidate = 0
-                        i = i + 1
-                        continue
-
-                newrhs = list(rhs)
-                newrhs[i] = self._NULLABLE+sym
-                newrule = (lhs, tuple(newrhs))
-                worklist.append((newrule, i+1,
-                                 candidate, oldrule))
-                candidate = 0
-                i = i + 1
-            else:
-                if candidate:
-                    lhs = self._NULLABLE+lhs
-                    rule = (lhs, rhs)
-                if lhs in self.newrules:
-                    self.newrules[lhs].append(rule)
-                else:
-                    self.newrules[lhs] = [ rule ]
-                self.new2old[rule] = oldrule
-
-    def typestring(self, token):
-        return None
-
-    def error(self, token):
-        print("Syntax error at or near `%s' token" % token)
-        raise SystemExit
-
-    def parse(self, tokens):
-        sets = [ [(1, 0), (2, 0)] ]
-        self.links = {}
-
-        if self.ruleschanged:
-            self.computeNull()
-            self.newrules = {}
-            self.new2old = {}
-            self.makeNewRules()
-            self.ruleschanged = False
-            self.edges, self.cores = {}, {}
-            self.states = { 0: self.makeState0() }
-            self.makeState(0, self._BOF)
-
-        for i in range(len(tokens)):
-            sets.append([])
-
-            if sets[i] == []:
-                break
-            self.makeSet(tokens[i], sets, i)
-        else:
-            sets.append([])
-            self.makeSet(None, sets, len(tokens))
-
-        finalitem = (self.finalState(tokens), 0)
-        if finalitem not in sets[-2]:
-            if len(tokens) > 0:
-                self.error(tokens[i-1])
-            else:
-                self.error(None)
-
-        return self.buildTree(self._START, finalitem,
-                    tokens, len(sets)-2)
-
-    def isnullable(self, sym):
-        #
-        #  For symbols in G_e only.  If we weren't supporting 1.5,
-        #  could just use sym.startswith().
-        #
-        return self._NULLABLE == sym[0:len(self._NULLABLE)]
-
-    def skip(self, xxx_todo_changeme, pos=0):
-        (lhs, rhs) = xxx_todo_changeme
-        n = len(rhs)
-        while pos < n:
-            if not self.isnullable(rhs[pos]):
-                break
-            pos = pos + 1
-        return pos
-
-    def makeState(self, state, sym):
-        assert sym is not None
-        # print(sym) # debug
-        #
-        #  Compute \epsilon-kernel state's core and see if
-        #  it exists already.
-        #
-        kitems = []
-        for rule, pos in self.states[state].items:
-            lhs, rhs = rule
-            if rhs[pos:pos+1] == (sym,):
-                kitems.append((rule, self.skip(rule, pos+1)))
-
-        tcore = tuple(sorted(kitems))
-        if tcore in self.cores:
-            return self.cores[tcore]
-        #
-        #  Nope, doesn't exist.  Compute it and the associated
-        #  \epsilon-nonkernel state together; we'll need it right away.
-        #
-        k = self.cores[tcore] = len(self.states)
-        K, NK = _State(k, kitems), _State(k+1, [])
-        self.states[k] = K
-        predicted = {}
-
-        edges = self.edges
-        rules = self.newrules
-        for X in K, NK:
-            worklist = X.items
-            for item in worklist:
-                rule, pos = item
-                lhs, rhs = rule
-                if pos == len(rhs):
-                    X.complete.append(rule)
-                    continue
-
-                nextSym = rhs[pos]
-                key = (X.stateno, nextSym)
-                if nextSym not in rules:
-                    if key not in edges:
-                        edges[key] = None
-                        X.T.append(nextSym)
-                else:
-                    edges[key] = None
-                    if nextSym not in predicted:
-                        predicted[nextSym] = 1
-                        for prule in rules[nextSym]:
-                            ppos = self.skip(prule)
-                            new = (prule, ppos)
-                            NK.items.append(new)
-            #
-            #  Problem: we know K needs generating, but we
-            #  don't yet know about NK.  Can't commit anything
-            #  regarding NK to self.edges until we're sure.  Should
-            #  we delay committing on both K and NK to avoid this
-            #  hacky code?  This creates other problems..
-            #
-            if X is K:
-                edges = {}
-
-        if NK.items == []:
-            return k
-
-        #
-        #  Check for \epsilon-nonkernel's core.  Unfortunately we
-        #  need to know the entire set of predicted nonterminals
-        #  to do this without accidentally duplicating states.
-        #
-        tcore = tuple(sorted(predicted.keys()))
-        if tcore in self.cores:
-            self.edges[(k, None)] = self.cores[tcore]
-            return k
-
-        nk = self.cores[tcore] = self.edges[(k, None)] = NK.stateno
-        self.edges.update(edges)
-        self.states[nk] = NK
-        return k
-
-    def goto(self, state, sym):
-        key = (state, sym)
-        if key not in self.edges:
-            #
-            #  No transitions from state on sym.
-            #
-            return None
-
-        rv = self.edges[key]
-        if rv is None:
-            #
-            #  Target state isn't generated yet.  Remedy this.
-            #
-            rv = self.makeState(state, sym)
-            self.edges[key] = rv
-        return rv
-
-    def gotoT(self, state, t):
-        if self.debug['rules']: print("Terminal", t, state)
-        return [self.goto(state, t)]
-
-    def gotoST(self, state, st):
-        if self.debug['transition']: print("GotoST", st, state)
-        rv = []
-        for t in self.states[state].T:
-            if st == t:
-                rv.append(self.goto(state, t))
-        return rv
-
-    def add(self, set, item, i=None, predecessor=None, causal=None):
-        if predecessor is None:
-            if item not in set:
-                set.append(item)
-        else:
-            key = (item, i)
-            if item not in set:
-                self.links[key] = []
-                set.append(item)
-            self.links[key].append((predecessor, causal))
-
-    def makeSet(self, token, sets, i):
-        cur, next = sets[i], sets[i+1]
-
-        ttype = token is not None and self.typestring(token) or None
-        if ttype is not None:
-            fn, arg = self.gotoT, ttype
-        else:
-            fn, arg = self.gotoST, token
-
-        for item in cur:
-            ptr = (item, i)
-            state, parent = item
-            add = fn(state, arg)
-            for k in add:
-                if k is not None:
-                    self.add(next, (k, parent), i+1, ptr)
-                    nk = self.goto(k, None)
-                    if nk is not None:
-                        self.add(next, (nk, i+1))
-
-            if parent == i:
-                continue
-
-            for rule in self.states[state].complete:
-                lhs, rhs = rule
-                if self.debug['reduce']:
-                    print("%s ::= %s" % (lhs, ' '.join(rhs)))
-                for pitem in sets[parent]:
-                    pstate, pparent = pitem
-                    k = self.goto(pstate, lhs)
-                    if k is not None:
-                        why = (item, i, rule)
-                        pptr = (pitem, parent)
-                        self.add(cur, (k, pparent),
-                                i, pptr, why)
-                        nk = self.goto(k, None)
-                        if nk is not None:
-                            self.add(cur, (nk, i))
-
-    def makeSet_fast(self, token, sets, i):
-        #
-        #  Call *only* when the entire state machine has been built!
-        #  It relies on self.edges being filled in completely, and
-        #  then duplicates and inlines code to boost speed at the
-        #  cost of extreme ugliness.
-        #
-        cur, next = sets[i], sets[i+1]
-        ttype = token is not None and self.typestring(token) or None
-
-        for item in cur:
-            ptr = (item, i)
-            state, parent = item
-            if ttype is not None:
-                k = self.edges.get((state, ttype), None)
-                if k is not None:
-                    # self.add(next, (k, parent), i+1, ptr)
-                    # INLINED --------v
-                    new = (k, parent)
-                    key = (new, i+1)
-                    if new not in next:
-                        self.links[key] = []
-                        next.append(new)
-                    self.links[key].append((ptr, None))
-                    # INLINED --------^
-                    # nk = self.goto(k, None)
-                    nk = self.edges.get((k, None), None)
-                    if nk is not None:
-                        # self.add(next, (nk, i+1))
-                        # INLINED -------------v
-                        new = (nk, i+1)
-                        if new not in next:
-                            next.append(new)
-                        # INLINED ---------------^
-            else:
-                add = self.gotoST(state, token)
-                for k in add:
-                    if k is not None:
-                        self.add(next, (k, parent), i+1, ptr)
-                        # nk = self.goto(k, None)
-                        nk = self.edges.get((k, None), None)
-                        if nk is not None:
-                            self.add(next, (nk, i+1))
-
-            if parent == i:
-                continue
-
-            for rule in self.states[state].complete:
-                lhs, rhs = rule
-                for pitem in sets[parent]:
-                    pstate, pparent = pitem
-                    # k = self.goto(pstate, lhs)
-                    k = self.edges.get((pstate, lhs), None)
-                    if k is not None:
-                        why = (item, i, rule)
-                        pptr = (pitem, parent)
-                        # self.add(cur, (k, pparent), i, pptr, why)
-                        # INLINED ---------v
-                        new = (k, pparent)
-                        key = (new, i)
-                        if new not in cur:
-                            self.links[key] = []
-                            cur.append(new)
-                        self.links[key].append((pptr, why))
-                        # INLINED ----------^
-                        # nk = self.goto(k, None)
-                        nk = self.edges.get((k, None), None)
-                        if nk is not None:
-                            # self.add(cur, (nk, i))
-                            # INLINED ---------v
-                            new = (nk, i)
-                            if new not in cur:
-                                cur.append(new)
-                            # INLINED ----------^
-
-    def predecessor(self, key, causal):
-        for p, c in self.links[key]:
-            if c == causal:
-                return p
-        assert 0
-
-    def causal(self, key):
-        links = self.links[key]
-        if len(links) == 1:
-            return links[0][1]
-        choices = []
-        rule2cause = {}
-        for p, c in links:
-            rule = c[2]
-            choices.append(rule)
-            rule2cause[rule] = c
-        return rule2cause[self.ambiguity(choices)]
-
-    def deriveEpsilon(self, nt):
-        if len(self.newrules[nt]) > 1:
-            rule = self.ambiguity(self.newrules[nt])
-        else:
-            rule = self.newrules[nt][0]
-        # print(rule) # debug
-
-        rhs = rule[1]
-        attr = [None] * len(rhs)
-
-        for i in range(len(rhs)-1, -1, -1):
-            attr[i] = self.deriveEpsilon(rhs[i])
-        return self.rule2func[self.new2old[rule]](attr)
-
-    def buildTree(self, nt, item, tokens, k):
-        if self.debug['rules']:
-            print("NT", nt)
-        state, parent = item
-
-        choices = []
-        for rule in self.states[state].complete:
-            if rule[0] == nt:
-                choices.append(rule)
-        rule = choices[0]
-        if len(choices) > 1:
-            rule = self.ambiguity(choices)
-        # print(rule) # debug
-
-        rhs = rule[1]
-        attr = [None] * len(rhs)
-
-        for i in range(len(rhs)-1, -1, -1):
-            sym = rhs[i]
-            if sym not in self.newrules:
-                if sym != self._BOF:
-                    attr[i] = tokens[k-1]
-                    key = (item, k)
-                    item, k = self.predecessor(key, None)
-            # elif self.isnullable(sym):
-            elif self._NULLABLE == sym[0:len(self._NULLABLE)]:
-                attr[i] = self.deriveEpsilon(sym)
-            else:
-                key = (item, k)
-                why = self.causal(key)
-                attr[i] = self.buildTree(sym, why[0],
-                            tokens, why[1])
-                item, k = self.predecessor(key, why)
-        return self.rule2func[self.new2old[rule]](attr)
-
-    def ambiguity(self, rules):
-        #
-        #  XXX - problem here and in collectRules() if the same rule
-        #  appears in >1 method.  Also undefined results if rules
-        #  causing the ambiguity appear in the same method.
-        #
-        sortlist = []
-        name2index = {}
-        for i in range(len(rules)):
-            lhs, rhs = rule = rules[i]
-            name = self.rule2name[self.new2old[rule]]
-            sortlist.append((len(rhs), name))
-            name2index[name] = i
-        sortlist.sort()
-        list = [a_b[1] for a_b in sortlist]
-        return rules[name2index[self.resolve(list)]]
-
-    def resolve(self, list):
-        '''
-        Resolve ambiguity in favor of the shortest RHS.
-        Since we walk the tree from the top down, this
-        should effectively resolve in favor of a "shift".
-        '''
-        return list[0]
-
-#
-#  GenericASTBuilder automagically constructs a concrete/abstract syntax tree
-#  for a given input.  The extra argument is a class (not an instance!)
-#  which supports the "__setslice__" and "__len__" methods.
-#
-#  XXX - silently overrides any user code in methods.
-#
-
-class GenericASTBuilder(GenericParser):
-    def __init__(self, AST, start, debug=DEFAULT_DEBUG):
-        GenericParser.__init__(self, start, debug=debug)
-        self.AST = AST
-
-    def preprocess(self, rule, func):
-        rebind = lambda lhs, self=self: \
-                lambda args, lhs=lhs, self=self: \
-                self.buildASTNode(args, lhs)
-        lhs, rhs = rule
-        return rule, rebind(lhs)
-
-    def buildASTNode(self, args, lhs):
-        children = []
-        for arg in args:
-            if isinstance(arg, self.AST):
-                children.append(arg)
-            else:
-                children.append(self.terminal(arg))
-        return self.nonterminal(lhs, children)
-
-    def terminal(self, token):
-        return token
-
-    def nonterminal(self, type, args):
-        rv = self.AST(type)
-        rv[:len(args)] = args
-        return rv
-
-class GenericASTTraversalPruningException(BaseException):
-    pass
-
-class GenericASTTraversal:
-    '''
-    GenericASTTraversal is a Visitor pattern according to Design Patterns.  For
-    each node it attempts to invoke the method n_<node type>, falling
-    back onto the default() method if the n_* can't be found.  The preorder
-    traversal also looks for an exit hook named n_<node type>_exit (no default
-    routine is called if it's not found).  To prematurely halt traversal
-    of a subtree, call the prune() method -- this only makes sense for a
-    preorder traversal.  Node type is determined via the typestring() method.
-    '''
-    def __init__(self, ast):
-        self.ast = ast
-
-    def typestring(self, node):
-        return node.type
-
-    def prune(self):
-        raise GenericASTTraversalPruningException
-
-    def preorder(self, node=None):
-        """Walk the tree. For each node with typestring name *name* if the
-        node has a method called n_*name*, call that before walking
-        children. If there is no method define, call a
-        self.default(node) instead. Subclasses of GenericASTTtraversal
-        ill probably want to override this method.
-
-        If the node has a method called *name*_exit, that is called
-        after all children have been called.  So in this sense this
-        function is both preorder and postorder combined.
-        """
-        if node is None:
-            node = self.ast
-
-        try:
-            name = 'n_' + self.typestring(node)
-            if hasattr(self, name):
-                func = getattr(self, name)
-                func(node)
-            else:
-                self.default(node)
-        except GenericASTTraversalPruningException:
-            return
-
-        for kid in node:
-            self.preorder(kid)
-
-        name = name + '_exit'
-        if hasattr(self, name):
-            func = getattr(self, name)
-            func(node)
-
-    def default(self, node):
-        """Default acttion to take on an ASTNode. Our defualt is to do nothing.
-        Subclasses will probably want to define this for other behavior."""
-        pass

uncompyle6/scanner.py

@@ -29,7 +29,8 @@ if PYTHON3:
 else:
     L65536 = long(65536) # NOQA
 
-from uncompyle6.opcodes import opcode_25, opcode_26, opcode_27, opcode_32, opcode_33, opcode_34
+from uncompyle6.opcodes import (opcode_25, opcode_26, opcode_27,
+                                opcode_32, opcode_33, opcode_34, opcode_35)
 
 
 class Code:
@@ -61,6 +62,8 @@ class Scanner(object):
             self.opc = opcode_33
         elif version == 3.4:
             self.opc = opcode_34
+        elif version == 3.5:
+            self.opc = opcode_35
         else:
             raise TypeError("%s is not a Python version I know about" % version)
 
@@ -289,13 +292,13 @@ def get_scanner(version):
 
     if version == 2.7:
         import uncompyle6.scanners.scanner27 as scan
-        scanner = scan.Scanner27()
+        scanner = scan.Scanner27(version)
     elif version == 2.6:
         import uncompyle6.scanners.scanner26 as scan
-        scanner = scan.Scanner26()
+        scanner = scan.Scanner26(version)
     elif version == 2.5:
         import uncompyle6.scanners.scanner25 as scan
-        scanner = scan.Scanner25()
+        scanner = scan.Scanner25(version)
     elif version == 3.2:
         import uncompyle6.scanners.scanner32 as scan
         scanner = scan.Scanner32(version)
@@ -305,6 +308,9 @@ def get_scanner(version):
     elif version == 3.4:
         import uncompyle6.scanners.scanner34 as scan
         scanner = scan.Scanner34(version)
+    elif version == 3.5:
+        import uncompyle6.scanners.scanner35 as scan
+        scanner = scan.Scanner35(version)
     else:
         raise RuntimeError("Unsupported Python version %d" % version)
     return scanner

uncompyle6/scanners/scanner25.py

@@ -20,7 +20,7 @@ from uncompyle6.opcodes.opcode_25 import *
 import uncompyle6.scanner as scan
 
 class Scanner25(scan.Scanner):
-    def __init__(self):
+    def __init__(self, version):
         scan.Scanner.__init__(self, 2.5) # check
 
     def disassemble(self, co, classname=None, code_objects={}):

uncompyle6/scanners/scanner26.py

@@ -19,7 +19,7 @@ import dis
 import uncompyle6.scanner as scan
 
 class Scanner26(scan.Scanner):
-    def __init__(self):
+    def __init__(self, version):
         scan.Scanner.__init__(self, 2.5) # check
 
     def disassemble(self, co, classname=None, code_objects={}):

uncompyle6/scanners/scanner27.py

@@ -22,7 +22,7 @@ from uncompyle6.opcodes.opcode_27 import * # NOQA
 import uncompyle6.scanner as scan
 
 class Scanner27(scan.Scanner):
-    def __init__(self):
+    def __init__(self, version):
         scan.Scanner.__init__(self, 2.7) # check
 
     def disassemble(self, co, classname=None, code_objects={}):

uncompyle6/scanners/scanner3.py

@@ -156,25 +156,20 @@ class Scanner3(scan.Scanner):
                 elif op in op3.hasfree:
                     pattr = free[oparg]
 
-            if op in (BUILD_LIST, BUILD_TUPLE, BUILD_SET, BUILD_SLICE,
-                            UNPACK_SEQUENCE,
-                            MAKE_FUNCTION, CALL_FUNCTION, MAKE_CLOSURE,
-                            CALL_FUNCTION_VAR, CALL_FUNCTION_KW,
-                            CALL_FUNCTION_VAR_KW, RAISE_VARARGS
+            if op_name in ('BUILD_LIST', 'BUILD_TUPLE', 'BUILD_SET', 'BUILD_SLICE',
+                            'UNPACK_SEQUENCE',
+                            'MAKE_FUNCTION', 'CALL_FUNCTION', 'MAKE_CLOSURE',
+                            'CALL_FUNCTION_VAR', 'CALL_FUNCTION_KW',
+                            'CALL_FUNCTION_VAR_KW', 'RAISE_VARARGS'
                             ):
-                # As of Python 2.5, values loaded via LOAD_CLOSURE are packed into
-                #   a tuple before calling MAKE_CLOSURE.
-                if (op == BUILD_TUPLE and
-                    self.code[self.prev_op[offset]] == LOAD_CLOSURE):
-                    continue
-                else:
-                    # CALL_FUNCTION OP renaming is done as a custom rule in parse3
-                    if op_name not in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
-                                       'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_KW'):
-                        op_name = '%s_%d' % (op_name, oparg)
-                    if op != BUILD_SLICE:
+                # CALL_FUNCTION OP renaming is done as a custom rule in parse3
+                if op_name not in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
+                                   'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_KW',
+                                   ):
+                    op_name = '%s_%d' % (op_name, oparg)
+                    if op_name != 'BUILD_SLICE':
                         customize[op_name] = oparg
-            elif op == JUMP_ABSOLUTE:
+            elif op_name == 'JUMP_ABSOLUTE':
                 target = self.get_target(offset)
                 if target < offset:
                     if (offset in self.stmts
@@ -184,10 +179,10 @@ class Scanner3(scan.Scanner):
                     else:
                         op_name = 'JUMP_BACK'
 
-            elif op == LOAD_GLOBAL:
+            elif op_name == 'LOAD_GLOBAL':
                 if offset in self.load_asserts:
                     op_name = 'LOAD_ASSERT'
-            elif op == RETURN_VALUE:
+            elif op_name == 'RETURN_VALUE':
                 if offset in self.return_end_ifs:
                     op_name = 'RETURN_END_IF'
 

uncompyle6/scanners/scanner32.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015 by Rocky Bernstein
+#  Copyright (c) 2015-2016 by Rocky Bernstein
 """
 Python 3.2 bytecode scanner/deparser
 
@@ -23,7 +23,7 @@ class Scanner32(scan3.Scanner3):
 if __name__ == "__main__":
     import inspect
     co = inspect.currentframe().f_code
-    tokens, customize = Scanner32().disassemble(co)
+    tokens, customize = Scanner32(3.2).disassemble(co)
     for t in tokens:
         print(t)
     pass

uncompyle6/scanners/scanner33.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015 by Rocky Bernstein
+#  Copyright (c) 2015-2016 by Rocky Bernstein
 """
 Python 3 bytecode scanner/deparser
 
@@ -23,7 +23,7 @@ class Scanner33(scan3.Scanner3):
 if __name__ == "__main__":
     import inspect
     co = inspect.currentframe().f_code
-    tokens, customize = Scanner33().disassemble(co)
+    tokens, customize = Scanner33(3.3).disassemble(co)
     for t in tokens:
         print(t)
     pass

uncompyle6/scanners/scanner34.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015 by Rocky Bernstein
+#  Copyright (c) 2015-2016 by Rocky Bernstein
 """
 Python 3.4 bytecode scanner/deparser
 
@@ -327,7 +327,7 @@ class Scanner34(scan3.Scanner3):
 
 if __name__ == "__main__":
     co = inspect.currentframe().f_code
-    tokens, customize = Scanner34().disassemble(co)
+    tokens, customize = Scanner34(3.4).disassemble(co)
     for t in tokens:
         print(t)
     pass

uncompyle6/scanners/scanner35.py

@@ -0,0 +1,333 @@
+#  Copyright (c) 2016 by Rocky Bernstein
+"""
+Python 3.5 bytecode scanner/deparser
+
+This overlaps Python's 3.5's dis module, and in fact in some cases
+we just fall back to that. But the intent is that it can be run from
+Python 2 and other versions of Python. Also, we save token information
+for later use in deparsing.
+"""
+
+from __future__ import print_function
+
+import dis, inspect
+from array import array
+import uncompyle6.scanners.scanner3 as scan3
+
+from uncompyle6 import PYTHON_VERSION
+from uncompyle6.code import iscode
+from uncompyle6.scanner import Token
+
+# Get all the opcodes into globals
+globals().update(dis.opmap)
+
+import uncompyle6.opcodes.opcode_35
+# verify uses JUMP_OPs from here
+JUMP_OPs = uncompyle6.opcodes.opcode_35.JUMP_OPs
+
+from uncompyle6.opcodes.opcode_35 import *
+
+class Scanner35(scan3.Scanner3):
+
+    def disassemble(self, co, classname=None, code_objects={}):
+        fn = self.disassemble_built_in if PYTHON_VERSION == 3.4 \
+            else self.disassemble_generic
+        return fn(co, classname, code_objects=code_objects)
+
+    def disassemble_built_in(self, co, classname=None,
+                             code_objects={}):
+        # Container for tokens
+        tokens = []
+        customize = {}
+        self.code = array('B', co.co_code)
+        self.build_lines_data(co)
+        self.build_prev_op()
+
+        # Get jump targets
+        # Format: {target offset: [jump offsets]}
+        jump_targets = self.find_jump_targets()
+        bytecode = dis.Bytecode(co)
+
+        # self.lines contains (block,addrLastInstr)
+        if classname:
+            classname = '_' + classname.lstrip('_') + '__'
+
+            def unmangle(name):
+                if name.startswith(classname) and name[-2:] != '__':
+                    return name[len(classname) - 2:]
+                return name
+
+            # free = [ unmangle(name) for name in (co.co_cellvars + co.co_freevars) ]
+            # names = [ unmangle(name) for name in co.co_names ]
+            # varnames = [ unmangle(name) for name in co.co_varnames ]
+        else:
+            # free = co.co_cellvars + co.co_freevars
+            # names = co.co_names
+            # varnames = co.co_varnames
+            pass
+
+        # Scan for assertions. Later we will
+        # turn 'LOAD_GLOBAL' to 'LOAD_ASSERT' for those
+        # assertions
+        self.load_asserts = set()
+        bs = list(bytecode)
+        n = len(bs)
+        for i in range(n):
+            inst = bs[i]
+            if inst.opname == 'POP_JUMP_IF_TRUE' and  i+1 < n:
+                next_inst = bs[i+1]
+                if (next_inst.opname == 'LOAD_GLOBAL' and
+                    next_inst.argval == 'AssertionError'):
+                    self.load_asserts.add(next_inst.offset)
+
+        for inst in bytecode:
+            if inst.offset in jump_targets:
+                jump_idx = 0
+                for jump_offset in jump_targets[inst.offset]:
+                    tokens.append(Token('COME_FROM', None, repr(jump_offset),
+                                        offset='%s_%s' % (inst.offset, jump_idx)))
+                    jump_idx += 1
+                    pass
+                pass
+
+            pattr =  inst.argrepr
+            opname = inst.opname
+
+            if opname in ['LOAD_CONST']:
+                const = inst.argval
+                if iscode(const):
+                    if const.co_name == '<lambda>':
+                        opname = 'LOAD_LAMBDA'
+                    elif const.co_name == '<genexpr>':
+                        opname = 'LOAD_GENEXPR'
+                    elif const.co_name == '<dictcomp>':
+                        opname = 'LOAD_DICTCOMP'
+                    elif const.co_name == '<setcomp>':
+                        opname = 'LOAD_SETCOMP'
+                    elif const.co_name == '<listcomp>':
+                        opname = 'LOAD_LISTCOMP'
+                    # verify() uses 'pattr' for comparison, since 'attr'
+                    # now holds Code(const) and thus can not be used
+                    # for comparison (todo: think about changing this)
+                    # pattr = 'code_object @ 0x%x %s->%s' %\
+                    # (id(const), const.co_filename, const.co_name)
+                    pattr = '<code_object ' + const.co_name + '>'
+                else:
+                    pattr = const
+                    pass
+            elif opname in ('BUILD_LIST', 'BUILD_TUPLE', 'BUILD_SET', 'BUILD_SLICE',
+                            'UNPACK_SEQUENCE',
+                            'MAKE_FUNCTION', 'MAKE_CLOSURE',
+                            'DUP_TOPX', 'RAISE_VARARGS'
+                            ):
+                # if opname == 'BUILD_TUPLE' and \
+                #     self.code[self.prev[offset]] == LOAD_CLOSURE:
+                #     continue
+                # else:
+                #     op_name = '%s_%d' % (op_name, oparg)
+                #     if opname != BUILD_SLICE:
+                #         customize[op_name] = oparg
+                opname = '%s_%d' % (opname, inst.argval)
+                if inst.opname != 'BUILD_SLICE':
+                    customize[opname] = inst.argval
+
+            elif opname == 'JUMP_ABSOLUTE':
+                pattr = inst.argval
+                target = self.get_target(inst.offset)
+                if target < inst.offset:
+                    if (inst.offset in self.stmts and
+                        self.code[inst.offset+3] not in (END_FINALLY, POP_BLOCK)
+                        and offset not in self.not_continue):
+                        opname = 'CONTINUE'
+                    else:
+                        opname = 'JUMP_BACK'
+
+            elif inst.offset in self.load_asserts:
+                opname = 'LOAD_ASSERT'
+
+            tokens.append(
+                Token(
+                    type_ = opname,
+                    attr = inst.argval,
+                    pattr = pattr,
+                    offset = inst.offset,
+                    linestart = inst.starts_line,
+                    )
+                )
+            pass
+        return tokens, {}
+
+    # FIXME: merge with scanner3 code
+    def detect_structure(self, offset):
+        """
+        Detect structures and their boundaries to fix optimizied jumps
+        in python2.3+
+        """
+        code = self.code
+        op = code[offset]
+        # Detect parent structure
+        parent = self.structs[0]
+        start = parent['start']
+        end = parent['end']
+
+        # Pick inner-most parent for our offset
+        for struct in self.structs:
+            curent_start = struct['start']
+            curent_end   = struct['end']
+            if (curent_start <= offset < curent_end) and (curent_start >= start and curent_end <= end):
+                start = curent_start
+                end = curent_end
+                parent = struct
+                pass
+
+        if op in (POP_JUMP_IF_FALSE, POP_JUMP_IF_TRUE):
+            start = offset + self.op_size(op)
+            target = self.get_target(offset)
+            rtarget = self.restrict_to_parent(target, parent)
+            prev_op = self.prev_op
+
+            # Do not let jump to go out of parent struct bounds
+            if target != rtarget and parent['type'] == 'and/or':
+                self.fixed_jumps[offset] = rtarget
+                return
+
+            # Does this jump to right after another cond jump?
+            # If so, it's part of a larger conditional
+            if (code[prev_op[target]] in (JUMP_IF_FALSE_OR_POP, JUMP_IF_TRUE_OR_POP,
+                                          POP_JUMP_IF_FALSE, POP_JUMP_IF_TRUE)) and (target > offset):
+                self.fixed_jumps[offset] = prev_op[target]
+                self.structs.append({'type': 'and/or',
+                                     'start': start,
+                                     'end': prev_op[target]})
+                return
+            # Is it an and inside if block
+            if op == POP_JUMP_IF_FALSE:
+                # Search for other POP_JUMP_IF_FALSE targetting the same op,
+                # in current statement, starting from current offset, and filter
+                # everything inside inner 'or' jumps and midline ifs
+                match = self.rem_or(start, self.next_stmt[offset], POP_JUMP_IF_FALSE, target)
+                match = self.remove_mid_line_ifs(match)
+                # If we still have any offsets in set, start working on it
+                if match:
+                    if (code[prev_op[rtarget]] in (JUMP_FORWARD, JUMP_ABSOLUTE) and prev_op[rtarget] not in self.stmts and
+                        self.restrict_to_parent(self.get_target(prev_op[rtarget]), parent) == rtarget):
+                        if (code[prev_op[prev_op[rtarget]]] == JUMP_ABSOLUTE and self.remove_mid_line_ifs([offset]) and
+                            target == self.get_target(prev_op[prev_op[rtarget]]) and
+                            (prev_op[prev_op[rtarget]] not in self.stmts or self.get_target(prev_op[prev_op[rtarget]]) > prev_op[prev_op[rtarget]]) and
+                            1 == len(self.remove_mid_line_ifs(self.rem_or(start, prev_op[prev_op[rtarget]], (POP_JUMP_IF_FALSE, POP_JUMP_IF_TRUE), target)))):
+                            pass
+                        elif (code[prev_op[prev_op[rtarget]]] == RETURN_VALUE and self.remove_mid_line_ifs([offset]) and
+                              1 == (len(set(self.remove_mid_line_ifs(self.rem_or(start, prev_op[prev_op[rtarget]],
+                                                                                 (POP_JUMP_IF_FALSE, POP_JUMP_IF_TRUE), target))) |
+                                    set(self.remove_mid_line_ifs(self.rem_or(start, prev_op[prev_op[rtarget]],
+                                                                             (POP_JUMP_IF_FALSE, POP_JUMP_IF_TRUE, JUMP_ABSOLUTE),
+                                                                             prev_op[rtarget], True)))))):
+                            pass
+                        else:
+                            fix = None
+                            jump_ifs = self.all_instr(start, self.next_stmt[offset], POP_JUMP_IF_FALSE)
+                            last_jump_good = True
+                            for j in jump_ifs:
+                                if target == self.get_target(j):
+                                    if self.lines[j].next == j + 3 and last_jump_good:
+                                        fix = j
+                                        break
+                                else:
+                                    last_jump_good = False
+                            self.fixed_jumps[offset] = fix or match[-1]
+                            return
+                    else:
+                        self.fixed_jumps[offset] = match[-1]
+                        return
+            # op == POP_JUMP_IF_TRUE
+            else:
+                next = self.next_stmt[offset]
+                if prev_op[next] == offset:
+                    pass
+                elif code[next] in (JUMP_FORWARD, JUMP_ABSOLUTE) and target == self.get_target(next):
+                    if code[prev_op[next]] == POP_JUMP_IF_FALSE:
+                        if code[next] == JUMP_FORWARD or target != rtarget or code[prev_op[prev_op[rtarget]]] not in (JUMP_ABSOLUTE, RETURN_VALUE):
+                            self.fixed_jumps[offset] = prev_op[next]
+                            return
+                elif (code[next] == JUMP_ABSOLUTE and code[target] in (JUMP_ABSOLUTE, JUMP_FORWARD) and
+                      self.get_target(target) == self.get_target(next)):
+                    self.fixed_jumps[offset] = prev_op[next]
+                    return
+
+            # Don't add a struct for a while test, it's already taken care of
+            if offset in self.ignore_if:
+                return
+
+            if (code[prev_op[rtarget]] == JUMP_ABSOLUTE and prev_op[rtarget] in self.stmts and
+                prev_op[rtarget] != offset and prev_op[prev_op[rtarget]] != offset and
+                not (code[rtarget] == JUMP_ABSOLUTE and code[rtarget+3] == POP_BLOCK and code[prev_op[prev_op[rtarget]]] != JUMP_ABSOLUTE)):
+                rtarget = prev_op[rtarget]
+
+            # Does the if jump just beyond a jump op, then this is probably an if statement
+            if code[prev_op[rtarget]] in (JUMP_ABSOLUTE, JUMP_FORWARD):
+                if_end = self.get_target(prev_op[rtarget])
+
+                # Is this a loop not an if?
+                if (if_end < prev_op[rtarget]) and (code[prev_op[if_end]] == SETUP_LOOP):
+                    if(if_end > start):
+                        return
+
+                end = self.restrict_to_parent(if_end, parent)
+
+                self.structs.append({'type': 'if-then',
+                                     'start': start,
+                                     'end': prev_op[rtarget]})
+                self.not_continue.add(prev_op[rtarget])
+
+                if rtarget < end:
+                    self.structs.append({'type': 'if-else',
+                                         'start': rtarget,
+                                         'end': end})
+            elif code[prev_op[rtarget]] == RETURN_VALUE:
+                self.structs.append({'type': 'if-then',
+                                     'start': start,
+                                     'end': rtarget})
+                self.return_end_ifs.add(prev_op[rtarget])
+
+        elif op in (JUMP_IF_FALSE_OR_POP, JUMP_IF_TRUE_OR_POP):
+            target = self.get_target(offset)
+            if target > offset:
+                unop_target = self.last_instr(offset, target, JUMP_FORWARD, target)
+                if unop_target and code[unop_target+3] != ROT_TWO:
+                    self.fixed_jumps[offset] = unop_target
+                else:
+                    self.fixed_jumps[offset] = self.restrict_to_parent(target, parent)
+
+    def next_except_jump(self, start):
+        """
+        Return the next jump that was generated by an except SomeException:
+        construct in a try...except...else clause or None if not found.
+        """
+
+        if self.code[start] == DUP_TOP:
+            except_match = self.first_instr(start, len(self.code), POP_JUMP_IF_FALSE)
+            if except_match:
+                jmp = self.prev_op[self.get_target(except_match)]
+                self.ignore_if.add(except_match)
+                self.not_continue.add(jmp)
+                return jmp
+
+        count_END_FINALLY = 0
+        count_SETUP_ = 0
+        for i in self.op_range(start, len(self.code)):
+            op = self.code[i]
+            if op == END_FINALLY:
+                if count_END_FINALLY == count_SETUP_:
+                    assert self.code[self.prev_op[i]] in (JUMP_ABSOLUTE, JUMP_FORWARD, RETURN_VALUE)
+                    self.not_continue.add(self.prev_op[i])
+                    return self.prev_op[i]
+                count_END_FINALLY += 1
+            elif op in (SETUP_EXCEPT, SETUP_WITH, SETUP_FINALLY):
+                count_SETUP_ += 1
+
+if __name__ == "__main__":
+    co = inspect.currentframe().f_code
+    tokens, customize = Scanner35(3.5).disassemble(co)
+    for t in tokens:
+        print(t)
+    pass

uncompyle6/semantics/fragments.py

@@ -26,7 +26,7 @@ node 2 range information, it in %c, is copied to nodes 0 and 1.
 
 from __future__ import print_function
 
-import inspect, re, sys
+import re, sys
 
 from uncompyle6 import PYTHON3
 from uncompyle6.code import iscode
@@ -46,7 +46,7 @@ else:
     from StringIO import StringIO
 
 
-from uncompyle6.parsers.spark import GenericASTTraversal, GenericASTTraversalPruningException, \
+from spark_parser import GenericASTTraversal, GenericASTTraversalPruningException, \
      DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 
 from types import CodeType
@@ -864,8 +864,6 @@ class FragmentsWalker(pysource.SourceWalker, object):
         return self.extract_node_info(p), p
 
     def print_super_classes(self, node):
-        node[1][0].parent = node
-        node = node[1][0]
         if not (node == 'build_list'):
             return
 
@@ -889,12 +887,13 @@ class FragmentsWalker(pysource.SourceWalker, object):
         start = len(self.f.getvalue())
         n = len(node)-1
         assert node[n].type.startswith('CALL_FUNCTION')
-        for i in range(n-1, 0, -1):
+
+        for i in range(n-2, 0, -1):
             if not node[i].type in ['expr', 'LOAD_CLASSNAME']:
                 break
             pass
 
-        if i == n-1:
+        if i == n-2:
             return
         self.write('(')
         line_separator = ', '

uncompyle6/semantics/pysource.py

@@ -64,13 +64,13 @@ methods implement most of the below.
 """
 from __future__ import print_function
 
-import inspect, sys, re
+import sys, re
 
 from uncompyle6 import PYTHON3
 from uncompyle6.code import iscode
 from uncompyle6.parser import get_python_parser
 from uncompyle6.parsers.astnode import AST
-from uncompyle6.parsers.spark import GenericASTTraversal, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser import GenericASTTraversal, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.scanner import Code, get_scanner
 from uncompyle6.scanners.tok import Token, NoneToken
 import uncompyle6.parser as python_parser
@@ -93,7 +93,6 @@ RETURN_LOCALS = AST('return_stmt',
                     [ AST('ret_expr', [AST('expr', [ Token('LOAD_LOCALS') ])]),
                       Token('RETURN_VALUE')])
 
-
 NONE = AST('expr', [ NoneToken ] )
 
 RETURN_NONE = AST('stmt',
@@ -487,7 +486,8 @@ class SourceWalker(GenericASTTraversal, object):
     stacked_params = ('f', 'indent', 'isLambda', '_globals')
 
     def __init__(self, version, out, scanner, showast=False,
-                 debug_parser=PARSER_DEFAULT_DEBUG):
+                 debug_parser=PARSER_DEFAULT_DEBUG,
+                 compile_mode='exec'):
         GenericASTTraversal.__init__(self, ast=None)
         self.scanner = scanner
         params = {
@@ -495,7 +495,8 @@ class SourceWalker(GenericASTTraversal, object):
             'indent': '',
             }
         self.version = version
-        self.p = get_python_parser(version, debug_parser=debug_parser)
+        self.p = get_python_parser(version, debug_parser=debug_parser,
+                                   compile_mode=compile_mode)
         self.debug_parser = dict(debug_parser)
         self.showast = showast
         self.params = params
@@ -921,22 +922,17 @@ class SourceWalker(GenericASTTraversal, object):
 
     def n_mkfunc(self, node):
 
-        if self.version >= 3.0:
+        if self.version >= 3.3:
             # LOAD_CONST code object ..
             # LOAD_CONST        'x0'
             # MAKE_FUNCTION ..
-            if self.version >= 3.3:
-                func_name = node[-2].pattr
-                code_index = -3
-            else:
-                func_name = node[-2].attr.co_name
-                code_index = -2
-            pass
+            code_index = -3
         else:
             # LOAD_CONST code object ..
             # MAKE_FUNCTION ..
-            func_name =   node[-2].attr.co_name
             code_index = -2
+        code = node[code_index]
+        func_name = code.attr.co_name
         self.write(func_name)
 
         self.indentMore()
@@ -1123,22 +1119,20 @@ class SourceWalker(GenericASTTraversal, object):
 
     def print_super_classes3(self, node):
 
-        # FIXME: wrap superclasses onto a node
-        # as a custom rule
         n = len(node)-1
         assert node[n].type.startswith('CALL_FUNCTION')
 
-        for i in range(n-1, 0, -1):
+        for i in range(n-2, 0, -1):
             if not node[i].type in ['expr', 'LOAD_CLASSNAME']:
                 break
             pass
 
-        if i == n-1:
+        if i == n-2:
             return
         self.write('(')
         line_separator = ', '
         sep = ''
-        i += 1
+        i += 2
         while i < n:
             value = self.traverse(node[i])
             i += 1
@@ -1229,6 +1223,14 @@ class SourceWalker(GenericASTTraversal, object):
 
     n_unpack_w_parens = n_unpack
 
+    def n_assign(self, node):
+        # A horrible hack for Python 3.0 .. 3.2
+        if 3.0 <= self.version <= 3.2 and len(node) == 2:
+            if (node[0][0] == 'LOAD_FAST' and node[0][0].pattr == '__locals__' and
+                node[1][0].type == 'STORE_LOCALS'):
+                self.prune()
+        self.default(node)
+
     def n_assign2(self, node):
         for n in node[-2:]:
             if n[0] == 'unpack':
@@ -1543,12 +1545,20 @@ class SourceWalker(GenericASTTraversal, object):
             pass
 
         # if docstring exists, dump it
-        if (code.co_consts and code.co_consts[0] is not None
-            and len(ast) > 0 and ast[0][0] == ASSIGN_DOC_STRING(code.co_consts[0])):
-            if self.hide_internal:
+        if (code.co_consts and code.co_consts[0] is not None and len(ast) > 0):
+            do_doc = False
+            if (ast[0][0] == ASSIGN_DOC_STRING(code.co_consts[0])):
+                i = 0
+                do_doc = True
+            elif (len(ast) > 2 and 3.0 <= self.version <= 3.2 and
+                      ast[2][0] == ASSIGN_DOC_STRING(code.co_consts[0])):
+                i = 2
+                do_doc = True
+            if do_doc and self.hide_internal:
                 self.print_docstring(indent, code.co_consts[0])
                 self.print_()
-                del ast[0]
+                del ast[i]
+
 
         # the function defining a class normally returns locals(); we
         # don't want this to show up in the source, thus remove the node
@@ -1564,7 +1574,7 @@ class SourceWalker(GenericASTTraversal, object):
         code._tokens = None; code._customize = None # save memory
         self.classes.pop(-1)
 
-    def gen_source(self, ast, name, customize, isLambda=0, returnNone=False):
+    def gen_source(self, ast, name, customize, isLambda=False, returnNone=False):
         """convert AST to source code"""
 
         rn = self.return_none
@@ -1622,7 +1632,7 @@ class SourceWalker(GenericASTTraversal, object):
         return ast
 
 def deparse_code(version, co, out=sys.stdout, showasm=False, showast=False,
-                 showgrammar=False, code_objects={}):
+                 showgrammar=False, code_objects={}, compile_mode='exec'):
     """
     disassembles and deparses a given code block 'co'
     """
@@ -1640,7 +1650,8 @@ def deparse_code(version, co, out=sys.stdout, showasm=False, showast=False,
     debug_parser['reduce'] = showgrammar
 
     #  Build AST from disassembly.
-    deparsed = SourceWalker(version, out, scanner, showast=showast, debug_parser=debug_parser)
+    deparsed = SourceWalker(version, out, scanner, showast=showast,
+                            debug_parser=debug_parser, compile_mode=compile_mode)
 
     deparsed.ast = deparsed.build_ast(tokens, customize)
 

uncompyle6/verify.py

@@ -139,8 +139,10 @@ def cmp_code_objects(version, code_obj1, code_obj2, name=''):
     This is the main part of this module.
     """
     # print code_obj1, type(code_obj2)
-    assert iscode(code_obj1)
-    assert iscode(code_obj2)
+    assert iscode(code_obj1), \
+      "cmp_code_object first object type is %s, not code" % type(code_obj1)
+    assert iscode(code_obj2), \
+      "cmp_code_object second object type is %s, not code" % type(code_obj2)
     # print dir(code_obj1)
     if isinstance(code_obj1, object):
         # new style classes (Python 2.2)
@@ -180,22 +182,22 @@ def cmp_code_objects(version, code_obj1, code_obj2, name=''):
         elif member == 'co_code':
             if version == 2.5:
                 import uncompyle6.scanners.scanner25 as scan
-                scanner = scan.Scanner25()
+                scanner = scan.Scanner25(version)
             elif version == 2.6:
                 import uncompyle6.scanners.scanner26 as scan
-                scanner = scan.Scanner26()
+                scanner = scan.Scanner26(version)
             elif version == 2.7:
                 import uncompyle6.scanners.scanner27 as scan
-                scanner = scan.Scanner27()
+                scanner = scan.Scanner27(version)
             elif version == 3.2:
                 import uncompyle6.scanners.scanner32 as scan
-                scanner = scan.Scanner32()
+                scanner = scan.Scanner32(version)
             elif version == 3.3:
                 import uncompyle6.scanners.scanner33 as scan
-                scanner = scan.Scanner33()
+                scanner = scan.Scanner33(version)
             elif version == 3.4:
                 import uncompyle6.scanners.scanner34 as scan
-                scanner = scan.Scanner34()
+                scanner = scan.Scanner34(version)
 
             global JUMP_OPs
             JUMP_OPs = list(scan.JUMP_OPs) + ['JUMP_BACK']