Add -V | --version and simplfy changing it

Drop 2.5 test until we figure out what's wrong

.gitignore

@@ -3,9 +3,12 @@
 /.cache
 /.eggs
 /.python-version
+/.tox
+/README
 /__pkginfo__.pyc
 /dist
 /how-to-make-a-release.txt
+/tmp
 /uncompyle6.egg-info
 __pycache__
 build

.travis.yml

@@ -9,6 +9,7 @@ python:
   - '3.5'
 
 install:
+- pip install -r requirements.txt
 - pip install -r requirements-dev.txt
 
 script:

ChangeLog

@@ -1,6 +1,184 @@
+2016-05-02  rocky <rocky@gnu.org>
+
+	* __pkginfo__.py, bin/pydisassemble, bin/uncompyle6, setup.py,
+	uncompyle6/__init__.py, uncompyle6/version.py: Add -V | --version
+	and simplfy changing it
+
+2016-05-01  rocky <rocky@gnu.org>
+
+	* uncompyle6/__init__.py: Expose uncompyle_file
+
+2016-05-01  rocky <rocky@gnu.org>
+
+	* test/Makefile, uncompyle6/semantics/pysource.py: Bug
+
+2016-05-01  rocky <rocky@gnu.org>
+
+	* test/Makefile, test/simple_source/expression/05_const_map.py: Add
+	test for last fix.  Drop 2.5 test until we figure out what's wrong
+
+2016-05-01  rocky <rocky@gnu.org>
+
+	* uncompyle6/parsers/parse3.py, uncompyle6/scanners/scanner35.py,
+	uncompyle6/semantics/pysource.py: Bug in 3.5 constant map parsing
+
+2016-05-01  rocky <rocky@gnu.org>
+
+	* uncompyle6/__init__.py: Export module load and fns load_file,
+	load_module
+
+2016-05-01  rocky <rocky@gnu.org>
+
+	* __pkginfo__.py, setup.py, uncompyle6/marsh.py: License is MIT marsh.py: remove unused import
+
+2016-05-01  rocky <rocky@gnu.org>
+
+	* uncompyle6/parsers/parse3.py: Forgot to define Python3ParserSingle
+
+2016-05-01  rocky <rocky@gnu.org>
+
+	* uncompyle6/parser.py, uncompyle6/parsers/parse2.py,
+	uncompyle6/parsers/parse3.py: Start to DRY Python2 and Python3
+	grammars Separate out 3.2, and 3.5+ specific grammar code
+
+2016-04-30  rocky <rocky@gnu.org>
+
+	* ChangeLog, NEWS, README.rst, __pkginfo__.py: Get ready for release
+	2.3.1
+
+2016-04-30  rocky <rocky@gnu.org>
+
+	* uncompyle6/parsers/astnode.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/semantics/pysource.py: Python 3.0..3.2 bug in
+	LOAD_FAST/STORE_LOCAL LOAD_FAST         '__locals__' STORE_LOCALS      '' Also have to adjust doc constants for this crap astnode.py: minor format change
+
+2016-04-30  rocky <rocky@gnu.org>
+
+	* test/Makefile, test/simple_source/def/06_classbug.py,
+	test/test_pythonlib.py: Test optimized Python code and Python 3.2
+
+2016-04-30  rocky <rocky@gnu.org>
+
+	* uncompyle6/parsers/parse3.py: Pevious commit grammar change is
+	Python 3.5 and up
+
+2016-04-30  rocky <rocky@gnu.org>
+
+	* uncompyle6/parsers/parse3.py: Python 3.5 if statments decompyle Sometimes it doesn't need JUMP_FORWARD _come_from _come_from For example: def handle2(module):     if module == 'foo':         try:             module = 1         except ImportError as exc:             module = exc     return module And: if __name__:     for i in (1, 2):         x = 3
+
+2016-04-28  rocky <rocky@gnu.org>
+
+	* requirements.txt, uncompyle6/parser.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/semantics/fragments.py, uncompyle6/semantics/pysource.py: 
+	spark -> spark_parser
+
+2016-04-28  rocky <rocky@gnu.org>
+
+	* uncompyle6/parsers/spark.py: Really remove spark - Use external
+	package instead
+
+2016-04-27  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #8 from rocky/external-spark External spark
+
+2016-04-27  rocky <rocky@gnu.org>
+
+	* .travis.yml, circle.yml: Note dependencies on spark
+
+2016-04-27  rocky <rocky@gnu.org>
+
+	* .gitignore, README.rst, requirements.txt, uncompyle6/parser.py,
+	uncompyle6/parsers/astnode.py, uncompyle6/parsers/parse2.py,
+	uncompyle6/parsers/parse3.py, uncompyle6/semantics/fragments.py,
+	uncompyle6/semantics/pysource.py: Use external spark now.
+
+2016-04-20  rocky <rocky@gnu.org>
+
+	* circle.yml: Back to 2.7.8
+
+2016-04-20  rocky <rocky@gnu.org>
+
+	* circle.yml: Try python 2.7.10
+
+2016-04-19  rocky <rocky@gnu.org>
+
+	* README.rst: Remove link to Mysterie uncompyle2 per request
+
+2016-04-19  rocky <rocky@gnu.org>
+
+	* ChangeLog, NEWS, __pkginfo__.py: Get ready for release 2.2.0
+
+2016-04-18  rocky <rocky@gnu.org>
+
+	* README.rst: Another typo
+
+2016-04-18  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #7 from rocky/single-compile Support single-mode compile
+
+2016-04-18  rocky <rocky@gnu.org>
+
+	* README.rst: Doc typo
+
+2016-04-18  Thomas Grainger <tom.grainger@procensus.com>
+
+	* __pkginfo__.py, setup.cfg, setup.py: declare Python3 support in
+	wheel and trove
+
+2016-04-18  rocky <rocky@gnu.org>
+
+	* uncompyle6/parser.py, uncompyle6/parsers/parse2.py,
+	uncompyle6/parsers/parse3.py: Start to DRY Python 2 and Python 3
+	grammar code Move common code to parser.py
+
+2016-04-18  rocky <rocky@gnu.org>
+
+	* pytest/test_single_compile.py, uncompyle6/parser.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py: Add
+	simgle-mode compilation
+
+2016-04-17  rocky <rocky@gnu.org>
+
+	* pytest/test_single_compile.py, uncompyle6/parser.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/semantics/pysource.py: Towards single compilation
+
+2016-04-10  rocky <rb@dustyfeet.com>
+
+	* : Back off if_else_ternary pending Fails on Python 3.4 investigation Python 3.5 works though
+
+2016-04-10  rocky <rb@dustyfeet.com>
+
+	* .gitignore: Test administrivia
+
+2016-04-07  rocky <rb@dustyfeet.com>
+
+	* HISTORY.md, README.rst,
+	test/simple_source/branching/10_if_else_ternary.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py: Handle
+	Ternary "or". Remove mention of uncompyle3 uncompyle3 removed per Mysterie's request [Fixes Issue #5]
+
+2016-03-11  rocky <rb@dustyfeet.com>
+
+	* test/Makefile: remove uncompyle3 and make test work again * uncompyle3 removed by request * make test on python 2.7 is failing on some python3 and python3.5   bytecodes. Remove for now.
+
+2016-02-23  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #3 from lelicopter/master Bug correction (parse cmd options)
+
+2016-01-07  rocky <rb@dustyfeet.com>
+
+	* test/Makefile, test/bytecompile-tests, test/test_pythonlib.py: Add
+	Python 3.5 tests that we can do.
+
+2016-01-05  R. Bernstein <rocky@users.noreply.github.com>
+
+	* HISTORY.md: Grammar fixes
+
 2016-01-02  rocky <rb@dustyfeet.com>
 
-	* __pkginfo__.py: Get ready for release 2.1.3
+	* ChangeLog, NEWS, __pkginfo__.py: Get ready for release 2.1.3
 
 2016-01-02  rocky <rb@dustyfeet.com>
 

HISTORY.md

@@ -30,7 +30,7 @@ The last mention of a release of SPARK from John is around 2002.
 
 In the fall of 2000, Hartmut Goebel
 [took over maintaining the code](https://groups.google.com/forum/#!searchin/comp.lang.python/hartmut$20goebel/comp.lang.python/35s3mp4-nuY/UZALti6ujnQJ). The
-first subsequennt public release announcement that I can find is
+first subsequent public release announcement that I can find is
 ["decompyle - A byte-code-decompiler version 2.2 beta 1"](https://mail.python.org/pipermail/python-announce-list/2002-February/001272.html).
 
 From the CHANGES file found in
@@ -38,9 +38,8 @@ From the CHANGES file found in
 it appears that Hartmut did most of the work to get this code to
 accept the full Python language. He added precidence to the table
 specifiers, support for multiple versions of Python, the
-pretty-printing of docstrings, lists and hashes. He also wrote
-extensive tests and routines to the testing and verification of
-decompiled bytecode.
+pretty-printing of docstrings, lists, and hashes. He also wrote test and verification routines of
+deparsed bytecode, and used this in an extensive set of tests that he also wrote. He could verify against the entire Python library.
 
 decompyle2.2 was packaged for Debian (sarge) by
 [Ben Burton around 2002](https://packages.qa.debian.org/d/decompyle.html). As
@@ -91,10 +90,6 @@ Hartmut a decade an a half ago:
     NB. This is not a masterpiece of software, but became more like a hack.
     Probably a complete rewrite would be sensefull. hG/2000-12-27
 
-One of the attempts to modernize it and make it available for Python3
-is [the one by Anton Vorobyov (DarkFenX)](https://github.com/DarkFenX/uncompyle3). I've
-followed some of the ideas there in this project.
-
 Lastly, I should mention [unpyc](https://code.google.com/p/unpyc3/)
 and most especially [pycdc](https://github.com/zrax/pycdc), largely by
 Michael Hansen and Darryl Pogue. If they supported getting source-code

NEWS

@@ -1,3 +1,30 @@
+uncompyle6 2.3.2 2016-05-1
+
+- Add --version option standalone scripts
+- Correct License information in package
+- expose fns uncompyle_file, load_file, and load_module
+- Start to DRY Python2 and Python3 grammars Separate out 3.2, and 3.5+
+  specific grammar code
+- Fix bug in 3.5+ constant map parsing
+
+uncompyle6 2.3.0, 2.3.1 2016-04-30
+
+- Require spark_parser  >= 1.1.0
+
+uncompyle6 2.2.0 2016-04-30
+
+- Spark is no longer here but pulled separate package spark_parse
+- Python 3 parsing fixes
+- More tests
+
+uncompyle6 2.2.0 2016-04-02
+
+- Support single-mode (in addtion to exec-mode) compilation
+- Start to DRY Python 2 and Python 3 grammars
+- Fix bug in if else ternary  construct
+- Fix bug in uncomplye6 -d and -r options (via lelicopter)
+
+
 uncompyle6 2.1.3 2016-01-02
 
 - Limited support for decompiling Python 3.5

README.rst

@@ -1,4 +1,4 @@
-|downloads| |buildstatus|
+|buildstatus|
 
 uncompyle6
 ==========
@@ -18,11 +18,11 @@ so and has been tested on Python running versions 2.6, 2.7, 3.3,
 Why this?
 ---------
 
-What makes this different other CPython bytecode decompilers?  Its
+What makes this different from other CPython bytecode decompilers?  Its
 ability to deparse just fragments and give source-code information
 around a given bytecode offset.
 
-I using this to deparse fragments of code inside my trepan_
+I use this to deparse fragments of code inside my trepan_
 debuggers_. For that, I need to record text fragments for all
 bytecode offsets (of interest). This purpose although largely
 compatible with the original intention is yet a little bit different.
@@ -48,6 +48,8 @@ This uses setup.py, so it follows the standard Python routine:
 
 ::
 
+    pip install -r requirements.txt
+    pip install -r requirements-dev.txt
     python setup.py install # may need sudo
     # or if you have pyenv:
     python setup.py develop
@@ -77,7 +79,7 @@ Run
 ::
 
      ./bin/uncompyle6 -h
-     ./bin/pydisassemble -y
+     ./bin/pydisassemble -h
 
 for usage help
 
@@ -92,8 +94,6 @@ See Also
 --------
 
 * https://github.com/zrax/pycdc
-* https://github.com/Mysterie/uncompyle2
-* https://github.com/DarkFenX/uncompyle3
 * https://code.google.com/p/unpyc3/
 
 The HISTORY file.

__pkginfo__.py

@@ -9,13 +9,20 @@
 
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015, 2016 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
 classifiers =  ['Development Status :: 3 - Alpha',
                 'Intended Audience :: Developers',
                 'Operating System :: OS Independent',
                 'Programming Language :: Python',
+                'Programming Language :: Python :: 2',
+                'Programming Language :: Python :: 2.6',
+                'Programming Language :: Python :: 2.7',
+                'Programming Language :: Python :: 3',
+                'Programming Language :: Python :: 3.3',
+                'Programming Language :: Python :: 3.4',
+                'Programming Language :: Python :: 3.5',
                 'Topic :: Software Development :: Debuggers',
                 'Topic :: Software Development :: Libraries :: Python Modules',
                 ]
@@ -24,7 +31,10 @@ classifiers =  ['Development Status :: 3 - Alpha',
 author             = "Rocky Bernstein, Hartmut Goebel, John Aycock, and others"
 author_email       = "rb@dustyfeet.com"
 ftp_url            = None
-# license            = 'BSDish'
+install_requires   = ['python-spark >= 1.1.0']
+license            = 'GPL'
+
+license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'
 packages           = ['uncompyle6', 'uncompyle6.opcodes', 'uncompyle6.semantics', 'uncompyle6.scanners', 'uncompyle6.parsers']
@@ -40,7 +50,6 @@ def get_srcdir():
     return os.path.realpath(filename)
 
 ns = {}
-version            = '2.1.3'
 web                = 'https://github.com/rocky/python-uncompyle6/'
 
 # tracebacks in zip files are funky and not debuggable

bin/pydisassemble

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 # Mode: -*- python -*-
 #
-# Copyright (c) 2015 by Rocky Bernstein <rb@dustyfeet.com>
+# Copyright (c) 2015-2016 by Rocky Bernstein <rb@dustyfeet.com>
 #
 from __future__ import print_function
 import sys, os, getopt
@@ -9,11 +9,13 @@ import sys, os, getopt
 program =  os.path.basename(__file__)
 
 __doc__ = """
-Usage: %s [OPTIONS]... FILE
+Usage:
+  %s [OPTIONS]... FILE
+  %s [--help | -h | -V | --version]
 
 Examples:
-  %s      foo.pyc
-  %s      foo.py
+  %s foo.pyc
+  %s foo.py
   %s -o foo.pydis foo.pyc
   %s -o /tmp foo.pyc
 
@@ -24,7 +26,7 @@ Options:
                 <path>
   --help        show this message
 
-""" % ((program,) * 5)
+""" % ((program,) * 6)
 
 
 Usage_short = \
@@ -32,6 +34,7 @@ Usage_short = \
 
 from uncompyle6 import check_python_version
 from uncompyle6.disas import disassemble_files
+from uncompyle6.version import VERSION
 
 check_python_version(program)
 
@@ -40,7 +43,7 @@ out_base = None
 
 
 try:
-    opts, files = getopt.getopt(sys.argv[1:], 'ho:', ['help'])
+    opts, files = getopt.getopt(sys.argv[1:], 'hVo:', ['help', 'version'])
 except getopt.GetoptError as e:
     print('%s: %s' % (os.path.basename(sys.argv[0]), e),  file=sys.stderr)
     sys.exit(-1)
@@ -48,6 +51,9 @@ except getopt.GetoptError as e:
 for opt, val in opts:
     if opt in ('-h', '--help'):
         print(__doc__)
+        sys.exit(1)
+    elif opt in ('-V', '--version'):
+        print("%s %s" % (program, VERSION))
         sys.exit(0)
     elif opt == '-o':
         outfile = val

bin/uncompyle6

@@ -1,16 +1,23 @@
 #!/usr/bin/env python
 # Mode: -*- python -*-
 #
+# Copyright (c) 2015-2016 by Rocky Bernstein
 # Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
-# Copyright (c) 2015 by Rocky Bernstein
+#
+from __future__ import print_function
+import sys, os, getopt, time
 
-"""
-Usage: uncompyle6 [OPTIONS]... [ FILE | DIR]...
+program =  os.path.basename(__file__)
+
+__doc__ = """
+Usage:
+  %s [OPTIONS]... [ FILE | DIR]...
+  %s [--help | -h | --V | --version]
 
 Examples:
-  uncompyle6      foo.pyc bar.pyc       # decompile foo.pyc, bar.pyc to stdout
-  uncompyle6 -o . foo.pyc bar.pyc       # decompile to ./foo.pyc_dis and ./bar.pyc_dis
-  uncompyle6 -o /tmp /usr/lib/python1.5 # decompile whole library
+  %s      foo.pyc bar.pyc       # decompile foo.pyc, bar.pyc to stdout
+  %s -o . foo.pyc bar.pyc       # decompile to ./foo.pyc_dis and ./bar.pyc_dis
+  %s -o /tmp /usr/lib/python1.5 # decompile whole library
 
 Options:
   -o <path>     output decompiled files to this path:
@@ -34,26 +41,25 @@ Options:
 Debugging Options:
   --asm     -a  include byte-code         (disables --verify)
   --grammar -g  show matching grammar
-  --treee   -t  include syntax tree       (disables --verify)
+  --tree    -t  include syntax tree       (disables --verify)
 
 Extensions of generated files:
   '.pyc_dis' '.pyo_dis'   successfully decompiled (and verified if --verify)
     + '_unverified'       successfully decompile but --verify failed
     + '_failed'           decompile failed (contact author for enhancement)
-"""
-
-from __future__ import print_function
-import sys, os, getopt, time
+""" % ((program,) * 6
 
 program =  os.path.basename(__file__)
 
 from uncompyle6 import verify, check_python_version
 from uncompyle6.main import main, status_msg
+from uncompyle6.version import VERSION
 
 def usage():
     print("""usage:
-   %s [--help] [--verify] [--asm] [--tree] [--grammar] [-o <path>] FILE|DIR...
-"""  % program)
+   %s [--verify] [--asm] [--tree] [--grammar] [-o <path>] FILE|DIR...
+   %s [--help | -h | --version | -V]
+"""  % (program, program))
     sys.exit(1)
 
 
@@ -68,8 +74,8 @@ timestamp = False
 timestampfmt = "# %Y.%m.%d %H:%M:%S %Z"
 
 try:
-    opts, files = getopt.getopt(sys.argv[1:], 'hagtdro:c:p:',
-                                'help asm grammar recurse timestamp tree verify '
+    opts, files = getopt.getopt(sys.argv[1:], 'hagtdrVo:c:p:',
+                                'help asm grammar recurse timestamp tree verify version '
                                 'showgrammar'.split(' '))
 except getopt.GetoptError as e:
     print('%s: %s' % (os.path.basename(sys.argv[0]), e),  file=sys.stderr)
@@ -80,6 +86,9 @@ for opt, val in opts:
     if opt in ('-h', '--help'):
         print(__doc__)
         sys.exit(0)
+    elif opt in ('-V', '--version'):
+        print("%s %s" % (program, VERSION))
+        sys.exit(0)
     elif opt == '--verify':
         options['do_verify'] = True
     elif opt in ('--asm', '-a'):
@@ -92,13 +101,13 @@ for opt, val in opts:
         options['showgrammar'] = True
     elif opt == '-o':
         outfile = val
-    elif opt == ('--timestamp', '-d'):
+    elif opt in ('--timestamp', '-d'):
         timestamp = True
     elif opt == '-c':
         codes.append(val)
     elif opt == '-p':
         numproc = int(val)
-    elif opt == ('--recurse', '-r'):
+    elif opt in ('--recurse', '-r'):
         recurse_dirs = True
     else:
         print(opt, file=sys.stderr)

circle.yml

@@ -6,7 +6,8 @@ machine:
 
 dependencies:
     override:
-        - pip install -r test-requirements.txt
+        - pip install -r requirements.txt
+        - pip install -r requirements-dev.txt
 test:
     override:
         - python ./setup.py develop && make check-2.7

pytest/test_single_compile.py

@@ -0,0 +1,21 @@
+import pytest
+from uncompyle6 import PYTHON_VERSION, PYTHON3, deparse_code
+
+def test_single_mode():
+    single_expressions = (
+        'i = 1',
+        'i and (j or k)',
+        'i += 1',
+        'i = j % 4',
+        'i = {}',
+        'i = []',
+        'while i < 1 or stop:\n    i\n',
+        'while i < 1 or stop:\n    print%s\n' % ('(i)' if PYTHON3 else ' i'),
+        'for i in range(10):\n    i\n',
+        'for i in range(10):\n    for j in range(10):\n        i + j\n',
+        'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
+    )
+
+    for expr in single_expressions:
+        code = compile(expr + '\n', '<string>', 'single')
+        assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'

requirements.txt

@@ -0,0 +1 @@
+spark_parser >= 1.1.0

setup.cfg

@@ -6,3 +6,6 @@ doc_files = README
 #		USAGE.txt
 #		doc/
 #		examples/
+
+[bdist_wheel]
+universal=1

setup.py

@@ -1,32 +1,25 @@
-#! python
+#!/usr/bin/env python
 
 """Setup script for the 'uncompyle6' distribution."""
 
-from distutils.core import setup, Extension
-
-# Get the package information used in setup().
-# from __pkginfo__ import \
-#     author,           author_email,       classifiers,                    \
-#     install_requires, license,            long_description,               \
-#     modname,          packages,           py_modules,   \
-#     short_desc,       version,            web,              zip_safe
-
 from __pkginfo__ import \
     author,           author_email,                                  \
-    long_description,                                                \
+    license,          long_description,                              \
     modname,          packages,           py_modules,       scripts, \
-    short_desc,       version,            web,              zip_safe
+    short_desc,       web,                zip_safe
 
 __import__('pkg_resources')
 from setuptools import setup
 
+exec(open('uncompyle6/version.py').read())
+
 setup(
        author             = author,
        author_email       = author_email,
        # classifiers        = classifiers,
        description        = short_desc,
        # install_requires   = install_requires,
-       # license            = license,
+       license            = license,
        long_description   = long_description,
        py_modules         = py_modules,
        name               = modname,
@@ -35,5 +28,5 @@ setup(
        url                = web,
        setup_requires     = ['nose>=1.0'],
        scripts            = scripts,
-       version            = version,
+       version            = VERSION,
        zip_safe           = zip_safe)

test-requirements.txt

@@ -1 +0,0 @@
-pytest

test/Makefile

@@ -20,7 +20,7 @@ check:
 	$(MAKE) check-$$PYTHON_VERSION
 
 #: Run working tests from Python 2.6 or 2.7
-check-2.6 check-2.7: check-bytecode check-2.7-ok
+check-2.6 check-2.7: check-bytecode-sans-3.5 check-2.7-ok
 
 #: Run working tests from Python 3.3
 check-3.3: check-bytecode
@@ -43,6 +43,11 @@ check-bytecode-2:
 
 #: Check deparsing bytecode only
 check-bytecode:
+	$(PYTHON) test_pythonlib.py --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
+        --bytecode-3.2 --bytecode-3.3 --bytecode-3.4 --bytecode-3.5
+
+#: Check deparsing bytecode only
+check-bytecode-sans-3.5:
 	$(PYTHON) test_pythonlib.py --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
         --bytecode-3.2 --bytecode-3.3 --bytecode-3.4
 
@@ -70,6 +75,10 @@ check-bytecode-3.3:
 check-bytecode-3.4:
 	$(PYTHON) test_pythonlib.py --bytecode-3.4
 
+#: Check deparsing Python 3.5
+check-bytecode-3.5:
+	$(PYTHON) test_pythonlib.py --bytecode-3.5
+
 #: short tests for bytecodes only for this version of Python
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --verify $(COMPILE)
@@ -78,7 +87,11 @@ check-native-short:
 check-2.7-ok:
 	$(PYTHON) test_pythonlib.py --ok-2.7 --verify $(COMPILE)
 
-#: Run longer Python 2.7's lib files known to be okay
+#: Run longer Python 3.2's lib files known to be okay
+check-3.2-ok:
+	$(PYTHON) test_pythonlib.py --ok-3.2 --verify $(COMPILE)
+
+#: Run longer Python 3.4's lib files known to be okay
 check-3.4-ok:
 	$(PYTHON) test_pythonlib.py --ok-3.4 --verify $(COMPILE)
 

test/bytecompile-tests

@@ -70,7 +70,8 @@ for root, dirs, basenames in os.walk('simple_source'):
             simple_source.append(os.path.join(root, basename)[0:-3])
     pass
 
-tests['2.6'] = tests['2.7'] = tests['3.2'] = tests['3.3'] = tests['3.4'] = simple_source
+tests['2.6'] = tests['2.7'] = tests['3.2'] = \
+  tests['3.3'] = tests['3.4'] = tests['3.5'] = simple_source
 
 total_tests = len(tests['2.7'])
 #tests['2.2'].sort(); print tests['2.2']

test/simple_source/branching/10_if_else_ternary.py

@@ -0,0 +1,11 @@
+# Tests:
+
+# ret_expr_or_cond ::= ret_expr
+# ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
+# ret_expr_or_cond ::= ret_cond
+# ret_or ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+
+# See https://github.com/rocky/python-uncompyle6/issues/5
+
+def minimize(x, y):
+    return x or (x if x < y else y)

test/simple_source/def/06_classbug.py

@@ -0,0 +1,13 @@
+# Python 3.2 Bug from abc.py
+
+# Make sure we handle:
+
+# LOAD_FAST         '__locals__'
+# STORE_LOCALS      ''
+
+class abstractclassmethod(object):
+    """A Python 3.2 STORE_LOCALS bug
+    """
+
+    def __init__(self, callable):
+        callable.__isabstractmethod__ = True

test/simple_source/expression/05_const_map.py

@@ -0,0 +1,7 @@
+# Addresses a bug in the way Python 3.5+ handles
+# creation of map constants
+opts = {'highlight': True,
+            'start_line': -1,
+            'end_line': None
+            }
+print(opts)

test/test_pythonlib.py

@@ -55,28 +55,31 @@ PYO = ('*.pyo', )
 PYOC = ('*.pyc', '*.pyo')
 
 test_options = {
-    # name:   (src_basedir, pattern, output_base_suffix, pythoin_version)
+    # name:   (src_basedir, pattern, output_base_suffix, python_version)
     'test':
         ('test', PYC, 'test'),
 
     'ok-2.6':
         (os.path.join(src_dir, 'ok_2.6'),
-         PYC, 'ok-2.6', 2.6),
+         PYOC, 'ok-2.6', 2.6),
 
     'ok-2.7':    (os.path.join(src_dir, 'ok_lib2.7'),
-                 PYC, 'ok-2.7', 2.7),
+                 PYOC, 'ok-2.7', 2.7),
+
+    'ok-3.2':    (os.path.join(src_dir, 'ok_lib3.2'),
+                 PYOC, 'ok-3.2', 3.5),
 
     'base-2.7':  (os.path.join(src_dir, 'base_tests', 'python2.7'),
-                  PYC, 'base_2.7', 2.7),
+                  PYOC, 'base_2.7', 2.7),
 }
 
-for  vers in (2.7, 3.4):
+for  vers in (2.7, 3.4, 3.5):
     pythonlib = "ok_lib%s" % vers
     key = "ok-%s" % vers
-    test_options[key] = (os.path.join(src_dir, pythonlib), PYC, key, vers)
+    test_options[key] = (os.path.join(src_dir, pythonlib), PYOC, key, vers)
     pass
 
-for  vers in (2.5, 2.6, 2.7, 3.2, 3.3, 3.4):
+for  vers in (2.5, 2.6, 2.7, 3.2, 3.3, 3.4, 3.5):
     bytecode = "bytecode_%s" % vers
     key = "bytecode-%s" % vers
     test_options[key] =  (bytecode, PYC, bytecode, vers)
@@ -84,7 +87,7 @@ for  vers in (2.5, 2.6, 2.7, 3.2, 3.3, 3.4):
     pythonlib = "python%s" % vers
     if vers >= 3.0:
         pythonlib = os.path.join(pythonlib, '__pycache__')
-    test_options[key] =  (os.path.join(lib_prefix, pythonlib), PYC, pythonlib, vers)
+    test_options[key] =  (os.path.join(lib_prefix, pythonlib), PYOC, pythonlib, vers)
 
 #-----
 

uncompyle6/__init__.py

@@ -30,6 +30,8 @@ from __future__ import print_function
 
 import sys
 
+__docformat__ = 'restructuredtext'
+
 PYTHON3 = (sys.version_info >= (3, 0))
 
 # We do this crazy way to support Python 2.6 which
@@ -50,6 +52,11 @@ def check_python_version(program):
 
 import uncompyle6.semantics.pysource
 import uncompyle6.semantics.fragments
+import uncompyle6.load
+
+# Export some functions
+from uncompyle6.load import load_module, load_file
+from uncompyle6.main import uncompyle_file
 
 # Conventience functions so you can say:
 # from uncompyle6 import deparse_code

uncompyle6/marsh.py

@@ -18,7 +18,6 @@ from __future__ import print_function
 import sys, types
 from struct import unpack
 
-import uncompyle6.scanners.scanner3 as scan3
 from uncompyle6.magics import PYTHON_MAGIC_INT
 from uncompyle6.code import Code3
 

uncompyle6/parser.py

@@ -3,7 +3,7 @@
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
 """
-Common spark parser routines Python.
+Common uncompyle parser routines.
 """
 
 from __future__ import print_function
@@ -11,7 +11,7 @@ from __future__ import print_function
 import sys
 
 from uncompyle6.code import iscode
-from uncompyle6.parsers.spark import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 
 class ParserError(Exception):
     def __init__(self, token, offset):
@@ -69,6 +69,175 @@ class PythonParser(GenericASTBuilder):
         # print >> sys.stderr, 'resolve', str(list)
         return GenericASTBuilder.resolve(self, list)
 
+    ##############################################
+    ## Common Python 2 and Python 3 grammar rules
+    ##############################################
+    def p_start(self, args):
+        '''
+        # The start or goal symbol
+        stmts ::= stmts sstmt
+        stmts ::= sstmt
+        '''
+
+    def p_call_stmt(self, args):
+        '''
+        # eval-mode compilation.  Single-mode interactive compilation
+        # adds another rule.
+        call_stmt ::= expr POP_TOP
+        '''
+
+    def p_funcdef(self, args):
+        '''
+        stmt ::= funcdef
+        funcdef ::= mkfunc designator
+        stmt ::= funcdefdeco
+        funcdefdeco ::= mkfuncdeco designator
+        mkfuncdeco ::= expr mkfuncdeco CALL_FUNCTION_1
+        mkfuncdeco ::= expr mkfuncdeco0 CALL_FUNCTION_1
+        mkfuncdeco0 ::= mkfunc
+        load_closure ::= load_closure LOAD_CLOSURE
+        load_closure ::= LOAD_CLOSURE
+        '''
+
+    def p_genexpr(self, args):
+        '''
+        expr ::= genexpr
+
+        genexpr ::= LOAD_GENEXPR MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
+
+        stmt ::= genexpr_func
+
+        genexpr_func ::= LOAD_FAST FOR_ITER designator comp_iter JUMP_BACK
+        '''
+
+    def p_dictcomp(self, args):
+        '''
+        expr ::= dictcomp
+        dictcomp ::= LOAD_DICTCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
+        stmt ::= dictcomp_func
+
+        dictcomp_func ::= BUILD_MAP LOAD_FAST FOR_ITER designator
+                comp_iter JUMP_BACK RETURN_VALUE RETURN_LAST
+
+        '''
+
+    def p_augmented_assign(self, args):
+        '''
+        stmt ::= augassign1
+        stmt ::= augassign2
+        augassign1 ::= expr expr inplace_op designator
+        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SUBSCR
+        augassign1 ::= expr expr inplace_op ROT_TWO   STORE_SLICE+0
+        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+1
+        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+2
+        augassign1 ::= expr expr inplace_op ROT_FOUR  STORE_SLICE+3
+        augassign2 ::= expr DUP_TOP LOAD_ATTR expr
+                inplace_op ROT_TWO   STORE_ATTR
+
+        inplace_op ::= INPLACE_ADD
+        inplace_op ::= INPLACE_SUBTRACT
+        inplace_op ::= INPLACE_MULTIPLY
+        inplace_op ::= INPLACE_DIVIDE
+        inplace_op ::= INPLACE_TRUE_DIVIDE
+        inplace_op ::= INPLACE_FLOOR_DIVIDE
+        inplace_op ::= INPLACE_MODULO
+        inplace_op ::= INPLACE_POWER
+        inplace_op ::= INPLACE_LSHIFT
+        inplace_op ::= INPLACE_RSHIFT
+        inplace_op ::= INPLACE_AND
+        inplace_op ::= INPLACE_XOR
+        inplace_op ::= INPLACE_OR
+        '''
+
+    def p_assign(self, args):
+        '''
+        stmt ::= assign
+        assign ::= expr DUP_TOP designList
+        assign ::= expr designator
+
+        stmt ::= assign2
+        stmt ::= assign3
+        assign2 ::= expr expr ROT_TWO designator designator
+        assign3 ::= expr expr expr ROT_THREE ROT_TWO designator designator designator
+        '''
+
+    def p_import20(self, args):
+        '''
+        stmt ::= importstmt
+        stmt ::= importfrom
+        stmt ::= importstar
+        stmt ::= importmultiple
+
+        importlist2 ::= importlist2 import_as
+        importlist2 ::= import_as
+        import_as ::= IMPORT_NAME designator
+        import_as ::= IMPORT_NAME load_attrs designator
+        import_as ::= IMPORT_FROM designator
+
+        importstmt ::= LOAD_CONST LOAD_CONST import_as
+        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME IMPORT_STAR
+        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist2 POP_TOP
+        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT IMPORT_STAR
+        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT importlist2 POP_TOP
+        importmultiple ::= LOAD_CONST LOAD_CONST import_as imports_cont
+
+        imports_cont ::= imports_cont import_cont
+        imports_cont ::= import_cont
+        import_cont ::= LOAD_CONST LOAD_CONST import_as_cont
+        import_as_cont ::= IMPORT_NAME_CONT designator
+        import_as_cont ::= IMPORT_NAME_CONT load_attrs designator
+        import_as_cont ::= IMPORT_FROM designator
+
+        load_attrs ::= LOAD_ATTR
+        load_attrs ::= load_attrs LOAD_ATTR
+        '''
+
+    def p_list_comprehension(self, args):
+        """
+        expr ::= list_compr
+        list_compr ::= BUILD_LIST_0 list_iter
+
+        list_iter ::= list_for
+        list_iter ::= list_if
+        list_iter ::= list_if_not
+        list_iter ::= lc_body
+
+        _come_from ::= COME_FROM
+        _come_from ::=
+
+        list_if ::= expr jmp_false list_iter
+        list_if_not ::= expr jmp_true list_iter
+
+        lc_body ::= expr LIST_APPEND
+        """
+
+    def p_setcomp(self, args):
+        """
+        expr ::= setcomp
+
+        setcomp ::= LOAD_SETCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
+
+        stmt ::= setcomp_func
+
+        setcomp_func ::= BUILD_SET_0 LOAD_FAST FOR_ITER designator comp_iter
+                JUMP_BACK RETURN_VALUE RETURN_LAST
+
+        comp_iter ::= comp_if
+        comp_iter ::= comp_ifnot
+        comp_iter ::= comp_for
+        comp_iter ::= comp_body
+        comp_body ::= set_comp_body
+        comp_body ::= gen_comp_body
+        comp_body ::= dict_comp_body
+        set_comp_body ::= expr SET_ADD
+        gen_comp_body ::= expr YIELD_VALUE POP_TOP
+        dict_comp_body ::= expr expr MAP_ADD
+
+        comp_if ::= expr jmp_false comp_iter
+        comp_ifnot ::= expr jmp_true comp_iter
+        """
+
+
 def parse(p, tokens, customize):
     p.add_custom_rules(tokens, customize)
     ast = p.parse(tokens)
@@ -76,20 +245,50 @@ def parse(p, tokens, customize):
     return ast
 
 
-def get_python_parser(version, debug_parser):
-    """
-    Returns parser object for Python version 2 or 3
-    depending on the parameter passed.
+def get_python_parser(version, debug_parser, compile_mode='exec'):
+    """Returns parser object for Python version 2 or 3, 3.2, 3.5on,
+    etc., depending on the parameters passed.  *compile_mode* is either
+    'exec', 'eval', or 'single'. See
+    https://docs.python.org/3.6/library/functions.html#compile for an
+    explanation of the different modes.
     """
+
     if version < 3.0:
         import uncompyle6.parsers.parse2 as parse2
-        p = parse2.Python2Parser(debug_parser)
+        if compile_mode == 'exec':
+            p = parse2.Python2Parser(debug_parser)
+        else:
+            p = parse2.Python2ParserSingle(debug_parser)
     else:
         import uncompyle6.parsers.parse3 as parse3
-        p = parse3.Python3Parser(debug_parser)
+        if version == 3.2:
+            if compile_mode == 'exec':
+                p = parse3.Python32Parser(debug_parser)
+            else:
+                p = parse3.Python32ParserSingle(debug_parser)
+        elif version >= 3.5:
+            if compile_mode == 'exec':
+                p = parse3.Python35onParser(debug_parser)
+            else:
+                p = parse3.Python35onParserSingle(debug_parser)
+        else:
+            if compile_mode == 'exec':
+                p = parse3.Python3Parser(debug_parser)
+            else:
+                p = parse3.Python3ParserSingle(debug_parser)
     p.version = version
     return p
 
+class PythonParserSingle(PythonParser):
+    def p_call_stmt(self, args):
+        '''
+        # single-mode compilation. Eval-mode interactive compilation
+        # drops the last rule.
+
+        call_stmt ::= expr POP_TOP
+        call_stmt ::= expr PRINT_EXPR
+        '''
+
 def python_parser(version, co, out=sys.stdout, showasm=False,
                   parser_debug=PARSER_DEFAULT_DEBUG):
     assert iscode(co)
@@ -105,7 +304,7 @@ def python_parser(version, co, out=sys.stdout, showasm=False,
 
 if __name__ == '__main__':
     def parse_test(co):
-        from uncompyl6 import PYTHON_VERSION
+        from uncompyle6 import PYTHON_VERSION
         ast = python_parser(PYTHON_VERSION, co, showasm=True)
         print(ast)
         return

uncompyle6/parsers/astnode.py

@@ -10,8 +10,8 @@ else:
 
 
 class AST(UserList):
-    def __init__(self, type, kids=[]):
-        self.type = intern(type)
+    def __init__(self, kind, kids=[]):
+        self.type = intern(kind)
         UserList.__init__(self, kids)
 
     def isNone(self):
@@ -28,7 +28,8 @@ class AST(UserList):
         else:
             return self.type == o
 
-    def __hash__(self):            return hash(self.type)
+    def __hash__(self):
+        return hash(self.type)
 
     def __repr__(self, indent=''):
         rv = str(self.type)

uncompyle6/parsers/parse2.py

@@ -17,138 +17,31 @@ that a later phase can tern into a sequence of ASCII text.
 
 from __future__ import print_function
 
-from uncompyle6.parser import PythonParser, nop_func
+from uncompyle6.parser import PythonParser, PythonParserSingle, nop_func
 from uncompyle6.parsers.astnode import AST
-from uncompyle6.parsers.spark import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6 import PYTHON3
 
 class Python2Parser(PythonParser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        GenericASTBuilder.__init__(self, AST, 'stmts', debug=debug_parser)
+        if PYTHON3:
+            super().__init__(AST, 'stmts', debug=debug_parser)
+        else:
+            super(Python2Parser, self).__init__(AST, 'stmts', debug=debug_parser)
         self.customized = {}
 
-    def p_funcdef(self, args):
-        '''
-        stmt ::= funcdef
-        funcdef ::= mkfunc designator
-        stmt ::= funcdefdeco
-        funcdefdeco ::= mkfuncdeco designator
-        mkfuncdeco ::= expr mkfuncdeco CALL_FUNCTION_1
-        mkfuncdeco ::= expr mkfuncdeco0 CALL_FUNCTION_1
-        mkfuncdeco0 ::= mkfunc
-        load_closure ::= load_closure LOAD_CLOSURE
-        load_closure ::= LOAD_CLOSURE
-        '''
-
-    def p_list_comprehension(self, args):
-        '''
-        expr ::= list_compr
-        list_compr ::= BUILD_LIST_0 list_iter
-
-        list_iter ::= list_for
-        list_iter ::= list_if
-        list_iter ::= list_if_not
-        list_iter ::= lc_body
-
-        _come_from ::= COME_FROM
-        _come_from ::=
-
+    def p_list_comprehension2(self, args):
+        """
         list_for ::= expr _for designator list_iter JUMP_BACK
-        list_if ::= expr jmp_false list_iter
-        list_if_not ::= expr jmp_true list_iter
+        """
 
-        lc_body ::= expr LIST_APPEND
+    def p_setcomp2(self, args):
         '''
-
-    def p_setcomp(self, args):
-        '''
-        expr ::= setcomp
-
-        setcomp ::= LOAD_SETCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-
-        stmt ::= setcomp_func
-
-        setcomp_func ::= BUILD_SET_0 LOAD_FAST FOR_ITER designator comp_iter
-                JUMP_BACK RETURN_VALUE RETURN_LAST
-
-        comp_iter ::= comp_if
-        comp_iter ::= comp_ifnot
-        comp_iter ::= comp_for
-        comp_iter ::= comp_body
-        comp_body ::= set_comp_body
-        comp_body ::= gen_comp_body
-        comp_body ::= dict_comp_body
-        set_comp_body ::= expr SET_ADD
-        gen_comp_body ::= expr YIELD_VALUE POP_TOP
-        dict_comp_body ::= expr expr MAP_ADD
-
-        comp_if ::= expr jmp_false comp_iter
-        comp_ifnot ::= expr jmp_true comp_iter
+        # This is different in python3 - should it be?
         comp_for ::= expr _for designator comp_iter JUMP_BACK
         '''
 
-    def p_genexpr(self, args):
-        '''
-        expr ::= genexpr
-
-        genexpr ::= LOAD_GENEXPR MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-
-        stmt ::= genexpr_func
-
-        genexpr_func ::= LOAD_FAST FOR_ITER designator comp_iter JUMP_BACK
-        '''
-
-    def p_dictcomp(self, args):
-        '''
-        expr ::= dictcomp
-        dictcomp ::= LOAD_DICTCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-        stmt ::= dictcomp_func
-
-        dictcomp_func ::= BUILD_MAP LOAD_FAST FOR_ITER designator
-                comp_iter JUMP_BACK RETURN_VALUE RETURN_LAST
-
-        '''
-
-    def p_augmented_assign(self, args):
-        '''
-        stmt ::= augassign1
-        stmt ::= augassign2
-        augassign1 ::= expr expr inplace_op designator
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SUBSCR
-        augassign1 ::= expr expr inplace_op ROT_TWO   STORE_SLICE+0
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+1
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+2
-        augassign1 ::= expr expr inplace_op ROT_FOUR  STORE_SLICE+3
-        augassign2 ::= expr DUP_TOP LOAD_ATTR expr
-                inplace_op ROT_TWO   STORE_ATTR
-
-        inplace_op ::= INPLACE_ADD
-        inplace_op ::= INPLACE_SUBTRACT
-        inplace_op ::= INPLACE_MULTIPLY
-        inplace_op ::= INPLACE_DIVIDE
-        inplace_op ::= INPLACE_TRUE_DIVIDE
-        inplace_op ::= INPLACE_FLOOR_DIVIDE
-        inplace_op ::= INPLACE_MODULO
-        inplace_op ::= INPLACE_POWER
-        inplace_op ::= INPLACE_LSHIFT
-        inplace_op ::= INPLACE_RSHIFT
-        inplace_op ::= INPLACE_AND
-        inplace_op ::= INPLACE_XOR
-        inplace_op ::= INPLACE_OR
-        '''
-
-    def p_assign(self, args):
-        '''
-        stmt ::= assign
-        assign ::= expr DUP_TOP designList
-        assign ::= expr designator
-
-        stmt ::= assign2
-        stmt ::= assign3
-        assign2 ::= expr expr ROT_TWO designator designator
-        assign3 ::= expr expr expr ROT_THREE ROT_TWO designator designator designator
-        '''
-
     def p_print(self, args):
         '''
         stmt ::= print_items_stmt
@@ -178,41 +71,8 @@ class Python2Parser(PythonParser):
         print_to_item ::= DUP_TOP expr ROT_TWO PRINT_ITEM_TO
         '''
 
-    def p_import20(self, args):
-        '''
-        stmt ::= importstmt
-        stmt ::= importfrom
-        stmt ::= importstar
-        stmt ::= importmultiple
-
-        importlist2 ::= importlist2 import_as
-        importlist2 ::= import_as
-        import_as ::= IMPORT_NAME designator
-        import_as ::= IMPORT_NAME load_attrs designator
-        import_as ::= IMPORT_FROM designator
-
-        importstmt ::= LOAD_CONST LOAD_CONST import_as
-        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME IMPORT_STAR
-        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist2 POP_TOP
-        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT IMPORT_STAR
-        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT importlist2 POP_TOP
-        importmultiple ::= LOAD_CONST LOAD_CONST import_as imports_cont
-
-        imports_cont ::= imports_cont import_cont
-        imports_cont ::= import_cont
-        import_cont ::= LOAD_CONST LOAD_CONST import_as_cont
-        import_as_cont ::= IMPORT_NAME_CONT designator
-        import_as_cont ::= IMPORT_NAME_CONT load_attrs designator
-        import_as_cont ::= IMPORT_FROM designator
-
-        load_attrs ::= LOAD_ATTR
-        load_attrs ::= load_attrs LOAD_ATTR
-        '''
-
     def p_grammar(self, args):
         '''
-        stmts ::= stmts sstmt
-        stmts ::= sstmt
         sstmt ::= stmt
         sstmt ::= ifelsestmtr
         sstmt ::= return_stmt RETURN_LAST
@@ -581,8 +441,8 @@ class Python2Parser(PythonParser):
         _mklambda ::= load_closure mklambda
         _mklambda ::= mklambda
 
-        or   ::= expr jmp_true expr _come_from
         or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
+        or   ::= expr jmp_true expr _come_from
         and  ::= expr jmp_false expr _come_from
         and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
         and2 ::= _jump jmp_false COME_FROM expr COME_FROM
@@ -601,10 +461,10 @@ class Python2Parser(PythonParser):
         ret_expr_or_cond ::= ret_cond
         ret_expr_or_cond ::= ret_cond_not
 
-        ret_and  ::= expr jmp_false ret_expr_or_cond COME_FROM
-        ret_or   ::= expr jmp_true ret_expr_or_cond COME_FROM
-        ret_cond ::= expr jmp_false expr RETURN_END_IF ret_expr_or_cond
-        ret_cond_not ::= expr jmp_true expr RETURN_END_IF ret_expr_or_cond
+        ret_and  ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
+        ret_or   ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+        ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
+        ret_cond_not ::= expr POP_JUMP_IF_TRUE expr RETURN_END_IF ret_expr_or_cond
 
         stmt ::= return_lambda
         stmt ::= conditional_lambda
@@ -710,3 +570,6 @@ class Python2Parser(PythonParser):
             else:
                 raise Exception('unknown customize token %s' % k)
             self.addRule(rule, nop_func)
+
+class Python2ParserSingle(Python2Parser, PythonParserSingle):
+    pass

uncompyle6/parsers/parse3.py

@@ -17,15 +17,19 @@ that a later phase can tern into a sequence of ASCII text.
 
 from __future__ import print_function
 
-from uncompyle6.parser import PythonParser, nop_func
+from uncompyle6.parser import PythonParser, PythonParserSingle, nop_func
 from uncompyle6.parsers.astnode import AST
-from uncompyle6.parsers.spark import GenericASTBuilder, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6 import PYTHON3
 
 class Python3Parser(PythonParser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
         self.added_rules = set()
-        GenericASTBuilder.__init__(self, AST, 'stmts', debug=debug_parser)
+        if PYTHON3:
+            super().__init__(AST, 'stmts', debug=debug_parser)
+        else:
+            super(Python3Parser, self).__init__(AST, 'stmts', debug=debug_parser)
         self.new_rules = set()
 
     def add_unique_rule(self, rule, opname, count, customize):
@@ -38,198 +42,29 @@ class Python3Parser(PythonParser):
             pass
         return
 
-    def p_funcdef(self, args):
-        '''
-        stmt ::= funcdef
-        funcdef ::= mkfunc designator
-        stmt ::= funcdefdeco
-        funcdefdeco ::= mkfuncdeco designator
-        mkfuncdeco ::= expr mkfuncdeco CALL_FUNCTION_1
-        mkfuncdeco ::= expr mkfuncdeco0 CALL_FUNCTION_1
-        mkfuncdeco0 ::= mkfunc
-        load_closure ::= load_closure LOAD_CLOSURE
-        load_closure ::= LOAD_CLOSURE
-        '''
-
-    def p_list_comprehension(self, args):
-        '''
+    def p_list_comprehension3(self, args):
+        """
         # Python3 scanner adds LOAD_LISTCOMP. Python3 does list comprehension like
         # other comprehensions (set, dictionary).
 
-        # listcomp is a custom rule
+        # listcomp is a custom Python3 rule
         expr ::= listcomp
 
-        expr ::= list_compr
-        list_compr ::= BUILD_LIST_0 list_iter
-
-        list_iter ::= list_for
-        list_iter ::= list_if
-        list_iter ::= list_if_not
-        list_iter ::= lc_body
-
-        _come_from ::= COME_FROM
-        _come_from ::=
-
         list_for ::= expr FOR_ITER designator list_iter JUMP_BACK
-        list_if ::= expr jmp_false list_iter
-        list_if_not ::= expr jmp_true list_iter
 
-        lc_body ::= expr LIST_APPEND
-        '''
+        # See also common Python p_list_comprehension
+        """
 
-    def p_setcomp(self, args):
-        '''
-        expr ::= setcomp
-
-        setcomp ::= LOAD_SETCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-
-        stmt ::= setcomp_func
-
-        setcomp_func ::= BUILD_SET_0 LOAD_FAST FOR_ITER designator comp_iter
-                JUMP_BACK RETURN_VALUE RETURN_LAST
-
-        comp_iter ::= comp_if
-        comp_iter ::= comp_ifnot
-        comp_iter ::= comp_for
-        comp_iter ::= comp_body
-        comp_body ::= set_comp_body
-        comp_body ::= gen_comp_body
-        comp_body ::= dict_comp_body
-        set_comp_body ::= expr SET_ADD
-        gen_comp_body ::= expr YIELD_VALUE POP_TOP
-        dict_comp_body ::= expr expr MAP_ADD
-
-        comp_if ::= expr jmp_false comp_iter
-        comp_ifnot ::= expr jmp_true comp_iter
+    def p_setcomp3(self, args):
+        """
+        # This is different in Python 2 - should it be?
         comp_for ::= expr _for designator comp_iter JUMP_ABSOLUTE
-        '''
 
-    def p_genexpr(self, args):
-        '''
-        expr ::= genexpr
-
-        genexpr ::= LOAD_GENEXPR MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-
-        stmt ::= genexpr_func
-
-        genexpr_func ::= LOAD_FAST FOR_ITER designator comp_iter JUMP_BACK
-        '''
-
-    def p_dictcomp(self, args):
-        '''
-        expr ::= dictcomp
-        dictcomp ::= LOAD_DICTCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-        stmt ::= dictcomp_func
-
-        dictcomp_func ::= BUILD_MAP LOAD_FAST FOR_ITER designator
-                comp_iter JUMP_BACK RETURN_VALUE RETURN_LAST
-
-        '''
-
-    def p_augmented_assign(self, args):
-        '''
-        stmt ::= augassign1
-        stmt ::= augassign2
-        augassign1 ::= expr expr inplace_op designator
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SUBSCR
-        augassign1 ::= expr expr inplace_op ROT_TWO   STORE_SLICE+0
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+1
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+2
-        augassign1 ::= expr expr inplace_op ROT_FOUR  STORE_SLICE+3
-        augassign2 ::= expr DUP_TOP LOAD_ATTR expr
-                inplace_op ROT_TWO   STORE_ATTR
-
-        inplace_op ::= INPLACE_ADD
-        inplace_op ::= INPLACE_SUBTRACT
-        inplace_op ::= INPLACE_MULTIPLY
-        inplace_op ::= INPLACE_DIVIDE
-        inplace_op ::= INPLACE_TRUE_DIVIDE
-        inplace_op ::= INPLACE_FLOOR_DIVIDE
-        inplace_op ::= INPLACE_MODULO
-        inplace_op ::= INPLACE_POWER
-        inplace_op ::= INPLACE_LSHIFT
-        inplace_op ::= INPLACE_RSHIFT
-        inplace_op ::= INPLACE_AND
-        inplace_op ::= INPLACE_XOR
-        inplace_op ::= INPLACE_OR
-        '''
-
-    def p_assign(self, args):
-        '''
-        stmt ::= assign
-        assign ::= expr DUP_TOP designList
-        assign ::= expr designator
-
-        stmt ::= assign2
-        stmt ::= assign3
-        assign2 ::= expr expr ROT_TWO designator designator
-        assign3 ::= expr expr expr ROT_THREE ROT_TWO designator designator designator
-        '''
-
-    # Python3 doesn't have a built-in print.
-    # def p_print(self, args):
-    #     '''
-    #     stmt ::= print_items_stmt
-    #     stmt ::= print_nl
-    #     stmt ::= print_items_nl_stmt
-
-    #     print_items_stmt ::= expr PRINT_ITEM print_items_opt
-    #     print_items_nl_stmt ::= expr PRINT_ITEM print_items_opt PRINT_NEWLINE_CONT
-    #     print_items_opt ::= print_items
-    #     print_items_opt ::=
-    #     print_items ::= print_items print_item
-    #     print_items ::= print_item
-    #     print_item ::= expr PRINT_ITEM_CONT
-    #     print_nl ::= PRINT_NEWLINE
-    #     '''
-
-    # def p_print_to(self, args):
-    #     '''
-    #     stmt ::= print_to
-    #     stmt ::= print_to_nl
-    #     stmt ::= print_nl_to
-    #     print_to ::= expr print_to_items POP_TOP
-    #     print_to_nl ::= expr print_to_items PRINT_NEWLINE_TO
-    #     print_nl_to ::= expr PRINT_NEWLINE_TO
-    #     print_to_items ::= print_to_items print_to_item
-    #     print_to_items ::= print_to_item
-    #     print_to_item ::= DUP_TOP expr ROT_TWO PRINT_ITEM_TO
-    #     '''
-
-    def p_import20(self, args):
-        '''
-        stmt ::= importstmt
-        stmt ::= importfrom
-        stmt ::= importstar
-        stmt ::= importmultiple
-
-        importlist2 ::= importlist2 import_as
-        importlist2 ::= import_as
-        import_as ::= IMPORT_NAME designator
-        import_as ::= IMPORT_NAME load_attrs designator
-        import_as ::= IMPORT_FROM designator
-
-        importstmt ::= LOAD_CONST LOAD_CONST import_as
-        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME IMPORT_STAR
-        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist2 POP_TOP
-        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT IMPORT_STAR
-        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT importlist2 POP_TOP
-        importmultiple ::= LOAD_CONST LOAD_CONST import_as imports_cont
-
-        imports_cont ::= imports_cont import_cont
-        imports_cont ::= import_cont
-        import_cont ::= LOAD_CONST LOAD_CONST import_as_cont
-        import_as_cont ::= IMPORT_NAME_CONT designator
-        import_as_cont ::= IMPORT_NAME_CONT load_attrs designator
-        import_as_cont ::= IMPORT_FROM designator
-
-        load_attrs ::= LOAD_ATTR
-        load_attrs ::= load_attrs LOAD_ATTR
-        '''
+        # See also common Python p_setcomp
+        """
 
     def p_grammar(self, args):
-        '''stmts ::= stmts sstmt
-        stmts ::= sstmt
+        '''
         sstmt ::= stmt
         sstmt ::= ifelsestmtr
         sstmt ::= return_stmt RETURN_LAST
@@ -301,7 +136,6 @@ class Python3Parser(PythonParser):
 
         stmt ::= classdef
         stmt ::= call_stmt
-        call_stmt ::= expr POP_TOP
 
         stmt ::= return_stmt
         return_stmt ::= ret_expr RETURN_VALUE
@@ -631,10 +465,10 @@ class Python3Parser(PythonParser):
         ret_expr_or_cond ::= ret_cond
         ret_expr_or_cond ::= ret_cond_not
 
-        ret_and  ::= expr jmp_false ret_expr_or_cond COME_FROM
-        ret_or   ::= expr jmp_true ret_expr_or_cond COME_FROM
-        ret_cond ::= expr jmp_false expr RETURN_END_IF ret_expr_or_cond
-        ret_cond_not ::= expr jmp_true expr RETURN_END_IF ret_expr_or_cond
+        ret_and  ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
+        ret_or   ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+        ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
+        ret_cond_not ::= expr POP_JUMP_IF_TRUE expr RETURN_END_IF ret_expr_or_cond
 
         stmt ::= return_lambda
         stmt ::= conditional_lambda
@@ -781,6 +615,12 @@ class Python3Parser(PythonParser):
             elif opname_base in ('BUILD_LIST', 'BUILD_TUPLE', 'BUILD_SET'):
                 rule = 'build_list ::= ' + 'expr ' * token.attr + opname
                 self.add_unique_rule(rule, opname, token.attr, customize)
+            elif self.version >= 3.5 and opname_base == 'BUILD_MAP':
+                kvlist_n = "kvlist_%s" % token.attr
+                rule = kvlist_n + ' ::= ' + 'expr ' * (token.attr*2)
+                self.add_unique_rule(rule, opname, token.attr, customize)
+                rule = "mapexpr ::=  %s %s" % (kvlist_n, opname)
+                self.add_unique_rule(rule, opname, token.attr, customize)
             elif opname_base in ('UNPACK_TUPLE', 'UNPACK_SEQUENCE'):
                 rule = 'unpack ::= ' + opname + ' designator' * token.attr
                 self.add_unique_rule(rule, opname, token.attr, customize)
@@ -818,3 +658,26 @@ class Python3Parser(PythonParser):
                         % ('expr ' * token.attr, opname))
                 self.add_unique_rule(rule, opname, token.attr, customize)
         return
+
+class Python32Parser(Python3Parser):
+    def p_32(self, args):
+        """
+        # Store locals is only used in Python 3.2
+        designator ::= STORE_LOCALS
+        """
+
+class Python3ParserSingle(Python3Parser, PythonParserSingle):
+    pass
+
+class Python32ParserSingle(Python32Parser, PythonParserSingle):
+    pass
+
+class Python35onParser(Python3Parser):
+    def p_35on(self, args):
+        """
+        # this optimization is only used in Python 3.5 and beyond
+        _ifstmts_jump ::= c_stmts_opt
+        """
+
+class Python35onParserSingle(Python35onParser, PythonParserSingle):
+    pass

uncompyle6/parsers/spark.py

@@ -1,737 +0,0 @@
-"""
-Copyright (c) 1998-2002 John Aycock
-Copyright (c) 2015 Rocky Bernstein
-
-  Permission is hereby granted, free of charge, to any person obtaining
-  a copy of this software and associated documentation files (the
-  "Software"), to deal in the Software without restriction, including
-  without limitation the rights to use, copy, modify, merge, publish,
-  distribute, sublicense, and/or sell copies of the Software, and to
-  permit persons to whom the Software is furnished to do so, subject to
-  the following conditions:
-
-  The above copyright notice and this permission notice shall be
-  included in all copies or substantial portions of the Software.
-
-  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-  IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-  CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-  TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-  SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-"""
-from __future__ import print_function
-
-import os, re
-
-__version__ = 'SPARK-1.0 Python3 compatible'
-
-def _namelist(instance):
-    namelist, namedict, classlist = [], {}, [instance.__class__]
-    for c in classlist:
-        for b in c.__bases__:
-            classlist.append(b)
-        for name in list(c.__dict__.keys()):
-            if name not in namedict:
-                namelist.append(name)
-                namedict[name] = 1
-    return namelist
-
-class _State:
-    '''
-    Extracted from GenericParser and made global so that [un]picking works.
-    '''
-    def __init__(self, stateno, items):
-        self.T, self.complete, self.items = [], [], items
-        self.stateno = stateno
-
-# DEFAULT_DEBUG = {'rules': True, 'transition': True, 'reduce' : True}
-# DEFAULT_DEBUG = {'rules': False, 'transition': False, 'reduce' : True}
-DEFAULT_DEBUG = {'rules': False, 'transition': False, 'reduce': False}
-
-class GenericParser:
-    '''
-    An Earley parser, as per J. Earley, "An Efficient Context-Free
-    Parsing Algorithm", CACM 13(2), pp. 94-102.  Also J. C. Earley,
-    "An Efficient Context-Free Parsing Algorithm", Ph.D. thesis,
-    Carnegie-Mellon University, August 1968.  New formulation of
-    the parser according to J. Aycock, "Practical Earley Parsing
-    and the SPARK Toolkit", Ph.D. thesis, University of Victoria,
-    2001, and J. Aycock and R. N. Horspool, "Practical Earley
-    Parsing", unpublished paper, 2001.
-    '''
-
-    def __init__(self, start, debug=DEFAULT_DEBUG):
-        self.rules = {}
-        self.rule2func = {}
-        self.rule2name = {}
-        self.collectRules()
-        self.augment(start)
-        self.ruleschanged = True
-        self.debug = debug
-
-    _NULLABLE = '\e_'
-    _START = 'START'
-    _BOF = '|-'
-
-    #
-    #  When pickling, take the time to generate the full state machine;
-    #  some information is then extraneous, too.  Unfortunately we
-    #  can't save the rule2func map.
-    #
-    def __getstate__(self):
-        if self.ruleschanged:
-            #
-            #  XXX - duplicated from parse()
-            #
-            self.computeNull()
-            self.newrules = {}
-            self.new2old = {}
-            self.makeNewRules()
-            self.ruleschanged = False
-            self.edges, self.cores = {}, {}
-            self.states = { 0: self.makeState0() }
-            self.makeState(0, self._BOF)
-        #
-        #  XXX - should find a better way to do this..
-        #
-        changes = 1
-        while changes:
-            changes = 0
-            for k, v in list(self.edges.items()):
-                if v is None:
-                    state, sym = k
-                    if state in self.states:
-                        self.goto(state, sym)
-                        changes = 1
-        rv = self.__dict__.copy()
-        for s in list(self.states.values()):
-            del s.items
-        del rv['rule2func']
-        del rv['nullable']
-        del rv['cores']
-        return rv
-
-    def __setstate__(self, D):
-        self.rules = {}
-        self.rule2func = {}
-        self.rule2name = {}
-        self.collectRules()
-        start = D['rules'][self._START][0][1][1] # Blech.
-        self.augment(start)
-        D['rule2func'] = self.rule2func
-        D['makeSet'] = self.makeSet_fast
-        self.__dict__ = D
-
-    #
-    #  A hook for GenericASTBuilder and GenericASTMatcher.  Mess
-    #  thee not with this; nor shall thee toucheth the _preprocess
-    #  argument to addRule.
-    #
-    def preprocess(self, rule, func):
-        return rule, func
-
-    def addRule(self, doc, func, _preprocess=True):
-        """Add a grammar rules to self.rules, self.rule2func and self.rule2name"""
-        fn = func
-
-        # remove blanks lines and comment lines, e.g. lines starting with "#"
-        doc = os.linesep.join([s for s in doc.splitlines() if s and not re.match("^\s*#", s)])
-
-        rules = doc.split()
-
-        index = []
-        for i in range(len(rules)):
-            if rules[i] == '::=':
-                index.append(i-1)
-        index.append(len(rules))
-
-        for i in range(len(index)-1):
-            lhs = rules[index[i]]
-            rhs = rules[index[i]+2:index[i+1]]
-            rule = (lhs, tuple(rhs))
-
-            if _preprocess:
-                rule, fn = self.preprocess(rule, func)
-
-            if lhs in self.rules:
-                self.rules[lhs].append(rule)
-            else:
-                self.rules[lhs] = [ rule ]
-            self.rule2func[rule] = fn
-            self.rule2name[rule] = func.__name__[2:]
-        self.ruleschanged = True
-
-    def collectRules(self):
-        for name in _namelist(self):
-            if name[:2] == 'p_':
-                func = getattr(self, name)
-                doc = func.__doc__
-                self.addRule(doc, func)
-
-    def augment(self, start):
-        rule = '%s ::= %s %s' % (self._START, self._BOF, start)
-        self.addRule(rule, lambda args: args[1], 0)
-
-    def computeNull(self):
-        self.nullable = {}
-        tbd = []
-
-        for rulelist in list(self.rules.values()):
-            lhs = rulelist[0][0]
-            self.nullable[lhs] = 0
-            for rule in rulelist:
-                rhs = rule[1]
-                if len(rhs) == 0:
-                    self.nullable[lhs] = 1
-                    continue
-                #
-                #  We only need to consider rules which
-                #  consist entirely of nonterminal symbols.
-                #  This should be a savings on typical
-                #  grammars.
-                #
-                for sym in rhs:
-                    if sym not in self.rules:
-                        break
-                else:
-                    tbd.append(rule)
-        changes = 1
-        while changes:
-            changes = 0
-            for lhs, rhs in tbd:
-                if self.nullable[lhs]:
-                    continue
-                for sym in rhs:
-                    if not self.nullable[sym]:
-                        break
-                else:
-                    self.nullable[lhs] = 1
-                    changes = 1
-
-    def makeState0(self):
-        s0 = _State(0, [])
-        for rule in self.newrules[self._START]:
-            s0.items.append((rule, 0))
-        return s0
-
-    def finalState(self, tokens):
-        #
-        #  Yuck.
-        #
-        if len(self.newrules[self._START]) == 2 and len(tokens) == 0:
-            return 1
-        start = self.rules[self._START][0][1][1]
-        return self.goto(1, start)
-
-    def makeNewRules(self):
-        worklist = []
-        for rulelist in list(self.rules.values()):
-            for rule in rulelist:
-                worklist.append((rule, 0, 1, rule))
-
-        for rule, i, candidate, oldrule in worklist:
-            lhs, rhs = rule
-            n = len(rhs)
-            while i < n:
-                sym = rhs[i]
-                if sym not in self.rules or \
-                    not self.nullable[sym]:
-                        candidate = 0
-                        i = i + 1
-                        continue
-
-                newrhs = list(rhs)
-                newrhs[i] = self._NULLABLE+sym
-                newrule = (lhs, tuple(newrhs))
-                worklist.append((newrule, i+1,
-                                 candidate, oldrule))
-                candidate = 0
-                i = i + 1
-            else:
-                if candidate:
-                    lhs = self._NULLABLE+lhs
-                    rule = (lhs, rhs)
-                if lhs in self.newrules:
-                    self.newrules[lhs].append(rule)
-                else:
-                    self.newrules[lhs] = [ rule ]
-                self.new2old[rule] = oldrule
-
-    def typestring(self, token):
-        return None
-
-    def error(self, token):
-        print("Syntax error at or near `%s' token" % token)
-        raise SystemExit
-
-    def parse(self, tokens):
-        sets = [ [(1, 0), (2, 0)] ]
-        self.links = {}
-
-        if self.ruleschanged:
-            self.computeNull()
-            self.newrules = {}
-            self.new2old = {}
-            self.makeNewRules()
-            self.ruleschanged = False
-            self.edges, self.cores = {}, {}
-            self.states = { 0: self.makeState0() }
-            self.makeState(0, self._BOF)
-
-        for i in range(len(tokens)):
-            sets.append([])
-
-            if sets[i] == []:
-                break
-            self.makeSet(tokens[i], sets, i)
-        else:
-            sets.append([])
-            self.makeSet(None, sets, len(tokens))
-
-        finalitem = (self.finalState(tokens), 0)
-        if finalitem not in sets[-2]:
-            if len(tokens) > 0:
-                self.error(tokens[i-1])
-            else:
-                self.error(None)
-
-        return self.buildTree(self._START, finalitem,
-                    tokens, len(sets)-2)
-
-    def isnullable(self, sym):
-        #
-        #  For symbols in G_e only.  If we weren't supporting 1.5,
-        #  could just use sym.startswith().
-        #
-        return self._NULLABLE == sym[0:len(self._NULLABLE)]
-
-    def skip(self, xxx_todo_changeme, pos=0):
-        (lhs, rhs) = xxx_todo_changeme
-        n = len(rhs)
-        while pos < n:
-            if not self.isnullable(rhs[pos]):
-                break
-            pos = pos + 1
-        return pos
-
-    def makeState(self, state, sym):
-        assert sym is not None
-        # print(sym) # debug
-        #
-        #  Compute \epsilon-kernel state's core and see if
-        #  it exists already.
-        #
-        kitems = []
-        for rule, pos in self.states[state].items:
-            lhs, rhs = rule
-            if rhs[pos:pos+1] == (sym,):
-                kitems.append((rule, self.skip(rule, pos+1)))
-
-        tcore = tuple(sorted(kitems))
-        if tcore in self.cores:
-            return self.cores[tcore]
-        #
-        #  Nope, doesn't exist.  Compute it and the associated
-        #  \epsilon-nonkernel state together; we'll need it right away.
-        #
-        k = self.cores[tcore] = len(self.states)
-        K, NK = _State(k, kitems), _State(k+1, [])
-        self.states[k] = K
-        predicted = {}
-
-        edges = self.edges
-        rules = self.newrules
-        for X in K, NK:
-            worklist = X.items
-            for item in worklist:
-                rule, pos = item
-                lhs, rhs = rule
-                if pos == len(rhs):
-                    X.complete.append(rule)
-                    continue
-
-                nextSym = rhs[pos]
-                key = (X.stateno, nextSym)
-                if nextSym not in rules:
-                    if key not in edges:
-                        edges[key] = None
-                        X.T.append(nextSym)
-                else:
-                    edges[key] = None
-                    if nextSym not in predicted:
-                        predicted[nextSym] = 1
-                        for prule in rules[nextSym]:
-                            ppos = self.skip(prule)
-                            new = (prule, ppos)
-                            NK.items.append(new)
-            #
-            #  Problem: we know K needs generating, but we
-            #  don't yet know about NK.  Can't commit anything
-            #  regarding NK to self.edges until we're sure.  Should
-            #  we delay committing on both K and NK to avoid this
-            #  hacky code?  This creates other problems..
-            #
-            if X is K:
-                edges = {}
-
-        if NK.items == []:
-            return k
-
-        #
-        #  Check for \epsilon-nonkernel's core.  Unfortunately we
-        #  need to know the entire set of predicted nonterminals
-        #  to do this without accidentally duplicating states.
-        #
-        tcore = tuple(sorted(predicted.keys()))
-        if tcore in self.cores:
-            self.edges[(k, None)] = self.cores[tcore]
-            return k
-
-        nk = self.cores[tcore] = self.edges[(k, None)] = NK.stateno
-        self.edges.update(edges)
-        self.states[nk] = NK
-        return k
-
-    def goto(self, state, sym):
-        key = (state, sym)
-        if key not in self.edges:
-            #
-            #  No transitions from state on sym.
-            #
-            return None
-
-        rv = self.edges[key]
-        if rv is None:
-            #
-            #  Target state isn't generated yet.  Remedy this.
-            #
-            rv = self.makeState(state, sym)
-            self.edges[key] = rv
-        return rv
-
-    def gotoT(self, state, t):
-        if self.debug['rules']: print("Terminal", t, state)
-        return [self.goto(state, t)]
-
-    def gotoST(self, state, st):
-        if self.debug['transition']: print("GotoST", st, state)
-        rv = []
-        for t in self.states[state].T:
-            if st == t:
-                rv.append(self.goto(state, t))
-        return rv
-
-    def add(self, set, item, i=None, predecessor=None, causal=None):
-        if predecessor is None:
-            if item not in set:
-                set.append(item)
-        else:
-            key = (item, i)
-            if item not in set:
-                self.links[key] = []
-                set.append(item)
-            self.links[key].append((predecessor, causal))
-
-    def makeSet(self, token, sets, i):
-        cur, next = sets[i], sets[i+1]
-
-        ttype = token is not None and self.typestring(token) or None
-        if ttype is not None:
-            fn, arg = self.gotoT, ttype
-        else:
-            fn, arg = self.gotoST, token
-
-        for item in cur:
-            ptr = (item, i)
-            state, parent = item
-            add = fn(state, arg)
-            for k in add:
-                if k is not None:
-                    self.add(next, (k, parent), i+1, ptr)
-                    nk = self.goto(k, None)
-                    if nk is not None:
-                        self.add(next, (nk, i+1))
-
-            if parent == i:
-                continue
-
-            for rule in self.states[state].complete:
-                lhs, rhs = rule
-                if self.debug['reduce']:
-                    print("%s ::= %s" % (lhs, ' '.join(rhs)))
-                for pitem in sets[parent]:
-                    pstate, pparent = pitem
-                    k = self.goto(pstate, lhs)
-                    if k is not None:
-                        why = (item, i, rule)
-                        pptr = (pitem, parent)
-                        self.add(cur, (k, pparent),
-                                i, pptr, why)
-                        nk = self.goto(k, None)
-                        if nk is not None:
-                            self.add(cur, (nk, i))
-
-    def makeSet_fast(self, token, sets, i):
-        #
-        #  Call *only* when the entire state machine has been built!
-        #  It relies on self.edges being filled in completely, and
-        #  then duplicates and inlines code to boost speed at the
-        #  cost of extreme ugliness.
-        #
-        cur, next = sets[i], sets[i+1]
-        ttype = token is not None and self.typestring(token) or None
-
-        for item in cur:
-            ptr = (item, i)
-            state, parent = item
-            if ttype is not None:
-                k = self.edges.get((state, ttype), None)
-                if k is not None:
-                    # self.add(next, (k, parent), i+1, ptr)
-                    # INLINED --------v
-                    new = (k, parent)
-                    key = (new, i+1)
-                    if new not in next:
-                        self.links[key] = []
-                        next.append(new)
-                    self.links[key].append((ptr, None))
-                    # INLINED --------^
-                    # nk = self.goto(k, None)
-                    nk = self.edges.get((k, None), None)
-                    if nk is not None:
-                        # self.add(next, (nk, i+1))
-                        # INLINED -------------v
-                        new = (nk, i+1)
-                        if new not in next:
-                            next.append(new)
-                        # INLINED ---------------^
-            else:
-                add = self.gotoST(state, token)
-                for k in add:
-                    if k is not None:
-                        self.add(next, (k, parent), i+1, ptr)
-                        # nk = self.goto(k, None)
-                        nk = self.edges.get((k, None), None)
-                        if nk is not None:
-                            self.add(next, (nk, i+1))
-
-            if parent == i:
-                continue
-
-            for rule in self.states[state].complete:
-                lhs, rhs = rule
-                for pitem in sets[parent]:
-                    pstate, pparent = pitem
-                    # k = self.goto(pstate, lhs)
-                    k = self.edges.get((pstate, lhs), None)
-                    if k is not None:
-                        why = (item, i, rule)
-                        pptr = (pitem, parent)
-                        # self.add(cur, (k, pparent), i, pptr, why)
-                        # INLINED ---------v
-                        new = (k, pparent)
-                        key = (new, i)
-                        if new not in cur:
-                            self.links[key] = []
-                            cur.append(new)
-                        self.links[key].append((pptr, why))
-                        # INLINED ----------^
-                        # nk = self.goto(k, None)
-                        nk = self.edges.get((k, None), None)
-                        if nk is not None:
-                            # self.add(cur, (nk, i))
-                            # INLINED ---------v
-                            new = (nk, i)
-                            if new not in cur:
-                                cur.append(new)
-                            # INLINED ----------^
-
-    def predecessor(self, key, causal):
-        for p, c in self.links[key]:
-            if c == causal:
-                return p
-        assert 0
-
-    def causal(self, key):
-        links = self.links[key]
-        if len(links) == 1:
-            return links[0][1]
-        choices = []
-        rule2cause = {}
-        for p, c in links:
-            rule = c[2]
-            choices.append(rule)
-            rule2cause[rule] = c
-        return rule2cause[self.ambiguity(choices)]
-
-    def deriveEpsilon(self, nt):
-        if len(self.newrules[nt]) > 1:
-            rule = self.ambiguity(self.newrules[nt])
-        else:
-            rule = self.newrules[nt][0]
-        # print(rule) # debug
-
-        rhs = rule[1]
-        attr = [None] * len(rhs)
-
-        for i in range(len(rhs)-1, -1, -1):
-            attr[i] = self.deriveEpsilon(rhs[i])
-        return self.rule2func[self.new2old[rule]](attr)
-
-    def buildTree(self, nt, item, tokens, k):
-        if self.debug['rules']:
-            print("NT", nt)
-        state, parent = item
-
-        choices = []
-        for rule in self.states[state].complete:
-            if rule[0] == nt:
-                choices.append(rule)
-        rule = choices[0]
-        if len(choices) > 1:
-            rule = self.ambiguity(choices)
-        # print(rule) # debug
-
-        rhs = rule[1]
-        attr = [None] * len(rhs)
-
-        for i in range(len(rhs)-1, -1, -1):
-            sym = rhs[i]
-            if sym not in self.newrules:
-                if sym != self._BOF:
-                    attr[i] = tokens[k-1]
-                    key = (item, k)
-                    item, k = self.predecessor(key, None)
-            # elif self.isnullable(sym):
-            elif self._NULLABLE == sym[0:len(self._NULLABLE)]:
-                attr[i] = self.deriveEpsilon(sym)
-            else:
-                key = (item, k)
-                why = self.causal(key)
-                attr[i] = self.buildTree(sym, why[0],
-                            tokens, why[1])
-                item, k = self.predecessor(key, why)
-        return self.rule2func[self.new2old[rule]](attr)
-
-    def ambiguity(self, rules):
-        #
-        #  XXX - problem here and in collectRules() if the same rule
-        #  appears in >1 method.  Also undefined results if rules
-        #  causing the ambiguity appear in the same method.
-        #
-        sortlist = []
-        name2index = {}
-        for i in range(len(rules)):
-            lhs, rhs = rule = rules[i]
-            name = self.rule2name[self.new2old[rule]]
-            sortlist.append((len(rhs), name))
-            name2index[name] = i
-        sortlist.sort()
-        list = [a_b[1] for a_b in sortlist]
-        return rules[name2index[self.resolve(list)]]
-
-    def resolve(self, list):
-        '''
-        Resolve ambiguity in favor of the shortest RHS.
-        Since we walk the tree from the top down, this
-        should effectively resolve in favor of a "shift".
-        '''
-        return list[0]
-
-#
-#  GenericASTBuilder automagically constructs a concrete/abstract syntax tree
-#  for a given input.  The extra argument is a class (not an instance!)
-#  which supports the "__setslice__" and "__len__" methods.
-#
-#  XXX - silently overrides any user code in methods.
-#
-
-class GenericASTBuilder(GenericParser):
-    def __init__(self, AST, start, debug=DEFAULT_DEBUG):
-        GenericParser.__init__(self, start, debug=debug)
-        self.AST = AST
-
-    def preprocess(self, rule, func):
-        rebind = lambda lhs, self=self: \
-                lambda args, lhs=lhs, self=self: \
-                self.buildASTNode(args, lhs)
-        lhs, rhs = rule
-        return rule, rebind(lhs)
-
-    def buildASTNode(self, args, lhs):
-        children = []
-        for arg in args:
-            if isinstance(arg, self.AST):
-                children.append(arg)
-            else:
-                children.append(self.terminal(arg))
-        return self.nonterminal(lhs, children)
-
-    def terminal(self, token):
-        return token
-
-    def nonterminal(self, type, args):
-        rv = self.AST(type)
-        rv[:len(args)] = args
-        return rv
-
-class GenericASTTraversalPruningException(BaseException):
-    pass
-
-class GenericASTTraversal:
-    '''
-    GenericASTTraversal is a Visitor pattern according to Design Patterns.  For
-    each node it attempts to invoke the method n_<node type>, falling
-    back onto the default() method if the n_* can't be found.  The preorder
-    traversal also looks for an exit hook named n_<node type>_exit (no default
-    routine is called if it's not found).  To prematurely halt traversal
-    of a subtree, call the prune() method -- this only makes sense for a
-    preorder traversal.  Node type is determined via the typestring() method.
-    '''
-    def __init__(self, ast):
-        self.ast = ast
-
-    def typestring(self, node):
-        return node.type
-
-    def prune(self):
-        raise GenericASTTraversalPruningException
-
-    def preorder(self, node=None):
-        """Walk the tree. For each node with typestring name *name* if the
-        node has a method called n_*name*, call that before walking
-        children. If there is no method define, call a
-        self.default(node) instead. Subclasses of GenericASTTtraversal
-        ill probably want to override this method.
-
-        If the node has a method called *name*_exit, that is called
-        after all children have been called.  So in this sense this
-        function is both preorder and postorder combined.
-        """
-        if node is None:
-            node = self.ast
-
-        try:
-            name = 'n_' + self.typestring(node)
-            if hasattr(self, name):
-                func = getattr(self, name)
-                func(node)
-            else:
-                self.default(node)
-        except GenericASTTraversalPruningException:
-            return
-
-        for kid in node:
-            self.preorder(kid)
-
-        name = name + '_exit'
-        if hasattr(self, name):
-            func = getattr(self, name)
-            func(node)
-
-    def default(self, node):
-        """Default acttion to take on an ASTNode. Our defualt is to do nothing.
-        Subclasses will probably want to define this for other behavior."""
-        pass

uncompyle6/scanners/scanner35.py

@@ -14,7 +14,6 @@ import dis, inspect
 from array import array
 import uncompyle6.scanners.scanner3 as scan3
 
-from uncompyle6 import PYTHON_VERSION
 from uncompyle6.code import iscode
 from uncompyle6.scanner import Token
 
@@ -29,13 +28,10 @@ from uncompyle6.opcodes.opcode_35 import *
 
 class Scanner35(scan3.Scanner3):
 
-    def disassemble(self, co, classname=None, code_objects={}):
-        fn = self.disassemble_built_in if PYTHON_VERSION == 3.4 \
-            else self.disassemble_generic
-        return fn(co, classname, code_objects=code_objects)
-
-    def disassemble_built_in(self, co, classname=None,
-                             code_objects={}):
+    # Note: we can't use built-in disassembly routines, unless
+    # we do post-processing like we do here.
+    def disassemble(self, co, classname=None,
+                    code_objects={}):
         # Container for tokens
         tokens = []
         customize = {}
@@ -116,17 +112,11 @@ class Scanner35(scan3.Scanner3):
                     pattr = const
                     pass
             elif opname in ('BUILD_LIST', 'BUILD_TUPLE', 'BUILD_SET', 'BUILD_SLICE',
+                            'BUILD_MAP',
                             'UNPACK_SEQUENCE',
                             'MAKE_FUNCTION', 'MAKE_CLOSURE',
                             'DUP_TOPX', 'RAISE_VARARGS'
                             ):
-                # if opname == 'BUILD_TUPLE' and \
-                #     self.code[self.prev[offset]] == LOAD_CLOSURE:
-                #     continue
-                # else:
-                #     op_name = '%s_%d' % (op_name, oparg)
-                #     if opname != BUILD_SLICE:
-                #         customize[op_name] = oparg
                 opname = '%s_%d' % (opname, inst.argval)
                 if inst.opname != 'BUILD_SLICE':
                     customize[opname] = inst.argval

uncompyle6/semantics/fragments.py

@@ -46,7 +46,7 @@ else:
     from StringIO import StringIO
 
 
-from uncompyle6.parsers.spark import GenericASTTraversal, GenericASTTraversalPruningException, \
+from spark_parser import GenericASTTraversal, GenericASTTraversalPruningException, \
      DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 
 from types import CodeType

uncompyle6/semantics/pysource.py

@@ -70,7 +70,7 @@ from uncompyle6 import PYTHON3
 from uncompyle6.code import iscode
 from uncompyle6.parser import get_python_parser
 from uncompyle6.parsers.astnode import AST
-from uncompyle6.parsers.spark import GenericASTTraversal, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from spark_parser import GenericASTTraversal, DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 from uncompyle6.scanner import Code, get_scanner
 from uncompyle6.scanners.tok import Token, NoneToken
 import uncompyle6.parser as python_parser
@@ -93,7 +93,6 @@ RETURN_LOCALS = AST('return_stmt',
                     [ AST('ret_expr', [AST('expr', [ Token('LOAD_LOCALS') ])]),
                       Token('RETURN_VALUE')])
 
-
 NONE = AST('expr', [ NoneToken ] )
 
 RETURN_NONE = AST('stmt',
@@ -487,7 +486,8 @@ class SourceWalker(GenericASTTraversal, object):
     stacked_params = ('f', 'indent', 'isLambda', '_globals')
 
     def __init__(self, version, out, scanner, showast=False,
-                 debug_parser=PARSER_DEFAULT_DEBUG):
+                 debug_parser=PARSER_DEFAULT_DEBUG,
+                 compile_mode='exec'):
         GenericASTTraversal.__init__(self, ast=None)
         self.scanner = scanner
         params = {
@@ -495,7 +495,8 @@ class SourceWalker(GenericASTTraversal, object):
             'indent': '',
             }
         self.version = version
-        self.p = get_python_parser(version, debug_parser=debug_parser)
+        self.p = get_python_parser(version, debug_parser=debug_parser,
+                                   compile_mode=compile_mode)
         self.debug_parser = dict(debug_parser)
         self.showast = showast
         self.params = params
@@ -1147,29 +1148,42 @@ class SourceWalker(GenericASTTraversal, object):
         """
         p = self.prec
         self.prec = 100
-        assert node[-1] == 'kvlist'
-        node = node[-1] # goto kvlist
 
         self.indentMore(INDENT_PER_LEVEL)
         line_seperator = ',\n' + self.indent
         sep = INDENT_PER_LEVEL[:-1]
         self.write('{')
-        for kv in node:
-            assert kv in ('kv', 'kv2', 'kv3')
-            # kv ::= DUP_TOP expr ROT_TWO expr STORE_SUBSCR
-            # kv2 ::= DUP_TOP expr expr ROT_THREE STORE_SUBSCR
-            # kv3 ::= expr expr STORE_MAP
-            if kv == 'kv':
-                name = self.traverse(kv[-2], indent='')
-                value = self.traverse(kv[1], indent=self.indent+(len(name)+2)*' ')
-            elif kv == 'kv2':
-                name = self.traverse(kv[1], indent='')
-                value = self.traverse(kv[-3], indent=self.indent+(len(name)+2)*' ')
-            elif kv == 'kv3':
-                name = self.traverse(kv[-2], indent='')
-                value = self.traverse(kv[0], indent=self.indent+(len(name)+2)*' ')
-            self.write(sep, name, ': ', value)
-            sep = line_seperator
+
+        if node[0].type.startswith('kvlist'):
+            # Python 3.5 style key/value list in mapexpr
+            l = list(node[0])
+            i = 0
+            while i < len(l):
+                name = self.traverse(l[i], indent='')
+                value = self.traverse(l[i+1], indent=self.indent+(len(name)+2)*' ')
+                self.write(sep, name, ': ', value)
+                sep = line_seperator
+                i += 2
+        else:
+            assert node[-1] == 'kvlist'
+            node = node[-1] # goto kvlist
+
+            for kv in node:
+                assert kv in ('kv', 'kv2', 'kv3')
+                # kv ::= DUP_TOP expr ROT_TWO expr STORE_SUBSCR
+                # kv2 ::= DUP_TOP expr expr ROT_THREE STORE_SUBSCR
+                # kv3 ::= expr expr STORE_MAP
+                if kv == 'kv':
+                    name = self.traverse(kv[-2], indent='')
+                    value = self.traverse(kv[1], indent=self.indent+(len(name)+2)*' ')
+                elif kv == 'kv2':
+                    name = self.traverse(kv[1], indent='')
+                    value = self.traverse(kv[-3], indent=self.indent+(len(name)+2)*' ')
+                elif kv == 'kv3':
+                    name = self.traverse(kv[-2], indent='')
+                    value = self.traverse(kv[0], indent=self.indent+(len(name)+2)*' ')
+                    self.write(sep, name, ': ', value)
+                    sep = line_seperator
         self.write('}')
         self.indentLess(INDENT_PER_LEVEL)
         self.prec = p
@@ -1222,6 +1236,14 @@ class SourceWalker(GenericASTTraversal, object):
 
     n_unpack_w_parens = n_unpack
 
+    def n_assign(self, node):
+        # A horrible hack for Python 3.0 .. 3.2
+        if 3.0 <= self.version <= 3.2 and len(node) == 2:
+            if (node[0][0] == 'LOAD_FAST' and node[0][0].pattr == '__locals__' and
+                node[1][0].type == 'STORE_LOCALS'):
+                self.prune()
+        self.default(node)
+
     def n_assign2(self, node):
         for n in node[-2:]:
             if n[0] == 'unpack':
@@ -1536,12 +1558,20 @@ class SourceWalker(GenericASTTraversal, object):
             pass
 
         # if docstring exists, dump it
-        if (code.co_consts and code.co_consts[0] is not None
-            and len(ast) > 0 and ast[0][0] == ASSIGN_DOC_STRING(code.co_consts[0])):
-            if self.hide_internal:
+        if (code.co_consts and code.co_consts[0] is not None and len(ast) > 0):
+            do_doc = False
+            if (ast[0][0] == ASSIGN_DOC_STRING(code.co_consts[0])):
+                i = 0
+                do_doc = True
+            elif (len(ast) > 2 and 3.0 <= self.version <= 3.2 and
+                      ast[2][0] == ASSIGN_DOC_STRING(code.co_consts[0])):
+                i = 2
+                do_doc = True
+            if do_doc and self.hide_internal:
                 self.print_docstring(indent, code.co_consts[0])
                 self.print_()
-                del ast[0]
+                del ast[i]
+
 
         # the function defining a class normally returns locals(); we
         # don't want this to show up in the source, thus remove the node
@@ -1615,7 +1645,7 @@ class SourceWalker(GenericASTTraversal, object):
         return ast
 
 def deparse_code(version, co, out=sys.stdout, showasm=False, showast=False,
-                 showgrammar=False, code_objects={}):
+                 showgrammar=False, code_objects={}, compile_mode='exec'):
     """
     disassembles and deparses a given code block 'co'
     """
@@ -1633,7 +1663,8 @@ def deparse_code(version, co, out=sys.stdout, showasm=False, showast=False,
     debug_parser['reduce'] = showgrammar
 
     #  Build AST from disassembly.
-    deparsed = SourceWalker(version, out, scanner, showast=showast, debug_parser=debug_parser)
+    deparsed = SourceWalker(version, out, scanner, showast=showast,
+                            debug_parser=debug_parser, compile_mode=compile_mode)
 
     deparsed.ast = deparsed.build_ast(tokens, customize)
 

uncompyle6/version.py

@@ -0,0 +1,3 @@
+# This file is suitable for sourcing inside bash as
+# well as importing into Python
+VERSION='2.3.2'