Get ready for release 2.8.3

"<= 2.6" -> "< 2.7" since python 2.6's version is 2.6000001

.travis.yml

@@ -4,8 +4,9 @@ sudo: false
 
 python:
   - '3.5'
-  - '2.7.11'
+  - '2.7.12'
   - '2.6'
+  - '3.3'
   - '3.4'
 
 install:

ChangeLog

@@ -1,6 +1,498 @@
+2016-09-11  rocky <rb@dustyfeet.com>
+
+	* __pkginfo__.py, uncompyle6/version.py: Get ready for release 2.8.3
+
+2016-09-11  rocky <rb@dustyfeet.com>
+
+	* test/bytecode_3.6/fstring.py,
+	test/bytecode_3.6/fstring_single.py, uncompyle6/parsers/parse35.py: 
+	Tidy a bit
+
+2016-09-09  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/looping/10_while.py,
+	uncompyle6/parsers/parse3.py: 3.0 .. 3.4 bug in whileTrue
+
+2016-09-09  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner31.py: 3.1 scanner small fixes
+
+2016-09-09  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner2.py, uncompyle6/semantics/pysource.py: 
+	ret_cond adjustment for < 2.7 and ...  "<= 2.6" -> "< 2.7" since python 2.6's version is 2.6000001
+
+2016-09-09  rocky <rb@dustyfeet.com>
+
+	* test/Makefile, test/test_pyenvlib.py, test/test_pythonlib.py,
+	uncompyle6/scanners/scanner31.py, uncompyle6/verify.py: Start
+	accepting Python 3.1 bytecode
+
+2016-09-08  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parser.py: Add 3.6 parser
+
+2016-09-08  rocky <rb@dustyfeet.com>
+
+	* .travis.yml, test/Makefile,
+	test/simple_source/expression/06_frozenset.py,
+	test/test_pythonlib.py: More testing - travis:    * 2.7.12 - bump from 2.7.11    * 3.3.6 test_pyenvlib: add --weak-verify to Makefile: check-3.2.6 needs
+	--weak-verify
+
+2016-09-08  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse3.py, uncompyle6/semantics/pysource.py: 
+	Python 3.0-3.2 *args processing
+
+2016-09-08  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug32/01_named_and_kwargs.py,
+	uncompyle6/semantics/pysource.py: 3.2 (and down to 3.0?) bug in fn
+	name and kwargs
+
+2016-09-07  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug35/05_return_in_else.py,
+	uncompyle6/scanners/scanner3.py: Another 3.5+ erroneous
+	RETURN_END_IF misclassify
+
+2016-09-07  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug35/05_return_in_else.py,
+	uncompyle6/scanners/scanner3.py: But in Python 3.5+ erroneosly
+	adding RETURN_END_IF
+
+2016-09-05  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug33/02_named_and_kwargs.py,
+	test/test_pyenvlib.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/semantics/pysource.py: Fix 3.3 named bug and ...  Parse 3.4 parameters correctly.  Allow test_pyenvlib to do 3.3.6
+
+2016-09-05  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug33/02_named_and_kwargs.py: Python 3.x bug in
+	getting parameter of ** argument
+
+2016-09-05  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse26.py,
+	uncompyle6/parsers/parse27.py, uncompyle6/scanners/scanner2.py,
+	uncompyle6/scanners/scanner26.py, uncompyle6/scanners/scanner3.py: 
+	Python 2.6- try/except control flow detection
+
+2016-09-04  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/disas.py, uncompyle6/main.py, uncompyle6/parser.py,
+	uncompyle6/scanner.py, uncompyle6/scanners/scanner2.py,
+	uncompyle6/scanners/scanner22.py, uncompyle6/scanners/scanner26.py,
+	uncompyle6/scanners/scanner27.py, uncompyle6/scanners/scanner3.py,
+	uncompyle6/scanners/scanner32.py, uncompyle6/scanners/scanner33.py,
+	uncompyle6/scanners/scanner34.py, uncompyle6/scanners/scanner35.py,
+	uncompyle6/scanners/scanner36.py, uncompyle6/semantics/aligner.py,
+	uncompyle6/semantics/fragments.py,
+	uncompyle6/semantics/pysource.py, uncompyle6/show.py,
+	uncompyle6/verify.py: disassemble -> ingest where appropriate As part of tokenization for (de)parsing, we need to do something
+	like a disassembly, but is is really a little different.  Disassembly, strictly speaking, is done by the xdis module now.
+	What "ingestion" does is massage the instruction tokens to a form
+	that is more amenable for parsing.  In sum, ingestion is different than disassembly, although
+	disassembly is generally the first part of ingestion.
+
+2016-09-04  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: Small cleanup/code optimization
+
+2016-09-04  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse26.py: Python 2.3--2.6 bug in parsing
+	try-middle Note: it looks like COME_FROMs need to be classified better with
+	respect to whether they are from an except or not we are getting
+	if/else vs except nesting errors.
+
+2016-09-04  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug26/07_generator_return.py,
+	uncompyle6/scanners/scanner2.py, uncompyle6/semantics/pysource.py: A
+	couple more 2.6 (and below) bugs fixed * Detect "return None" inside if statement * another case of triple ==, ==, ==      scanner2.py: detect_structure: descriminate more on parent type
+
+2016-09-03  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug26/05_generator.py,
+	uncompyle6/semantics/pysource.py: Python 2.5-2.6 generator bug Adjust for variable position of list iteration node in a genexpr
+	node
+
+2016-09-03  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: Python 2.7 "return None" bug Same as 2.3-2.6 "return None".
+
+2016-09-03  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug26/08_triple_equals.py,
+	uncompyle6/scanners/scanner2.py: Python 2.2..2.6 bug in a == b == c
+	== d Fix was to remove some come froms. Feels a little hacky though.
+
+2016-09-03  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug26/07_generator_return.py,
+	uncompyle6/semantics/pysource.py: Python 2.3..2.6 "return" bug In Python 2.6 and possibly down to 2.3 we need to issue "return" not
+	"return None" inside a generator. So check for that "return None"
+	inside n_return and issue "return" for that.
+
+2016-09-03  rocky <rb@dustyfeet.com>
+
+	* DECOMPYLE-2.4-CHANGELOG.txt: Add hartmut Goebel's changes before
+	2.4
+
+2016-09-03  rocky <rb@dustyfeet.com>
+
+	* Makefile, test/simple_source/bug26/04_comp_for.py,
+	uncompyle6/semantics/pysource.py: Fix Python 2.4-2.6 comp_for text
+	generation...  Makefile: tolerate pypy 5.3.x Rest: fix semantic action rule for
+	comp_for and test this
+
+2016-09-02  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug26/06_return_pop.py,
+	uncompyle6/parsers/parse26.py, uncompyle6/scanners/scanner2.py: 
+	Python 2.6- bug: RETURN_ENDIF, POP_TOP ..  POP_TOP should be excluded as a potentional statement beginning
+
+2016-09-02  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug33/02_named_and_kwargs.py,
+	uncompyle6/scanners/scanner2.py, uncompyle6/semantics/pysource.py: 
+	Fix Python 3.x named param and kwargs bug
+
+2016-09-01  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug26/04_while_and_stmt_one_line.py,
+	uncompyle6/scanners/scanner2.py, uncompyle6/scanners/scanner26.py: 
+	2.6- bug: while..and: stmt - on one line If 2.6 or before POP_BLOCK after a JUMP_IF_FALSE does not constitute
+	a new statement. The POP_BLOCK is really part of the JUMP_IF_FALSE.
+	In Python 2.7+ it's a single op.
+
+2016-09-01  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug26/02_except_as.py,
+	uncompyle6/parsers/parse26.py, uncompyle6/semantics/pysource.py: 
+	Handle Python 2.6 and below "except <cond>, <var>"
+
+2016-08-31  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug35/02_fn_varargs.py,
+	uncompyle6/semantics/pysource.py: Python 3.x bug in handling var
+	number of args
+
+2016-08-31  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug33/04_aug_assign.py, test/test_pyenvlib.py,
+	uncompyle6/main.py, uncompyle6/scanners/scanner2.py,
+	uncompyle6/scanners/scanner3.py, uncompyle6/verify.py: Bug in 3.x
+	detecting "if" structure and ...  scanner3.py: bug in 3.x detecting "if" structure Make scanner2.py
+	look more like scanner3.py verify.py: add weak-verify which tests
+	Pytyon syntax, but not code
+
+2016-08-30  rocky <rb@dustyfeet.com>
+
+	* NEWS, test/simple_source/bug26/03_raise_from.py,
+	test/simple_source/bug33/03_raise_from.py,
+	uncompyle6/semantics/pysource.py: Handle Python 2 vs 3 raise syntax
+	change    raise_stmt ::=  "raise" expression "," expression becomes:    raise_stmt ::=  "raise" expression from expression raise expr, expr -> raise
+
+2016-08-29  rocky <rb@dustyfeet.com>
+
+	* ChangeLog, __pkginfo__.py, uncompyle6/version.py: Get ready for
+	release 2.8.2
+
+2016-08-27  rocky <rb@dustyfeet.com>
+
+	* test/test_pyenvlib.py, uncompyle6/verify.py: Correct PYPY bit
+	logic in previous commit
+
+2016-08-27  rocky <rb@dustyfeet.com>
+
+	* test/test_pyenvlib.py, uncompyle6/semantics/pysource.py,
+	uncompyle6/verify.py: PYPY bugs and inspired changes ...  verify.py: Show co_flags when different.  pysource.py: PYPY also
+	generates normal tryfinallystmt code test_pyenvlib.py: allow
+	pypy-5.3.1
+
+2016-08-26  rocky <rb@dustyfeet.com>
+
+	* Makefile, test/Makefile, uncompyle6/parser.py,
+	uncompyle6/parsers/parse3.py, uncompyle6/scanner.py,
+	uncompyle6/scanners/scanner3.py: Start to handle Python 3.1 bytecode
+
+2016-08-26  rocky <rb@dustyfeet.com>
+
+	* __pkginfo__.py, requirements.txt: Bump min requirement versions xdis we need increased so we don't catch old xdis bugs
+
+2016-08-25  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug36/01_fstring.py,
+	uncompyle6/semantics/pysource.py: Handle 3.6 Format String
+	conversions !r, !s, !a
+
+2016-08-20  rocky <rb@dustyfeet.com>
+
+	* ChangeLog, NEWS, uncompyle6/version.py: Get ready for release
+	2.8.1
+
+2016-08-16  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug22/05_test_yield.py,
+	uncompyle6/scanners/scanner2.py: Python 2.2 doesn't have opcode
+	LIST_APPEND
+
+2016-08-16  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner26.py: Python 2.2 scanner bug: don't
+	mung IMPORT_NAME op
+
+2016-08-16  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug_pypy27/02_call_method.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py: Small
+	pypy LOOKUP_METHOD cleanups
+
+2016-08-16  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #49 from moagstar/master Fixed a bug with FORMAT_VALUE with sub expressions.
+
+2016-08-16  DanielBradburn <moagstar@gmail.com>
+
+	* pytest/test_fstring.py, uncompyle6/parsers/parse3.py: Fixed bug
+	with FORMAT_VALUE where a sub expression would not be correctly
+	interpreted
+
+2016-08-14  rocky <rb@dustyfeet.com>
+
+	* __pkginfo__.py: xdis 2.1.1 removes some bugs encountered here
+
+2016-08-14  rocky <rb@dustyfeet.com>
+
+	* : commit a65443ee0225933367be9c7640629298882532c9 Author: rocky
+	<rb@dustyfeet.com> Date:   Sun Aug 14 22:59:34 2016 -0400
+
+2016-08-14  DanielBradburn <moagstar@gmail.com>
+
+	* pytest/test_fstring.py: added examples for known failures
+
+2016-08-14  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #45 from rocky/revert-43-patch-1 Revert "Cache pip installation in travis"
+
+2016-08-14  R. Bernstein <rocky@users.noreply.github.com>
+
+	* .travis.yml: Revert "Test with latest PyPy in Travis"
+
+2016-08-14  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #44 from thedrow/patch-2 Test with latest PyPy in Travis
+
+2016-08-14  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #43 from thedrow/patch-1 Cache pip installation in travis
+
+2016-08-13  rocky <rb@dustyfeet.com>
+
+	* test/Makefile: Back off of 3.6 testing for now
+
+2016-08-13  rocky <rb@dustyfeet.com>
+
+	* : commit 7ccbd419c6b26e8ae9d0929f1bfddedebce6bbaf Author: rocky
+	<rb@dustyfeet.com> Date:   Sat Aug 13 20:25:19 2016 -0400
+
+2016-08-13  rocky <rb@dustyfeet.com>
+
+	* test/Makefile: I said - we test 2.2 now.
+
+2016-08-13  rocky <rb@dustyfeet.com>
+
+	* test/Makefile, test/simple_source/stmts/00_import.py,
+	test/test_pythonlib.py: Include Python 2.2 in testing
+
+2016-08-13  rocky <rb@dustyfeet.com>
+
+	* README.rst, uncompyle6/parser.py, uncompyle6/parsers/parse22.py,
+	uncompyle6/scanner.py, uncompyle6/scanners/scanner22.py,
+	uncompyle6/scanners/scanner23.py, uncompyle6/scanners/scanner24.py,
+	uncompyle6/scanners/scanner25.py, uncompyle6/semantics/pysource.py: 
+	Start handling Python 2.2 bytecode and...  Fix some bugs in Python 2.3-2.5 bytecode handling
+
+2016-08-11  Omer Katz <omer.drow@gmail.com>
+
+	* .travis.yml: Test with latest PyPy.
+
+2016-08-11  Omer Katz <omer.drow@gmail.com>
+
+	* .travis.yml: Cache pip installation
+
+2016-08-10  DanielBradburn <moagstar@gmail.com>
+
+	* pytest/test_fstring.py: small formatting change
+
+2016-08-10  DanielBradburn <moagstar@gmail.com>
+
+	* pytest/test_fstring.py, requirements-dev.txt: added hypothesis to
+	requirements-dev
+
+2016-08-10  DanielBradburn <moagstar@gmail.com>
+
+	* pytest/test_fstring.py: added hypothesis test (currently failing
+	due to limited support) for testing fstring uncompyling
+
+2016-08-10  Daniel Bradburn <moagstar@gmail.com>
+
+	* : Merge pull request #2 from rocky/master Merging pypy and cpython 3.6 from rocky
+
+2016-08-03  rocky <rb@dustyfeet.com>
+
+	* : commit 109e813058380630bda82014eee94a9089cc4666 Author: rocky
+	<rb@dustyfeet.com> Date:   Wed Aug 3 08:07:47 2016 -0400
+
+2016-08-01  rocky <rb@dustyfeet.com>
+
+	* __pkginfo__.py, requirements.txt: Need recent xdis fix for 3.6
+	wordcode
+
+2016-08-01  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/bug36/01_fstring.py: Add Python 3.6
+	formatted_str test
+
+2016-08-01  R. Bernstein <rocky@users.noreply.github.com>
+
+	* : Merge pull request #41 from rocky/3.6 Move forward on moagstar's Python 3.6 support
+
+2016-08-01  rocky <rb@dustyfeet.com>
+
+	* Makefile, README.rst, test/Makefile, test/test_pythonlib.py,
+	uncompyle6/bin/uncompile.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/parsers/parse36.py, uncompyle6/scanner.py,
+	uncompyle6/scanners/scanner36.py, uncompyle6/semantics/aligner.py,
+	uncompyle6/semantics/pysource.py, uncompyle6/verify.py: Moagstar's
+	3.6 wordcode + formattedValue rules
+
+2016-07-30  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/aligner.py: WIP try to keep line numbers the
+	same
+
+2016-07-29  rocky <rb@dustyfeet.com>
+
+	* test/Makefile, uncompyle6/semantics/fragments.py: Small changes
+
+2016-07-29  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse35.py, uncompyle6/scanner.py,
+	uncompyle6/scanners/scanner3.py, uncompyle6/semantics/pysource.py: 
+	Fix 3.5 misclassifying RETURN_VALUE We use location of SETUP_EXCEPT instructions to disambiguate.
+
+2016-07-28  Daniel Bradburn <moagstar@gmail.com>
+
+	* README.rst, test/Makefile, test/bytecode_3.6/fstring.py,
+	test/bytecode_3.6/fstring_single.py, test/test_pythonlib.py,
+	uncompyle6/bin/uncompile.py, uncompyle6/parser.py,
+	uncompyle6/parsers/parse3.py, uncompyle6/scanner.py,
+	uncompyle6/scanners/scanner36.py, uncompyle6/semantics/pysource.py,
+	uncompyle6/verify.py: Starting adding python 3.6 support to
+	uncompyle
+
+2016-07-28  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse23.py,
+	uncompyle6/parsers/parse24.py: while1 bug applied to Python 2.3 and
+	2.4
+
+2016-07-28  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/scanners/scanner3.py, uncompyle6/scanners/tok.py: PyPy
+	3.2 bug confusing RETURN_END_IF for except Also fix a instruction formatting bug
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parser.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/parsers/parse35.py: Split out 3.5 parser
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* pytest/test_grammar.py, uncompyle6/parser.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/parsers/parse34.py: Add python 3.4 grammar checking DRY grammar testing
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse25.py, uncompyle6/parsers/parse26.py,
+	uncompyle6/parsers/parse27.py, uncompyle6/parsers/parse3.py,
+	uncompyle6/parsers/parse34.py: Clean and check Python 2.6 grammar
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* pytest/test_grammar.py, uncompyle6/parser.py,
+	uncompyle6/parsers/parse2.py, uncompyle6/parsers/parse26.py,
+	uncompyle6/parsers/parse27.py, uncompyle6/parsers/parse3.py: Start
+	to segregate and clean up grammar
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* pytest/test_grammar.py, uncompyle6/disas.py,
+	uncompyle6/scanner.py, uncompyle6/semantics/fragments.py: Add
+	is_pypy parameter to places that need it
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* test/simple_source/stmts/09_whiletrue_bug.py,
+	uncompyle6/parser.py, uncompyle6/parsers/parse2.py,
+	uncompyle6/parsers/parse26.py, uncompyle6/parsers/parse27.py,
+	uncompyle6/scanners/scanner2.py, uncompyle6/scanners/scanner3.py,
+	uncompyle6/semantics/pysource.py: 2.6 and 2.7 while1 grammar rule Fixes issue #40
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* pytest/test_grammar.py, uncompyle6/parser.py,
+	uncompyle6/parsers/parse3.py: Start grammar checker
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/main.py, uncompyle6/show.py: Show magic number in
+	output Fix bugs due to removal of token.format()
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/disas.py, uncompyle6/parsers/parse2.py,
+	uncompyle6/parsers/parse3.py, uncompyle6/scanners/scanner2.py,
+	uncompyle6/scanners/scanner27.py, uncompyle6/scanners/scanner3.py,
+	uncompyle6/scanners/scanner35.py, uncompyle6/scanners/tok.py,
+	uncompyle6/show.py: tok.format -> tok.__str__; simplify pypy code
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py, uncompyle6/verify.py: Python 2.7
+	set comprehension bug
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/pysource.py: separate semantic action version
+	differences Added customize_for_version which uses is_pypy and version to adjust
+	tables
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/semantics/fragments.py,
+	uncompyle6/semantics/pysource.py: Customize tables better for
+	specific Python versions
+
+2016-07-27  rocky <rb@dustyfeet.com>
+
+	* uncompyle6/parsers/parse2.py, uncompyle6/semantics/pysource.py: 
+	Small code clean up
+
 2016-07-26  rocky <rb@dustyfeet.com>
 
-	* README.rst, __pkginfo__.py, requirements.txt,
+	* uncompyle6/scanners/tok.py, uncompyle6/semantics/fragments.py,
+	uncompyle6/verify.py: Usuability fixes * try using format for __str__ * Explicitly nuke self.attr and self.pattr when no arg * Sync pysource and format wrt make_function
+
+2016-07-26  rocky <rb@dustyfeet.com>
+
+	* ChangeLog, NEWS, README.rst, __pkginfo__.py, requirements.txt,
 	test/test_pyenvlib.py, uncompyle6/version.py: Get ready for release
 	2.7.1
 
@@ -105,9 +597,9 @@
 	uncompyle6/scanners/scanner3.py, uncompyle6/semantics/pysource.py: 
 	PyPy support * Use proper PYPY 32 opcodes * handle opcodes LOOKUP_METHOD and CALL_METHOD * Administrative stuff for PyPy
 
-2016-07-24  rocky <rb@dustyfeet.com>
+2016-07-24  Daniel Bradburn <moagstar@gmail.com>
 
-	* test/add-test.py: add-test: Make sure PyPy bytecode is separated
+	* : Merge pull request #1 from rocky/master Syncing with rocky
 
 2016-07-24  rocky <rb@dustyfeet.com>
 

DECOMPYLE-2.4-CHANGELOG.txt

@@ -1,4 +1,5 @@
-This is the changelog from *decompyle*'s release 2.4 passed on by Dan Pascu
+This is the changelog from *decompyle*'s release 2.4 and before
+passed on by Dan Pascu
 
 
 release 2.4 (Dan Pascu)
@@ -74,3 +75,57 @@ release 2.2beta1 (hartmut Goebel)
   - use a single Walker and a single Parser, thus saving time and memory
   - use augmented assign and 'print >>' internally
   - optimized 'Walker.engine', the main part of code generation
+release 0.6.0: (hartmut Goebel)
+  - extended print (Python 2.0)
+  - extended import (Python 2.0) (may not cover all cases)
+  - augmented assign (Python 2.0) (may not cover all cases)
+  - list comprehensions (Python 2.0)
+  - equivalent for 'apply' (Python 1.6)
+  - if .. elif .. else are now nested as expected
+  - assert test, data
+  - unpack list corrected (was the same as unpack tuple)
+  - fixed unpack tuple (trailing semicolon was missing)
+  - major speed up :-)
+  - reduced memory usage (pre-alpha-0.5 has increased it a lot)
+  - still missing: EXTENDED_ARG
+
+pre-alpha-0.5: (hartmut Goebel)
+  - *args, **kwargs
+  - global
+  - formal tuple parameters (eg. def a(self, (x,y,z)) )
+  - actual lambda parameters (eg.  X(lambda z: z**2) )
+  - remove last 'return None' in procedures
+  - remove last 'return locals()' in class definitions
+  - docstrings
+
+pre-alpha-0.4: (hartmut Goebel)
+  - assert
+  - try/except/finally
+  - parentheses in expressions
+  - nested expressions
+  - extracted dissassemble() from module dis and
+    removed ugly redirect of stdout, thus saved a lot of
+    ugly code and a lot of memory
+
+pre-alpha-0.3: (hartmut Goebel)
+  - keyword arguments
+  - some boolean expressions
+  - and/or
+  - complex conditions in if/while
+  - read byte-code from .pyc without importing
+  - access to the body of classes and modules
+  - class and function definitions
+  - a = b = c = xxx
+
+pre-alpha-0.1 -> pre-alpha-0.2:
+  -  SET_LINENO filtered out in lexer now
+  -  added support for subscripts (just for Christian Tismer :-)
+  -  fixed bug with handling of BUILD_{LIST,TUPLE} & CALL_FUNCTION
+  -  dict-building support
+  -  comparison support
+  -  exec support
+  -  del support
+  -  pass support
+  -  slice support
+  -  no more extraneous (albeit legal) commas
+  -  finally, it excepts try [sic] but not all 42 variations of it

Makefile

@@ -33,7 +33,7 @@ check-2.7 check-3.3 check-3.4: pytest
 
 #: Tests for Python 3.2 and 3.5 - pytest doesn't work here
 # Or rather 3.5 doesn't work not on Travis
-check-3.2 check-3.5:
+check-3.1 check-3.2 check-3.5 check-3.6:
 	$(MAKE) -C test $@
 
 #:Tests for Python 2.6 (doesn't have pytest)
@@ -42,7 +42,7 @@ check-2.6:
 
 #:PyPy 2.6.1 or PyPy 5.0.1
 # Skip for now
-2.6 5.0:
+2.6 5.0 5.3:
 
 #:PyPy pypy3-2.4.0 Python 3:
 pypy-3.2 2.4:

NEWS

@@ -1,3 +1,57 @@
+uncompyle6 2.832 2016-09-11 live from NYC!
+
+NOTE: this is possibly the last release before a major reworking of
+control-flow structure detection is done.
+
+- Lots of bug fixes decompilations:
+  * 3.0 .. 3.4 whileTrue bug
+  * 3.x function declaration deparsing:
+    . 3.0 .. 3.2 *args processing
+    . 3.0 .. 3.2 call name and kwargs bug
+    . 3.0 .. getting parameter of *
+    . 3.0 .. handling varible number of args
+    . 3.0 .. "if" structure bugs
+  * 3.5+ if/else bugs
+  * 2.2-2.6 bugs
+    . try/except control flow
+    . a == b == c -like detection
+    . generator detection
+    . "while .. and" statement bugs
+    . handle "except <cond>, <var>"
+    . use older raise format in 2.x
+- scanner "disassemble" is now "ingest". True disassembly is done by xdis
+- Start accepting Python 3.1 bytecode
+- Add --weak-verify option on test_pyenvlib and test_pythonlib. This
+  catches more bugs more easily
+- bump xdis requirement so we can deparse dropbox 2.5 code
+- Added H. Goebel's changes before 2.4 in DECOMPYLE-2.4-CHANGELOG.txt
+
+uncompyle6 2.8.2 2016-08-29
+
+- Handle Python 3.6 format string conversions !r, !s, !a
+- Start to handle 3.1 bytecode
+- Fix some PyPy translation bugs
+- We now only handle 3.6.0a3+ since that is incompatible with 3.6 before that
+
+uncompyle6 2.8.1 2016-08-20
+
+- Add Python 2.2 decompilation
+
+- Fix bugs
+ * PyPy LOOKUP_METHOD bug
+ * Python 3.6 FORMAT_VALUE handles expressions now
+
+uncompyle6 2.8.0 2016-08-03
+
+- Start Python 3.6 support (moagstar)
+  more work on PEP 498 needed
+- tidy bytecode/word output
+- numerous decompiling bugs fixed
+- grammar testing started
+- show magic number in deparsed output
+- better grammar and semantic action segregation based
+  on python bytecode version
+
 uncompyle6 2.7.1 2016-07-26
 
 - PyPy bytecodes for 2.7 and 3.2 added

README.rst

@@ -11,7 +11,7 @@ Introduction
 ------------
 
 *uncompyle6* translates Python bytecode back into equivalent Python
-source code. It accepts bytecodes from Python version 2.3 to 3.5 or
+source code. It accepts bytecodes from Python version 2.2 to 3.6 or
 so, including PyPy bytecode.
 
 Why this?
@@ -45,7 +45,7 @@ Requirements
 
 This project requires Python 2.6 or later, PyPy 3-2.4, or PyPy-5.0.1.
 The bytecode files it can read has been tested on Python bytecodes from
-versions 2.3-2.7, and 3.2-3.5 and the above-mentioned PyPy versions.
+versions 2.2-2.7, and 3.2-3.6 and the above-mentioned PyPy versions.
 
 Installation
 ------------

__pkginfo__.py

@@ -23,6 +23,7 @@ classifiers =  ['Development Status :: 4 - Beta',
                 'Programming Language :: Python :: 3.3',
                 'Programming Language :: Python :: 3.4',
                 'Programming Language :: Python :: 3.5',
+                'Programming Language :: Python :: 3.6',
                 'Topic :: Software Development :: Debuggers',
                 'Topic :: Software Development :: Libraries :: Python Modules',
                 ]
@@ -37,7 +38,7 @@ entry_points={
     ]}
 ftp_url            = None
 install_requires   = ['spark-parser >= 1.4.0',
-                      'xdis >= 2.0.3']
+                      'xdis >= 2.3.0']
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'

pytest/test_fstring.py

@@ -0,0 +1,136 @@
+# std
+import os
+# test
+import pytest
+import hypothesis
+from hypothesis import strategies as st
+# uncompyle6
+from uncompyle6 import PYTHON_VERSION, deparse_code
+
+
+@st.composite
+def expressions(draw):
+    # todo : would be nice to generate expressions using hypothesis however
+    # this is pretty involved so for now just use a corpus of expressions
+    # from which to select.
+    return draw(st.sampled_from((
+        'abc',
+        'len(items)',
+        'x + 1',
+        'lineno',
+        'container',
+        'self.attribute',
+        'self.method()',
+        'sorted(items, key=lambda x: x.name)',
+        'func(*args, **kwargs)',
+        'text or default',
+    )))
+
+
+@st.composite
+def format_specifiers(draw):
+    """
+    Generate a valid format specifier using the rules:
+
+    format_spec ::=  [[fill]align][sign][#][0][width][,][.precision][type]
+    fill        ::=  <any character>
+    align       ::=  "<" | ">" | "=" | "^"
+    sign        ::=  "+" | "-" | " "
+    width       ::=  integer
+    precision   ::=  integer
+    type        ::=  "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"
+
+    See https://docs.python.org/2/library/string.html
+
+    :param draw: Let hypothesis draw from other strategies.
+
+    :return: An example format_specifier.
+    """
+    alphabet_strategy = st.characters(min_codepoint=ord('a'), max_codepoint=ord('z'))
+    fill = draw(st.one_of(alphabet_strategy, st.none()))
+    align = draw(st.sampled_from(list('<>=^')))
+    fill_align = (fill + align or '') if fill else ''
+
+    type_ = draw(st.sampled_from('bcdeEfFgGnosxX%'))
+    can_have_sign = type_ in 'deEfFgGnoxX%'
+    can_have_comma = type_ in 'deEfFgG%'
+    can_have_precision = type_ in 'fFgG'
+    can_have_pound = type_ in 'boxX%'
+    can_have_zero = type_ in 'oxX'
+
+    sign = draw(st.sampled_from(list('+- ') + [''])) if can_have_sign else ''
+    pound = draw(st.sampled_from(('#', '',))) if can_have_pound else ''
+    zero = draw(st.sampled_from(('0', '',))) if can_have_zero else ''
+
+    int_strategy = st.integers(min_value=1, max_value=1000)
+
+    width = draw(st.one_of(int_strategy, st.none()))
+    width = str(width) if width is not None else ''
+
+    comma = draw(st.sampled_from((',', '',))) if can_have_comma else ''
+    if can_have_precision:
+        precision = draw(st.one_of(int_strategy, st.none()))
+        precision = '.' + str(precision) if precision else ''
+    else:
+        precision = ''
+
+    return ''.join((fill_align, sign, pound, zero, width, comma, precision, type_,))
+
+
+@st.composite
+def fstrings(draw):
+    """
+    Generate a valid f-string.
+    See https://www.python.org/dev/peps/pep-0498/#specification
+
+    :param draw: Let hypothsis draw from other strategies.
+
+    :return: A valid f-string.
+    """
+    is_raw = draw(st.booleans())
+    integer_strategy = st.integers(min_value=0, max_value=3)
+    expression_count = draw(integer_strategy)
+    content = []
+    for _ in range(expression_count):
+        expression = draw(expressions())
+        # not yet : conversion not supported
+        conversion = ''#draw(st.sampled_from(('', '!s', '!r', '!a',)))
+        has_specifier = draw(st.booleans())
+        specifier = ':' + draw(format_specifiers()) if has_specifier else ''
+        content.append('{{{}{}}}'.format(expression, conversion, specifier))
+    content = ''.join(content)
+
+    return "f{}'{}'".format('r' if is_raw else '', content)
+
+
+@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+@hypothesis.given(format_specifiers())
+def test_format_specifiers(format_specifier):
+    """Verify that format_specifiers generates valid specifiers"""
+    try:
+        exec('"{:' + format_specifier + '}".format(0)')
+    except ValueError as e:
+        if 'Unknown format code' not in str(e):
+            raise
+
+
+@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+@hypothesis.given(fstrings())
+def test_uncompyle_fstring(fstring):
+    """Verify uncompyling fstring bytecode"""
+
+    # ignore fstring with no expressions an fsring with
+    # no expressions just gets compiled to a normal string.
+    hypothesis.assume('{' in fstring)
+
+    # BUG : At the moment a single expression is not supported
+    # for example f'{abc}'.
+    hypothesis.assume(fstring.count('{') > 1)
+
+    expr = fstring + '\n'
+    code = compile(expr, '<string>', 'single')
+    deparsed = deparse_code(PYTHON_VERSION, code, compile_mode='single')
+    recompiled = compile(deparsed.text, '<string>', 'single')
+
+    if recompiled != code:
+        assert deparsed.text == expr

pytest/test_grammar.py

@@ -0,0 +1,44 @@
+import pytest, re
+from uncompyle6 import PYTHON_VERSION, PYTHON3, IS_PYPY # , PYTHON_VERSION
+from uncompyle6.parser import get_python_parser
+from uncompyle6.scanner import get_scanner
+
+def test_grammar():
+
+    def check_tokens(tokens, opcode_set):
+        remain_tokens = set(tokens) - opcode_set
+        remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+        remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+        remain_tokens = set(remain_tokens) - opcode_set
+        assert remain_tokens == set([]), \
+            "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dumpGrammar())
+
+    p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
+    lhs, rhs, tokens, right_recursive = p.checkSets()
+    expect_lhs = set(['expr1024', 'pos_arg'])
+    unused_rhs = set(['build_list', 'call_function', 'mkfunc', 'mklambda',
+                      'unpack', 'unpack_list'])
+    expect_right_recursive = [['designList', ('designator', 'DUP_TOP', 'designList')]]
+    if PYTHON3:
+        expect_lhs.add('load_genexpr')
+        unused_rhs = unused_rhs.union(set("""
+        except_pop_except genexpr classdefdeco2 listcomp
+        """.split()))
+    else:
+        expect_lhs.add('kwarg')
+    assert expect_lhs == set(lhs)
+    assert unused_rhs == set(rhs)
+    assert expect_right_recursive == right_recursive
+    s = get_scanner(PYTHON_VERSION, IS_PYPY)
+    ignore_set = set(
+            """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
+               LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP
+               LAMBDA_MARKER RETURN_LAST
+            """.split())
+    if 2.6 <= PYTHON_VERSION <= 2.7:
+        opcode_set = set(s.opc.opname).union(ignore_set)
+        check_tokens(tokens, opcode_set)
+    elif PYTHON_VERSION == 3.4:
+        ignore_set.add('LOAD_CLASSNAME')
+        opcode_set = set(s.opc.opname).union(ignore_set)
+        check_tokens(tokens, opcode_set)

requirements-dev.txt

@@ -1,2 +1,3 @@
 pytest
 flake8
+hypothesis

requirements.txt

@@ -1,2 +1,2 @@
-spark-parser >= 1.2.1
-xdis >= 2.0.3
+spark-parser >= 1.4.0
+xdis >= 2.2.2

test/Makefile

@@ -22,9 +22,13 @@ check:
 #: Run working tests from Python 2.6 or 2.7
 check-2.6 check-2.7: check-bytecode-2 check-bytecode-3  check-2.7-ok
 
+#: Run working tests from Python 3.1
+check-3.1: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.1 --verify $(COMPILE)
+
 #: Run working tests from Python 3.2
 check-3.2: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.2 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.3
 check-3.3: check-bytecode
@@ -38,13 +42,18 @@ check-3.4: check-bytecode check-3.4-ok check-2.7-ok
 check-3.5: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.5 --verify $(COMPILE)
 
+#: Run working tests from Python 3.6
+check-3.6: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.6 --verify $(COMPILE)
+
 #: Check deparsing only, but from a different Python version
 check-disasm:
 	$(PYTHON) dis-compare.py
 
 #: Check deparsing bytecode 2.x only
 check-bytecode-2:
-	$(PYTHON) test_pythonlib.py --bytecode-2.3 --bytecode-2.4 \
+	$(PYTHON) test_pythonlib.py \
+	--bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
         --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
 
 #: Check deparsing bytecode 3.x only
@@ -54,9 +63,14 @@ check-bytecode-3:
 
 #: Check deparsing bytecode that works running Python 2 and Python 3
 check-bytecode: check-bytecode-3
-	$(PYTHON) test_pythonlib.py --bytecode-2.3 --bytecode-2.4 \
+	$(PYTHON) test_pythonlib.py \
+	--bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
         --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
 
+#: Check deparsing Python 2.2
+check-bytecode-2.3:
+	$(PYTHON) test_pythonlib.py --bytecode-2.2
+
 #: Check deparsing Python 2.3
 check-bytecode-2.3:
 	$(PYTHON) test_pythonlib.py --bytecode-2.3
@@ -77,6 +91,10 @@ check-bytecode-2.6:
 check-bytecode-2.7:
 	$(PYTHON) test_pythonlib.py --bytecode-2.7
 
+#: Check deparsing Python 3.1
+check-bytecode-3.1:
+	$(PYTHON) test_pythonlib.py --bytecode-3.1
+
 #: Check deparsing Python 3.2
 check-bytecode-3.2:
 	$(PYTHON) test_pythonlib.py --bytecode-3.2
@@ -93,6 +111,10 @@ check-bytecode-3.4:
 check-bytecode-3.5:
 	$(PYTHON) test_pythonlib.py --bytecode-3.5
 
+#: Check deparsing Python 3.6
+check-bytecode-3.6:
+	$(PYTHON) test_pythonlib.py --bytecode-3.6
+
 #: short tests for bytecodes only for this version of Python
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --verify $(COMPILE)
@@ -118,7 +140,7 @@ check-3.4-ok:
 2.6:
 
 #: PyPy 5.0.x with Python 2.7 ...
-pypy-2.7 5.0:
+pypy-2.7 5.0 5.3:
 	$(PYTHON) test_pythonlib.py --bytecode-pypy2.7 --verify
 
 #: PyPy 2.4.x with Python 3.2 ...

test/simple_source/bug22/02_apply_equiv.py

@@ -0,0 +1,27 @@
+# decompyle's test_appyEquiv.py
+
+def kwfunc(**kwargs):
+    print kwargs.items()
+
+
+def argsfunc(*args):
+    print args
+
+
+def no_apply(*args, **kwargs):
+    print args
+    print kwargs.items()
+    argsfunc(34)
+    foo = argsfunc(*args)
+    argsfunc(*args)
+    argsfunc(34, *args)
+    kwfunc(**None)
+    kwfunc(x = 11, **None)
+    no_apply(*args, **args)
+    no_apply(34, *args, **args)
+    no_apply(x = 11, *args, **args)
+    no_apply(34, x = 11, *args, **args)
+    no_apply(42, 34, x = 11, *args, **args)
+    return foo
+
+no_apply(1, 2, 4, 8, a = 2, b = 3, c = 5)

test/simple_source/bug22/05_test_yield.py

@@ -0,0 +1,24 @@
+# From decompyle
+# In Python 2.2 we don't have op LIST_APPEND while in > 2.3 we do.
+
+from __future__ import generators
+
+def inorder(t):
+    if t:
+        for x in inorder(t.left):
+            yield x
+
+        yield t.label
+        for x in inorder(t.right):
+            yield x
+
+def generate_ints(n):
+    for i in range(n):
+        yield i * 2
+
+for i in generate_ints(5):
+    print i,
+
+print
+gen = generate_ints(3)
+print gen.next(), gen.next(), gen.next(), gen.next()

test/simple_source/bug26/02_except_as.py

@@ -0,0 +1,18 @@
+# From 2.6.9 ConfigParser.py
+# Bug was being able to handle:
+#   except KeyError, e
+# vs 2.6+.
+#   except KeyError as e
+#
+# In terms of table syntax:
+# 2.7+:
+# 'except_cond2':	( '%|except %c as %c:\n', 1, 5 )
+# vs 2.6 and before
+# 'except_cond3':	( '%|except %c, %c:\n', 1, 6 )
+#
+# Python 2.6 allows both, but we use the older form since
+# that matches the grammar for how this gets parsed
+try:
+    value = "foo"
+except KeyError, e:
+    raise RuntimeError('foo')

test/simple_source/bug26/03_raise_from.py

@@ -0,0 +1,9 @@
+# From 3.4 _pyio.py
+# Bug is change in syntax between Python 2 and 3:
+#    raise_stmt ::=  "raise" expression "," expression
+# becomes:
+#    raise_stmt ::=  "raise" expression from expression
+try:
+    x = 1
+except AttributeError as err:
+    raise TypeError("an integer is required"), err

test/simple_source/bug26/04_comp_for.py

@@ -0,0 +1,4 @@
+# From python2.6/_abcoll.py
+# Bug was wrong code for "comp_for" giving
+# "for  in x" instead of: "for x in y"
+chain = (e for s in (self, other) for x in y)

test/simple_source/bug26/04_while_and_stmt_one_line.py

@@ -0,0 +1,9 @@
+# From Python 2.6/MimeWriter.py
+#
+# Bug is detecting that "del" starts a statement
+# and is not: del not lines[-1] and lines[-1]
+#
+# A complicating factor is that the "while"
+# and statement are on the same line.
+lines = __file__.split("\n")
+while lines and not lines[-1]: del lines[-1]

test/simple_source/bug26/05_generator.py

@@ -0,0 +1,9 @@
+# From Python 2.6 ast.py
+# Bug was finding the last iteration expression in generator-expression handling
+#
+fields = (1, 2)
+rv = '%s(%s' % (__name__.__class__.__name__, ', '.join(
+    ('%s=%s' % field for field in fields)
+    if __file__ else
+    (b for a, b in fields)
+))

test/simple_source/bug26/06_return_pop.py

@@ -0,0 +1,34 @@
+# From python 2.6 StringIO.py
+# Bug was turning emitting code like:
+#     if spos == slen:
+#          self.len = self.pos = spos + len(s)
+#         return None
+#         # wrong indent below
+#         if spos > slen:
+#            ...
+#         if self.buflist:
+#            # Invalid "and" below
+#            self.buflist and self.buf += ''.join(self.buflist)
+#
+# This was caused by POP_TOP in RETURN_VALUE, POP_TOP getting treated
+# as the beginning of a statement
+def write(self, s, spos):
+    if not s: return
+    # Force s to be a string or unicode
+    if not isinstance(s, basestring):
+        s = str(s)
+    slen = self.len
+    if spos == slen:
+        self.len = self.pos = spos + len(s)
+        return
+    if spos > slen:
+        slen = spos
+    newpos = spos + len(s)
+    if spos < slen:
+        if self.buflist:
+            self.buf += ''.join(self.buflist)
+        self.buflist = [self.buf[:spos], s, self.buf[newpos:]]
+        if newpos > slen:
+            slen = newpos
+    else:
+        self.buflist.append(s)

test/simple_source/bug26/07_generator_return.py

@@ -0,0 +1,29 @@
+# From python2.6/_abcoll.py
+# Bug was producing "return None" which isn't
+# allowed in a generator, instead of "return"
+def __iter__(self):
+    i = 0
+    try:
+        while True:
+            v = self[i]
+            yield v
+            i += 1
+    except IndexError:
+        return
+
+
+# From 2.6 bsddb/__init.py
+# Bug is return None in a generator inside
+# an if.
+def iteritems(self):
+    if not self.db:
+        return
+    try:
+        try:
+            yield self.kv
+        except:
+            # the database was modified during iteration.  abort.
+            pass
+    except:
+        self._in_iter -= 1
+        raise

test/simple_source/bug26/08_triple_equals.py

@@ -0,0 +1,7 @@
+# From Python 2.6 aifc.py
+# Bug was handling triple ==
+# Fixed by removing some COME_FROMs inside
+if expon == himant == lomant == 0:
+    f = 0.0
+else:
+    f = 1.1

test/simple_source/bug32/01_named_and_kwargs.py

@@ -0,0 +1,20 @@
+# python 3.2 configparser.py
+
+# Bug was emitting in 3.2
+# def __init__(self, defaults=delimiters=('=', ':'),
+#            comment_prefixes=('#', ';'), inline_comment_prefixes=None,
+#            strict=True, empty_lines_in_values=True,
+#            default_section=DEFAULTSECT, interpolation=_UNSET, dict_type=None,
+#            allow_no_value=_default_dict, *, delimiters=('=', ':'),
+#            comment_prefixes=('#', ';'), inline_comment_prefixes=None,
+#            strict=True, empty_lines_in_values=True,
+#            default_section=DEFAULTSECT,
+#            interpolation=_UNSET):
+
+def __init__(self, defaults=None, dict_type=_default_dict,
+             allow_no_value=False, *, delimiters=('=', ':'),
+             comment_prefixes=('#', ';'), inline_comment_prefixes=None,
+             strict=True, empty_lines_in_values=True,
+             default_section=DEFAULTSECT,
+             interpolation=_UNSET):
+    pass

test/simple_source/bug33/02_named_and_kwargs.py

@@ -0,0 +1,19 @@
+# Python 3.6 subprocess.py bug
+# Bug is getting params correct: timeout before **kwargs
+def call(*popenargs, timeout=None, **kwargs):
+    return
+
+# From 3.4 asyncio/base_events.py
+# Bug was in pickin up name of ** (kwargs)
+def subprocess_shell(self, protocol_factory, cmd, *, stdin=subprocess.PIPE,
+                     stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+                     universal_newlines=False, shell=True, bufsize=0,
+                     **kwargs):
+    return
+
+# From 3.4 asyncio/locks.py
+# Bug was handling" "value=1, *"
+
+class BoundedSemaphore(Semaphore):
+    def __init__(self, value=1, *, loop=None):
+        super().__init__(value, loop=loop)

test/simple_source/bug33/03_raise_from.py

@@ -0,0 +1,9 @@
+# From 3.4 _pyio.py
+# Bug is change in syntax between Python 2 and 3:
+#    raise_stmt ::=  "raise" expression "," expression
+# becomes:
+#    raise_stmt ::=  "raise" expression from expression
+try:
+    x = 1
+except AttributeError as err:
+    raise TypeError("an integer is required") from err

test/simple_source/bug33/04_aug_assign.py

@@ -0,0 +1,11 @@
+# From python 3.4 difflib
+# Bug in 3.x is combining bestsize +=1 with while condition
+bestsize = 5
+b = [2]
+def isbjunk(x):
+    return False
+
+while 1 < bestsize and \
+      not isbjunk(b[0]) and \
+      b[0]:
+    bestsize += 1

test/simple_source/bug35/02_fn_varargs.py

@@ -0,0 +1,5 @@
+# From python 3.4 pstats.py
+# Bug was not adding *, since *args covers that. And getting stream=None
+# without *
+def __init__(self, *args, stream=None):
+    pass

test/simple_source/bug35/05_return_in_else.py

@@ -0,0 +1,24 @@
+# Bug in 3.5 _pydecimal.py was erroriously thinking the
+# return after the "else" was an "end if"
+
+def parseline(self, line):
+    if not line:
+        return 5
+    elif line:
+        if hasattr(self, 'do_shell'):
+            line = 'shell'
+        else:
+            return 3 if line[3] else 4
+    return 6
+
+# From 3.5 gettext.py
+# In the below code, the "return" was erroneously
+# classifying RETURN_END_IF instead of RETURN_VALUE
+def find(domain):
+    for lang in domain:
+        if lang:
+            if all:
+                domain.append(5)
+            else:
+                return lang
+    return domain

test/simple_source/bug36/01_fstring.py

@@ -0,0 +1,3 @@
+var1 = 'x'
+var2 = 'y'
+print(f'interpolate {var1} strings {var2!r} {var2!s} py36')

test/simple_source/bug_pypy27/02_call_method.py

@@ -0,0 +1,4 @@
+# Bug in PyPy was not handling CALL_METHOD_xxx like
+# CALL_FUNCTION_XXX
+def truncate(self, size=None):
+    self.db.put(self.key, '', txn=self.txn, dlen=self.len - size, doff=size)

test/simple_source/expression/06_frozenset.py

@@ -1,5 +1,5 @@
 # Bug from Python 3.3 _markupbase.py cross compilatin
 # error in unmarshaling a frozenset
 import sys
-if sys.argv[0] in {"attlist", "linktype", "link", "element"}:
+if sys.argv[0] in frozenset({"attlist", "linktype", "link", "element"}):
     print("Yep")

test/simple_source/looping/10_while.py

@@ -1,2 +1,13 @@
+# From 3.2 text_file.py
+def readline(self, line):
+    while True:
+        if self.join_lines:
+            if line:
+                continue
+            if self:
+                continue
+
+        return line
+
 while __name__ != '__main__':
     b = 4

test/simple_source/stmts/00_import.py

@@ -3,3 +3,4 @@
 import sys
 from os import path
 from os import *
+import time as time1, os as os1

test/simple_source/stmts/09_whiletrue_bug.py

@@ -1,9 +1,12 @@
-if args == ['-']:
+if __file__ == ['-']:
     while True:
         try:
-            compile(filename, doraise=True)
+            compile(__file__, doraise=True)
         except RuntimeError:
             rv = 1
 else:
     rv = 1
 print(rv)
+
+
+while 1:pass

test/test_pyenvlib.py

@@ -27,10 +27,12 @@ from fnmatch import fnmatch
 
 #----- configure this for your needs
 
-TEST_VERSIONS=('2.3.7', '2.4.6', '2.5.6', '2.6.9', 'pypy-2.6.1',
-               'pypy-5.0.1',
-               '2.7.10', '2.7.11',
-               '3.2.6', '3.3.5', '3.4.2', '3.5.1')
+TEST_VERSIONS=('2.3.7', '2.4.6', '2.5.6', '2.6.9',
+               'pypy-2.4.0', 'pypy-2.6.1',
+               'pypy-5.0.1', 'pypy-5.3.1',
+               '2.7.10', '2.7.11', '2.7.12',
+               '3.1.5', '3.2.6', '3.3.5', '3.3.6',
+               '3.4.2', '3.5.1')
 
 target_base = '/tmp/py-dis/'
 lib_prefix = os.path.join(os.environ['HOME'], '.pyenv/versions')
@@ -110,11 +112,13 @@ if __name__ == '__main__':
     test_options_keys = list(test_options.keys())
     test_options_keys.sort()
     opts, args = getopt.getopt(sys.argv[1:], '',
-                               ['start-with=', 'verify', 'all', ] \
+                               ['start-with=', 'verify', 'weak-verify', 'all', ] \
                                + test_options_keys )
     for opt, val in opts:
         if opt == '--verify':
             do_verify = True
+        if opt == '--weak-verify':
+            do_verify = 'weak'
         elif opt == '--start-with':
             start_with = val
         elif opt[2:] in test_options_keys:

test/test_pythonlib.py

@@ -72,13 +72,15 @@ test_options = {
                   PYOC, 'base_2.7', 2.7),
 }
 
-for  vers in (2.7, 3.4, 3.5):
+for  vers in (2.7, 3.4, 3.5, 3.6):
     pythonlib = "ok_lib%s" % vers
     key = "ok-%s" % vers
     test_options[key] = (os.path.join(src_dir, pythonlib), PYOC, key, vers)
     pass
 
-for  vers in (2.3, 2.4, 2.5, 2.6, 2.7, 3.2, 3.3, 3.4, 3.5, 'pypy3.2', 'pypy2.7'):
+for  vers in (2.2, 2.3, 2.4, 2.5, 2.6, 2.7,
+              3.1, 3.2, 3.3,
+              3.4, 3.5, 3.6, 'pypy3.2', 'pypy2.7'):
     bytecode = "bytecode_%s" % vers
     key = "bytecode-%s" % vers
     test_options[key] =  (bytecode, PYC, bytecode, vers)
@@ -186,7 +188,7 @@ if __name__ == '__main__':
     test_options_keys = list(test_options.keys())
     test_options_keys.sort()
     opts, args = getopt.getopt(sys.argv[1:], '',
-                               ['start-with=', 'verify', 'all', 'compile',
+                               ['start-with=', 'verify', 'weak-verify', 'all', 'compile',
                                 'no-rm'] \
                                + test_options_keys )
     if not opts: help()
@@ -201,6 +203,8 @@ if __name__ == '__main__':
     for opt, val in opts:
         if opt == '--verify':
             test_opts['do_verify'] = True
+        elif opt == '--weak-verify':
+            test_opts['do_verify'] = 'weak'
         elif opt == '--compile':
             test_opts['do_compile'] = True
         elif opt == '--start-with':

uncompyle6/bin/uncompile.py

@@ -64,8 +64,8 @@ def usage():
 
 
 def main_bin():
-    if not (sys.version_info[0:2] in ((2, 6), (2, 7), (3, 2), (3, 3), (3, 4), (3, 5))):
-        print('Error: %s requires Python 2.6, 2.7, 3.2, 3.3, 3.4 or 3.5' % program,
+    if not (sys.version_info[0:2] in ((2, 6), (2, 7), (3, 2), (3, 3), (3, 4), (3, 5), (3, 6))):
+        print('Error: %s requires Python 2.6, 2.7, 3.2, 3.3, 3.4, 3.5, or 3.6' % program,
               file=sys.stderr)
         sys.exit(-1)
 

uncompyle6/disas.py

@@ -18,7 +18,7 @@ want to run on Python 2.7.
 
 from __future__ import print_function
 
-import os, sys
+import sys
 from collections import deque
 
 import uncompyle6
@@ -28,7 +28,7 @@ from xdis.code import iscode
 from xdis.load import check_object_path, load_module
 from uncompyle6.scanner import get_scanner
 
-def disco(version, co, out=None):
+def disco(version, co, out=None, is_pypy=False):
     """
     diassembles and deparses a given code block 'co'
     """
@@ -42,10 +42,10 @@ def disco(version, co, out=None):
         print('# Embedded file name: %s' % co.co_filename,
               file=real_out)
 
-    scanner = get_scanner(version)
+    scanner = get_scanner(version, is_pypy=is_pypy)
 
     queue = deque([co])
-    disco_loop(scanner.disassemble, queue, real_out)
+    disco_loop(scanner.ingest, queue, real_out)
 
 
 def disco_loop(disasm, queue, real_out):
@@ -61,7 +61,7 @@ def disco_loop(disasm, queue, real_out):
                 queue.append(t.pattr)
             elif iscode(t.attr):
                 queue.append(t.attr)
-            print(t.format(), file=real_out)
+            print(t, file=real_out)
             pass
         pass
 
@@ -82,7 +82,7 @@ def disassemble_file(filename, outstream=None, native=False):
         for con in co:
             disco(version, con, outstream)
     else:
-        disco(version, co, outstream)
+        disco(version, co, outstream, is_pypy=is_pypy)
     co = None
 
 def _test():

uncompyle6/main.py

@@ -12,9 +12,9 @@ from xdis.load import load_module
 def uncompyle(
         version, co, out=None, showasm=False, showast=False,
         timestamp=None, showgrammar=False, code_objects={},
-        is_pypy=False):
+        is_pypy=False, magic_int=None):
     """
-    disassembles and deparses a given code block 'co'
+    ingests and deparses a given code block 'co'
     """
     assert iscode(co)
 
@@ -22,8 +22,10 @@ def uncompyle(
     real_out = out or sys.stdout
     co_pypy_str = 'PyPy ' if is_pypy else ''
     run_pypy_str = 'PyPy ' if IS_PYPY else ''
-    print('# %sPython bytecode %s (disassembled from %sPython %s)\n' %
-              (co_pypy_str, version, run_pypy_str, PYTHON_VERSION),
+    print('# %sPython bytecode %s%s disassembled from %sPython %s' %
+          (co_pypy_str, version,
+           " (%d)" % magic_int if magic_int else "",
+           run_pypy_str, PYTHON_VERSION),
           file=real_out)
     if co.co_filename:
         print('# Embedded file name: %s' % co.co_filename,
@@ -60,11 +62,11 @@ def uncompyle_file(filename, outstream=None, showasm=False, showast=False,
         for con in co:
             uncompyle(version, con, outstream, showasm, showast,
                       timestamp, showgrammar, code_objects=code_objects,
-                      is_pypy=is_pypy)
+                      is_pypy=is_pypy, magic_int=magic_int)
     else:
         uncompyle(version, co, outstream, showasm, showast,
                   timestamp, showgrammar, code_objects=code_objects,
-                  is_pypy=is_pypy)
+                  is_pypy=is_pypy, magic_int=magic_int)
     co = None
 
 # FIXME: combine into an options parameter
@@ -145,8 +147,9 @@ def main(in_base, out_base, files, codes, outfile=None,
             if outfile:
                 outstream.close()
                 if do_verify:
+                    weak_verify = do_verify == 'weak'
                     try:
-                        msg = verify.compare_code_with_srcfile(infile, outfile)
+                        msg = verify.compare_code_with_srcfile(infile, outfile, weak_verify=weak_verify)
                         if not outfile:
                             if not msg:
                                 print('\n# okay decompiling %s' % infile)

uncompyle6/parser.py

@@ -72,7 +72,7 @@ class PythonParser(GenericASTBuilder):
         print("Instruction context:")
         for i in range(start, finish):
             indent = '   ' if i != index else '-> '
-            print("%s%s" % (indent, instructions[i].format()))
+            print("%s%s" % (indent, instructions[i]))
         raise ParserError(err_token, err_token.offset)
 
     def typestring(self, token):
@@ -236,17 +236,12 @@ class PythonParser(GenericASTBuilder):
         stmt ::= augassign2
         augassign1 ::= expr expr inplace_op designator
         augassign1 ::= expr expr inplace_op ROT_THREE STORE_SUBSCR
-        augassign1 ::= expr expr inplace_op ROT_TWO   STORE_SLICE+0
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+1
-        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+2
-        augassign1 ::= expr expr inplace_op ROT_FOUR  STORE_SLICE+3
         augassign2 ::= expr DUP_TOP LOAD_ATTR expr
                 inplace_op ROT_TWO   STORE_ATTR
 
         inplace_op ::= INPLACE_ADD
         inplace_op ::= INPLACE_SUBTRACT
         inplace_op ::= INPLACE_MULTIPLY
-        inplace_op ::= INPLACE_DIVIDE
         inplace_op ::= INPLACE_TRUE_DIVIDE
         inplace_op ::= INPLACE_FLOOR_DIVIDE
         inplace_op ::= INPLACE_MODULO
@@ -273,7 +268,6 @@ class PythonParser(GenericASTBuilder):
     def p_forstmt(self, args):
         """
         _for ::= GET_ITER FOR_ITER
-        _for ::= LOAD_CONST FOR_LOOP
 
         for_block ::= l_stmts_opt _come_from JUMP_BACK
         for_block ::= return_stmts _come_from
@@ -339,8 +333,6 @@ class PythonParser(GenericASTBuilder):
         imports_cont ::= imports_cont import_cont
         imports_cont ::= import_cont
         import_cont ::= LOAD_CONST LOAD_CONST import_as_cont
-        import_as_cont ::= IMPORT_NAME_CONT designator
-        import_as_cont ::= IMPORT_NAME_CONT load_attrs designator
         import_as_cont ::= IMPORT_FROM designator
 
         load_attrs ::= LOAD_ATTR
@@ -371,9 +363,6 @@ class PythonParser(GenericASTBuilder):
 
         stmt ::= setcomp_func
 
-        setcomp_func ::= BUILD_SET_0 LOAD_FAST FOR_ITER designator comp_iter
-                JUMP_BACK RETURN_VALUE RETURN_LAST
-
         comp_iter ::= comp_if
         comp_iter ::= comp_ifnot
         comp_iter ::= comp_for
@@ -381,9 +370,7 @@ class PythonParser(GenericASTBuilder):
         comp_body ::= set_comp_body
         comp_body ::= gen_comp_body
         comp_body ::= dict_comp_body
-        set_comp_body ::= expr SET_ADD
         gen_comp_body ::= expr YIELD_VALUE POP_TOP
-        dict_comp_body ::= expr expr MAP_ADD
 
         comp_if ::= expr jmp_false comp_iter
         comp_ifnot ::= expr jmp_true comp_iter
@@ -394,7 +381,6 @@ class PythonParser(GenericASTBuilder):
     def p_expr(self, args):
         '''
         expr ::= _mklambda
-        expr ::= SET_LINENO
         expr ::= LOAD_FAST
         expr ::= LOAD_NAME
         expr ::= LOAD_CONST
@@ -411,19 +397,17 @@ class PythonParser(GenericASTBuilder):
         expr ::= unary_expr
         expr ::= call_function
         expr ::= unary_not
-        expr ::= unary_convert
         expr ::= binary_subscr
         expr ::= binary_subscr2
         expr ::= load_attr
         expr ::= get_iter
-        expr ::= slice0
-        expr ::= slice1
-        expr ::= slice2
-        expr ::= slice3
         expr ::= buildslice2
         expr ::= buildslice3
         expr ::= yield
 
+        # Possibly Python < 2.3
+        # expr ::= SET_LINENO
+
         binary_expr ::= expr expr binary_op
         binary_op ::= BINARY_ADD
         binary_op ::= BINARY_MULTIPLY
@@ -431,7 +415,6 @@ class PythonParser(GenericASTBuilder):
         binary_op ::= BINARY_OR
         binary_op ::= BINARY_XOR
         binary_op ::= BINARY_SUBTRACT
-        binary_op ::= BINARY_DIVIDE
         binary_op ::= BINARY_TRUE_DIVIDE
         binary_op ::= BINARY_FLOOR_DIVIDE
         binary_op ::= BINARY_MODULO
@@ -445,21 +428,11 @@ class PythonParser(GenericASTBuilder):
         unary_op ::= UNARY_INVERT
 
         unary_not ::= expr UNARY_NOT
-        unary_convert ::= expr UNARY_CONVERT
 
         binary_subscr ::= expr expr BINARY_SUBSCR
-        binary_subscr2 ::= expr expr DUP_TOPX_2 BINARY_SUBSCR
 
         load_attr ::= expr LOAD_ATTR
         get_iter ::= expr GET_ITER
-        slice0 ::= expr SLICE+0
-        slice0 ::= expr DUP_TOP SLICE+0
-        slice1 ::= expr expr SLICE+1
-        slice1 ::= expr expr DUP_TOPX_2 SLICE+1
-        slice2 ::= expr expr SLICE+2
-        slice2 ::= expr expr DUP_TOPX_2 SLICE+2
-        slice3 ::= expr expr expr SLICE+3
-        slice3 ::= expr expr expr DUP_TOPX_3 SLICE+3
         buildslice3 ::= expr expr expr BUILD_SLICE_3
         buildslice2 ::= expr expr BUILD_SLICE_2
 
@@ -468,12 +441,6 @@ class PythonParser(GenericASTBuilder):
         _mklambda ::= load_closure mklambda
         _mklambda ::= mklambda
 
-        # Note: Python < 2.7 doesn't have *POP* or this. Remove from here?
-        # FIXME: segregate 2.7+
-
-        or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
-        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
-
         or   ::= expr jmp_true expr come_from_opt
         and  ::= expr jmp_false expr come_from_opt
         and2 ::= _jump jmp_false COME_FROM expr COME_FROM
@@ -492,14 +459,6 @@ class PythonParser(GenericASTBuilder):
         ret_expr_or_cond ::= ret_cond
         ret_expr_or_cond ::= ret_cond_not
 
-        # Note: Python < 2.7 doesn't have *POP* or this. Remove from here?
-        # FIXME: segregate 2.7+
-
-        ret_and  ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
-        ret_or   ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
-        ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
-        ret_cond_not ::= expr POP_JUMP_IF_TRUE expr RETURN_END_IF ret_expr_or_cond
-
         stmt ::= return_lambda
         stmt ::= conditional_lambda
 
@@ -511,15 +470,9 @@ class PythonParser(GenericASTBuilder):
         compare ::= expr expr COMPARE_OP
         cmp_list ::= expr cmp_list1 ROT_TWO POP_TOP
                 _come_from
-        cmp_list1 ::= expr DUP_TOP ROT_THREE
-                COMPARE_OP JUMP_IF_FALSE_OR_POP
-                cmp_list1 COME_FROM
         cmp_list1 ::= expr DUP_TOP ROT_THREE
                 COMPARE_OP jmp_false
                 cmp_list1 _come_from
-        cmp_list1 ::= expr DUP_TOP ROT_THREE
-                COMPARE_OP JUMP_IF_FALSE_OR_POP
-                cmp_list2 COME_FROM
         cmp_list1 ::= expr DUP_TOP ROT_THREE
                 COMPARE_OP jmp_false
                 cmp_list2 _come_from
@@ -564,10 +517,6 @@ class PythonParser(GenericASTBuilder):
         designator ::= STORE_GLOBAL
         designator ::= STORE_DEREF
         designator ::= expr STORE_ATTR
-        designator ::= expr STORE_SLICE+0
-        designator ::= expr expr STORE_SLICE+1
-        designator ::= expr expr STORE_SLICE+2
-        designator ::= expr expr expr STORE_SLICE+3
         designator ::= store_subscr
         store_subscr ::= expr expr STORE_SUBSCR
         designator ::= unpack
@@ -583,7 +532,7 @@ def parse(p, tokens, customize):
 
 
 def get_python_parser(
-        version, debug_parser, compile_mode='exec',
+        version, debug_parser={}, compile_mode='exec',
         is_pypy = False):
     """Returns parser object for Python version 2 or 3, 3.2, 3.5on,
     etc., depending on the parameters passed.  *compile_mode* is either
@@ -594,7 +543,13 @@ def get_python_parser(
 
     # FIXME: there has to be a better way...
     if version < 3.0:
-        if version == 2.3:
+        if version == 2.2:
+            import uncompyle6.parsers.parse22 as parse22
+            if compile_mode == 'exec':
+                p = parse22.Python22Parser(debug_parser)
+            else:
+                p = parse22.Python22ParserSingle(debug_parser)
+        elif version == 2.3:
             import uncompyle6.parsers.parse23 as parse23
             if compile_mode == 'exec':
                 p = parse23.Python23Parser(debug_parser)
@@ -635,7 +590,12 @@ def get_python_parser(
         pass
     else:
         import uncompyle6.parsers.parse3 as parse3
-        if version == 3.2:
+        if version == 3.1:
+            if compile_mode == 'exec':
+                p = parse3.Python31Parser(debug_parser)
+            else:
+                p = parse3.Python31ParserSingle(debug_parser)
+        elif version == 3.2:
             if compile_mode == 'exec':
                 p = parse3.Python32Parser(debug_parser)
             else:
@@ -651,11 +611,18 @@ def get_python_parser(
                 p = parse34.Python34Parser(debug_parser)
             else:
                 p = parse34.Python34ParserSingle(debug_parser)
-        elif version >= 3.5:
+        elif version == 3.5:
+            import uncompyle6.parsers.parse35 as parse35
             if compile_mode == 'exec':
-                p = parse3.Python35onParser(debug_parser)
+                p = parse35.Python35Parser(debug_parser)
             else:
-                p = parse3.Python35onParserSingle(debug_parser)
+                p = parse35.Python35ParserSingle(debug_parser)
+        elif version == 3.6:
+            import uncompyle6.parsers.parse36 as parse36
+            if compile_mode == 'exec':
+                p = parse36.Python36Parser(debug_parser)
+            else:
+                p = parse36.Python36ParserSingle(debug_parser)
         else:
             if compile_mode == 'exec':
                 p = parse3.Python3Parser(debug_parser)
@@ -684,10 +651,8 @@ def python_parser(version, co, out=sys.stdout, showasm=False,
                             example 2.6, 2.7, 3.2, 3.3, 3.4, 3.5 etc.
     :param co:              The code object to parse.
     :param out:             File like object to write the output to.
-    :param showasm:         Flag which determines whether the disassembled code
-                            is written to sys.stdout or not. (It is also to
-                            pass a file like object, into which the asm will be
-                            written).
+    :param showasm:         Flag which determines whether the disassembled and
+                            ingested code is written to sys.stdout or not.
     :param parser_debug:    dict containing debug flags for the spark parser.
 
     :return: Abstract syntax tree representation of the code object.
@@ -696,7 +661,7 @@ def python_parser(version, co, out=sys.stdout, showasm=False,
     assert iscode(co)
     from uncompyle6.scanner import get_scanner
     scanner = get_scanner(version, is_pypy)
-    tokens, customize = scanner.disassemble(co)
+    tokens, customize = scanner.ingest(co)
     maybe_show_asm(showasm, tokens)
 
     # For heavy grammar debugging

uncompyle6/parsers/parse2.py

@@ -40,6 +40,15 @@ class Python2Parser(PythonParser):
         print_nl ::= PRINT_NEWLINE
         '''
 
+    def p_stmt2(self, args):
+        """
+        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_BLOCK COME_FROM
+
+        exec_stmt ::= expr exprlist DUP_TOP EXEC_STMT
+        exec_stmt ::= expr exprlist EXEC_STMT
+
+        """
+
     def p_print_to(self, args):
         '''
         stmt ::= print_to
@@ -84,8 +93,6 @@ class Python2Parser(PythonParser):
         raise_stmt3 ::= expr expr expr RAISE_VARARGS_3
 
         stmt ::= exec_stmt
-        exec_stmt ::= expr exprlist DUP_TOP EXEC_STMT
-        exec_stmt ::= expr exprlist EXEC_STMT
 
         stmt ::= assert
         stmt ::= assert2
@@ -174,6 +181,11 @@ class Python2Parser(PythonParser):
         trystmt      ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
                          try_middle COME_FROM
 
+        try_middle   ::= JUMP_FORWARD COME_FROM except_stmts
+                         END_FINALLY COME_FROM
+        try_middle   ::= jmp_abs COME_FROM except_stmts
+                         END_FINALLY
+
         except_stmts ::= except_stmts except_stmt
         except_stmts ::= except_stmt
 
@@ -202,18 +214,24 @@ class Python2Parser(PythonParser):
         genexpr ::= LOAD_GENEXPR MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
         '''
 
-    def p_import2(self, args):
-        '''
-        # These might be relevant for only Python 2.0 or so.
-        # Not relevant for Python 3.
-        importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT IMPORT_STAR
-        importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT importlist2 POP_TOP
-        '''
+    # def p_import2(self, args):
+    #     '''
+    #     # These might be relevant for only Python 2.0 or so.
+    #     importstar ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT IMPORT_STAR
+    #     importfrom ::= LOAD_CONST LOAD_CONST IMPORT_NAME_CONT importlist2 POP_TOP
+    #     import_as_cont ::= IMPORT_NAME_CONT designator
+    #     import_as_cont ::= IMPORT_NAME_CONT load_attrs designator
+    #     '''
 
 
     def p_expr2(self, args):
-        '''
+        """
         expr ::= LOAD_LOCALS
+        expr ::= slice0
+        expr ::= slice1
+        expr ::= slice2
+        expr ::= slice3
+        expr ::= unary_convert
 
         slice0 ::= expr SLICE+0
         slice0 ::= expr DUP_TOP SLICE+0
@@ -223,10 +241,38 @@ class Python2Parser(PythonParser):
         slice2 ::= expr expr DUP_TOPX_2 SLICE+2
         slice3 ::= expr expr expr SLICE+3
         slice3 ::= expr expr expr DUP_TOPX_3 SLICE+3
+        unary_convert ::= expr UNARY_CONVERT
 
         # In Python 3, DUP_TOPX_2 is DUP_TOP_TWO
         binary_subscr2 ::= expr expr DUP_TOPX_2 BINARY_SUBSCR
-        '''
+        """
+
+    def p_slice2(self, args):
+        """
+        designator ::= expr STORE_SLICE+0
+        designator ::= expr expr STORE_SLICE+1
+        designator ::= expr expr STORE_SLICE+2
+        designator ::= expr expr expr STORE_SLICE+3
+        augassign1 ::= expr expr inplace_op ROT_TWO   STORE_SLICE+0
+        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+1
+        augassign1 ::= expr expr inplace_op ROT_THREE STORE_SLICE+2
+        augassign1 ::= expr expr inplace_op ROT_FOUR  STORE_SLICE+3
+        slice0 ::= expr SLICE+0
+        slice0 ::= expr DUP_TOP SLICE+0
+        slice1 ::= expr expr SLICE+1
+        slice1 ::= expr expr DUP_TOPX_2 SLICE+1
+        slice2 ::= expr expr SLICE+2
+        slice2 ::= expr expr DUP_TOPX_2 SLICE+2
+        slice3 ::= expr expr expr SLICE+3
+        slice3 ::= expr expr expr DUP_TOPX_3 SLICE+3
+        """
+
+    def p_op2(self, args):
+        """
+        inplace_op ::= INPLACE_DIVIDE
+        binary_op  ::= BINARY_DIVIDE
+        binary_subscr2 ::= expr expr DUP_TOPX_2 BINARY_SUBSCR
+        """
 
     def add_custom_rules(self, tokens, customize):
         '''
@@ -252,12 +298,14 @@ class Python2Parser(PythonParser):
         for opname, v in list(customize.items()):
             opname_base = opname[:opname.rfind('_')]
             if opname == 'PyPy':
-                self.add_unique_rules([
-                    'stmt ::= assign3_pypy',
-                    'stmt ::= assign2_pypy',
-                    'assign3_pypy ::= expr expr expr designator designator designator',
-                    'assign2_pypy ::= expr expr designator designator'
-                    ], customize)
+                self.addRule("""
+                    stmt ::= assign3_pypy
+                    stmt ::= assign2_pypy
+                    assign3_pypy ::= expr expr expr designator designator designator
+                    assign2_pypy ::= expr expr designator designator
+                    list_compr ::= expr  BUILD_LIST_FROM_ARG _for designator list_iter
+                                         JUMP_BACK
+                """, nop_func)
                 continue
             elif opname_base in ('BUILD_LIST', 'BUILD_TUPLE', 'BUILD_SET'):
                 thousands = (v//1024)
@@ -331,25 +379,31 @@ class Python2Parser(PythonParser):
             elif opname_base == 'UNPACK_LIST':
                 rule = 'unpack_list ::= ' + opname + ' designator'*v
             elif opname_base in ('DUP_TOPX', 'RAISE_VARARGS'):
-                # no need to add a rule
+                # FIXME: remove these conditions if they are not needed.
+                # no longer need to add a rule
                 continue
-                # rule = 'dup_topx ::= ' + 'expr '*v + opname
             elif opname_base == 'MAKE_FUNCTION':
                 self.addRule('mklambda ::= %s LOAD_LAMBDA %s' %
                       ('pos_arg '*v, opname), nop_func)
                 rule = 'mkfunc ::= %s LOAD_CONST %s' % ('expr '*v, opname)
             elif opname_base == 'MAKE_CLOSURE':
                 # FIXME: use add_unique_rules to tidy this up.
-                self.addRule('mklambda ::= %s load_closure LOAD_LAMBDA %s' %
-                      ('expr '*v, opname), nop_func)
-                self.addRule('genexpr ::= %s load_closure LOAD_GENEXPR %s expr GET_ITER CALL_FUNCTION_1' %
-                      ('expr '*v, opname), nop_func)
-                self.addRule('setcomp ::= %s load_closure LOAD_SETCOMP %s expr GET_ITER CALL_FUNCTION_1' %
-                      ('expr '*v, opname), nop_func)
-                self.addRule('dictcomp ::= %s load_closure LOAD_DICTCOMP %s expr GET_ITER CALL_FUNCTION_1' %
-                      ('expr '*v, opname), nop_func)
-                rule = 'mkfunc ::= %s load_closure LOAD_CONST %s' % ('expr '*v, opname)
-                # rule = 'mkfunc ::= %s closure_list LOAD_CONST %s' % ('expr '*v, opname)
+                self.add_unique_rules([
+                    ('mklambda ::= %s load_closure LOAD_LAMBDA %s' %
+                     ('expr '*v, opname)),
+                    ('genexpr ::= %s load_closure LOAD_GENEXPR %s expr'
+                     ' GET_ITER CALL_FUNCTION_1' %
+                     ('expr '*v, opname)),
+                    ('setcomp ::= %s load_closure LOAD_SETCOMP %s expr'
+                     ' GET_ITER CALL_FUNCTION_1' %
+                      ('expr '*v, opname)),
+                    ('dictcomp ::= %s load_closure LOAD_DICTCOMP %s expr'
+                     ' GET_ITER CALL_FUNCTION_1' %
+                      ('expr '*v, opname)),
+                    ('mkfunc ::= %s load_closure LOAD_CONST %s' %
+                     ('expr '*v, opname))],
+                    customize)
+                continue
             elif opname_base in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
                                  'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_KW'):
                 args_pos = (v & 0xff)          # positional parameters
@@ -360,7 +414,7 @@ class Python2Parser(PythonParser):
                        + 'expr ' * nak + opname
             elif opname_base == 'CALL_METHOD':
                 # PyPy only - DRY with parse3
-                args_pos = (v & 0xff)           # positional parameters
+                args_pos = (v & 0xff)          # positional parameters
                 args_kw = (v >> 8) & 0xff      # keyword parameters
                 # number of apply equiv arguments:
                 nak = ( len(opname_base)-len('CALL_METHOD') ) // 3

uncompyle6/parsers/parse22.py

@@ -0,0 +1,29 @@
+#  Copyright (c) 2016 Rocky Bernstein
+#  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
+
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6.parser import PythonParserSingle
+from uncompyle6.parsers.parse23 import Python23Parser
+
+class Python22Parser(Python23Parser):
+
+    def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
+        super(Python23Parser, self).__init__(debug_parser)
+        self.customized = {}
+
+    def p_misc22(self, args):
+        '''
+        _for ::= LOAD_CONST FOR_LOOP
+        '''
+
+class Python22ParserSingle(Python23Parser, PythonParserSingle):
+    pass
+
+if __name__ == '__main__':
+    # Check grammar
+    p = Python22Parser()
+    p.checkGrammar()
+    p.dumpGrammar()
+
+# local variables:
+# tab-width: 4

uncompyle6/parsers/parse23.py

@@ -14,9 +14,11 @@ class Python23Parser(Python24Parser):
 
     def p_misc23(self, args):
         '''
-        _while1test ::= JUMP_FORWARD JUMP_IF_FALSE POP_TOP COME_FROM
+        # Used to keep semantic positions the same across later versions
+        # of Python
+        _while1test ::= SETUP_LOOP JUMP_FORWARD JUMP_IF_FALSE POP_TOP COME_FROM
 
-        while1stmt ::= SETUP_LOOP _while1test l_stmts JUMP_BACK
+        while1stmt ::= _while1test l_stmts_opt JUMP_BACK
                        COME_FROM POP_TOP POP_BLOCK COME_FROM
 
         list_compr ::= BUILD_LIST_0 DUP_TOP LOAD_ATTR designator list_iter del_stmt

uncompyle6/parsers/parse24.py

@@ -26,6 +26,7 @@ class Python24Parser(Python25Parser):
 
         # Python 2.5+ omits POP_TOP POP_BLOCK
         while1stmt ::= SETUP_LOOP l_stmts JUMP_BACK POP_TOP POP_BLOCK COME_FROM
+        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_TOP POP_BLOCK COME_FROM
 
         # Python 2.5+:
         #  call_stmt ::= expr POP_TOP

uncompyle6/parsers/parse25.py

@@ -14,7 +14,6 @@ class Python25Parser(Python26Parser):
 
     def p_misc25(self, args):
         '''
-
         # If "return_if_stmt" is in a loop, a JUMP_BACK can be emitted. In 2.6 the
         # JUMP_BACK doesn't appear
 

uncompyle6/parsers/parse26.py

@@ -16,9 +16,10 @@ class Python26Parser(Python2Parser):
 
     def p_try_except26(self, args):
         """
+        except_stmt  ::= except_cond3 except_suite
         except_cond1 ::= DUP_TOP expr COMPARE_OP
                          JUMP_IF_FALSE POP_TOP POP_TOP POP_TOP POP_TOP
-        except_cond2 ::= DUP_TOP expr COMPARE_OP
+        except_cond3 ::= DUP_TOP expr COMPARE_OP
                          JUMP_IF_FALSE POP_TOP POP_TOP designator POP_TOP
 
         # Might be a bug from when COME_FROM wasn't properly handled
@@ -26,11 +27,7 @@ class Python26Parser(Python2Parser):
                          POP_TOP END_FINALLY come_froms
 
         try_middle   ::= JUMP_FORWARD COME_FROM except_stmts
-                         END_FINALLY come_froms
-        try_middle   ::= JUMP_FORWARD COME_FROM except_stmts
-                         come_from_pop END_FINALLY come_froms
-        try_middle   ::= JUMP_FORWARD COME_FROM except_stmts
-                         END_FINALLY come_froms
+                         come_from_pop END_FINALLY COME_FROM
 
         try_middle   ::= jmp_abs COME_FROM except_stmts
                          POP_TOP END_FINALLY
@@ -38,9 +35,6 @@ class Python26Parser(Python2Parser):
         try_middle   ::= jmp_abs COME_FROM except_stmts
                          come_from_pop END_FINALLY
 
-        try_middle   ::= jmp_abs COME_FROM except_stmts
-                         END_FINALLY
-
         trystmt      ::= SETUP_EXCEPT suite_stmts_opt come_from_pop
                          try_middle
 
@@ -114,7 +108,7 @@ class Python26Parser(Python2Parser):
         break_stmt ::= BREAK_LOOP JUMP_BACK
 
         # Semantic actions want the else to be at position 3
-        ifelsestmt  ::= testexpr c_stmts_opt jf_cf_pop else_suite COME_FROM
+        ifelsestmt  ::= testexpr c_stmts_opt jf_cf_pop else_suite come_froms
         ifelsestmt  ::= testexpr c_stmts_opt filler else_suitel come_froms POP_TOP
 
         # Semantic actions want else_suitel to be at index 3
@@ -157,6 +151,8 @@ class Python26Parser(Python2Parser):
         iflaststmtl ::= testexpr c_stmts_opt JUMP_BACK come_from_pop
         iflaststmt  ::= testexpr c_stmts_opt JUMP_ABSOLUTE come_from_pop
 
+        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK COME_FROM
+
         # Common with 2.7
         while1stmt ::= SETUP_LOOP return_stmts bp_come_from
         while1stmt ::= SETUP_LOOP return_stmts COME_FROM
@@ -196,12 +192,14 @@ class Python26Parser(Python2Parser):
         '''
         ret_and  ::= expr jmp_false ret_expr_or_cond COME_FROM
         ret_or   ::= expr jmp_true ret_expr_or_cond COME_FROM
-        ret_cond ::= expr jmp_false expr RETURN_END_IF come_from_pop ret_expr_or_cond
+        ret_cond ::= expr jmp_false expr RETURN_END_IF POP_TOP ret_expr_or_cond
         ret_cond ::= expr jmp_false expr ret_expr_or_cond
-        ret_cond_not ::= expr jmp_true expr RETURN_END_IF come_from_pop ret_expr_or_cond
+        ret_cond_not ::= expr jmp_true expr RETURN_END_IF POP_TOP ret_expr_or_cond
+
+        return_if_stmt ::= ret_expr RETURN_END_IF POP_TOP
+        return_stmt ::= ret_expr RETURN_VALUE POP_TOP
 
         # FIXME: split into Python 2.5
-        ret_cond ::= expr jmp_false expr JUMP_RETURN come_from_pop ret_expr_or_cond
         ret_or   ::= expr jmp_true ret_expr_or_cond come_froms
         '''
 
@@ -224,3 +222,20 @@ if __name__ == '__main__':
     # Check grammar
     p = Python26Parser()
     p.checkGrammar()
+    from uncompyle6 import PYTHON_VERSION, IS_PYPY
+    if PYTHON_VERSION == 2.6:
+        lhs, rhs, tokens, right_recursive = p.checkSets()
+        from uncompyle6.scanner import get_scanner
+        s = get_scanner(PYTHON_VERSION, IS_PYPY)
+        opcode_set = set(s.opc.opname).union(set(
+            """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
+               LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP
+               LAMBDA_MARKER RETURN_LAST
+            """.split()))
+        remain_tokens = set(tokens) - opcode_set
+        import re
+        remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+        remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+        remain_tokens = set(remain_tokens) - opcode_set
+        print(remain_tokens)
+        # print(sorted(p.rule2name.items()))

uncompyle6/parsers/parse27.py

@@ -12,20 +12,21 @@ class Python27Parser(Python2Parser):
         super(Python27Parser, self).__init__(debug_parser)
         self.customized = {}
 
-    def p_list_comprehension27(self, args):
+    def p_comprehension27(self, args):
         """
         list_for ::= expr _for designator list_iter JUMP_BACK
 
-        list_compr ::= expr  BUILD_LIST_FROM_ARG _for designator list_iter JUMP_BACK
+        setcomp_func ::= BUILD_SET_0 LOAD_FAST FOR_ITER designator comp_iter
+                JUMP_BACK RETURN_VALUE RETURN_LAST
+
+        dict_comp_body ::= expr expr MAP_ADD
+        set_comp_body ::= expr SET_ADD
+
+        # See also common Python p_list_comprehension
         """
 
     def p_try27(self, args):
         """
-        try_middle   ::= JUMP_FORWARD COME_FROM except_stmts
-                         END_FINALLY COME_FROM
-        try_middle   ::= jmp_abs COME_FROM except_stmts
-                         END_FINALLY
-
         tryelsestmt    ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
                            try_middle else_suite COME_FROM
 
@@ -39,11 +40,27 @@ class Python27Parser(Python2Parser):
     def p_jump27(self, args):
         """
         _ifstmts_jump ::= c_stmts_opt JUMP_FORWARD COME_FROM
+        bp_come_from    ::= POP_BLOCK COME_FROM
+
+        # FIXME: Common with 3.0+
         jmp_false ::= POP_JUMP_IF_FALSE
         jmp_true  ::= POP_JUMP_IF_TRUE
-        bp_come_from    ::= POP_BLOCK COME_FROM
-        """
 
+        ret_and  ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
+        ret_or   ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+        ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
+        ret_cond_not ::= expr POP_JUMP_IF_TRUE expr RETURN_END_IF ret_expr_or_cond
+
+        or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
+        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
+
+        cmp_list1 ::= expr DUP_TOP ROT_THREE
+                COMPARE_OP JUMP_IF_FALSE_OR_POP
+                cmp_list1 COME_FROM
+        cmp_list1 ::= expr DUP_TOP ROT_THREE
+                COMPARE_OP JUMP_IF_FALSE_OR_POP
+                cmp_list2 COME_FROM
+        """
 
     def p_stmt27(self, args):
         """
@@ -61,6 +78,8 @@ class Python27Parser(Python2Parser):
                 POP_BLOCK LOAD_CONST COME_FROM
                 WITH_CLEANUP END_FINALLY
 
+        while1stmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_BLOCK COME_FROM
+
         # Common with 2.6
         while1stmt ::= SETUP_LOOP return_stmts bp_come_from
         while1stmt ::= SETUP_LOOP return_stmts COME_FROM
@@ -73,3 +92,20 @@ if __name__ == '__main__':
     # Check grammar
     p = Python27Parser()
     p.checkGrammar()
+    from uncompyle6 import PYTHON_VERSION, IS_PYPY
+    if PYTHON_VERSION == 2.7:
+        lhs, rhs, tokens, right_recursive = p.checkSets()
+        from uncompyle6.scanner import get_scanner
+        s = get_scanner(PYTHON_VERSION, IS_PYPY)
+        opcode_set = set(s.opc.opname).union(set(
+            """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
+               LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP
+               LAMBDA_MARKER RETURN_LAST
+            """.split()))
+        remain_tokens = set(tokens) - opcode_set
+        import re
+        remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+        remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+        remain_tokens = set(remain_tokens) - opcode_set
+        print(remain_tokens)
+        # p.dumpGrammar()

uncompyle6/parsers/parse3.py

@@ -17,22 +17,18 @@ that a later phase can turn into a sequence of ASCII text.
 
 from __future__ import print_function
 
-from uncompyle6.parser import PythonParser, PythonParserSingle
+from uncompyle6.parser import PythonParser, PythonParserSingle, nop_func
 from uncompyle6.parsers.astnode import AST
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
-from uncompyle6 import PYTHON3
 
 class Python3Parser(PythonParser):
 
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
         self.added_rules = set()
-        if PYTHON3:
-            super().__init__(AST, 'stmts', debug=debug_parser)
-        else:
-            super(Python3Parser, self).__init__(AST, 'stmts', debug=debug_parser)
+        super(Python3Parser, self).__init__(AST, 'stmts', debug=debug_parser)
         self.new_rules = set()
 
-    def p_list_comprehension3(self, args):
+    def p_comprehension3(self, args):
         """
         # Python3 scanner adds LOAD_LISTCOMP. Python3 does list comprehension like
         # other comprehensions (set, dictionary).
@@ -51,6 +47,11 @@ class Python3Parser(PythonParser):
         jb_or_c ::= JUMP_BACK
         jb_or_c ::= CONTINUE
 
+        setcomp_func ::= BUILD_SET_0 LOAD_FAST FOR_ITER designator comp_iter
+                JUMP_BACK RETURN_VALUE RETURN_LAST
+        dict_comp_body ::= expr expr MAP_ADD
+        set_comp_body ::= expr SET_ADD
+
         # See also common Python p_list_comprehension
         """
 
@@ -89,10 +90,6 @@ class Python3Parser(PythonParser):
         raise_stmt2 ::= expr expr RAISE_VARARGS_2
         raise_stmt3 ::= expr expr expr RAISE_VARARGS_3
 
-        stmt ::= exec_stmt
-        exec_stmt ::= expr exprlist DUP_TOP EXEC_STMT
-        exec_stmt ::= expr exprlist EXEC_STMT
-
         stmt ::= assert
         stmt ::= assert2
         stmt ::= ifstmt
@@ -114,10 +111,6 @@ class Python3Parser(PythonParser):
         del_stmt ::= DELETE_FAST
         del_stmt ::= DELETE_NAME
         del_stmt ::= DELETE_GLOBAL
-        del_stmt ::= expr DELETE_SLICE+0
-        del_stmt ::= expr expr DELETE_SLICE+1
-        del_stmt ::= expr expr DELETE_SLICE+2
-        del_stmt ::= expr expr expr DELETE_SLICE+3
         del_stmt ::= delete_subscr
         delete_subscr ::= expr expr DELETE_SUBSCR
         del_stmt ::= expr DELETE_ATTR
@@ -262,6 +255,22 @@ class Python3Parser(PythonParser):
         come_froms ::= COME_FROM
         jmp_false ::= POP_JUMP_IF_FALSE
         jmp_true  ::= POP_JUMP_IF_TRUE
+
+        # FIXME: Common with 2.7
+        ret_and  ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
+        ret_or   ::= expr JUMP_IF_TRUE_OR_POP ret_expr_or_cond COME_FROM
+        ret_cond ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF ret_expr_or_cond
+        ret_cond_not ::= expr POP_JUMP_IF_TRUE expr RETURN_END_IF ret_expr_or_cond
+
+        or   ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
+        and  ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
+
+        cmp_list1 ::= expr DUP_TOP ROT_THREE
+                COMPARE_OP JUMP_IF_FALSE_OR_POP
+                cmp_list1 COME_FROM
+        cmp_list1 ::= expr DUP_TOP ROT_THREE
+                COMPARE_OP JUMP_IF_FALSE_OR_POP
+                cmp_list2 COME_FROM
         """
 
     def p_stmt3(self, args):
@@ -277,6 +286,7 @@ class Python3Parser(PythonParser):
         # Python < 3.5 no POP BLOCK
         whileTruestmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK _come_from
         whileTruestmt ::= SETUP_LOOP l_stmts_opt JUMP_BACK NOP _come_from
+        whileTruestmt ::= SETUP_LOOP return_stmts _come_from
         while1stmt ::= SETUP_LOOP l_stmts _come_from JUMP_BACK _come_from
         """
 
@@ -356,7 +366,7 @@ class Python3Parser(PythonParser):
         call_function ::= expr {expr}^n CALL_FUNCTION_KW_n POP_TOP
 
         classdefdeco2 ::= LOAD_BUILD_CLASS mkfunc {expr}^n-1 CALL_FUNCTION_n
-       """
+        """
         # Low byte indicates number of positional paramters,
         # high byte number of positional parameters
         args_pos = token.attr & 0xff
@@ -435,13 +445,33 @@ class Python3Parser(PythonParser):
         For PYPY:
             load_attr ::= expr LOOKUP_METHOD
             call_function ::= expr CALL_METHOD
-       """
+        """
+        saw_format_value = False
         for i, token in enumerate(tokens):
             opname = token.type
             opname_base = opname[:opname.rfind('_')]
 
-            if opname in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
-                          'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_KW'):
+            if opname == 'PyPy':
+                self.addRule("""
+                    stmt ::= assign3_pypy
+                    stmt ::= assign2_pypy
+                    assign3_pypy ::= expr expr expr designator designator designator
+                    assign2_pypy ::= expr expr designator designator
+                """, nop_func)
+                continue
+            elif opname == 'FORMAT_VALUE':
+                # Python 3.6+
+                self.addRule("""
+                formatted_value ::= expr FORMAT_VALUE
+                formatted_value ::= expr FORMAT_VALUE
+                str ::= LOAD_CONST
+                formatted_value_or_str ::= formatted_value
+                formatted_value_or_str ::= str
+                """, nop_func)
+                saw_format_value = True
+
+            elif opname in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
+                            'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_KW'):
                 self.custom_classfunc_rule(opname, token, customize)
             elif opname == 'LOAD_DICTCOMP':
                 rule_pat = ("dictcomp ::= LOAD_DICTCOMP %sMAKE_FUNCTION_0 expr "
@@ -462,6 +492,16 @@ class Python3Parser(PythonParser):
                 if opname_base == 'BUILD_TUPLE':
                     rule = ('load_closure ::= %s%s' % (('LOAD_CLOSURE ' * v), opname))
                     self.add_unique_rule(rule, opname, token.attr, customize)
+                if opname_base == 'BUILD_LIST' and saw_format_value:
+                    saw_format_value = False
+                    format_or_str_n = "formatted_value_or_str_%s" % v
+                    self.addRule("""
+                    expr ::= joined_str
+                    joined_str ::=  LOAD_CONST LOAD_ATTR %s CALL_FUNCTION_1
+                    %s ::= %s%s
+                    """ % (format_or_str_n, format_or_str_n, ("formatted_value_or_str " *v), opname),
+                                 nop_func)
+
             elif opname == 'LOOKUP_METHOD':
                 # A PyPy speciality - DRY with parse2
                 self.add_unique_rule("load_attr ::= expr LOOKUP_METHOD",
@@ -542,14 +582,17 @@ class Python3Parser(PythonParser):
                 self.add_unique_rule(rule, opname, token.attr, customize)
             elif opname_base == 'CALL_METHOD':
                 # PyPy only - DRY with parse2
+
+                # FIXME: The below argument parsing will be wrong when PyPy gets to 3.6
                 args_pos = (token.attr & 0xff)          # positional parameters
                 args_kw = (token.attr >> 8) & 0xff      # keyword parameters
+
                 # number of apply equiv arguments:
                 nak = ( len(opname_base)-len('CALL_METHOD') ) // 3
                 rule = ('call_function ::= expr '
                         + ('pos_arg ' * args_pos)
                         + ('kwarg ' * args_kw)
-                        + 'expr ' * nak + token.type)
+                        + 'expr ' * nak + opname)
                 self.add_unique_rule(rule, opname, token.attr, customize)
             elif opname.startswith('MAKE_CLOSURE'):
                 # DRY with MAKE_FUNCTION
@@ -577,11 +620,14 @@ class Python3Parser(PythonParser):
                 # FIXME: kwarg processing is missing here.
                 # Note order of kwargs and pos args changed between 3.3-3.4
                 if self.version <= 3.2:
-                    rule = ('mkfunc ::= %sload_closure LOAD_CONST kwargs %s'
+                    rule = ('mkfunc ::= kwargs %sload_closure LOAD_CONST kwargs %s'
                             % ('expr ' * args_pos, opname))
-                elif self.version >= 3.3:
+                elif self.version == 3.3:
                     rule = ('mkfunc ::= kwargs %sload_closure LOAD_CONST LOAD_CONST %s'
                             % ('expr ' * args_pos, opname))
+                elif self.version >= 3.4:
+                    rule = ('mkfunc ::= %skwargs load_closure LOAD_CONST LOAD_CONST %s'
+                            % ('expr ' * args_pos, opname))
 
                 self.add_unique_rule(rule, opname, token.attr, customize)
                 rule = ('mkfunc ::= %sload_closure load_genexpr %s'
@@ -593,7 +639,17 @@ class Python3Parser(PythonParser):
         return
 
 
+class Python31Parser(Python3Parser):
+
+    def p_31(self, args):
+        """
+        # Store locals is only in Python 3.0 to 3.3
+        stmt ::= store_locals
+        store_locals ::= LOAD_FAST STORE_LOCALS
+        """
+
 class Python32Parser(Python3Parser):
+
     def p_32(self, args):
         """
         # Store locals is only in Python 3.0 to 3.3
@@ -613,53 +669,13 @@ class Python33Parser(Python3Parser):
         yield_from ::= expr expr YIELD_FROM
         """
 
-class Python35onParser(Python3Parser):
-    def p_35on(self, args):
-        """
-        # Python 3.5+ has WITH_CLEANUP_START/FINISH
-
-        withstmt ::= expr SETUP_WITH exprlist suite_stmts_opt
-                    POP_BLOCK LOAD_CONST COME_FROM
-                    WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
-
-        withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt
-                     POP_BLOCK LOAD_CONST COME_FROM
-                     WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
-
-        withasstmt ::= expr SETUP_WITH designator suite_stmts_opt
-                POP_BLOCK LOAD_CONST COME_FROM
-                WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
-
-        inplace_op ::= INPLACE_MATRIX_MULTIPLY
-        binary_op  ::= BINARY_MATRIX_MULTIPLY
-
-        # Python 3.5+ does jump optimization
-        # In <.3.5 the below is a JUMP_FORWARD to a JUMP_ABSOLUTE.
-        # in return_stmt, we will need the semantic actions in pysource.py
-        # to work out whether to dedent or not based on the presence of
-        # RETURN_END_IF vs RETURN_VALUE
-
-        ifelsestmtc ::= testexpr c_stmts_opt JUMP_FORWARD else_suitec
-        return_stmt ::= ret_expr RETURN_END_IF
-
-
-        # Python 3.3+ also has yield from. 3.5 does it
-        # differently than 3.3, 3.4
-
-        expr ::= yield_from
-        yield_from ::= expr GET_YIELD_FROM_ITER LOAD_CONST YIELD_FROM
-
-        # Python 3.4+ has more loop optimization that removes
-        # JUMP_FORWARD in some cases, and hence we also don't
-        # see COME_FROM
-        _ifstmts_jump ::= c_stmts_opt
-
-        """
-
 class Python3ParserSingle(Python3Parser, PythonParserSingle):
     pass
 
 
+class Python31ParserSingle(Python31Parser, PythonParserSingle):
+    pass
+
 class Python32ParserSingle(Python32Parser, PythonParserSingle):
     pass
 
@@ -667,18 +683,15 @@ class Python32ParserSingle(Python32Parser, PythonParserSingle):
 class Python33ParserSingle(Python33Parser, PythonParserSingle):
     pass
 
-class Python35onParserSingle(Python35onParser, PythonParserSingle):
-    pass
-
 def info(args):
     # Check grammar
     # Should also add a way to dump grammar
-    import sys
     p = Python3Parser()
     if len(args) > 0:
         arg = args[0]
         if arg == '3.5':
-            p = Python35onParser()
+            from uncompyle6.parser.parse35 import Python35Parser
+            p = Python35Parser()
         elif arg == '3.3':
             p = Python33Parser()
         elif arg == '3.2':

uncompyle6/parsers/parse34.py

@@ -46,13 +46,24 @@ class Python34ParserSingle(Python34Parser, PythonParserSingle):
     pass
 
 
-def info(args):
+if __name__ == '__main__':
     # Check grammar
-    # Should also add a way to dump grammar
     p = Python34Parser()
     p.checkGrammar()
-
-
-if __name__ == '__main__':
-    import sys
-    info(sys.argv)
+    from uncompyle6 import PYTHON_VERSION, IS_PYPY
+    if PYTHON_VERSION == 3.4:
+        lhs, rhs, tokens, right_recursive = p.checkSets()
+        from uncompyle6.scanner import get_scanner
+        s = get_scanner(PYTHON_VERSION, IS_PYPY)
+        opcode_set = set(s.opc.opname).union(set(
+            """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
+               LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_CLASSNAME
+               LAMBDA_MARKER RETURN_LAST
+            """.split()))
+        remain_tokens = set(tokens) - opcode_set
+        import re
+        remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+        remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+        remain_tokens = set(remain_tokens) - opcode_set
+        print(remain_tokens)
+        # print(sorted(p.rule2name.items()))

uncompyle6/parsers/parse35.py

@@ -0,0 +1,79 @@
+#  Copyright (c) 2016 Rocky Bernstein
+"""
+spark grammar differences over Python3 for Python 3.5.
+"""
+from __future__ import print_function
+
+from uncompyle6.parser import PythonParserSingle
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6.parsers.parse3 import Python3Parser
+
+class Python35Parser(Python3Parser):
+
+    def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
+        super(Python35Parser, self).__init__(debug_parser)
+        self.customized = {}
+
+    def p_35on(self, args):
+        """
+        # Python 3.5+ has WITH_CLEANUP_START/FINISH
+
+        withstmt ::= expr SETUP_WITH exprlist suite_stmts_opt
+                    POP_BLOCK LOAD_CONST COME_FROM
+                    WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+
+        withstmt ::= expr SETUP_WITH POP_TOP suite_stmts_opt
+                     POP_BLOCK LOAD_CONST COME_FROM
+                     WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+
+        withasstmt ::= expr SETUP_WITH designator suite_stmts_opt
+                POP_BLOCK LOAD_CONST COME_FROM
+                WITH_CLEANUP_START WITH_CLEANUP_FINISH END_FINALLY
+
+        inplace_op ::= INPLACE_MATRIX_MULTIPLY
+        binary_op  ::= BINARY_MATRIX_MULTIPLY
+
+        # Python 3.5+ does jump optimization
+        # In <.3.5 the below is a JUMP_FORWARD to a JUMP_ABSOLUTE.
+        # in return_stmt, we will need the semantic actions in pysource.py
+        # to work out whether to dedent or not based on the presence of
+        # RETURN_END_IF vs RETURN_VALUE
+
+        ifelsestmtc ::= testexpr c_stmts_opt JUMP_FORWARD else_suitec
+        # ifstmt ::= testexpr c_stmts_opt
+
+        # Python 3.3+ also has yield from. 3.5 does it
+        # differently than 3.3, 3.4
+
+        expr ::= yield_from
+        yield_from ::= expr GET_YIELD_FROM_ITER LOAD_CONST YIELD_FROM
+
+        # Python 3.4+ has more loop optimization that removes
+        # JUMP_FORWARD in some cases, and hence we also don't
+        # see COME_FROM
+        _ifstmts_jump ::= c_stmts_opt
+        """
+class Python35ParserSingle(Python35Parser, PythonParserSingle):
+    pass
+
+if __name__ == '__main__':
+    # Check grammar
+    p = Python35Parser()
+    p.checkGrammar()
+    from uncompyle6 import PYTHON_VERSION, IS_PYPY
+    if PYTHON_VERSION == 3.5:
+        lhs, rhs, tokens, right_recursive = p.checkSets()
+        from uncompyle6.scanner import get_scanner
+        s = get_scanner(PYTHON_VERSION, IS_PYPY)
+        opcode_set = set(s.opc.opname).union(set(
+            """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
+               LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_CLASSNAME
+               LAMBDA_MARKER RETURN_LAST
+            """.split()))
+        remain_tokens = set(tokens) - opcode_set
+        import re
+        remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+        remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+        remain_tokens = set(remain_tokens) - opcode_set
+        print(remain_tokens)
+        # print(sorted(p.rule2name.items()))

uncompyle6/parsers/parse36.py

@@ -0,0 +1,52 @@
+#  Copyright (c) 2016 Rocky Bernstein
+"""
+spark grammar differences over Python 3.5 for Python 3.6.
+"""
+from __future__ import print_function
+
+from uncompyle6.parser import PythonParserSingle
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6.parsers.parse35 import Python35Parser
+
+class Python36Parser(Python35Parser):
+
+    def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
+        super(Python36Parser, self).__init__(debug_parser)
+        self.customized = {}
+
+    def p_36misc(self, args):
+        """
+        formatted_value ::= LOAD_FAST FORMAT_VALUE
+        str ::= LOAD_CONST
+        joined_str ::=  LOAD_CONST LOAD_ATTR format_value_or_strs
+                        BUILD_LIST CALL_FUNCTION
+        format_value_or_strs ::= format_value_or_strs format_value_or_str
+        format_value_or_strs ::= format_value_or_str
+        format_value_or_str ::= format_value
+        format_value_or_str ::= str
+        """
+
+class Python36ParserSingle(Python36Parser, PythonParserSingle):
+    pass
+
+if __name__ == '__main__':
+    # Check grammar
+    p = Python36Parser()
+    p.checkGrammar()
+    from uncompyle6 import PYTHON_VERSION, IS_PYPY
+    if PYTHON_VERSION == 3.6:
+        lhs, rhs, tokens, right_recursive = p.checkSets()
+        from uncompyle6.scanner import get_scanner
+        s = get_scanner(PYTHON_VERSION, IS_PYPY)
+        opcode_set = set(s.opc.opname).union(set(
+            """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
+               LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_CLASSNAME
+               LAMBDA_MARKER RETURN_LAST
+            """.split()))
+        remain_tokens = set(tokens) - opcode_set
+        import re
+        remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+        remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+        remain_tokens = set(remain_tokens) - opcode_set
+        print(remain_tokens)
+        # print(sorted(p.rule2name.items()))

uncompyle6/scanner.py

@@ -6,23 +6,20 @@
 #  See LICENSE
 #
 """
-scanner/disassembler module. From here we call various version-specific
+scanner/ingestion module. From here we call various version-specific
 scanners, e.g. for Python 2.7 or 3.4.
-
-This overlaps Python's dis module, but it can be run from Python 2 or
-Python 3 and other versions of Python. Also, we save token information
-for later use in deparsing.
 """
 
 from __future__ import print_function
 
 import sys
 
-from uncompyle6 import PYTHON3
+from uncompyle6 import PYTHON3, IS_PYPY
 from uncompyle6.scanners.tok import Token
 
 # The byte code versions we support
-PYTHON_VERSIONS = (2.3, 2.4, 2.5, 2.6, 2.7, 3.2, 3.3, 3.4, 3.5)
+PYTHON_VERSIONS = (2.2, 2.3, 2.4, 2.5, 2.6, 2.7,
+                   3.1, 3.2, 3.3, 3.4, 3.5, 3.6)
 
 # FIXME: DRY
 if PYTHON3:
@@ -44,7 +41,7 @@ class Code(object):
         for i in dir(co):
             if i.startswith('co_'):
                 setattr(self, i, getattr(co, i))
-        self._tokens, self._customize = scanner.disassemble(co, classname)
+        self._tokens, self._customize = scanner.ingest(co, classname)
 
 class Scanner(object):
 
@@ -251,11 +248,13 @@ class Scanner(object):
         self.Token = tokenClass
         return self.Token
 
+def op_has_argument(op, opc):
+    return op >= opc.HAVE_ARGUMENT
 
 def parse_fn_counts(argc):
     return ((argc & 0xFF), (argc >> 8) & 0xFF, (argc >> 16) & 0x7FFF)
 
-def get_scanner(version, show_asm=None, is_pypy=False):
+def get_scanner(version, is_pypy=False, show_asm=None):
     # Pick up appropriate scanner
     if version in PYTHON_VERSIONS:
         v_str = "%s" % (int(version * 10))
@@ -282,5 +281,5 @@ def get_scanner(version, show_asm=None, is_pypy=False):
 if __name__ == "__main__":
     import inspect, uncompyle6
     co = inspect.currentframe().f_code
-    scanner = get_scanner(uncompyle6.PYTHON_VERSION, True)
-    tokens, customize = scanner.disassemble(co, {})
+    scanner = get_scanner(uncompyle6.PYTHON_VERSION, IS_PYPY, True)
+    tokens, customize = scanner.ingest(co, {})

uncompyle6/scanners/scanner2.py

@@ -39,12 +39,12 @@ class Scanner2(scan.Scanner):
         # For <2.5 it is <generator expression>
         self.genexpr_name = '<genexpr>';
 
-    def disassemble(self, co, classname=None, code_objects={}, show_asm=None):
+    def ingest(self, co, classname=None, code_objects={}, show_asm=None):
         """
         Pick out tokens from an uncompyle6 code object, and transform them,
         returning a list of uncompyle6 'Token's.
 
-        The tranformations are made to assist the deparsing grammar.
+        The transformations are made to assist the deparsing grammar.
         Specificially:
            -  various types of LOAD_CONST's are categorized in terms of what they load
            -  COME_FROM instructions are added to assist parsing control structures
@@ -56,13 +56,18 @@ class Scanner2(scan.Scanner):
         """
 
         show_asm = self.show_asm if not show_asm else show_asm
-        # show_asm = 'before'
+        # show_asm = 'after'
         if show_asm in ('both', 'before'):
             from xdis.bytecode import Bytecode
             bytecode = Bytecode(co, self.opc)
             for instr in bytecode.get_instructions(co):
                 print(instr._disassemble())
 
+        # from xdis.bytecode import Bytecode
+        # bytecode = Bytecode(co, self.opc)
+        # for instr in bytecode.get_instructions(co):
+        #     print(instr._disassemble())
+
         # Container for tokens
         tokens = []
 
@@ -77,7 +82,7 @@ class Scanner2(scan.Scanner):
         self.build_lines_data(co, n)
         self.build_prev_op(n)
 
-        # self.lines contains (block,addrLastInstr)
+        # class and names
         if classname:
             classname = '_' + classname.lstrip('_') + '__'
 
@@ -120,6 +125,7 @@ class Scanner2(scan.Scanner):
 
         cf = self.find_jump_targets()
         # contains (code, [addrRefToCode])
+
         last_stmt = self.next_stmt[0]
         i = self.next_stmt[last_stmt]
         replace = {}
@@ -218,8 +224,19 @@ class Scanner2(scan.Scanner):
                 # in arbitrary value 0.
                 customize[opname] = 0
             elif op == self.opc.JUMP_ABSOLUTE:
+                # Further classify JUMP_ABSOLUTE into backward jumps
+                # which are used in loops, and "CONTINUE" jumps which
+                # may appear in a "continue" statement.  The loop-type
+                # and continue-type jumps will help us classify loop
+                # boundaries The continue-type jumps help us get
+                # "continue" statements with would otherwise be turned
+                # into a "pass" statement because JUMPs are sometimes
+                # ignored in rules as just boundary overhead. In
+                # comprehensions we might sometimes classify JUMP_BACK
+                # as CONTINUE, but that's okay since we add a grammar
+                # rule for that.
                 target = self.get_target(offset)
-                if target < offset:
+                if target <= offset:
                     if (offset in self.stmts
                         and self.code[offset+3] not in (self.opc.END_FINALLY,
                                                         self.opc.POP_BLOCK)
@@ -253,7 +270,7 @@ class Scanner2(scan.Scanner):
 
         if show_asm in ('both', 'after'):
             for t in tokens:
-                print(t.format())
+                print(t)
             print()
         return tokens, customize
 
@@ -305,7 +322,7 @@ class Scanner2(scan.Scanner):
         self.lines = []
         linetuple = namedtuple('linetuple', ['l_no', 'next'])
 
-        # linestarts is a tuple of (offset, line number).
+        # self.linestarts is a tuple of (offset, line number).
         # Turn that in a has that we can index
         self.linestarts = list(self.opc.findlinestarts(co))
         self.linestartoffsets = {}
@@ -370,12 +387,24 @@ class Scanner2(scan.Scanner):
                 j = self.prev[s]
                 while code[j] == self.opc.JUMP_ABSOLUTE:
                     j = self.prev[j]
-                if code[j] == self.opc.LIST_APPEND: # list comprehension
+                if (self.version >= 2.3 and
+                    self.opc.opname[code[j]] == 'LIST_APPEND'): # list comprehension
+                    stmts.remove(s)
+                    continue
+            elif code[s] == self.opc.POP_TOP:
+                # The POP_TOP in:
+                #   ROT_TWO, POP_TOP,
+                #   RETURN_xxx, POP_TOP (in 2.6-), or
+                #   JUMP_IF_{FALSE,TRUE}, POP_TOP  (in 2.6-)
+                # is part of the previous instruction and not the
+                # beginning of a new statement
+                prev = code[self.prev[s]]
+                if (prev == self.opc.ROT_TWO or
+                    self.version < 2.7 and prev in
+                    (self.opc.JUMP_IF_FALSE, self.opc.JUMP_IF_TRUE,
+                     self.opc.RETURN_VALUE)):
                     stmts.remove(s)
                     continue
-            elif code[s] == self.opc.POP_TOP and code[self.prev[s]] == self.opc.ROT_TWO:
-                stmts.remove(s)
-                continue
             elif code[s] in self.designator_ops:
                 j = self.prev[s]
                 while code[j] in self.designator_ops:
@@ -399,8 +428,8 @@ class Scanner2(scan.Scanner):
             if except_match:
                 jmp = self.prev[self.get_target(except_match)]
 
-                # In Python <= 2.6 we may have jumps to jumps
-                if self.version <= 2.6 and self.code[jmp] in self.jump_forward:
+                # In Python < 2.7 we may have jumps to jumps
+                if self.version < 2.7 and self.code[jmp] in self.jump_forward:
                     self.not_continue.add(jmp)
                     jmp = self.get_target(jmp)
                     if jmp not in self.pop_jump_if | self.jump_forward:
@@ -426,7 +455,7 @@ class Scanner2(scan.Scanner):
             elif op in self.setup_ops:
                 count_SETUP_ += 1
 
-    def detect_structure(self, pos, op=None):
+    def detect_structure(self, pos, op):
         '''
         Detect type of block structures and their boundaries to fix optimized jumps
         in python2.3+
@@ -435,26 +464,23 @@ class Scanner2(scan.Scanner):
         # TODO: check the struct boundaries more precisely -Dan
 
         code = self.code
-        # Ev remove this test and make op a mandatory argument -Dan
-        if op is None:
-            op = code[pos]
 
         # Detect parent structure
         parent = self.structs[0]
         start  = parent['start']
         end    = parent['end']
-        for s in self.structs:
-            _start = s['start']
-            _end   = s['end']
+        for struct in self.structs:
+            _start = struct['start']
+            _end   = struct['end']
             if (_start <= pos < _end) and (_start >= start and _end <= end):
                 start  = _start
                 end    = _end
-                parent = s
+                parent = struct
 
         if op == self.opc.SETUP_LOOP:
 
             # We categorize loop types: 'for', 'while', 'while 1' with
-            # possibly suffices '-loop' and '-else'
+            # possibly suffixes '-loop' and '-else'
             # Try to find the jump_back instruction of the loop.
             # It could be a return instruction.
 
@@ -616,16 +642,17 @@ class Scanner2(scan.Scanner):
                 self.fixed_jumps[i] = i+1
 
         elif op in self.pop_jump_if:
-            start = pos+3
             target = self.get_target(pos, op)
             rtarget = self.restrict_to_parent(target, parent)
-            pre = self.prev
 
             # Do not let jump to go out of parent struct bounds
             if target != rtarget and parent['type'] == 'and/or':
                 self.fixed_jumps[pos] = rtarget
                 return
 
+            start = pos+3
+            pre = self.prev
+
             # Does this jump to right after another conditional jump that is
             # not myself?  If so, it's part of a larger conditional.
             # rocky: if we have a conditional jump to the next instruction, then
@@ -645,15 +672,13 @@ class Scanner2(scan.Scanner):
                                        'end':   pre[target]})
                 return
 
-            # Is it an "and" inside an "if" block
+            # Is it an "and" inside an "if" or "while" block
             if op == self.opc.PJIF:
+
                 # Search for other POP_JUMP_IF_FALSE targetting the same op,
                 # in current statement, starting from current offset, and filter
                 # everything inside inner 'or' jumps and midline ifs
                 match = self.rem_or(start, self.next_stmt[pos], self.opc.PJIF, target)
-                ## We can't remove mid-line ifs because line structures have changed
-                ## from restructBytecode().
-                ##  match = self.remove_mid_line_ifs(match)
 
                 # If we still have any offsets in set, start working on it
                 if match:
@@ -688,9 +713,18 @@ class Scanner2(scan.Scanner):
                             self.fixed_jumps[pos] = fix or match[-1]
                             return
                     else:
-                        self.fixed_jumps[pos] = match[-1]
+                        if (self.version < 2.7
+                            and parent['type'] in ('root', 'for-loop', 'if-then',
+                                                   'if-else', 'try')):
+                            self.fixed_jumps[pos] = rtarget
+                        else:
+                            # note test for < 2.7 might be superflous although informative
+                            # for 2.7 a different branch is taken and the below code is handled
+                            # under: elif op in self.pop_jump_if_or_pop
+                            # below
+                            self.fixed_jumps[pos] = match[-1]
                         return
-            else: # op == self.opc.PJIT
+            else: # op != self.opc.PJIT
                 if self.version < 2.7 and code[pos+3] == self.opc.POP_TOP:
                     assert_pos = pos + 4
                 else:
@@ -769,12 +803,11 @@ class Scanner2(scan.Scanner):
 
     def find_jump_targets(self):
         '''
-        Detect all offsets in a byte code which are jump targets.
+        Detect all offsets in a byte code which are jump targets
+        where we might insert a COME_FROM instruction.
 
-        Return the list of offsets.
-
-        This procedure is modelled after dis.findlabels(), but here
-        for each target the number of jumps are counted.
+        Return the list of offsets. An instruction can be jumped
+        to in from multiple instructions.
         '''
 
         n = len(self.code)
@@ -791,30 +824,53 @@ class Scanner2(scan.Scanner):
         self.return_end_ifs = set()
 
         targets = {}
-        for i in self.op_range(0, n):
-            op = self.code[i]
+        for offset in self.op_range(0, n):
+            op = self.code[offset]
 
             # Determine structures and fix jumps in Python versions
             # since 2.3
-            self.detect_structure(i, op)
+            self.detect_structure(offset, op)
 
             if op >= self.opc.HAVE_ARGUMENT:
-                label = self.fixed_jumps.get(i)
-                oparg = self.get_argument(i)
+                label = self.fixed_jumps.get(offset)
+                oparg = self.get_argument(offset)
+
                 if label is None:
                     if op in self.opc.hasjrel and op != self.opc.FOR_ITER:
-                        label = i + 3 + oparg
+                        label = offset + 3 + oparg
                     elif self.version == 2.7 and op in self.opc.hasjabs:
                         if op in (self.opc.JUMP_IF_FALSE_OR_POP,
                                   self.opc.JUMP_IF_TRUE_OR_POP):
-                            if (oparg > i):
+                            if (oparg > offset):
                                 label = oparg
+                                pass
+                            pass
 
+
+                # FIXME: All the < 2.7 conditions are is horrible. We need a better way.
                 if label is not None and label != -1:
-                    targets[label] = targets.get(label, []) + [i]
-            elif op == self.opc.END_FINALLY and i in self.fixed_jumps:
-                label = self.fixed_jumps[i]
-                targets[label] = targets.get(label, []) + [i]
+                    # In Python < 2.7, the POP_TOP in:
+                    #   RETURN_VALUE, POP_TOP
+                    # does now start a new statement
+                    # Otherwise, we have want to add a "COME_FROM"
+                    if not (self.version < 2.7 and
+                            self.code[label] == self.opc.POP_TOP and
+                            self.code[self.prev[label]] == self.opc.RETURN_VALUE):
+                        # In Python < 2.7, don't add a COME_FROM, for:
+                        #     JUMP_FORWARD, END_FINALLY
+                        # or:
+                        #     JUMP_FORWARD, POP_TOP, END_FINALLY
+                        if not (self.version < 2.7 and op == self.opc.JUMP_FORWARD
+                                and ((self.code[offset+3] == self.opc.END_FINALLY)
+                                     or (self.code[offset+3] == self.opc.POP_TOP
+                                         and self.code[offset+4] == self.opc.END_FINALLY))):
+                            targets[label] = targets.get(label, []) + [offset]
+                            pass
+                        pass
+                    pass
+            elif op == self.opc.END_FINALLY and offset in self.fixed_jumps and self.version == 2.7:
+                label = self.fixed_jumps[offset]
+                targets[label] = targets.get(label, []) + [offset]
         return targets
 
     # FIXME: combine with scanner3.py code and put into scanner.py
@@ -863,9 +919,9 @@ if __name__ == "__main__":
     if PYTHON_VERSION >= 2.3:
         co = inspect.currentframe().f_code
         from uncompyle6 import PYTHON_VERSION
-        tokens, customize = Scanner2(PYTHON_VERSION).disassemble(co)
+        tokens, customize = Scanner2(PYTHON_VERSION).ingest(co)
         for t in tokens:
-            print(t.format())
+            print(t)
     else:
         print("Need to be Python 3.2 or greater to demo; I am %s." %
               PYTHON_VERSION)

uncompyle6/scanners/scanner22.py

@@ -0,0 +1,35 @@
+#  Copyright (c) 2016 by Rocky Bernstein
+"""
+Python 2.2 bytecode scanner/deparser
+
+This overlaps Python's 2.2's dis module, but it can be run from
+Python 3 and other versions of Python. Also, we save token
+information for later use in deparsing.
+"""
+
+import uncompyle6.scanners.scanner23 as scan
+# from uncompyle6.scanners.scanner26 import ingest as  ingest26
+
+# bytecode verification, verify(), uses JUMP_OPs from here
+from xdis.opcodes import opcode_22
+JUMP_OPs = opcode_22.JUMP_OPs
+
+# We base this off of 2.3 instead of the other way around
+# because we cleaned things up this way.
+# The history is that 2.7 support is the cleanest,
+# then from that we got 2.6 and so on.
+class Scanner22(scan.Scanner23):
+    def __init__(self, show_asm=False):
+        scan.Scanner23.__init__(self, show_asm)
+        self.opc = opcode_22
+        self.opname = opcode_22.opname
+        self.version = 2.2
+        self.genexpr_name = '<generator expression>';
+        self.parent_ingest = self.ingest
+        self.ingest = self.ingest22
+        return
+
+    def ingest22(self, co, classname=None, code_objects={}, show_asm=None):
+        tokens, customize = self.parent_ingest(co, classname, code_objects, show_asm)
+        tokens = [t for t in tokens if t.type != 'SET_LINENO']
+        return tokens, customize

uncompyle6/scanners/scanner23.py

@@ -1,8 +1,8 @@
 #  Copyright (c) 2016 by Rocky Bernstein
 """
-Python 2.4 bytecode scanner/deparser
+Python 2.3 bytecode scanner/deparser
 
-This overlaps Python's 2.4's dis module, but it can be run from
+This overlaps Python's 2.3's dis module, but it can be run from
 Python 3 and other versions of Python. Also, we save token
 information for later use in deparsing.
 """
@@ -20,6 +20,8 @@ JUMP_OPs = opcode_23.JUMP_OPs
 class Scanner23(scan.Scanner24):
     def __init__(self, show_asm):
         scan.Scanner24.__init__(self, show_asm)
+        self.opc = opcode_23
+        self.opname = opcode_23.opname
         # These are the only differences in initialization between
         # 2.3-2.6
         self.version = 2.3

uncompyle6/scanners/scanner24.py

@@ -22,6 +22,8 @@ class Scanner24(scan.Scanner25):
         scan.Scanner25.__init__(self, show_asm)
         # These are the only differences in initialization between
         # 2.4, 2.5 and 2.6
+        self.opc = opcode_24
+        self.opname = opcode_24.opname
         self.version = 2.4
         self.genexpr_name = '<generator expression>';
         return

uncompyle6/scanners/scanner25.py

@@ -21,6 +21,8 @@ class Scanner25(scan.Scanner26):
     def __init__(self, show_asm):
         # There are no differences in initialization between
         # 2.5 and 2.6
+        self.opc = opcode_25
+        self.opname = opcode_25.opname
         scan.Scanner26.__init__(self, show_asm)
         self.version = 2.5
         return

uncompyle6/scanners/scanner26.py

@@ -70,15 +70,24 @@ class Scanner26(scan.Scanner2):
         self.pop_jump_if_or_pop = frozenset([])
         return
 
-    def disassemble(self, co, classname=None, code_objects={}, show_asm=None):
-        '''
-        Disassemble a code object, returning a list of 'Token'.
+    def ingest(self, co, classname=None, code_objects={}, show_asm=None):
+        """
+        Pick out tokens from an uncompyle6 code object, and transform them,
+        returning a list of uncompyle6 'Token's.
 
-        The main part of this procedure is modelled after
-        dis.disassemble().
-        '''
+        The transformations are made to assist the deparsing grammar.
+        Specificially:
+           -  various types of LOAD_CONST's are categorized in terms of what they load
+           -  COME_FROM instructions are added to assist parsing control structures
+           -  MAKE_FUNCTION and FUNCTION_CALLS append the number of positional arguments
+
+        Also, when we encounter certain tokens, we add them to a set which will cause custom
+        grammar rules. Specifically, variable arg tokens like MAKE_FUNCTION or BUILD_LIST
+        cause specific rules for the specific number of arguments they take.
+        """
 
         show_asm = self.show_asm if not show_asm else show_asm
+        # show_asm = 'after'
         if show_asm in ('both', 'before'):
             from xdis.bytecode import Bytecode
             bytecode = Bytecode(co, self.opc)
@@ -101,9 +110,6 @@ class Scanner26(scan.Scanner2):
         self.build_lines_data(co, n)
         self.build_prev_op(n)
 
-        # linestarts contains block code adresses (addr,block)
-        self.linestarts = list(self.opc.findlinestarts(co))
-
         # class and names
         if classname:
             classname = '_' + classname.lstrip('_') + '__'
@@ -160,7 +166,8 @@ class Scanner26(scan.Scanner2):
                                  self.opc.IMPORT_STAR))
         # Changes IMPORT_NAME to IMPORT_NAME_CONT.
         # Possibly a Python 2.0 hangover
-        if len(imports) > 1 and self.version < 2.3:
+        # FIXME: Move into a < 2.2 scanner.
+        if len(imports) > 1 and self.version < 2.2:
             last_import = imports[0]
             for i in imports[1:]:
                 if self.lines[last_import].next > i:
@@ -304,7 +311,7 @@ if __name__ == "__main__":
     if PYTHON_VERSION == 2.6:
         import inspect
         co = inspect.currentframe().f_code
-        tokens, customize = Scanner26(show_asm=True).disassemble(co)
+        tokens, customize = Scanner26(show_asm=True).ingest(co)
     else:
         print("Need to be Python 2.6 to demo; I am %s." %
               PYTHON_VERSION)