jinx/python-uncompyle6
1
Fork 0
mirror of https://github.com/rocky/python-uncompyle6.git synced 2025-08-04 01:09:52 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into python-2.4

Browse Source
This commit is contained in:
rocky
2019-07-04 10:23:41 -04:00
parent 0639fdbbb7 c7124ad9ca
commit cb5d4f4989
5 changed files with 427 additions and 324 deletions
Download Patch File Download Diff File Expand all files Collapse all files

174
NEWS.md
View File

@@ -3,13 +3,13 @@
Again, most of the work in this is release is thanks to x0ret. Again, most of the work in this is release is thanks to x0ret.
- Handle annotation args in Python 3.x - Handle annotation arguments in Python 3.x
- Fix vararg and function signatures in 3.x - Fix _vararg_ and function signatures in 3.x
- Some 3.x < 3.6 while(1)/if fixes - others remain - Some 3.x < 3.6 `while` (1)/`if` fixes &mdash; others remain
- Start reinstating else if -> elif - Start reinstating `else if` -> `elif`
- LOAD_CONST -> LOAD_CODE where appropriate - `LOAD_CONST` -> `LOAD_CODE` where appropriate
- option `weak-verify` is now `syntax-verify` - option `--weak-verify` is now `--syntax-verify`
- code cleanups, start using "blacken" to reformat text - code cleanups, start using [black](https://github.com/python/black) to reformat text
3.3.4 2019-06-19 Fleetwood at 65 3.3.4 2019-06-19 Fleetwood at 65
@@ -18,12 +18,12 @@ Again, most of the work in this is release is thanks to x0ret.
Most of the work in this is release is thanks to x0ret. Most of the work in this is release is thanks to x0ret.
- Major work was done by x0ret to correct function signatures and include annotation types - Major work was done by x0ret to correct function signatures and include annotation types
- Handle Python 3.6 STORE_ANNOTATION [#58](https://github.com/rocky/python-uncompyle6/issues/58) - Handle Python 3.6 `STORE_ANNOTATION` [#58](https://github.com/rocky/python-uncompyle6/issues/58)
- Friendlier assembly output - Friendlier assembly output
- `LOAD_CONST` replaced by `LOAD_STR` where appropriate to simplify parsing and improve clarity - `LOAD_CONST` replaced by `LOAD_STR` where appropriate to simplify parsing and improve clarity
- remove unneeded parenthesis in a generator expression when it is the single argument to the function [#247](https://github.com/rocky/python-uncompyle6/issues/246) - remove unneeded parenthesis in a generator expression when it is the single argument to the function [#247](https://github.com/rocky/python-uncompyle6/issues/246)
- Bug in noting an async function [#246](https://github.com/rocky/python-uncompyle6/issues/246) - Bug in noting an async function [#246](https://github.com/rocky/python-uncompyle6/issues/246)
- Handle unicode docstrings and fix docstring bugs [#241](https://github.com/rocky/python-uncompyle6/issues/241) - Handle Unicode docstrings and fix docstring bugs [#241](https://github.com/rocky/python-uncompyle6/issues/241)
- Add short option -T as an alternate for --tree+ - Add short option -T as an alternate for --tree+
- Some grammar cleanup - Some grammar cleanup
@@ -67,7 +67,7 @@ some span back as far as 2.x
But as before, many more remain in the 3.7 and 3.8 range which will But as before, many more remain in the 3.7 and 3.8 range which will
get addressed in future releases get addressed in future releases
Pypy 3.6 support was started. Pypy 3.x detection fixed (via xdis) Pypy 3.6 support was started. Pypy 3.x detection fixed (via `xdis`)
3.3.1 2019-04-19 Good Friday 3.3.1 2019-04-19 Good Friday
========================== ==========================
@@ -150,7 +150,7 @@ Pull Requests
- Add rudimentary 1.4 support (still a bit buggy) - Add rudimentary 1.4 support (still a bit buggy)
- add --tree+ option to show formatting rule, when it is constant - add --tree+ option to show formatting rule, when it is constant
- Python 2.7.15candidate1 support (via xdis) - Python 2.7.15candidate1 support (via `xdis`)
- bug fixes, especially for 3.7 (but 2.7 and 3.6 and others as well) - bug fixes, especially for 3.7 (but 2.7 and 3.6 and others as well)
3.1.3 2018-04-16 3.1.3 2018-04-16
@@ -255,24 +255,24 @@ function calls and definitions.
2.15.1 2018-01-27 2.15.1 2018-01-27
===================== =====================
- Add --linemap option to give line correspondences - Add `--linemap` option to give line correspondences
between original source lines and reconstructed line sources. between original source lines and reconstructed line sources.
It is far from perfect, but it is a start It is far from perfect, but it is a start
- Add a new class of tests: tests which when decompiled check themselves - Add a new class of tests: tests which when decompiled check themselves
- Split off Python version semantic action customizations into its own file - Split off Python version semantic action customizations into its own file
- Fix 2.7 bug in ifelse loop statement - Fix 2.7 bug in `if`/`else` loop statement
- Handle 3.6+ EXTENDED_ARGs for POP_JUMP_IF... instructions - Handle 3.6+ `EXTENDED_ARG`s for `POP_JUMP_IF..` instructions
- Correct 3.6+ calls with kwargs - Correct 3.6+ calls with `kwargs`
- Describe the difficulty of 3.6 in README - Describe the difficulty of 3.6 in README
2.14.3 2018-01-19 2.14.3 2018-01-19
===================== =====================
- Fix bug in 3.5+ await stmt - Fix bug in 3.5+ `await` statement
- Better version to magic handling; handle 3.5.2 .. 3.5.4 versions - Better version to magic handling; handle 3.5.2 .. 3.5.4 versions
- Improve/correct test_pyenvlib.py status messages - Improve/correct test_pyenvlib.py status messages
- Fix some 2.7 and 2.6 parser bugs - Fix some 2.7 and 2.6 parser bugs
- Fix whilelse parsing bugs - Fix `whilelse` parsing bugs
- Correct 2.5- decorator parsing - Correct 2.5- decorator parsing
- grammar for decorators matches AST a little more - grammar for decorators matches AST a little more
- better tests in setup.py for running the right version of Python - better tests in setup.py for running the right version of Python
@@ -283,15 +283,15 @@ function calls and definitions.
Decompilation bug fixes, mostly 3.6 and pre 2.7 Decompilation bug fixes, mostly 3.6 and pre 2.7
- 3.6 FUNCTION_EX (somewhat) - 3.6 `FUNCTION_EX` (somewhat)
- 3.6 FUNCTION_EX_KW fixes - 3.6 `FUNCTION_EX_KW` fixes
- 3.6 MAKE_FUNCTION fixes - 3.6 `MAKE_FUNCTION` fixes
- correct 3.5 CALL_FUNCTION_VAR - correct 3.5 `CALL_FUNCTION_VAR`
- stronger 3.x "while 1" testing - stronger 3.x "while 1" testing
- Fix bug in if's with "pass" bodies. Fixes #104 - Fix bug in if's with "pass" bodies. Fixes #104
- try/else and try/finally fixes on 2.6- - try/else and try/finally fixes on 2.6-
- limit pypy customization to pypy - limit pypy customization to pypy
- Add addr fields in COME_FROMS - Add addr fields in `COME_FROM`S
- Allow use of full instructions in parser reduction routines - Allow use of full instructions in parser reduction routines
- Reduce grammar in Python 3 by specialization more to specific - Reduce grammar in Python 3 by specialization more to specific
Python versions Python versions
@@ -300,11 +300,11 @@ Decompilation bug fixes, mostly 3.6 and pre 2.7
2.14.1 2017-12-10 Dr. Gecko 2.14.1 2017-12-10 Dr. Gecko
=================================== ===================================
- Many decompilation bugfixes - Many decompilation bug fixes
- Grammar rule reduction and version isolation - Grammar rule reduction and version isolation
- Match higher-level nonterminal names more closely - Match higher-level nonterminal names more closely
with Python AST with Python AST
- Start automated Python stdlib testing - full round trip - Start automated Python _stdlib_ testing &mdash; full round trip
2.14.0 2017-11-26 johnnybamazing 2.14.0 2017-11-26 johnnybamazing
========================================= =========================================
@@ -313,7 +313,7 @@ Decompilation bug fixes, mostly 3.6 and pre 2.7
and remove used grammar rules and remove used grammar rules
- Fix a number of bytecode decompile problems - Fix a number of bytecode decompile problems
(many more remain) (many more remain)
- Add stdlib/runtests.sh for even more rigorous testing - Add `stdlib/runtests.sh` for even more rigorous testing
2.13.3 2017-11-13 2.13.3 2017-11-13
===================== =====================
@@ -329,16 +329,16 @@ Overall: better 3.6 decompiling and some much needed code refactoring and cleanu
rather trying to parse the bytecode array. This largely been done in for versions 3.x; rather trying to parse the bytecode array. This largely been done in for versions 3.x;
3.0 custom mangling code has been reduced; 3.0 custom mangling code has been reduced;
some 2.x conversion has been done, but more is desired. This make it possible to... some 2.x conversion has been done, but more is desired. This make it possible to...
- Handle EXTENDED_ARGS better. While relevant to all Python versions it is most noticeable in - Handle `EXTENDED_ARGS` better. While relevant to all Python versions it is most noticeable in
version 3.6+ where in switching to wordcodes the size of operands has been reduced from 2**16 version 3.6+ where in switching to wordcodes the size of operands has been reduced from 2^16
to 2**8. JUMP instruction then often need EXTENDED_ARGS. to 2^8. `JUMP` instruction then often need EXTENDED_ARGS.
- Refactor find_jump_targets() with via working of of instructions rather the bytecode array. - Refactor find_jump_targets() with via working of of instructions rather the bytecode array.
- use --weak-verify more and additional fuzzing on verify() - use `--weak-verify` more and additional fuzzing on verify()
- fragment parser now ignores errors in nested function definitions; an parameter was - fragment parser now ignores errors in nested function definitions; an parameter was
added to assist here. Ignoring errors may be okay because the fragment parser often just needs, added to assist here. Ignoring errors may be okay because the fragment parser often just needs,
well, *fragments*. well, *fragments*.
- Distinguish RETURN_VALUE from RETURN_END_IF in exception bodies better in 3.6 - Distinguish `RETURN_VALUE` from `RETURN_END_IF` in exception bodies better in 3.6
- bug in 3.x language changes: import queue via import Queue - bug in 3.x language changes: import queue via `import Queue`
- reinstate some bytecode tests since decompiling has gotten better - reinstate some bytecode tests since decompiling has gotten better
- Revise how to report a bug - Revise how to report a bug
@@ -358,12 +358,12 @@ Overall: better 3.6 decompiling and some much needed code refactoring and cleanu
- Fixes in deparsing lambda expressions - Fixes in deparsing lambda expressions
- Improve table-semantics descriptions - Improve table-semantics descriptions
- Document hacky customize arg count better (until we can remove it) - Document hacky customize arg count better (until we can remove it)
- Update to use xdis 3.7.0 or greater - Update to use `xdis` 3.7.0 or greater
2.12.0 2017-09-26 2.12.0 2017-09-26
===================== =====================
- Use xdis 3.6.0 or greater now - Use `xdis` 3.6.0 or greater now
- Small semantic table cleanups - Small semantic table cleanups
- Python 3.4's terms a little names better - Python 3.4's terms a little names better
- Slightly more Python 3.7, but still failing a lot - Slightly more Python 3.7, but still failing a lot
@@ -377,13 +377,13 @@ Overall: better 3.6 decompiling and some much needed code refactoring and cleanu
2.11.4 2017-08-15 2.11.4 2017-08-15
===================== =====================
* scanner and parser now allow 3-part version string lookups, * scanner and parser now allow 3-part version string look ups,
e.g. 2.7.1 We allow a float here, but if passed a string like '2.7'. or e.g. 2.7.1 We allow a float here, but if passed a string like '2.7'. or
* unpin 3.5.1. xdis 3.5.4 has been release and fixes the problems we had. Use that. * unpin 3.5.1. `xdis` 3.5.4 has been release and fixes the problems we had. Use that.
* some routines here moved to xdis. Use the xdis version * some routines here moved to `xdis`. Use the `xdis` version
* README.rst: Link typo Name is trepan2 now not trepan * `README.rst`: Link typo Name is _trepan2_ now not _trepan_
* xdis-forced change adjust for COMPARE_OP "is-not" in * xdis-forced change adjust for `COMPARE_OP` "is-not" in
semanatic routines. We need "is not". semantic routines. We need "is not".
* Some PyPy tolerance in validate testing. * Some PyPy tolerance in validate testing.
* Some pyston tolerance * Some pyston tolerance
@@ -393,15 +393,15 @@ Overall: better 3.6 decompiling and some much needed code refactoring and cleanu
Very minor changes Very minor changes
- RsT doc fixes and updates - RsT doc fixes and updates
- use newer xdis, but not too new; 3.5.2 breaks uncompyle6 - use newer `xdis`, but not too new; 3.5.2 breaks uncompyle6
- use xdis opcode sets - use `xdis` opcode sets
- xdis "exception match" is now "exception-match" - `xdis` "exception match" is now "exception-match"
2.11.2 2017-07-09 2.11.2 2017-07-09
===================== =====================
- Start supporting Pypy 3.5 (5.7.1-beta) - Start supporting Pypy 3.5 (5.7.1-beta)
- use xdis 3.5.0's opcode sets and require xdis 3.5.0 - use `xdis` 3.5.0's opcode sets and require `xdis` 3.5.0
- Correct some Python 2.4-2.6 loop detection - Correct some Python 2.4-2.6 loop detection
- guard against badly formatted bytecode - guard against badly formatted bytecode
@@ -409,55 +409,55 @@ Very minor changes
===================== =====================
- Python 3.x annotation and function signature fixes - Python 3.x annotation and function signature fixes
- Bump xdis version - Bump `xdis` version
- Small pysource bug fixes - Small `pysource.py` bug fixes
2.11.0 2017-06-18 Fleetwood 2.11.0 2017-06-18 Fleetwood
================================== ==================================
- Major improvements in fragment tracking - Major improvements in fragment tracking
* Add nonterminal node in extractInfo * Add nonterminal node in `extractInfo()`
* tag more offsets in expressions * tag more offsets in expressions
* tag array subscripts * tag array subscripts
* set YIELD value offset in a <yield> expr * set `YIELD` value offset in a _yield expr_
* fix a long-standing bug in not adjusting final AST when melding other deparse ASTs * fix a long-standing bug in not adjusting final AST when melding other deparse ASTs
- Fixes yet again for make_function node handling; document what's up here - Fixes yet again for make_function node handling; document what's up here
- Fix bug in snowflake Python 3.5 *args kwargs - Fix bug in snowflake Python 3.5 `*args`, `kwargs`
2.10.1 2017-06-3 Marylin Frankel 2.10.1 2017-06-3 Marylin Frankel
======================================== ========================================
- fix some fragments parsing bugs - fix some fragments parsing bugs
- was returning the wrong type sometimes in deparse_code_around_offset() - was returning the wrong type sometimes in `deparse_code_around_offset()`
- capture function name in offsets - capture function name in offsets
- track changes to ifelstrmtr node from pysource into fragments - track changes to `ifelstrmtr` node from `pysource.py` into fragments
2.10.0 2017-05-30 Elaine Gordon 2.10.0 2017-05-30 Elaine Gordon
======================================= =======================================
- Add fuzzy offset deparse look up - Add fuzzy offset deparse look up
- 3.6 bug fixes - 3.6 bug fixes
- fix EXTENDED_ARGS handling (and in 2.6 and others) - fix `EXTENDED_ARGS` handling (and in 2.6 and others)
- semantic routine make_function fragments.py - semantic routine make_function fragments.py
- MAKE_FUNCTION handling - `MAKE_FUNCTION` handling
- CALL_FUNCTION_EX handling - `CALL_FUNCTION_EX` handling
- async property on defs - `async` property on `defs`
- support for CALL_FUNCTION_KW (moagstar) - support for `CALL_FUNCTION_KW` (moagstar)
- 3.5+ UNMAP_PACK and BUILD_UNMAP_PACK handling - 3.5+ `UNMAP_PACK` and` BUILD_UNMAP_PACK` handling
- 3.5 FUNCTION_VAR bug - 3.5 FUNCTION_VAR bug
- 3.x pass statement insdie while True - 3.x pass statement inside `while True`
- Improve 3.2 decompilation - Improve 3.2 decompilation
- Fixed -o argument processing (grkov90) - Fixed `-o` argument processing (grkov90)
- Reduce scope of LOAD_ASSERT as expr to 3.4+ - Reduce scope of LOAD_ASSERT as expr to 3.4+
- "await" statement fixes - `await` statement fixes
- 2.3, 2.4 "if 1 .." fixes - 2.3, 2.4 "if 1 .." fixes
- 3.x annotation fixes - 3.x annotation fixes
2.9.11 2017-04-06 2.9.11 2017-04-06
===================== =====================
- Better support for Python 3.5+ BUILD_MAP_UNPACK - Better support for Python 3.5+ `BUILD_MAP_UNPACK`
- Start 3.6 CALL_FUNCTION_EX support - Start 3.6 `CALL_FUNCTION_EX` support
- Many decompilation bug fixes. (Many more remain). See ChangeLog - Many decompilation bug fixes. (Many more remain). See ChangeLog
2.9.10 2017-02-25 2.9.10 2017-02-25
@@ -465,7 +465,7 @@ Very minor changes
- Python grammar rule fixes - Python grammar rule fixes
- Add ability to get grammar coverage on runs - Add ability to get grammar coverage on runs
- Handle Python 3.6 opcode BUILD_CONST_KEYMAP - Handle Python 3.6 opcode `BUILD_CONST_KEYMAP`
2.9.9 2016-12-16 2.9.9 2016-12-16
@@ -481,13 +481,13 @@ Very minor changes
==================== ====================
- Better control-flow detection - Better control-flow detection
- pseudo instruction THEN in 2.x - pseudo instruction `THEN` in 2.x
to disambiguate if from and to disambiguate if from and
- fix bug in --verify option - fix bug in `--verify` option
- DRY (a little) control-flow detection - DRY (a little) control-flow detection
- fix syntax in tuples with one element - fix syntax in tuples with one element
- if AST rule inheritance in Python 2.5 - if AST rule inheritance in Python 2.5
- NAME_MODULE removal for Python <= 2.4 - `NAME_MODULE` removal for Python <= 2.4
- verify call fixes for Python <= 2.4 - verify call fixes for Python <= 2.4
- more Python lint - more Python lint
@@ -498,9 +498,9 @@ Very minor changes
- Some Python 3.6 bytecode to wordcode conversion fixes - Some Python 3.6 bytecode to wordcode conversion fixes
- option -g: show start-end range when possible - option -g: show start-end range when possible
- track print_docstring move to help (used in python 3.1) - track print_docstring move to help (used in python 3.1)
- verify: allow RETURN_VALUE to match RETURN_END_IF - verify: allow `RETURN_VALUE` to match `RETURN_END_IF`
- some 3.2 compatibility - some 3.2 compatibility
- Better Python 3 control flow detection by adding Pseudo ELSE opcodes - Better Python 3 control flow detection by adding Pseudo `ELSE` opcodes
2.9.6 2016-12-04 2.9.6 2016-12-04
==================== ====================
@@ -516,7 +516,7 @@ Very minor changes
- Correct MANIFEST.in - Correct MANIFEST.in
- More AST grammar checking - More AST grammar checking
- --linemapping option or linenumbers.line_number_mapping() - `--linemapping` option or _linenumbers.line_number_mapping()_
Shows correspondence of lines between source Shows correspondence of lines between source
and decompiled source and decompiled source
- Some control flow adjustments in code for 2.x. - Some control flow adjustments in code for 2.x.
@@ -536,7 +536,7 @@ Very minor changes
* improper while 1 else * improper while 1 else
* docstring indent * docstring indent
* 3.3 default values in lambda expressions * 3.3 default values in lambda expressions
* start 3.0 decompilation (needs newer xdis) * start 3.0 decompilation (needs newer `xdis`)
- Start grammar misparse checking - Start grammar misparse checking
@@ -550,12 +550,12 @@ Very minor changes
2.9.3 2016-10-26 2.9.3 2016-10-26
==================== ====================
Release forced by incompatibility change in xdis 3.2.0. Release forced by incompatibility change in` xdis` 3.2.0.
- Python 3.1 bugs: - Python 3.1 bugs:
* handle "with ... as" * handle `with`... `as`
* handle "with" * handle `with`
* Start handling def (...) -> yy (has bugs still) * Start handling `def` (...) -> _yy_ (has bugs still)
- DRY Python 3.x via inheritance - DRY Python 3.x via inheritance
- Python 3.6 work (from Daniel Bradburn) - Python 3.6 work (from Daniel Bradburn)
@@ -581,12 +581,12 @@ Release forced by incompatibility change in xdis 3.2.0.
2.9.0 2016-10-09 2.9.0 2016-10-09
==================== ====================
- Use xdis 3.0.0 protocol load_module. - Use `xdis` 3.0.0 protocol `load_module`.
this Forces change in requirements.txt and _pkg_info_.py this Forces change in requirements.txt and _pkg_info_.py
- Start Python 1.5 decompiling; another round of work is needed to - Start Python 1.5 decompiling; another round of work is needed to
remove bugs remove bugs
- Simplify python 2.1 grammar - Simplify python 2.1 grammar
- Fix bug with -t ... Wasn't showing source text when -t option was given - Fix bug with `-t` ... Wasn't showing source text when `-t` option was given
- Fix 2.1-2.6 bug in list comprehension - Fix 2.1-2.6 bug in list comprehension
2.8.4 2016-10-08 2.8.4 2016-10-08
@@ -595,8 +595,8 @@ Release forced by incompatibility change in xdis 3.2.0.
- Python 3 disassembly bug fixes - Python 3 disassembly bug fixes
- Python 3.6 fstring bug fixes (from moagstar) - Python 3.6 fstring bug fixes (from moagstar)
- Python 2.1 disassembly - Python 2.1 disassembly
- COME_FROM suffixes added in Python3 - `COME_FROM` suffixes added in Python3
- use .py extension in verification disassembly - use `.py` extension in verification disassembly
2.8.3 2016-09-11 live from NYC! 2.8.3 2016-09-11 live from NYC!
======================================= =======================================
@@ -641,8 +641,8 @@ control-flow structure detection is done.
- Add Python 2.2 decompilation - Add Python 2.2 decompilation
- Fix bugs - Fix bugs
* PyPy LOOKUP_METHOD bug * PyPy `LOOKUP_METHOD` bug
* Python 3.6 FORMAT_VALUE handles expressions now * Python 3.6 `FORMAT_VALUE` handles expressions now
2.8.0 2016-08-03 2.8.0 2016-08-03
==================== ====================
@@ -692,7 +692,7 @@ control-flow structure detection is done.
==================== ====================
- Improve Python 2.6 bytecode deparsing: - Improve Python 2.6 bytecode deparsing:
stdlib now will deparse something _stdlib_ now will deparse something
- Better <2.6 vs. 2.7 grammar separation - Better <2.6 vs. 2.7 grammar separation
- Fix some 2.7 deparsing bugs - Fix some 2.7 deparsing bugs
- Fix bug in installing uncompyle6 script - Fix bug in installing uncompyle6 script
@@ -755,9 +755,9 @@ control-flow structure detection is done.
2.3.2 2016-05-1 2.3.2 2016-05-1
=================== ===================
- Add --version option standalone scripts - Add `--version` option standalone scripts
- Correct License information in package - Correct License information in package
- expose fns uncompyle_file, load_file, and load_module - expose functions `uncompyle_file()`, `load_file()`, and `load_module()`
- Start to DRY Python2 and Python3 grammars Separate out 3.2, and 3.5+ - Start to DRY Python2 and Python3 grammars Separate out 3.2, and 3.5+
specific grammar code specific grammar code
- Fix bug in 3.5+ constant map parsing - Fix bug in 3.5+ constant map parsing
@@ -765,12 +765,12 @@ control-flow structure detection is done.
2.3.0, 2.3.1 2016-04-30 2.3.0, 2.3.1 2016-04-30
============================= =============================
- Require spark_parser >= 1.1.0 - Require `spark_parser` >= 1.1.0
2.2.0 2016-04-30 2.2.0 2016-04-30
==================== ====================
- Spark is no longer here but pulled separate package spark_parse - Spark is no longer here but pulled separate package [spark_parser](https://pypi.org/project/spark_parser/)
- Python 3 parsing fixes - Python 3 parsing fixes
- More tests - More tests
@@ -780,7 +780,7 @@ control-flow structure detection is done.
- Support single-mode (in addition to exec-mode) compilation - Support single-mode (in addition to exec-mode) compilation
- Start to DRY Python 2 and Python 3 grammars - Start to DRY Python 2 and Python 3 grammars
- Fix bug in if else ternary construct - Fix bug in if else ternary construct
- Fix bug in uncomplye6 -d and -r options (via lelicopter) - Fix bug in uncomplye6 `-d` and `-r` options (via lelicopter)
2.1.3 2016-01-02 2.1.3 2016-01-02
@@ -788,7 +788,7 @@ control-flow structure detection is done.
- Limited support for decompiling Python 3.5 - Limited support for decompiling Python 3.5
- Improve Python 3 class deparsing - Improve Python 3 class deparsing
- Handle MAKE_CLOSURE opcode - Handle `MAKE_CLOSURE` opcode
- Start to DRY opcode code. - Start to DRY opcode code.
- increase test coverage - increase test coverage
- fix misc small bugs and some improvements - fix misc small bugs and some improvements
@@ -830,5 +830,5 @@ Changes from uncompyle2
SPARK: SPARK:
add option to show grammar rules applied add option to show grammar rules applied
allow Python-style # comments in grammar allow Python-style `#` comments in grammar
Runs on Python 3 and Python 2 Runs on Python 3 and Python 2

16
README.rst
View File

@@ -1,5 +1,7 @@
|buildstatus| |Latest Version| |Supported Python Versions| |buildstatus| |Latest Version| |Supported Python Versions|
|packagestatus|
uncompyle6 uncompyle6
========== ==========
@@ -132,7 +134,7 @@ could be compared with the original bytecode. However as Python's code
generation got better, this no longer was feasible. generation got better, this no longer was feasible.
If you want Python syntax verification of the correctness of the If you want Python syntax verification of the correctness of the
decompilation process, add the `--syntax-verify` option. However since decompilation process, add the :code:`--syntax-verify` option. However since
Python syntax changes, you should use this option if the bytecode is Python syntax changes, you should use this option if the bytecode is
the right bytecode for the Python interpreter that will be checking the right bytecode for the Python interpreter that will be checking
the syntax. the syntax.
@@ -146,7 +148,7 @@ available give stronger verification: those programs that when run
test themselves. Our test suite includes these. test themselves. Our test suite includes these.
And Python comes with another a set of programs like this: its test And Python comes with another a set of programs like this: its test
suite for the standard library. We have some code in `test/stdlib` to suite for the standard library. We have some code in :code:`test/stdlib` to
facilitate this kind of checking too. facilitate this kind of checking too.
Known Bugs/Restrictions Known Bugs/Restrictions
@@ -177,15 +179,15 @@ In the Python 3 series, Python support is is strongest around 3.4 or
3.0 is weird in that it in some ways resembles 2.6 more than it does 3.0 is weird in that it in some ways resembles 2.6 more than it does
3.1 or 2.7. Python 3.6 changes things drastically by using word codes 3.1 or 2.7. Python 3.6 changes things drastically by using word codes
rather than byte codes. As a result, the jump offset field in a jump rather than byte codes. As a result, the jump offset field in a jump
instruction argument has been reduced. This makes the `EXTENDED_ARG` instruction argument has been reduced. This makes the :code:`EXTENDED_ARG`
instructions are now more prevalent in jump instruction; previously instructions are now more prevalent in jump instruction; previously
they had been rare. Perhaps to compensate for the additional they had been rare. Perhaps to compensate for the additional
`EXTENDED_ARG` instructions, additional jump optimization has been :code:`EXTENDED_ARG` instructions, additional jump optimization has been
added. So in sum handling control flow by ad hoc means as is currently added. So in sum handling control flow by ad hoc means as is currently
done is worse. done is worse.
Between Python 3.5, 3.6 and 3.7 there have been major changes to the Between Python 3.5, 3.6 and 3.7 there have been major changes to the
`MAKE_FUNCTION` and `CALL_FUNCTION` instructions. :code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.
Currently not all Python magic numbers are supported. Specifically in Currently not all Python magic numbers are supported. Specifically in
some versions of Python, notably Python 3.6, the magic number has some versions of Python, notably Python 3.6, the magic number has
@@ -217,7 +219,7 @@ See Also
* https://github.com/zrax/pycdc : purports to support all versions of Python. It is written in C++ and is most accurate for Python versions around 2.7 and 3.3 when the code was more actively developed. Accuracy for more recent versions of Python 3 and early versions of Python are especially lacking. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained. * https://github.com/zrax/pycdc : purports to support all versions of Python. It is written in C++ and is most accurate for Python versions around 2.7 and 3.3 when the code was more actively developed. Accuracy for more recent versions of Python 3 and early versions of Python are especially lacking. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained. * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained. * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
* https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where `uncompyle6` results are incorrect while `uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because `uncompyle6` adheres to accuracy over idiomatic Python, `uncompyle2` can produce more natural-looking code when it is correct. Currently `uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details * https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details
* `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_ * `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_
* The HISTORY_ file. * The HISTORY_ file.
* https://github.com/rocky/python-xdis : Cross Python version disassembler * https://github.com/rocky/python-xdis : Cross Python version disassembler
@@ -234,6 +236,8 @@ See Also
.. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting .. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
.. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg .. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
:target: https://travis-ci.org/rocky/python-uncompyle6 :target: https://travis-ci.org/rocky/python-uncompyle6
.. |packagestatus| image:: https://repology.org/badge/vertical-allrepos/python:uncompyle6.svg
:target: https://repology.org/project/python:uncompyle6/versions
.. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84 .. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
.. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator .. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator
.. _Py2EXE: https://en.wikipedia.org/wiki/Py2exe .. _Py2EXE: https://en.wikipedia.org/wiki/Py2exe

163
test/test_pyenvlib.py
View File

@@ -27,64 +27,77 @@ from fnmatch import fnmatch
from uncompyle6 import main, PYTHON3 from uncompyle6 import main, PYTHON3
import xdis.magics as magics import xdis.magics as magics
#----- configure this for your needs # ----- configure this for your needs
python_versions = [v for v in magics.python_versions if python_versions = [v for v in magics.python_versions if re.match("^[0-9.]+$", v)]
re.match('^[0-9.]+$', v)]
# FIXME: we should remove Python versions that we don't support. # FIXME: we should remove Python versions that we don't support.
# These include Jython, and Python bytecode changes pre release. # These include Jython, and Python bytecode changes pre release.
TEST_VERSIONS = ( TEST_VERSIONS = (
'pypy3-2.4.0', 'pypy-2.6.1', "pypy3-2.4.0",
'pypy-5.0.1', 'pypy-5.3.1', 'pypy3.5-5.7.1-beta', "pypy-2.6.1",
'pypy3.5-5.9.0', 'pypy3.5-6.0.0', "pypy-5.0.1",
'native') + tuple(python_versions) "pypy-5.3.1",
"pypy3.5-5.7.1-beta",
"pypy3.5-5.9.0",
"pypy3.5-6.0.0",
"native",
) + tuple(python_versions)
target_base = '/tmp/py-dis/' target_base = "/tmp/py-dis/"
lib_prefix = os.path.join(os.environ['HOME'], '.pyenv/versions') lib_prefix = os.path.join(os.environ["HOME"], ".pyenv/versions")
PYC = ('*.pyc', ) PYC = ("*.pyc",)
PYO = ('*.pyo', ) PYO = ("*.pyo",)
PYOC = ('*.pyc', '*.pyo') PYOC = ("*.pyc", "*.pyo")
#----- # -----
test_options = { test_options = {
# name: (src_basedir, pattern, output_base_suffix) # name: (src_basedir, pattern, output_base_suffix)
'test': ('./test', PYOC, 'test'), "test": ("./test", PYOC, "test"),
'max=': 200, "max=": 200,
} }
for vers in TEST_VERSIONS: for vers in TEST_VERSIONS:
if vers.startswith('pypy'): if vers.startswith("pypy"):
if vers.startswith('pypy3.'): if vers.startswith("pypy3."):
short_vers = vers[4:6] short_vers = vers[4:6]
else: else:
short_vers = vers[0:-2] short_vers = vers[0:-2]
test_options[vers] = (os.path.join(lib_prefix, vers, 'lib_pypy'), test_options[vers] = (
PYC, 'python-lib'+short_vers) os.path.join(lib_prefix, vers, "lib_pypy"),
PYC,
"python-lib" + short_vers,
)
else: else:
if vers == 'native': if vers == "native":
short_vers = os.path.basename(sys.path[-1]) short_vers = os.path.basename(sys.path[-1])
test_options[vers] = (sys.path[-1], test_options[vers] = (sys.path[-1], PYC, short_vers)
PYC, short_vers)
else: else:
short_vers = vers[:3] short_vers = vers[:3]
test_options[vers] = (os.path.join(lib_prefix, vers, 'lib', 'python'+short_vers), test_options[vers] = (
PYC, 'python-lib'+short_vers) os.path.join(lib_prefix, vers, "lib", "python" + short_vers),
PYC,
"python-lib" + short_vers,
)
def do_tests(src_dir, patterns, target_dir, start_with=None,
do_verify=False, max_files=200):
def do_tests(
src_dir, patterns, target_dir, start_with=None, do_verify=False, max_files=200
):
def visitor(files, dirname, names): def visitor(files, dirname, names):
files.extend( files.extend(
[os.path.normpath(os.path.join(dirname, n)) [
for n in names os.path.normpath(os.path.join(dirname, n))
for pat in patterns for n in names
if fnmatch(n, pat)]) for pat in patterns
if fnmatch(n, pat)
]
)
files = [] files = []
cwd = os.getcwd() cwd = os.getcwd()
@@ -92,10 +105,13 @@ def do_tests(src_dir, patterns, target_dir, start_with=None,
if PYTHON3: if PYTHON3:
for root, dirname, names in os.walk(os.curdir): for root, dirname, names in os.walk(os.curdir):
files.extend( files.extend(
[os.path.normpath(os.path.join(root, n)) [
for n in names os.path.normpath(os.path.join(root, n))
for pat in patterns for n in names
if fnmatch(n, pat)]) for pat in patterns
if fnmatch(n, pat)
]
)
pass pass
pass pass
else: else:
@@ -107,26 +123,29 @@ def do_tests(src_dir, patterns, target_dir, start_with=None,
try: try:
start_with = files.index(start_with) start_with = files.index(start_with)
files = files[start_with:] files = files[start_with:]
print('>>> starting with file', files[0]) print(">>> starting with file", files[0])
except ValueError: except ValueError:
pass pass
if len(files) > max_files: if len(files) > max_files:
files = [file for file in files if not 'site-packages' in file] files = [file for file in files if not "site-packages" in file]
files = [file for file in files if not 'test' in file] files = [file for file in files if not "test" in file]
if len(files) > max_files: if len(files) > max_files:
# print("Number of files %d - truncating to last 200" % len(files)) # print("Number of files %d - truncating to last 200" % len(files))
print("Number of files %d - truncating to first %s" % print(
(len(files), max_files)) "Number of files %d - truncating to first %s" % (len(files), max_files)
)
files = files[:max_files] files = files[:max_files]
print(time.ctime()) print(time.ctime())
(tot_files, okay_files, failed_files, (tot_files, okay_files, failed_files, verify_failed_files) = main.main(
verify_failed_files) = main.main(src_dir, target_dir, files, [], do_verify=do_verify) src_dir, target_dir, files, [], do_verify=do_verify
)
print(time.ctime()) print(time.ctime())
return verify_failed_files + failed_files return verify_failed_files + failed_files
if __name__ == '__main__':
if __name__ == "__main__":
import getopt, sys import getopt, sys
do_coverage = do_verify = False do_coverage = do_verify = False
@@ -135,38 +154,46 @@ if __name__ == '__main__':
test_options_keys = list(test_options.keys()) test_options_keys = list(test_options.keys())
test_options_keys.sort() test_options_keys.sort()
opts, args = getopt.getopt(sys.argv[1:], '', opts, args = getopt.getopt(
['start-with=', 'verify', 'verify-run', 'syntax-verify', sys.argv[1:],
'max=', 'coverage', 'all', ] \ "",
+ test_options_keys ) [
vers = '' "start-with=",
"verify",
"verify-run",
"syntax-verify",
"max=",
"coverage",
"all",
]
+ test_options_keys,
)
vers = ""
for opt, val in opts: for opt, val in opts:
if opt == '--verify': if opt == "--verify":
do_verify = 'strong' do_verify = "strong"
elif opt == '--syntax-verify': elif opt == "--syntax-verify":
do_verify = 'weak' do_verify = "weak"
elif opt == '--verify-run': elif opt == "--verify-run":
do_verify = 'verify-run' do_verify = "verify-run"
elif opt == '--coverage': elif opt == "--coverage":
do_coverage = True do_coverage = True
elif opt == '--start-with': elif opt == "--start-with":
start_with = val start_with = val
elif opt[2:] in test_options_keys: elif opt[2:] in test_options_keys:
triple = test_options[opt[2:]] triple = test_options[opt[2:]]
vers = triple[-1] vers = triple[-1]
test_dirs.append(triple) test_dirs.append(triple)
elif opt == '--max': elif opt == "--max":
test_options['max='] = int(val) test_options["max="] = int(val)
elif opt == '--all': elif opt == "--all":
vers = 'all' vers = "all"
for val in test_options_keys: for val in test_options_keys:
test_dirs.append(test_options[val]) test_dirs.append(test_options[val])
if do_coverage: if do_coverage:
os.environ['SPARK_PARSER_COVERAGE'] = ( os.environ["SPARK_PARSER_COVERAGE"] = "/tmp/spark-grammar-%s.cover" % vers
'/tmp/spark-grammar-%s.cover' % vers
)
failed = 0 failed = 0
for src_dir, pattern, target_dir in test_dirs: for src_dir, pattern, target_dir in test_dirs:
@@ -174,8 +201,14 @@ if __name__ == '__main__':
target_dir = os.path.join(target_base, target_dir) target_dir = os.path.join(target_base, target_dir)
if os.path.exists(target_dir): if os.path.exists(target_dir):
shutil.rmtree(target_dir, ignore_errors=1) shutil.rmtree(target_dir, ignore_errors=1)
failed += do_tests(src_dir, pattern, target_dir, start_with, failed += do_tests(
do_verify, test_options['max=']) src_dir,
pattern,
target_dir,
start_with,
do_verify,
test_options["max="],
)
else: else:
print("### Path %s doesn't exist; skipping" % src_dir) print("### Path %s doesn't exist; skipping" % src_dir)
pass pass

206
test/test_pythonlib.py
View File

@@ -33,69 +33,89 @@ from uncompyle6 import PYTHON_VERSION
from uncompyle6.main import main from uncompyle6.main import main
from fnmatch import fnmatch from fnmatch import fnmatch
def get_srcdir(): def get_srcdir():
filename = os.path.normcase(os.path.dirname(__file__)) filename = os.path.normcase(os.path.dirname(__file__))
return os.path.realpath(filename) return os.path.realpath(filename)
src_dir = get_srcdir() src_dir = get_srcdir()
#----- configure this for your needs # ----- configure this for your needs
lib_prefix = '/usr/lib' lib_prefix = "/usr/lib"
#lib_prefix = [src_dir, '/usr/lib/', '/usr/local/lib/'] # lib_prefix = [src_dir, '/usr/lib/', '/usr/local/lib/']
target_base = tempfile.mkdtemp(prefix='py-dis-') target_base = tempfile.mkdtemp(prefix="py-dis-")
PY = ('*.py', ) PY = ("*.py",)
PYC = ('*.pyc', ) PYC = ("*.pyc",)
PYO = ('*.pyo', ) PYO = ("*.pyo",)
PYOC = ('*.pyc', '*.pyo') PYOC = ("*.pyc", "*.pyo")
test_options = { test_options = {
# name: (src_basedir, pattern, output_base_suffix, python_version) # name: (src_basedir, pattern, output_base_suffix, python_version)
'test': "test": ("test", PYC, "test"),
('test', PYC, 'test'), "ok-2.6": (os.path.join(src_dir, "ok_lib2.6"), PYOC, "ok-2.6", 2.6),
"ok-2.7": (os.path.join(src_dir, "ok_lib2.7"), PYOC, "ok-2.7", 2.7),
'ok-2.6': (os.path.join(src_dir, 'ok_lib2.6'), "ok-3.2": (os.path.join(src_dir, "ok_lib3.2"), PYOC, "ok-3.2", 3.2),
PYOC, 'ok-2.6', 2.6), "base-2.7": (
os.path.join(src_dir, "base_tests", "python2.7"),
'ok-2.7': (os.path.join(src_dir, 'ok_lib2.7'), PYOC,
PYOC, 'ok-2.7', 2.7), "base_2.7",
2.7,
'ok-3.2': (os.path.join(src_dir, 'ok_lib3.2'), ),
PYOC, 'ok-3.2', 3.2),
'base-2.7': (os.path.join(src_dir, 'base_tests', 'python2.7'),
PYOC, 'base_2.7', 2.7),
} }
for vers in (2.7, 3.4, 3.5, 3.6): for vers in (2.7, 3.4, 3.5, 3.6):
pythonlib = "ok_lib%s" % vers pythonlib = "ok_lib%s" % vers
key = "ok-%s" % vers key = "ok-%s" % vers
test_options[key] = (os.path.join(src_dir, pythonlib), PYOC, key, vers) test_options[key] = (os.path.join(src_dir, pythonlib), PYOC, key, vers)
pass pass
for vers in (1.3, 1.4, 1.5, for vers in (
2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 1.3,
3.0, 3.1, 3.2, 3.3, 1.4,
3.4, 3.5, 3.6, 3.7, 3.8, 'pypy3.2', 'pypy2.7', 'pypy3.6'): 1.5,
2.1,
2.2,
2.3,
2.4,
2.5,
2.6,
2.7,
3.0,
3.1,
3.2,
3.3,
3.4,
3.5,
3.6,
3.7,
3.8,
"pypy3.2",
"pypy2.7",
"pypy3.6",
):
bytecode = "bytecode_%s" % vers bytecode = "bytecode_%s" % vers
key = "bytecode-%s" % vers key = "bytecode-%s" % vers
test_options[key] = (bytecode, PYC, bytecode, vers) test_options[key] = (bytecode, PYC, bytecode, vers)
bytecode = "bytecode_%s_run" % vers bytecode = "bytecode_%s_run" % vers
key = "bytecode-%s-run" % vers key = "bytecode-%s-run" % vers
test_options[key] = (bytecode, PYC, bytecode, vers) test_options[key] = (bytecode, PYC, bytecode, vers)
key = "%s" % vers key = "%s" % vers
pythonlib = "python%s" % vers pythonlib = "python%s" % vers
if isinstance(vers, float) and vers >= 3.0: if isinstance(vers, float) and vers >= 3.0:
pythonlib = os.path.join(pythonlib, '__pycache__') pythonlib = os.path.join(pythonlib, "__pycache__")
test_options[key] = (os.path.join(lib_prefix, pythonlib), PYOC, pythonlib, vers) test_options[key] = (os.path.join(lib_prefix, pythonlib), PYOC, pythonlib, vers)
# -----
#-----
def help(): def help():
print("""Usage-Examples: print(
"""Usage-Examples:
# compile, decompyle and verify short tests for Python 2.7: # compile, decompyle and verify short tests for Python 2.7:
test_pythonlib.py --bytecode-2.7 --verify --compile test_pythonlib.py --bytecode-2.7 --verify --compile
@@ -105,18 +125,21 @@ def help():
# decompile and verify known good python 2.7 # decompile and verify known good python 2.7
test_pythonlib.py --ok-2.7 --verify test_pythonlib.py --ok-2.7 --verify
""") """
)
sys.exit(1) sys.exit(1)
def do_tests(src_dir, obj_patterns, target_dir, opts): def do_tests(src_dir, obj_patterns, target_dir, opts):
def file_matches(files, root, basenames, patterns): def file_matches(files, root, basenames, patterns):
files.extend( files.extend(
[os.path.normpath(os.path.join(root, n)) [
for n in basenames os.path.normpath(os.path.join(root, n))
for pat in patterns for n in basenames
if fnmatch(n, pat)]) for pat in patterns
if fnmatch(n, pat)
]
)
files = [] files = []
# Change directories so use relative rather than # Change directories so use relative rather than
@@ -125,8 +148,8 @@ def do_tests(src_dir, obj_patterns, target_dir, opts):
cwd = os.getcwd() cwd = os.getcwd()
os.chdir(src_dir) os.chdir(src_dir)
if opts['do_compile']: if opts["do_compile"]:
compiled_version = opts['compiled_version'] compiled_version = opts["compiled_version"]
if compiled_version and PYTHON_VERSION != compiled_version: if compiled_version and PYTHON_VERSION != compiled_version:
sys.stderr.write("Not compiling: " sys.stderr.write("Not compiling: "
"desired Python version is %s " "desired Python version is %s "
@@ -143,7 +166,7 @@ def do_tests(src_dir, obj_patterns, target_dir, opts):
pass pass
pass pass
for root, dirs, basenames in os.walk('.'): for root, dirs, basenames in os.walk("."):
# Turn root into a relative path # Turn root into a relative path
dirname = root[2:] # 2 = len('.') + 1 dirname = root[2:] # 2 = len('.') + 1
file_matches(files, dirname, basenames, obj_patterns) file_matches(files, dirname, basenames, obj_patterns)
@@ -156,21 +179,21 @@ def do_tests(src_dir, obj_patterns, target_dir, opts):
os.chdir(cwd) os.chdir(cwd)
files.sort() files.sort()
if opts['start_with']: if opts["start_with"]:
try: try:
start_with = files.index(opts['start_with']) start_with = files.index(opts["start_with"])
files = files[start_with:] files = files[start_with:]
print('>>> starting with file', files[0]) print(">>> starting with file", files[0])
except ValueError: except ValueError:
pass pass
print time.ctime() print(time.ctime())
print 'Source directory: ', src_dir print("Source directory: ", src_dir)
print 'Output directory: ', target_dir print("Output directory: ", target_dir)
try: try:
_, _, failed_files, failed_verify = \ _, _, failed_files, failed_verify = main(
main(src_dir, target_dir, files, [], src_dir, target_dir, files, [], do_verify=opts["do_verify"]
do_verify=opts['do_verify']) )
if failed_files != 0: if failed_files != 0:
sys.exit(2) sys.exit(2)
elif failed_verify != 0: elif failed_verify != 0:
@@ -179,63 +202,74 @@ def do_tests(src_dir, obj_patterns, target_dir, opts):
except (KeyboardInterrupt, OSError): except (KeyboardInterrupt, OSError):
print() print()
sys.exit(1) sys.exit(1)
if test_opts['rmtree']: if test_opts["rmtree"]:
parent_dir = os.path.dirname(target_dir) parent_dir = os.path.dirname(target_dir)
print("Everything good, removing %s" % parent_dir) print("Everything good, removing %s" % parent_dir)
shutil.rmtree(parent_dir) shutil.rmtree(parent_dir)
if __name__ == '__main__':
if __name__ == "__main__":
test_dirs = [] test_dirs = []
checked_dirs = [] checked_dirs = []
start_with = None start_with = None
test_options_keys = list(test_options.keys()) test_options_keys = list(test_options.keys())
test_options_keys.sort() test_options_keys.sort()
opts, args = getopt.getopt(sys.argv[1:], '', opts, args = getopt.getopt(
['start-with=', 'verify', 'verify-run', sys.argv[1:],
'syntax-verify', 'all', "",
'compile', 'coverage', [
'no-rm'] \ "start-with=",
+ test_options_keys ) "verify",
if not opts: help() "verify-run",
"syntax-verify",
"all",
"compile",
"coverage",
"no-rm",
]
+ test_options_keys,
)
if not opts:
help()
test_opts = { test_opts = {
'do_compile': False, "do_compile": False,
'do_verify': False, "do_verify": False,
'start_with': None, "start_with": None,
'rmtree' : True, "rmtree": True,
'coverage' : False "coverage": False,
} }
for opt, val in opts: for opt, val in opts:
if opt == '--verify': if opt == "--verify":
test_opts['do_verify'] = 'strong' test_opts["do_verify"] = "strong"
elif opt == '--syntax-verify': elif opt == "--syntax-verify":
test_opts['do_verify'] = 'weak' test_opts["do_verify"] = "weak"
elif opt == '--verify-run': elif opt == "--verify-run":
test_opts['do_verify'] = 'verify-run' test_opts["do_verify"] = "verify-run"
elif opt == '--compile': elif opt == "--compile":
test_opts['do_compile'] = True test_opts["do_compile"] = True
elif opt == '--start-with': elif opt == "--start-with":
test_opts['start_with'] = val test_opts["start_with"] = val
elif opt == '--no-rm': elif opt == "--no-rm":
test_opts['rmtree'] = False test_opts["rmtree"] = False
elif opt[2:] in test_options_keys: elif opt[2:] in test_options_keys:
test_dirs.append(test_options[opt[2:]]) test_dirs.append(test_options[opt[2:]])
elif opt == '--all': elif opt == "--all":
for val in test_options_keys: for val in test_options_keys:
test_dirs.append(test_options[val]) test_dirs.append(test_options[val])
elif opt == '--coverage': elif opt == "--coverage":
test_opts['coverage'] = True test_opts["coverage"] = True
else: else:
help() help()
pass pass
pass pass
if test_opts['coverage']: if test_opts["coverage"]:
os.environ['SPARK_PARSER_COVERAGE'] = ( os.environ["SPARK_PARSER_COVERAGE"] = (
'/tmp/spark-grammar-python-lib%s.cover' % test_dirs[0][-1] "/tmp/spark-grammar-python-lib%s.cover" % test_dirs[0][-1]
) )
last_compile_version = None last_compile_version = None
for src_dir, pattern, target_dir, compiled_version in test_dirs: for src_dir, pattern, target_dir, compiled_version in test_dirs:
@@ -251,7 +285,7 @@ if __name__ == '__main__':
sys.stderr.write("No directories found to check\n") sys.stderr.write("No directories found to check\n")
sys.exit(1) sys.exit(1)
test_opts['compiled_version'] = last_compile_version test_opts["compiled_version"] = last_compile_version
for src_dir, pattern, target_dir in checked_dirs: for src_dir, pattern, target_dir in checked_dirs:
target_dir = os.path.join(target_base, target_dir) target_dir = os.path.join(target_base, target_dir)

192
uncompyle6/semantics/customize.py
View File

@@ -16,122 +16,152 @@
"""Isolate Python version-specific semantic actions here. """Isolate Python version-specific semantic actions here.
""" """
from uncompyle6.semantics.consts import ( from uncompyle6.semantics.consts import TABLE_R, TABLE_DIRECT
TABLE_R, TABLE_DIRECT)
from uncompyle6.parsers.treenode import SyntaxTree from uncompyle6.parsers.treenode import SyntaxTree
from uncompyle6.scanners.tok import Token from uncompyle6.scanners.tok import Token
def customize_for_version(self, is_pypy, version): def customize_for_version(self, is_pypy, version):
if is_pypy: if is_pypy:
######################## ########################
# PyPy changes # PyPy changes
####################### #######################
TABLE_DIRECT.update({ TABLE_DIRECT.update(
'assert_pypy': ( '%|assert %c\n' , 1 ), {
'assert2_pypy': ( '%|assert %c, %c\n' , 1, 4 ), "assert2_pypy": ("%|assert %c, %c\n", (1, "assert_expr"), 4),
'try_except_pypy': ( '%|try:\n%+%c%-%c\n\n', 1, 2 ), "assert_pypy": ("%|assert %c\n", (1, "assert_expr")),
'tryfinallystmt_pypy': ( '%|try:\n%+%c%-%|finally:\n%+%c%-\n\n', 1, 3 ), "assign2_pypy": ("%|%c, %c = %c, %c\n", 3, 2, 0, 1),
'assign3_pypy': ( '%|%c, %c, %c = %c, %c, %c\n', 5, 4, 3, 0, 1, 2 ), "assign3_pypy": ("%|%c, %c, %c = %c, %c, %c\n", 5, 4, 3, 0, 1, 2),
'assign2_pypy': ( '%|%c, %c = %c, %c\n', 3, 2, 0, 1), "try_except_pypy": ("%|try:\n%+%c%-%c\n\n", 1, 2),
}) "tryfinallystmt_pypy": ("%|try:\n%+%c%-%|finally:\n%+%c%-\n\n", 1, 3),
}
)
else: else:
######################## ########################
# Without PyPy # Without PyPy
####################### #######################
TABLE_DIRECT.update({ TABLE_DIRECT.update(
'assert': ( '%|assert %c\n' , 0 ), {
'assert2': ( '%|assert %c, %c\n' , 0, 3 ), "assert": ("%|assert %c\n", (0, "assert_expr")),
'try_except': ( '%|try:\n%+%c%-%c\n\n', 1, 3 ), "assert2": ("%|assert %c, %c\n", (0, "assert_expr"), 3),
'assign2': ( '%|%c, %c = %c, %c\n', "assign2": ("%|%c, %c = %c, %c\n", 3, 4, 0, 1),
3, 4, 0, 1 ), "assign3": ("%|%c, %c, %c = %c, %c, %c\n", 5, 6, 7, 0, 1, 2),
'assign3': ( '%|%c, %c, %c = %c, %c, %c\n', "try_except": ("%|try:\n%+%c%-%c\n\n", 1, 3),
5, 6, 7, 0, 1, 2 ), }
}) )
if version >= 3.0: if version >= 3.0:
if version >= 3.2: if version >= 3.2:
TABLE_DIRECT.update({ TABLE_DIRECT.update(
'del_deref_stmt': ( '%|del %c\n', 0), {"del_deref_stmt": ("%|del %c\n", 0), "DELETE_DEREF": ("%{pattr}", 0)}
'DELETE_DEREF': ( '%{pattr}', 0 ), )
})
from uncompyle6.semantics.customize3 import customize_for_version3 from uncompyle6.semantics.customize3 import customize_for_version3
customize_for_version3(self, version) customize_for_version3(self, version)
else: # < 3.0 else: # < 3.0
TABLE_DIRECT.update({ TABLE_DIRECT.update(
'except_cond3' : ( '%|except %c, %c:\n', {"except_cond3": ("%|except %c, %c:\n", (1, "expr"), (-2, "store"))}
(1, 'expr'), (-2, 'store') ) )
})
if 2.4 <= version <= 2.6: if 2.4 <= version <= 2.6:
TABLE_DIRECT.update({ TABLE_DIRECT.update({"comp_for": (" for %c in %c", 3, 1)})
'comp_for': ( ' for %c in %c', 3, 1 ),
})
else: else:
TABLE_DIRECT.update({ TABLE_DIRECT.update({"comp_for": (" for %c in %c%c", 2, 0, 3)})
'comp_for': ( ' for %c in %c%c', 2, 0, 3 ),
})
if version >= 2.5: if version >= 2.5:
from uncompyle6.semantics.customize25 import customize_for_version25 from uncompyle6.semantics.customize25 import customize_for_version25
customize_for_version25(self, version) customize_for_version25(self, version)
if version >= 2.6: if version >= 2.6:
from uncompyle6.semantics.customize26_27 import customize_for_version26_27 from uncompyle6.semantics.customize26_27 import (
customize_for_version26_27,
)
customize_for_version26_27(self, version) customize_for_version26_27(self, version)
pass pass
else: # < 2.5 else: # < 2.5
global NAME_MODULE global NAME_MODULE
NAME_MODULE = SyntaxTree('stmt', NAME_MODULE = SyntaxTree(
[ SyntaxTree('assign', "stmt",
[ SyntaxTree('expr', [
[Token('LOAD_GLOBAL', pattr='__name__', SyntaxTree(
offset=0, has_arg=True)]), "assign",
SyntaxTree('store', [
[ Token('STORE_NAME', pattr='__module__', SyntaxTree(
offset=3, has_arg=True)]) "expr",
])]) [
TABLE_DIRECT.update({ Token(
'importmultiple': ( '%|import %c%c\n', 2, 3), "LOAD_GLOBAL",
'import_cont' : ( ', %c', 2), pattr="__name__",
'tryfinallystmt': ( '%|try:\n%+%c%-%|finally:\n%+%c%-', offset=0,
(1, 'suite_stmts_opt') , has_arg=True,
(5, 'suite_stmts_opt') ) )
}) ],
),
SyntaxTree(
"store",
[
Token(
"STORE_NAME",
pattr="__module__",
offset=3,
has_arg=True,
)
],
),
],
)
],
)
TABLE_DIRECT.update(
{
"importmultiple": ("%|import %c%c\n", 2, 3),
"import_cont": (", %c", 2),
"tryfinallystmt": (
"%|try:\n%+%c%-%|finally:\n%+%c%-",
(1, "suite_stmts_opt"),
(5, "suite_stmts_opt"),
),
}
)
if version == 2.4: if version == 2.4:
def n_iftrue_stmt24(node): def n_iftrue_stmt24(node):
self.template_engine(('%c', 0), node) self.template_engine(("%c", 0), node)
self.default(node) self.default(node)
self.prune() self.prune()
self.n_iftrue_stmt24 = n_iftrue_stmt24 self.n_iftrue_stmt24 = n_iftrue_stmt24
else: # version <= 2.3: else: # version <= 2.3:
TABLE_DIRECT.update({ TABLE_DIRECT.update({"if1_stmt": ("%|if 1\n%+%c%-", 5)})
'if1_stmt': ( '%|if 1\n%+%c%-', 5 )
})
if version <= 2.1: if version <= 2.1:
TABLE_DIRECT.update({ TABLE_DIRECT.update(
'importmultiple': ( '%c', 2 ), {
# FIXME: not quite right. We have indiividual imports "importmultiple": ("%c", 2),
# when there is in fact one: "import a, b, ..." # FIXME: not quite right. We have indiividual imports
'imports_cont': ( '%C%,', (1, 100, '\n') ), # when there is in fact one: "import a, b, ..."
}) "imports_cont": ("%C%,", (1, 100, "\n")),
}
)
pass pass
pass pass
pass # < 2.5 pass # < 2.5
# < 3.0 continues # < 3.0 continues
TABLE_R.update({ TABLE_R.update(
'STORE_SLICE+0': ( '%c[:]', 0 ), {
'STORE_SLICE+1': ( '%c[%p:]', 0, (1, -1) ), "STORE_SLICE+0": ("%c[:]", 0),
'STORE_SLICE+2': ( '%c[:%p]', 0, (1, -1) ), "STORE_SLICE+1": ("%c[%p:]", 0, (1, -1)),
'STORE_SLICE+3': ( '%c[%p:%p]', 0, (1, -1), (2, -1) ), "STORE_SLICE+2": ("%c[:%p]", 0, (1, -1)),
'DELETE_SLICE+0': ( '%|del %c[:]\n', 0 ), "STORE_SLICE+3": ("%c[%p:%p]", 0, (1, -1), (2, -1)),
'DELETE_SLICE+1': ( '%|del %c[%c:]\n', 0, 1 ), "DELETE_SLICE+0": ("%|del %c[:]\n", 0),
'DELETE_SLICE+2': ( '%|del %c[:%c]\n', 0, 1 ), "DELETE_SLICE+1": ("%|del %c[%c:]\n", 0, 1),
'DELETE_SLICE+3': ( '%|del %c[%c:%c]\n', 0, 1, 2 ), "DELETE_SLICE+2": ("%|del %c[:%c]\n", 0, 1),
}) "DELETE_SLICE+3": ("%|del %c[%c:%c]\n", 0, 1, 2),
TABLE_DIRECT.update({ }
'raise_stmt2': ( '%|raise %c, %c\n', 0, 1), )
}) TABLE_DIRECT.update({"raise_stmt2": ("%|raise %c, %c\n", 0, 1)})
# exec as a built-in statement is only in Python 2.x # exec as a built-in statement is only in Python 2.x
def n_exec_stmt(node): def n_exec_stmt(node):
@@ -139,17 +169,19 @@ def customize_for_version(self, is_pypy, version):
exec_stmt ::= expr exprlist DUP_TOP EXEC_STMT exec_stmt ::= expr exprlist DUP_TOP EXEC_STMT
exec_stmt ::= expr exprlist EXEC_STMT exec_stmt ::= expr exprlist EXEC_STMT
""" """
self.write(self.indent, 'exec ') self.write(self.indent, "exec ")
self.preorder(node[0]) self.preorder(node[0])
if not node[1][0].isNone(): if not node[1][0].isNone():
sep = ' in ' sep = " in "
for subnode in node[1]: for subnode in node[1]:
self.write(sep); sep = ", " self.write(sep)
sep = ", "
self.preorder(subnode) self.preorder(subnode)
self.println() self.println()
self.prune() # stop recursing self.prune() # stop recursing
self.n_exec_smt = n_exec_stmt self.n_exec_smt = n_exec_stmt
pass # < 3.0 pass # < 3.0
return return