jinx/python-uncompyle6
1
Fork 0
mirror of https://github.com/rocky/python-uncompyle6.git synced 2025-08-04 09:22:40 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into python-2.4

Browse Source
This commit is contained in:
rocky
2017-12-03 03:34:50 -05:00
parent ca7f267103 b1cdbe1656
commit 21a8726a47
21 changed files with 90 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
HISTORY.md
View File

@@ -64,14 +64,17 @@ success that his good work deserves.
Dan Pascu did a bit of work from late 2004 to early 2006 to get this
code to handle first Python 2.3 and then 2.4 bytecodes. Because of
jump optimization introduced in the CPython bytecode compiler at that
time, various JUMP instructions were classifed as going backwards, and
COME FROM instructions were reintroduced. See
time, various JUMP instructions were classified to assist parsing For
example, due to the way that code generation and line number table
work, jump instructions to an earlier offset must be looping jumps,
such as those found in a "continue" statement; "COME FROM"
instructions were reintroduced. See
[RELEASE-2.4-CHANGELOG.txt](https://github.com/rocky/python-uncompyle6/blob/master/DECOMPYLE-2.4-CHANGELOG.txt)
for more details here. There wasn't a public
release of RELEASE-2.4 and bytecodes other than Python 2.4 weren't
supported. Dan says the Python 2.3 version could verify the entire
Python library. But given subsequent bugs found like simply
recognizing complex-number constants in bytecode, decompilation wasn't perfect.
for more details here. There wasn't a public release of RELEASE-2.4
and bytecodes other than Python 2.4 weren't supported. Dan says the
Python 2.3 version could verify the entire Python library. But given
subsequent bugs found like simply recognizing complex-number constants
in bytecode, decompilation wasn't perfect.
Next we get to ["uncompyle" and
PyPI](https://pypi.python.org/pypi/uncompyle/1.1) and the era of
@@ -109,18 +112,21 @@ Given this, perhaps it is not surprising that subsequent changes
tended to shy away from using the built-in compiler technology
mechanisms and addressed problems and extensions by some other means.
Specifically, in `uncompyle`, decompilation of python bytecode 2.5 & 2.6
is done by transforming the byte code into a pseudo-2.7 Python
bytecode and is based on code from Eloi Vanderbeken.
Specifically, in `uncompyle`, decompilation of python bytecode 2.5 &
2.6 is done by transforming the byte code into a pseudo-2.7 Python
bytecode and is based on code from Eloi Vanderbeken. A bit of this
could have bene easily added by modifying grammar rules.
This project, `uncompyle6`, abandons that approach for various
reasons. Having a grammar per Python version is much cleaner and it
scales indefinitely. That said, we don't have entire copies of the
grammar, but work off of differences from some neighboring version.
And this too I find helpful. Should there be a desire to rebase or
start a new base version to work off of, say for some future Python
version, that can be done by dumping a grammar for a specific version
after it has been loaded incrementally.
Should there be a desire to rebase or start a new base version to work
off of, say for some future Python version, that can be done by
dumping a grammar for a specific version after it has been loaded
incrementally. You can get a full dump of the grammar by profiling the
grammar on a large body of Python source code.
Another problem with pseudo-2.7 bytecode is that that we need offsets
in fragment deparsing to be exactly the same as the bytecode; the
@@ -163,24 +169,26 @@ Hartmut a decade an a half ago:
This project deparses using an Earley-algorithm parse with lots of
massaging of tokens and the grammar in the scanner
phase. Earley-algorithm parsers are context free and tend to be linear
if the grammar is LR or left recursive.
if the grammar is LR or left recursive. There is a technique for
improving LL right recursion, but our parser doesn't have that yet.
Another approach that doesn't use grammars is to do something like
simulate execution symbolically and build expression trees off of
stack results. Control flow in that approach still needs to be handled
somewhat ad hoc. The two important projects that work this way are
[unpyc3](https://code.google.com/p/unpyc3/) and most especially
[pycdc](https://github.com/zrax/pycdc) The latter project is largely
by Michael Hansen and Darryl Pogue. If they supported getting
source-code fragments, did a better job in supporting Python more
fully, and had a way I could call it from Python, I'd probably would
have ditched this and used that. The code runs blindingly fast and
spans all versions of Python, although more recently Python 3 support
has been lagging. The code is impressive for its smallness given that
it covers many versions of Python. However, I think it has reached a
scalability issue, same as all the other efforts. For it to handle
Python versions more accurately, I think it will need to have a lot
more code specially which specialize for Python versions.
Another approach to decompiling, and one that doesn't use grammars is
to do something like simulate execution symbolically and build
expression trees off of stack results. Control flow in that approach
still needs to be handled somewhat ad hoc. The two important projects
that work this way are [unpyc3](https://code.google.com/p/unpyc3/) and
most especially [pycdc](https://github.com/zrax/pycdc) The latter
project is largely by Michael Hansen and Darryl Pogue. If they
supported getting source-code fragments, did a better job in
supporting Python more fully, and had a way I could call it from
Python, I'd probably would have ditched this and used that. The code
runs blindingly fast and spans all versions of Python, although more
recently Python 3 support has been lagging. The code is impressive for
its smallness given that it covers many versions of Python. However, I
think it has reached a scalability issue, same as all the other
efforts. To handle Python versions more accurately, I think that code
base will need to have a lot more code specially which specializes for
Python versions. And then it will run into a modularity problem.
Tests for the project have been, or are being, culled from all of the
projects mentioned. Quite a few have been added to improve grammar
@@ -190,11 +198,12 @@ If you think, as I am sure will happen in the future, "hey, I can just
write a decompiler from scratch and not have to deal with all all of
the complexity here", think again. What is likely to happen is that
you'll get at best a 90% solution working for a single Python release
that will be obsolete in about a year, and more obsolute each
that will be obsolete in about a year, and more obsolete each
subsequent year. Writing a decompiler for Python gets harder as it
Python progresses, so writing one for Python 3.7 isn't as easy as it
was for Python 2.2. That said, if you still feel you want to write a
single version decompiler, talk to me. I may have some ideas.
single version decompiler, look at the test cases in this project and
talk to me. I may have some ideas.
For a little bit of the history of changes to the Earley-algorithm parser,

6
README.rst
View File

@@ -75,11 +75,9 @@ This uses setup.py, so it follows the standard Python routine:
::
pip install -e .
pip install -r requirements-dev.txt
pip install -e . # set up to run from source tree
# Or if you want to install instead
python setup.py install # may need sudo
# or if you have pyenv:
python setup.py develop
A GNU makefile is also provided so :code:`make install` (possibly as root or
sudo) will do the steps above.

BIN
test/bytecode_2.5/02_except_as.pyc
View File

Binary file not shown.

BIN
test/bytecode_2.6/02_except_as.pyc
View File

Binary file not shown.

BIN
test/bytecode_2.7/02_except_as.pyc Normal file
View File

Binary file not shown.

7
test/simple_source/bug26/02_except_as.py
View File

@@ -1,4 +1,6 @@
# From 2.6.9 ConfigParser.py
# Note this can only be compiled in Python 2.x
#
# Bug was being able to handle:
# except KeyError, e
# vs 2.6+.
@@ -12,7 +14,12 @@
#
# Python 2.6 allows both, but we use the older form since
# that matches the grammar for how this gets parsed
try:
value = "foo"
except RuntimeError:
# Test:
# raise_stmt3 ::= expr expr expr RAISE_VARARGS_3
raise 1, 2, 3
except KeyError, e:
raise RuntimeError('foo')

2
test/simple_source/bug32/03_if_try_raise.py
View File

@@ -1,5 +1,5 @@
# From 3.2 distutils/core
# Ensure we handle funky trystmt
# Ensure we handle funky try_except
def setup (ok, dist, **attrs):
if ok:
try:

4
test/simple_source/bug_pypy27/03_try_return.py
View File

@@ -1,7 +1,7 @@
# From PyPy 2.7 argparse.py
# PyPY reduces branches as a result of the return statement
# So we need a new rules for trystmt and try_middle which we
# suffix with _pypy, e.g. trystmt_pypy, and try_middle_pypy
# So we need a new rules for try_except and try_middle which we
# suffix with _pypy, e.g. try_except_pypy, and try_middle_pypy
def call(self, string):
try:
return open(string, self, self._bufsize)

4
test/simple_source/exception/01_try_except.py
View File

@@ -1,5 +1,5 @@
# Tests:
# trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
# try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
# try_middle COME_FROM
# except_stmt ::= except
@@ -9,7 +9,7 @@ except:
pass
# Tests:
# trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
# try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
# try_middle COME_FROM
# except_stmt ::= except_cond1 except_suite
# except_suite ::= ...

2
test/simple_source/exception/20_try_except.py
View File

@@ -1,7 +1,7 @@
# Tests:
# forstmt ::= SETUP_LOOP expr _for store for_block POP_BLOCK COME_FROM
# for_block ::= l_stmts_opt JUMP_BACK
# trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK try_middle COME_FROM
# try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK try_middle COME_FROM
# try_middle ::= jmp_abs COME_FROM except_stmts END_FINALLY
# Had a bug with the end of the except matching the end of the

10
test/simple_source/exception/25_try_except.py
View File

@@ -2,11 +2,11 @@
#
# tryfinallystmt ::= SETUP_FINALLY suite_stmts POP_BLOCK LOAD_CONST COME_FROM suite_stmts_opt END_FINALLY
# suite_stmts_opt ::= suite_stmts
# trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK try_middle COME_FROM
# try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK try_middle COME_FROM
# try_middle ::= JUMP_FORWARD COME_FROM except_stmts END_FINALLY COME_FROM
# except_stmt ::= except_cond1 except_suite
# except_cond1 ::= DUP_TOP expr COMPARE_OP jmp_false POP_TOP POP_TOP POP_TOP
# trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK try_middle COME_FROM
# try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK try_middle COME_FROM
# try_middle ::= JUMP_FORWARD COME_FROM except_stmts END_FINALLY COME_FROM
# except_cond1 ::= DUP_TOP expr COMPARE_OP jmp_false POP_TOP POP_TOP POP_TOP
try:
@@ -21,13 +21,13 @@ finally:
x = 4
# Tests Python3:
# trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK try_middle come_froms
# try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK try_middle come_froms
# come_froms ::= COME_FROM COME_FROM
# START ::= |- stmts
# stmts ::= sstmt
# sstmt ::= stmt
# stmt ::= trystmt
# trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK try_middle come_froms
# stmt ::= try_except
# try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK try_middle come_froms
# come_froms ::= COME_FROM
# try_middle ::= JUMP_FORWARD COME_FROM except_stmts END_FINALLY COME_FROM
# Python2 doesn't have the come_froms (which allows for 3 successive COME_FROMs)

2
uncompyle6/parser.py
View File

@@ -252,7 +252,7 @@ class PythonParser(GenericASTBuilder):
stmt ::= while1elsestmt
stmt ::= forstmt
stmt ::= forelsestmt
stmt ::= trystmt
stmt ::= try_except
stmt ::= tryelsestmt
stmt ::= tryfinallystmt
stmt ::= withstmt

8
uncompyle6/parsers/parse2.py
View File

@@ -126,7 +126,7 @@ class Python2Parser(PythonParser):
iflaststmt ::= testexpr c_stmts_opt JUMP_ABSOLUTE
iflaststmtl ::= testexpr c_stmts_opt JUMP_BACK
# this is nested inside a trystmt
# this is nested inside a try_except
tryfinallystmt ::= SETUP_FINALLY suite_stmts_opt
POP_BLOCK LOAD_CONST
COME_FROM suite_stmts_opt END_FINALLY
@@ -138,7 +138,7 @@ class Python2Parser(PythonParser):
tryelsestmtl ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_middle else_suitel COME_FROM
trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_middle COME_FROM
try_middle ::= JUMP_FORWARD COME_FROM except_stmts
@@ -412,8 +412,8 @@ class Python2Parser(PythonParser):
# only have SETUP_EXCEPT customization for PyPy, but that might not
# always be the case.
self.add_unique_rules([
"stmt ::= trystmt_pypy",
"trystmt_pypy ::= SETUP_EXCEPT suite_stmts_opt try_middle_pypy",
"stmt ::= try_except_pypy",
"try_except_pypy ::= SETUP_EXCEPT suite_stmts_opt try_middle_pypy",
"try_middle_pypy ::= COME_FROM except_stmts END_FINALLY COME_FROM"
], customize)
continue

4
uncompyle6/parsers/parse26.py
View File

@@ -35,8 +35,8 @@ class Python26Parser(Python2Parser):
# Sometimes we don't put in COME_FROM to the next statement
# like we do in 2.7. Perhaps we should?
trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_middle
try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_middle
tryelsestmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_middle else_suite COME_FROM

6
uncompyle6/parsers/parse3.py
View File

@@ -97,12 +97,10 @@ class Python3Parser(PythonParser):
stmt ::= raise_stmt0
stmt ::= raise_stmt1
stmt ::= raise_stmt2
stmt ::= raise_stmt3
raise_stmt0 ::= RAISE_VARARGS_0
raise_stmt1 ::= expr RAISE_VARARGS_1
raise_stmt2 ::= expr expr RAISE_VARARGS_2
raise_stmt3 ::= expr expr expr RAISE_VARARGS_3
del_stmt ::= delete_subscr
delete_subscr ::= expr expr DELETE_SUBSCR
@@ -176,10 +174,10 @@ class Python3Parser(PythonParser):
# one COME_FROM for Python 2.7 seems to associate the
# COME_FROM targets from the wrong places
trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_middle opt_come_from_except
# this is nested inside a trystmt
# this is nested inside a try_except
tryfinallystmt ::= SETUP_FINALLY suite_stmts_opt
POP_BLOCK LOAD_CONST
COME_FROM_FINALLY suite_stmts_opt END_FINALLY

6
uncompyle6/parsers/parse32.py
View File

@@ -26,9 +26,9 @@ class Python32Parser(Python3Parser):
# Python 3.5+ has jump optimization to remove the redundant
# jump_excepts. But in 3.3 we need them added
trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_middle
jump_excepts come_from_except_clauses
try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_middle
jump_excepts come_from_except_clauses
try_middle ::= JUMP_FORWARD COME_FROM_EXCEPT except_stmts
END_FINALLY

2
uncompyle6/parsers/parse33.py
View File

@@ -20,7 +20,7 @@ class Python33Parser(Python32Parser):
# Python 3.5+ has jump optimization to remove the redundant
# jump_excepts. But in 3.3 we need them added
trystmt ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
try_middle
jump_excepts come_from_except_clauses
"""

4
uncompyle6/parsers/parse36.py
View File

@@ -70,8 +70,8 @@ class Python36Parser(Python35Parser):
# Try middle following a return_stmts
try_middle36 ::= COME_FROM_EXCEPT except_stmts END_FINALLY
stmt ::= trystmt36
trystmt36 ::= SETUP_EXCEPT return_stmts try_middle36 opt_come_from_except
stmt ::= try_except36
try_except36 ::= SETUP_EXCEPT return_stmts try_middle36 opt_come_from_except
"""
def add_custom_rules(self, tokens, customize):

4
uncompyle6/semantics/consts.py
View File

@@ -257,11 +257,11 @@ TABLE_DIRECT = {
'%|for %c in %c:\n%+%c%-%|else:\n%+%c%-', 3, 1, 4, -2 ),
'forelselaststmtl': (
'%|for %c in %c:\n%+%c%-%|else:\n%+%c%-\n\n', 3, 1, 4, -2 ),
'trystmt': ( '%|try:\n%+%c%-%c\n\n', 1, 3 ),
'try_except': ( '%|try:\n%+%c%-%c\n\n', 1, 3 ),
'tryelsestmt': ( '%|try:\n%+%c%-%c%|else:\n%+%c%-\n\n', 1, 3, 4 ),
'tryelsestmtc': ( '%|try:\n%+%c%-%c%|else:\n%+%c%-', 1, 3, 4 ),
'tryelsestmtl': ( '%|try:\n%+%c%-%c%|else:\n%+%c%-', 1, 3, 4 ),
'tf_trystmt': ( '%c%-%c%+', 1, 3 ),
'tf_try_except': ( '%c%-%c%+', 1, 3 ),
'tf_tryelsestmt': ( '%c%-%c%|else:\n%+%c', 1, 3, 4 ),
'tryfinallystmt': ( '%|try:\n%+%c%-%|finally:\n%+%c%-\n\n', 1, 5 ),
'except': ( '%|except:\n%+%c%-', 3 ),

4
uncompyle6/semantics/fragments.py
View File

@@ -202,12 +202,12 @@ class FragmentsWalker(pysource.SourceWalker, object):
self.set_pos_info(node, start, start+len("pass"))
self.default(node)
def n_trystmt(self, node):
def n_try_except(self, node):
start = len(self.f.getvalue()) + len(self.indent)
self.set_pos_info(node[0], start, start+len("try:"))
self.default(node)
n_tryelsestmt = n_tryelsestmtc = n_tryelsestmtl = n_tryfinallystmt = n_trystmt
n_tryelsestmt = n_tryelsestmtc = n_tryelsestmtl = n_tryfinallystmt = n_try_except
def n_return_stmt(self, node):
start = len(self.f.getvalue()) + len(self.indent)

18
uncompyle6/semantics/pysource.py
View File

@@ -242,10 +242,10 @@ class SourceWalker(GenericASTTraversal, object):
TABLE_DIRECT.update({
'assert_pypy': ( '%|assert %c\n' , 1 ),
'assert2_pypy': ( '%|assert %c, %c\n' , 1, 4 ),
'trystmt_pypy': ( '%|try:\n%+%c%-%c\n\n', 1, 2 ),
'try_except_pypy': ( '%|try:\n%+%c%-%c\n\n', 1, 2 ),
'tryfinallystmt_pypy': ( '%|try:\n%+%c%-%|finally:\n%+%c%-\n\n', 1, 3 ),
'assign3_pypy': ( '%|%c, %c, %c = %c, %c, %c\n', 5, 4, 3, 0, 1, 2 ),
'assign2_pypy': ( '%|%c, %c = %c, %c\n', 3, 2, 0, 1),
'assign3_pypy': ( '%|%c, %c, %c = %c, %c, %c\n', 5, 4, 3, 0, 1, 2 ),
'assign2_pypy': ( '%|%c, %c = %c, %c\n', 3, 2, 0, 1),
})
else:
########################
@@ -254,9 +254,9 @@ class SourceWalker(GenericASTTraversal, object):
TABLE_DIRECT.update({
'assert': ( '%|assert %c\n' , 0 ),
'assert2': ( '%|assert %c, %c\n' , 0, 3 ),
'trystmt': ( '%|try:\n%+%c%-%c\n\n', 1, 3 ),
'assign2': ( '%|%c, %c = %c, %c\n', 3, 4, 0, 1 ),
'assign3': ( '%|%c, %c, %c = %c, %c, %c\n', 5, 6, 7, 0, 1, 2 ),
'try_except': ( '%|try:\n%+%c%-%c\n\n', 1, 3 ),
'assign2': ( '%|%c, %c = %c, %c\n', 3, 4, 0, 1 ),
'assign3': ( '%|%c, %c, %c = %c, %c, %c\n', 5, 6, 7, 0, 1, 2 ),
})
if version < 3.0:
TABLE_R.update({
@@ -435,7 +435,7 @@ class SourceWalker(GenericASTTraversal, object):
})
if version >= 3.6:
TABLE_DIRECT.update({
'trystmt36': ( '%|try:\n%+%c%-%c\n\n', 1, 2 ),
'try_except36': ( '%|try:\n%+%c%-%c\n\n', 1, 2 ),
})
def n_async_call(node):
@@ -857,8 +857,8 @@ class SourceWalker(GenericASTTraversal, object):
# 'tryfinallystmt': ( '%|try:\n%+%c%-%|finally:\n%+%c%-', 1, 5 ),
def n_tryfinallystmt(self, node):
if len(node[1][0]) == 1 and node[1][0][0] == 'stmt':
if node[1][0][0][0] == 'trystmt':
node[1][0][0][0].kind = 'tf_trystmt'
if node[1][0][0][0] == 'try_except':
node[1][0][0][0].kind = 'tf_try_except'
if node[1][0][0][0] == 'tryelsestmt':
node[1][0][0][0].kind = 'tf_tryelsestmt'
self.default(node)