Compare commits

..

7 Commits

Author SHA1 Message Date
rocky
300d387349 Get ready for release 2.11.2 2017-07-09 01:44:55 -04:00
rocky
27ab6fe2f5 Use xdis 3.5.0's opcode sets 2017-07-08 20:41:46 -04:00
rocky
2e164763eb Start supporting Pypy 3.5 (5.7.1-beta) 2017-07-08 17:47:32 -04:00
rocky
d332bde104 Loops in Python 2.4-2.6 loop come_from
Looks like Python 2.4-2.6 may have a COME_FROM(_LOOP)
before the jump_back.

Fixes Issue #123
2017-07-05 06:12:14 -04:00
rocky
0893652943 Work around not having real flow-control analysis 2017-06-29 20:49:01 -04:00
rocky
6efd7afda3 continue non-detection in Python 2.7
fixes issue 122
2017-06-29 20:27:07 -04:00
rocky
ee3202779a A guard against badly formated bytecode 2017-06-28 18:39:05 -04:00
23 changed files with 183 additions and 74 deletions

View File

@@ -1,3 +1,46 @@
2017-07-09 rocky <rb@dustyfeet.com>
* HOW-TO-REPORT-A-BUG.md, uncompyle6/version.py: Get ready for
release 2.11.2
2017-07-08 rocky <rb@dustyfeet.com>
* __pkginfo__.py, uncompyle6/scanner.py,
uncompyle6/scanners/scanner2.py, uncompyle6/scanners/scanner26.py,
uncompyle6/scanners/scanner3.py, uncompyle6/scanners/scanner30.py,
uncompyle6/scanners/tok.py: Use xdis 3.5.0's opcode sets
2017-07-08 rocky <rb@dustyfeet.com>
* test/test_pyenvlib.py, uncompyle6/scanners/pypy32.py,
uncompyle6/scanners/pypy35.py, uncompyle6/scanners/scanner15.py,
uncompyle6/scanners/scanner32.py, uncompyle6/scanners/scanner34.py,
uncompyle6/scanners/scanner35.py, uncompyle6/scanners/scanner36.py:
Start supporting Pypy 3.5 (5.7.1-beta)
2017-07-05 rocky <rb@dustyfeet.com>
* test/simple_source/bug26/03_loop_if_cf.py,
uncompyle6/parsers/parse26.py: Loops in Python 2.4-2.6 loop
come_from Looks like Python 2.4-2.6 may have a COME_FROM(_LOOP) before the
jump_back. Fixes Issue #123
2017-06-29 rocky <rb@dustyfeet.com>
* : Work around not having real flow-control analysis
2017-06-28 rocky <rb@dustyfeet.com>
* uncompyle6/semantics/make_function.py: A guard against badly
formated bytecode
2017-06-25 rocky <rb@dustyfeet.com>
* ChangeLog, NEWS, test/simple_source/bug31/04_def_annotate.py,
uncompyle6/semantics/make_function.py,
uncompyle6/semantics/pysource.py: 3.x funciton and annotation bug
fixes
2017-06-25 rocky <rb@dustyfeet.com>
* uncompyle6/version.py: Get ready for release 2.11.1

View File

@@ -19,7 +19,7 @@ So it is likely you'll find a mistranslation in decompiling.
The basic requirement is pretty simple:
* Python bytecode
* Source text
* Python source text
## What to send (additional helpful information)
@@ -50,7 +50,7 @@ one fool can learn, so can another."
## Narrowing the problem
I don't need the entire source code base for which one file or module
I don't need or want the entire source code base for which one file or module
can't be decompiled. I just need that one file or module only. If
there are several files, file a bug report for each file.

47
NEWS
View File

@@ -1,10 +1,17 @@
uncompyle6 2.11.1 2016-06-22
uncompyle6 2.11.2 2017-07-09
- Python 3.x annotatoin and function signature fixes
- Start supporting Pypy 3.5 (5.7.1-beta)
- use xdis 3.5.0's opcode sets and require xdis 3.5.0
- Correct some Python 2.4-2.6 loop detection
- guard against badly formatted bytecode
uncompyle6 2.11.1 2017-06-25
- Python 3.x annotation and function signature fixes
- Bump xdis version
- Small pysource bug fixes
uncompyle6 2.11.0 2016-06-18 Fleetwood
uncompyle6 2.11.0 2017-06-18 Fleetwood
- Major improvements in fragment tracking
* Add nonterminal node in extractInfo
* tag more offsets in expressions
@@ -14,17 +21,17 @@ uncompyle6 2.11.0 2016-06-18 Fleetwood
- Fixes yet again for make_function node handling; document what's up here
- Fix bug in snowflake Python 3.5 *args kwargs
uncompyle6 2.10.1 2016-06-3 Marylin Frankel
uncompyle6 2.10.1 2017-06-3 Marylin Frankel
- fix some fragments parsing bugs
- was returning the wrong type sometimes in deparse_code_around_offset()
- capture function name in offsets
- track changes to ifelstrmtr node from pysource into fragments
uncompyle6 2.10.0 2016-05-30 Elaine Gordon
uncompyle6 2.10.0 2017-05-30 Elaine Gordon
- Add fuzzy offset deparse lookup
- 3.6 bugfixes
- Add fuzzy offset deparse look up
- 3.6 bug fixes
- fix EXTENDED_ARGS handling (and in 2.6 and others)
- semantic routine make_function fragments.py
- MAKE_FUNCTION handling
@@ -35,19 +42,19 @@ uncompyle6 2.10.0 2016-05-30 Elaine Gordon
- 3.5 FUNCTION_VAR bug
- 3.x pass statement insdie while True
- Improve 3.2 decompilation
- Fixed -o argument processing (Gregrory)
- Fixed -o argument processing (grkov90)
- Reduce scope of LOAD_ASSERT as expr to 3.4+
- "await" statement fixes
- 2.3, 2.4 "if 1 .." fixes
- 3.x annotation fixes
uncompyle6 2.9.11 2016-04-06
uncompyle6 2.9.11 2017-04-06
- Better support for Python 3.5+ BUILD_MAP_UNPACK
- Start 3.6 CALL_FUNCTION_EX support
- Many decompilation bug fixes. (Many more remain). See ChangeLog
uncompyle6 2.9.10 2016-02-25
uncompyle6 2.9.10 2017-02-25
- Python grammar rule fixes
- Add ability to get grammar coverage on runs
@@ -114,7 +121,7 @@ uncompyle6 2.9.6 2016-11-20
uncompyle6 2.9.5 2016-11-13
- Fix Python 3 bugs:
* improprer while 1 else
* improper while 1 else
* docstring indent
* 3.3 default values in lambda expressions
* start 3.0 decompilation (needs newer xdis)
@@ -124,12 +131,12 @@ uncompyle6 2.9.5 2016-11-13
uncompyle6 2.9.4 2016-11-02
- Handle Python 3.x function annotations
- track def keywoard-parameter line-splitting in source code better
- track def keyword-parameter line-splitting in source code better
- bump min xdis version to mask previous xdis bug
uncompyle6 2.9.3 2016-10-26
Release forced by incompatiblity change in xdis 3.2.0.
Release forced by incompatibility change in xdis 3.2.0.
- Python 3.1 bugs:
* handle "with ... as"
@@ -161,7 +168,7 @@ uncompyle6 2.9.0 2016-10-09
this Forces change in requirements.txt and _pkg_info_.py
- Start Python 1.5 decompiling; another round of work is needed to
remove bugs
- Simpify python 2.1 grammar
- Simplify python 2.1 grammar
- Fix bug with -t ... Wasn't showing source text when -t option was given
- Fix 2.1-2.6 bug in list comprehension
@@ -184,7 +191,7 @@ control-flow structure detection is done.
. 3.0 .. 3.2 *args processing
. 3.0 .. 3.2 call name and kwargs bug
. 3.0 .. getting parameter of *
. 3.0 .. handling varible number of args
. 3.0 .. handling variable number of args
. 3.0 .. "if" structure bugs
* 3.5+ if/else bugs
* 2.2-2.6 bugs
@@ -235,7 +242,7 @@ uncompyle6 2.7.1 2016-07-26
uncompyle6 2.7.0 2016-07-15
- Many Syntax and verifification bugs removed
- Many Syntax and verification bugs removed
tested on standard libraries from 2.3.7 to 3.5.1
and they all decompile and verify fine.
I'm sure there are more bugs though.
@@ -262,9 +269,9 @@ uncompyle6 2.6.0 2016-07-07
- Better <2.6 vs. 2.7 grammar separation
- Fix some 2.7 deparsing bugs
- Fix bug in installing uncompyle6 script
- Doc improvments
- Doc improvements
uncompyle6 2.5.0 2016-06-22 Summer Solstace
uncompyle6 2.5.0 2016-06-22 Summer Solstice
- Much better Python 3.2-3.5 coverage.
3.4.6 is probably the best;3.2 and 3.5 are weaker
@@ -276,7 +283,7 @@ uncompyle6 2.5.0 2016-06-22 Summer Solstace
uncompyle6 2.4.0 2016-05-18 (in memory of Lewis Bernstein)
- Many Python 3 bugs fixed:
* Python 3.2 to 3.5 libaries largely
* Python 3.2 to 3.5 libraries largely
uncompyle and most verify
- pydisassembler:
* disassembles all code objects in a file
@@ -334,7 +341,7 @@ uncompyle6 2.2.0 2016-04-30
uncompyle6 2.2.0 2016-04-02
- Support single-mode (in addtion to exec-mode) compilation
- Support single-mode (in addition to exec-mode) compilation
- Start to DRY Python 2 and Python 3 grammars
- Fix bug in if else ternary construct
- Fix bug in uncomplye6 -d and -r options (via lelicopter)

View File

@@ -40,7 +40,7 @@ entry_points = {
]}
ftp_url = None
install_requires = ['spark-parser >= 1.6.1, < 1.7.0',
'xdis >= 3.4.0, < 3.5.0', 'six']
'xdis >= 3.5.0, < 3.6.0', 'six']
license = 'MIT'
mailing_list = 'python-debugger@googlegroups.com'
modname = 'uncompyle6'

Binary file not shown.

View File

@@ -0,0 +1,19 @@
# Bug in < 2.6 is having a COME_FROM_LOOP (but we
# don't tag that so it is just COME_FROM *before*
# a jump back to the loop.
def pickup(self, open_players, open_buf, wrap_buf):
for aplayer in self._game.active_players:
if aplayer in open_players:
aplayer.send(open_players)
if self == aplayer:
for awatcher in self._watchers:
if awatcher._can_see_detail:
awatcher.send(open_buf)
else:
awatcher.send(wrap_buf)
else:
self._game.send(aplayer.side)
else:
self._game.send(aplayer.side, wrap_buf)

View File

@@ -29,7 +29,7 @@ from fnmatch import fnmatch
TEST_VERSIONS=('2.3.7', '2.4.6', '2.5.6', '2.6.9',
'pypy-2.4.0', 'pypy-2.6.1',
'pypy-5.0.1', 'pypy-5.3.1',
'pypy-5.0.1', 'pypy-5.3.1', 'pypy3.5-5.7.1-beta',
'2.7.10', '2.7.11', '2.7.12', '2.7.13',
'3.0.1', '3.1.5', '3.2.6',
'3.3.5', '3.3.6',

View File

@@ -84,6 +84,12 @@ class Python26Parser(Python2Parser):
ja_cf_pop ::= JUMP_ABSOLUTE come_froms POP_TOP
jf_cf_pop ::= JUMP_FORWARD come_froms POP_TOP
# The first optional COME_FROM when it appears is really
# COME_FROM_LOOP, but in <= 2.6 we don't distinguish
# this
cf_jb_cf_pop ::= _come_from JUMP_BACK come_froms POP_TOP
bp_come_from ::= POP_BLOCK COME_FROM
jb_bp_come_from ::= JUMP_BACK bp_come_from
@@ -111,7 +117,8 @@ class Python26Parser(Python2Parser):
break_stmt ::= BREAK_LOOP JUMP_BACK
# Semantic actions want else_suitel to be at index 3
ifelsestmtl ::= testexpr c_stmts_opt jb_cf_pop else_suitel
ifelsestmtl ::= testexpr c_stmts_opt cf_jb_cf_pop else_suitel
ifelsestmtc ::= testexpr c_stmts_opt ja_cf_pop else_suitec
# Semantic actions want suite_stmts_opt to be at index 3

View File

@@ -89,7 +89,7 @@ class Scanner(object):
if op is None:
op = self.code[pos]
target = self.get_argument(pos)
if op in self.opc.hasjrel:
if op in self.opc.JREL_OPS:
target += pos + 3
return target
@@ -100,7 +100,7 @@ class Scanner(object):
def print_bytecode(self):
for i in self.op_range(0, len(self.code)):
op = self.code[i]
if op in self.opc.hasjabs+self.opc.hasjrel:
if op in self.JUMP_OPs:
dest = self.get_target(i, op)
print('%i\t%s\t%i' % (i, self.opname[op], dest))
else:

View File

@@ -1,22 +1,18 @@
# Copyright (c) 2016 by Rocky Bernstein
# Copyright (c) 2017 by Rocky Bernstein
"""
Python PyPy 3.2 bytecode scanner/deparser
Python PyPy 3.2 decompiler scanner.
This overlaps Python's 3.2's dis module, but it can be run from
Python 3 and other versions of Python. Also, we save token
information for later use in deparsing.
Does some additional massaging of xdis-disassembled instructions to
make things easier for decompilation.
"""
import uncompyle6.scanners.scanner32 as scan
# bytecode verification, verify(), uses JUMP_OPs from here
from xdis.opcodes import opcode_32 as opc # is this rgith?
from xdis.opcodes import opcode_32 as opc # is this right?
JUMP_OPs = map(lambda op: opc.opname[op], opc.hasjrel + opc.hasjabs)
# We base this off of 2.6 instead of the other way around
# because we cleaned things up this way.
# The history is that 2.7 support is the cleanest,
# then from that we got 2.6 and so on.
# We base this off of 3.2
class ScannerPyPy32(scan.Scanner32):
def __init__(self, show_asm):
# There are no differences in initialization between

View File

@@ -0,0 +1,22 @@
# Copyright (c) 2017 by Rocky Bernstein
"""
Python PyPy 3.2 decompiler scanner.
Does some additional massaging of xdis-disassembled instructions to
make things easier for decompilation.
"""
import uncompyle6.scanners.scanner35 as scan
# bytecode verification, verify(), uses JUMP_OPs from here
from xdis.opcodes import opcode_35 as opc # is this right?
JUMP_OPs = map(lambda op: opc.opname[op], opc.hasjrel + opc.hasjabs)
# We base this off of 3.5
class ScannerPyPy35(scan.Scanner35):
def __init__(self, show_asm):
# There are no differences in initialization between
# pypy 3.5 and 3.5
scan.Scanner35.__init__(self, show_asm, is_pypy=True)
self.version = 3.5
return

View File

@@ -1,6 +1,6 @@
# Copyright (c) 2016 by Rocky Bernstein
# Copyright (c) 2016-2017 by Rocky Bernstein
"""
Python 1.5 bytecode scanner/deparser
Python 1.5 bytecode decompiler scanner.
This massages tokenized 1.5 bytecode to make it more amenable for
grammar parsing.

View File

@@ -189,7 +189,7 @@ class Scanner2(Scanner):
if op == self.opc.EXTENDED_ARG:
extended_arg = oparg * L65536
continue
if op in self.opc.hasconst:
if op in self.opc.CONST_OPS:
const = co.co_consts[oparg]
if iscode(const):
oparg = const
@@ -210,23 +210,23 @@ class Scanner2(Scanner):
pattr = '<code_object ' + const.co_name + '>'
else:
pattr = const
elif op in self.opc.hasname:
elif op in self.opc.NAME_OPS:
pattr = names[oparg]
elif op in self.opc.hasjrel:
elif op in self.opc.JREL_OPS:
# use instead: hasattr(self, 'patch_continue'): ?
if self.version == 2.7:
self.patch_continue(tokens, offset, op)
pattr = repr(offset + 3 + oparg)
elif op in self.opc.hasjabs:
elif op in self.opc.JABS_OPS:
# use instead: hasattr(self, 'patch_continue'): ?
if self.version == 2.7:
self.patch_continue(tokens, offset, op)
pattr = repr(oparg)
elif op in self.opc.haslocal:
elif op in self.opc.LOCAL_OPS:
pattr = varnames[oparg]
elif op in self.opc.hascompare:
elif op in self.opc.COMPARE_OPS:
pattr = self.opc.cmp_op[oparg]
elif op in self.opc.hasfree:
elif op in self.opc.FREE_OPS:
pattr = free[oparg]
if op in self.varargs_ops:
@@ -603,7 +603,7 @@ class Scanner2(Scanner):
if test == offset:
loop_type = 'while 1'
elif self.code[test] in self.opc.hasjabs + self.opc.hasjrel:
elif self.code[test] in self.opc.JUMP_OPs:
self.ignore_if.add(test)
test_target = self.get_target(test)
if test_target > (jump_back+3):
@@ -905,7 +905,9 @@ class Scanner2(Scanner):
'start': start-3,
'end': pre_rtarget})
self.not_continue.add(pre_rtarget)
# FIXME: this is yet another case were we need dominators.
if pre_rtarget not in self.linestartoffsets or self.version < 2.7:
self.not_continue.add(pre_rtarget)
if rtarget < end:
# We have an "else" block of some kind.
@@ -992,11 +994,11 @@ class Scanner2(Scanner):
oparg = self.get_argument(offset)
if label is None:
if op in self.opc.hasjrel and self.op_name(op) != 'FOR_ITER':
# if (op in self.opc.hasjrel and
if op in self.opc.JREL_OPS and self.op_name(op) != 'FOR_ITER':
# if (op in self.opc.JREL_OPS and
# (self.version < 2.0 or op != self.opc.FOR_ITER)):
label = offset + 3 + oparg
elif self.version == 2.7 and op in self.opc.hasjabs:
elif self.version == 2.7 and op in self.opc.JABS_OPS:
if op in (self.opc.JUMP_IF_FALSE_OR_POP,
self.opc.JUMP_IF_TRUE_OR_POP):
if (oparg > offset):

View File

@@ -181,7 +181,7 @@ class Scanner26(scan.Scanner2):
if op == self.opc.EXTENDED_ARG:
extended_arg = oparg * L65536
continue
if op in self.opc.hasconst:
if op in self.opc.CONST_OPS:
const = co.co_consts[oparg]
# We can't use inspect.iscode() because we may be
# using a different version of Python than the
@@ -206,9 +206,9 @@ class Scanner26(scan.Scanner2):
pattr = '<code_object ' + const.co_name + '>'
else:
pattr = const
elif op in self.opc.hasname:
elif op in self.opc.NAME_OPS:
pattr = names[oparg]
elif op in self.opc.hasjrel:
elif op in self.opc.JREL_OPS:
pattr = repr(offset + 3 + oparg)
if op == self.opc.JUMP_FORWARD:
target = self.get_target(offset)
@@ -218,13 +218,13 @@ class Scanner26(scan.Scanner2):
if len(tokens) and tokens[-1].type == 'JUMP_BACK':
tokens[-1].type = intern('CONTINUE')
elif op in self.opc.hasjabs:
elif op in self.opc.JABS_OPS:
pattr = repr(oparg)
elif op in self.opc.haslocal:
elif op in self.opc.LOCAL_OPS:
pattr = varnames[oparg]
elif op in self.opc.hascompare:
elif op in self.opc.COMPARE_OPS:
pattr = self.opc.cmp_op[oparg]
elif op in self.opc.hasfree:
elif op in self.opc.FREE_OPS:
pattr = free[oparg]
if op in self.varargs_ops:
# CE - Hack for >= 2.5

View File

@@ -633,11 +633,11 @@ class Scanner3(Scanner):
rel_offset = 0
if self.version >= 3.6:
target = self.code[offset+1]
if op in self.opc.hasjrel:
if op in self.opc.JREL_OPS:
rel_offset = offset + 2
else:
target = self.code[offset+1] + self.code[offset+2] * 256
if op in self.opc.hasjrel:
if op in self.opc.JREL_OPS:
rel_offset = offset + 3
pass
pass

View File

@@ -120,7 +120,7 @@ class Scanner30(Scanner3):
if test == offset:
loop_type = 'while 1'
elif self.code[test] in opc.hasjabs+opc.hasjrel:
elif self.code[test] in opc.JUMP_OPs:
self.ignore_if.add(test)
test_target = self.get_target(test)
if test_target > (jump_back+3):

View File

@@ -1,6 +1,9 @@
# Copyright (c) 2015-2016 by Rocky Bernstein
# Copyright (c) 2015-2017 by Rocky Bernstein
"""
Python 3.2 bytecode scanner/deparser
Python 3.2 bytecode decompiler scanner.
Does some additional massaging of xdis-disassembled instructions to
make things easier for decompilation.
This sets up opcodes Python's 3.2 and calls a generalized
scanner routine for Python 3.

View File

@@ -1,6 +1,9 @@
# Copyright (c) 2015-2016 by Rocky Bernstein
# Copyright (c) 2015-2017 by Rocky Bernstein
"""
Python 3.4 bytecode scanner/deparser
Python 3.4 bytecode decompiler scanner
Does some additional massaging of xdis-disassembled instructions to
make things easier for decompilation.
This sets up opcodes Python's 3.4 and calls a generalized
scanner routine for Python 3.

View File

@@ -1,6 +1,9 @@
# Copyright (c) 2016 by Rocky Bernstein
# Copyright (c) 2017 by Rocky Bernstein
"""
Python 3.5 bytecode scanner/deparser
Python 3.5 bytecode decompiler scanner
Does some additional massaging of xdis-disassembled instructions to
make things easier for decompilation.
This sets up opcodes Python's 3.5 and calls a generalized
scanner routine for Python 3.
@@ -16,8 +19,8 @@ JUMP_OPs = map(lambda op: opc.opname[op], opc.hasjrel + opc.hasjabs)
class Scanner35(Scanner3):
def __init__(self, show_asm=None):
Scanner3.__init__(self, 3.5, show_asm)
def __init__(self, show_asm=None, is_pypy=False):
Scanner3.__init__(self, 3.5, show_asm, is_pypy)
return
pass

View File

@@ -1,6 +1,9 @@
# Copyright (c) 2016-2017 by Rocky Bernstein
"""
Python 3.6 bytecode scanner/deparser
Python 3.6 bytecode decompiler scanner
Does some additional massaging of xdis-disassembled instructions to
make things easier for decompilation.
This sets up opcodes Python's 3.6 and calls a generalized
scanner routine for Python 3.

View File

@@ -65,10 +65,10 @@ class Token:
if self.pattr:
pattr = self.pattr
if self.opc:
if self.op in self.opc.hasjrel:
if self.op in self.opc.JREL_OPS:
if not self.pattr.startswith('to '):
pattr = "to " + self.pattr
elif self.op in self.opc.hasjabs:
elif self.op in self.opc.JABS_OPS:
self.pattr= str(self.pattr)
if not self.pattr.startswith('to '):
pattr = "to " + str(self.pattr)

View File

@@ -411,7 +411,8 @@ def make_function2(self, node, isLambda, nested=1, codeNode=None):
if code_has_star_star_arg(code):
if argc > 0:
self.write(', ')
self.write('**%s' % code.co_varnames[argc + kw_pairs])
if argc + kw_pairs > 0:
self.write('**%s' % code.co_varnames[argc + kw_pairs])
if isLambda:
self.write(": ")

View File

@@ -1,3 +1,3 @@
# This file is suitable for sourcing inside bash as
# well as importing into Python
VERSION='2.11.1'
VERSION='2.11.2'