Get ready for release 3.3.3

Fixes #239

HISTORY.md

@@ -115,7 +115,7 @@ mechanisms and addressed problems and extensions by some other means.
 Specifically, in `uncompyle`, decompilation of python bytecode 2.5 &
 2.6 is done by transforming the byte code into a pseudo-2.7 Python
 bytecode and is based on code from Eloi Vanderbeken. A bit of this
-could have bene easily added by modifying grammar rules.
+could have been easily added by modifying grammar rules.
 
 This project, `uncompyle6`, abandons that approach for various
 reasons. Having a grammar per Python version is much cleaner and it

NEWS.md

@@ -1,3 +1,31 @@
+3.3.3 2019-05-19 Henry and Lewis
+================================
+
+As before, decomplation bugs fixed. The focus has primarily been on
+Python 3.7. But with this release, releases will be put on hold,as a
+better control-flow detection is worked on . Tis has been needed for a
+while, and is long overdue. It will probably also take a while to get
+done as good as what we have now.
+
+However this work will be done in a new project
+[decompyle3](https://github.com/rocky/python-decompile3).  In contrast
+to _uncompyle6_ the code wil be written assuming a modern Python 3,
+e.g. 3.7. It is originally intended to decompile Python version 3.7
+and greater.
+
+* A number of Python 3.7+ chained comparisons were fixed
+* Revise Python 3.6ish format string handling
+* Go over operator precedence, e.g. for AST IfExp
+
+Reported Bug Fixes
+------------------
+
+* [#239: 3.7 handling of 4-level attribute import](https://github.com/rocky/python-uncompyle6/issues/239),
+* [#229: Inconsistent if block in python3.6](https://github.com/rocky/python-uncompyle6/issues/229),
+* [#227: Args not appearing in decompiled src when kwargs is specified explicitly (call_ex_kw)](https://github.com/rocky/python-uncompyle6/issues/227)
+2.7 confusion around "and" versus comprehension "if"
+* [#225: 2.7 confusion around "and" vs comprehension "if"](https://github.com/rocky/python-uncompyle6/issues/225)
+
 3.3.2 2019-05-03 Better Friday
 ==============================
 
@@ -12,7 +40,6 @@ get addressed in future releases
 
 Pypy 3.6 support was started. Pypy 3.x detection fixed (via xdis)
 
-
 3.3.1 2019-04-19 Good Friday
 ==========================
 
@@ -27,7 +54,7 @@ Lots of decomplation bugs, especially in the 3.x series fixed. Don't worry thoug
 * Fix some parser failures fixes in 3.4+ using test_pyenvlib
 * Add more run tests
 
-3.3.0 2019-43-14 Holy Week
+3.3.0 2019-04-14 Holy Week
 ==========================
 
 * First cut at Python 3.8 (many bugs remain)
@@ -42,11 +69,13 @@ Mostly more of the same: bug fixes and pull requests.
 Bug Fixes
 -----------
 
-* [#155: Python 3.x bytecode confusing "try/else" with "try" in a loop](https://github.com/rocky/python-uncompyle6/issues/155),
-* [#200: Python 3 bug in not detecting end bounds of an "if" ... "elif"](https://github.com/rocky/python-uncompyle6/issues/200),
-* [#208: Comma placement in 3.6 and 3.7 **kwargs](https://github.com/rocky/python-uncompyle6/issues/208),
-* [#209: Fix "if" return boundary in 3.6+](https://github.com/rocky/python-uncompyle6/issues/209),
+* [#221: Wrong grammar for nested ifelsestmt (in Python 3.7 at least)](https://github.com/rocky/python-uncompyle6/issues/221)
 * [#215: 2.7 can have two JUMP_BACKs at the end of a while loop](https://github.com/rocky/python-uncompyle6/issues/215)
+* [#209: Fix "if" return boundary in 3.6+](https://github.com/rocky/python-uncompyle6/issues/209),
+* [#208: Comma placement in 3.6 and 3.7 **kwargs](https://github.com/rocky/python-uncompyle6/issues/208),
+* [#200: Python 3 bug in not detecting end bounds of an "if" ... "elif"](https://github.com/rocky/python-uncompyle6/issues/200),
+* [#155: Python 3.x bytecode confusing "try/else" with "try" in a loop](https://github.com/rocky/python-uncompyle6/issues/155),
+
 
 Pull Requests
 ----------------

README.rst

@@ -152,7 +152,7 @@ for that bytecode version. Having done this the bytecode produced
 could be compared with the original bytecode. However as Python's code
 generation got better, this is no longer feasible.
 
-There verification that we use that doesn't check bytecode for
+The verification that we use that doesn't check bytecode for
 equivalence but does check to see if the resulting decompiled source
 is a valid Python program by running the Python interpreter. Because
 the Python language has changed so much, for best results you should
@@ -194,8 +194,12 @@ Between Python 3.5, 3.6 and 3.7 there have been major changes to the
 
 Currently not all Python magic numbers are supported. Specifically in
 some versions of Python, notably Python 3.6, the magic number has
-changes several times within a version. We support only the released
-magic. There are also customized Python interpreters, notably Dropbox,
+changes several times within a version. 
+
+**We support only released versions, not candidate versions.** Note however
+that the magic of a released version is usually the same as the *last* candidate version prior to release.
+
+There are also customized Python interpreters, notably Dropbox,
 which use their own magic and encrypt bytcode. With the exception of
 the Dropbox's old Python 2.5 interpreter this kind of thing is not
 handled.
@@ -226,7 +230,7 @@ See Also
 * https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
 
 
-.. _trepan: https://pypi.python.org/pypi/trepan2
+.. _trepan: https://pypi.python.org/pypi/trepan2g
 .. _compiler: https://pypi.python.org/pypi/spark_parser
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
 .. _debuggers: https://pypi.python.org/pypi/trepan3k

admin-tools/setup-master.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-PYTHON_VERSION=3.6.5
+PYTHON_VERSION=3.6.8
 
 # FIXME put some of the below in a common routine
 function finish {

test/Makefile

@@ -268,6 +268,7 @@ check-bytecode-3.6:
 
 #: Check deparsing Python 3.7
 check-bytecode-3.7:
+	$(PYTHON) test_pythonlib.py --bytecode-3.7-run --verify-run
 	$(PYTHON) test_pythonlib.py --bytecode-3.7 --weak-verify
 
 #: Check deparsing Python 3.8

test/simple_source/branching/02_ifelse_lambda.py

@@ -22,7 +22,7 @@ assert i[0]('a') == True
 assert i[0]('A') == False
 
 # Issue #170. Bug is needing an "conditional_not_lambda" grammar rule
-# in addition the the "conditional_lambda" rule
+# in addition the the "if_expr_lambda" rule
 j = lambda a: False if not a else True
 assert j(True) == True
 assert j(False) == False

test/simple_source/bug26/01_ifelse_listcomp.py

@@ -2,3 +2,10 @@
 # This is RUNNABLE!
 assert [False, True, True, True, True] == [False if not a else True for a in range(5)]
 assert [True, False, False, False, False] == [False if a else True for a in range(5)]
+
+# From bug #225
+m = ['hi', 'he', 'ih', 'who', 'ho']
+ms = {}
+for f in (f for f in m if f.startswith('h')):
+    ms[f] = 5
+assert ms == {'hi': 5, 'he': 5, 'ho': 5}

test/simple_source/bug26/03_weird26.py

@@ -8,7 +8,7 @@ list(x for x in range(10) if x % 2 if x % 3)
 
 # expresion which evaluates True unconditionally,
 # but leave dead code or junk around that we have to match on.
-# Tests "conditional_true" rule
+# Tests "if_expr_true" rule
 5 if 1 else 2
 
 0 or max(5, 3) if 0 else 3

test/simple_source/bug26/04_ifelse_parens.py

@@ -0,0 +1,9 @@
+# From 3.7.3 dataclasses.py
+# Bug was handling precedence. Need parenthesis before IfExp.
+#
+# RUNNABLE!
+def _hash_add(fields):
+    flds = [f for f in fields if (4 if f is None else f)]
+    return flds
+
+assert _hash_add([None, True, False, 3]) == [None, True, 3]

test/simple_source/bug27+/03_not_dead_code.py

@@ -1,6 +1,6 @@
 # Bug found in 2.7 test_itertools.py
-# Bug was erroneously using reduction to unconditional_true
-# A proper fix would be to use unconditional_true only when we
+# Bug was erroneously using reduction to if_expr_true
+# A proper fix would be to use if_expr_true only when we
 # can determine there is or was dead code.
 from itertools import izip_longest
 for args in [['abc', range(6)]]:

test/simple_source/bug36/01_fstring.py

@@ -4,8 +4,8 @@
 var1 = 'x'
 var2 = 'y'
 abc  = 'def'
-assert (f'interpolate {var1} strings {var2!r} {var2!s} py36' ==
-        "interpolate x strings 'y' y py36")
+assert (f"interpolate {var1} strings {var2!r} {var2!s} 'py36" ==
+        "interpolate x strings 'y' y 'py36")
 assert 'def0' == f'{abc}0'
 assert 'defdef' == f'{abc}{abc!s}'
 
@@ -38,4 +38,31 @@ filename = '.'
 source = 'foo'
 source = (f"__file__ = r'''{os.path.abspath(filename)}'''\n"
           + source + "\ndel __file__")
-print(source)
+
+# Note how { and } are *not* escaped here
+f = 'one'
+name = 'two'
+assert(f"{f}{'{{name}}'} {f}{'{name}'}") == 'one{{name}} one{name}'
+
+# From 3.7.3 dataclasses.py
+log_rounds  = 5
+assert "05$" == f'{log_rounds:02d}$'
+
+
+def testit(a, b, l):
+    # print(l)
+    return l
+
+# The call below shows the need for BUILD_STRING to count expr arguments.
+# Also note that we use {{ }} to escape braces in contrast to the example
+# above.
+def _repr_fn(fields):
+    return testit('__repr__',
+                  ('self',),
+                  ['return xx + f"(' +
+                   ', '.join([f"{f}={{self.{f}!r}}"
+                              for f in fields]) +
+                   ')"'])
+
+fields = ['a', 'b', 'c']
+assert _repr_fn(fields) == ['return xx + f"(a={self.a!r}, b={self.b!r}, c={self.c!r})"']

test/simple_source/bug36/02_call_ex_kw.py

@@ -0,0 +1,47 @@
+# From #227
+# Bug was not handling call_ex_kw correctly
+# This appears in
+#   showparams(c, test="A", **extra_args)
+# below
+
+def showparams(c, test, **extra_args):
+    return {'c': c, **extra_args, 'test': test}
+
+def f(c, **extra_args):
+    return showparams(c, test="A", **extra_args)
+
+def f1(c, d, **extra_args):
+    return showparams(c, test="B", **extra_args)
+
+def f2(**extra_args):
+    return showparams(1, test="C", **extra_args)
+
+def f3(c, *args, **extra_args):
+    return showparams(c, *args, **extra_args)
+
+assert f(1, a=2, b=3) == {'c': 1, 'a': 2, 'b': 3, 'test': 'A'}
+
+a = {'param1': 2}
+assert f1('2', '{\'test\': "4"}', test2='a', **a) \
+    == {'c': '2', 'test2': 'a', 'param1': 2, 'test': 'B'}
+assert f1(2, '"3"', test2='a', **a) \
+    == {'c': 2, 'test2': 'a', 'param1': 2, 'test': 'B'}
+assert f1(False, '"3"', test2='a', **a) \
+    == {'c': False, 'test2': 'a', 'param1': 2, 'test': 'B'}
+assert f(2, test2='A', **a) \
+    == {'c': 2, 'test2': 'A', 'param1': 2, 'test': 'A'}
+assert f(str(2) + str(1), test2='a', **a) \
+    == {'c': '21', 'test2': 'a', 'param1': 2, 'test': 'A'}
+assert f1((a.get('a'), a.get('b')), a, test3='A', **a) \
+    == {'c': (None, None), 'test3': 'A', 'param1': 2, 'test': 'B'}
+
+b = {'b1': 1, 'b2': 2}
+assert f2(**a, **b) == \
+    {'c': 1, 'param1': 2, 'b1': 1, 'b2': 2, 'test': 'C'}
+
+c = (2,)
+d = (2, 3)
+assert f(2, **a) == {'c': 2, 'param1': 2, 'test': 'A'}
+assert f3(2, *c, **a) == {'c': 2, 'param1': 2, 'test': 2}
+assert f3(*d, **a) == {'c': 2, 'param1': 2, 'test': 3}
+

test/simple_source/bug37/01_and_not_else.py

@@ -0,0 +1,16 @@
+# From 3.7.3 base64.py
+# Bug was handling "and not" in an
+# if/else in the presence of better Python bytecode generatation
+
+# RUNNABLE!
+def foo(foldnuls, word):
+    x = 5 if foldnuls and not word else 6
+    return x
+
+for expect, foldnuls, word in (
+        (6, True, True),
+        (5, True, False),
+        (6, False, True),
+        (6, False, False)
+        ):
+   assert foo(foldnuls, word) == expect

test/simple_source/bug37/01_chained_compare.py

@@ -11,9 +11,16 @@ def chained_compare_b(a, obj):
         if -0x80000000 <= obj <= 0x7fffffff:
             return 5
 
+def chained_compare_c(a, d):
+    for i in len(d):
+        if a == d[i] != 2:
+            return 5
+
 chained_compare_a(3)
 try:
     chained_compare_a(8)
 except ValueError:
     pass
 chained_compare_b(True, 0x0)
+
+chained_compare_c(3, [3])

test/simple_source/stmts/00_import.py

@@ -7,3 +7,4 @@ import http.client as httpclient
 if len(__file__) == 0:
     # a.b.c should force consecutive LOAD_ATTRs
     import a.b.c as d
+    import stuff0.stuff1.stuff2.stuff3 as stuff3

uncompyle6/parser.py

@@ -61,7 +61,6 @@ class PythonParser(GenericASTBuilder):
             'imports_cont',
             'kvlist_n',
             # Python 3.6+
-            'joined_str',
             'come_from_loops',
             ]
         self.collect = frozenset(nt_list)
@@ -83,7 +82,7 @@ class PythonParser(GenericASTBuilder):
         # FIXME: would love to do expr, sstmts, stmts and
         # so on but that would require major changes to the
         # semantic actions
-        self.singleton = frozenset(('str', 'joined_str', 'store', '_stmts', 'suite_stmts_opt',
+        self.singleton = frozenset(('str', 'store', '_stmts', 'suite_stmts_opt',
                                     'inplace_op'))
         # Instructions filled in from scanner
         self.insts = []

uncompyle6/parsers/parse2.py

@@ -98,9 +98,9 @@ class Python2Parser(PythonParser):
         for         ::= SETUP_LOOP expr for_iter store
                         for_block POP_BLOCK _come_froms
 
-        del_stmt ::= delete_subscr
-        delete_subscr ::= expr expr DELETE_SUBSCR
-        del_stmt ::= expr DELETE_ATTR
+        del_stmt         ::= delete_subscript
+        delete_subscript ::= expr expr DELETE_SUBSCR
+        del_stmt         ::= expr DELETE_ATTR
 
         _mklambda ::= load_closure mklambda
         kwarg     ::= LOAD_CONST expr
@@ -390,10 +390,10 @@ class Python2Parser(PythonParser):
                 continue
             elif opname == 'DELETE_SUBSCR':
                 self.addRule("""
-                    del_stmt ::= delete_subscr
-                    delete_subscr ::= expr expr DELETE_SUBSCR
+                    del_stmt ::= delete_subscript
+                    delete_subscript ::= expr expr DELETE_SUBSCR
                    """, nop_func)
-                self.check_reduce['delete_subscr'] = 'AST'
+                self.check_reduce['delete_subscript'] = 'AST'
                 custom_seen_ops.add(opname)
                 continue
             elif opname == 'GET_ITER':
@@ -549,7 +549,7 @@ class Python2Parser(PythonParser):
         elif rule == ('or', ('expr', 'jmp_true', 'expr', '\\e_come_from_opt')):
             expr2 = ast[2]
             return expr2 == 'expr' and expr2[0] == 'LOAD_ASSERT'
-        elif lhs in ('delete_subscr', 'del_expr'):
+        elif lhs in ('delete_subscript', 'del_expr'):
             op = ast[0][0]
             return op.kind in ('and', 'or')
 

uncompyle6/parsers/parse25.py

@@ -74,9 +74,9 @@ class Python25Parser(Python26Parser):
         return_stmt_lambda ::= ret_expr RETURN_VALUE_LAMBDA
         setupwithas      ::= DUP_TOP LOAD_ATTR ROT_TWO LOAD_ATTR CALL_FUNCTION_0 setup_finally
         stmt             ::= classdefdeco
-        stmt             ::= conditional_lambda
+        stmt             ::= if_expr_lambda
         stmt             ::= conditional_not_lambda
-        conditional_lambda ::= expr jmp_false_then expr return_if_lambda
+        if_expr_lambda   ::= expr jmp_false_then expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
         conditional_not_lambda
                            ::= expr jmp_true_then expr return_if_lambda

uncompyle6/parsers/parse26.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2017-2018 Rocky Bernstein
+#  Copyright (c) 2017-2019 Rocky Bernstein
 """
 spark grammar differences over Python2 for Python 2.6.
 """
@@ -293,19 +293,19 @@ class Python26Parser(Python2Parser):
         compare_chained2   ::= expr COMPARE_OP return_lambda
 
         return_if_lambda   ::= RETURN_END_IF_LAMBDA POP_TOP
-        stmt               ::= conditional_lambda
+        stmt               ::= if_expr_lambda
         stmt               ::= conditional_not_lambda
-        conditional_lambda ::= expr jmp_false_then expr return_if_lambda
+        if_expr_lambda     ::= expr jmp_false_then expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
         conditional_not_lambda ::=
                                expr jmp_true_then expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
 
-        # conditional_true are for conditions which always evaluate true
+        # if_expr_true are for conditions which always evaluate true
         # There is dead or non-optional remnants of the condition code though,
         # and we use that to match on to reconstruct the source more accurately
-        expr               ::= conditional_true
-        conditional_true   ::= expr jf_pop expr COME_FROM
+        expr               ::= if_expr_true
+        if_expr_true       ::= expr jf_pop expr COME_FROM
 
         # This comes from
         #   0 or max(5, 3) if 0 else 3

uncompyle6/parsers/parse27.py

@@ -112,14 +112,14 @@ class Python27Parser(Python2Parser):
         compare_chained2 ::= expr COMPARE_OP return_lambda
         compare_chained2 ::= expr COMPARE_OP return_lambda
 
-        # conditional_true are for conditions which always evaluate true
+        # if_expr_true are for conditions which always evaluate true
         # There is dead or non-optional remnants of the condition code though,
         # and we use that to match on to reconstruct the source more accurately.
         # FIXME: we should do analysis and reduce *only* if there is dead code?
         #        right now we check that expr is "or". Any other nodes types?
 
-        expr             ::= conditional_true
-        conditional_true ::= expr JUMP_FORWARD expr COME_FROM
+        expr             ::= if_expr_true
+        if_expr_true     ::= expr JUMP_FORWARD expr COME_FROM
 
         conditional      ::= expr jmp_false expr JUMP_FORWARD expr COME_FROM
         conditional      ::= expr jmp_false expr JUMP_ABSOLUTE expr
@@ -181,9 +181,9 @@ class Python27Parser(Python2Parser):
 
         # Common with 2.6
         return_if_lambda   ::= RETURN_END_IF_LAMBDA COME_FROM
-        stmt ::= conditional_lambda
-        stmt ::= conditional_not_lambda
-        conditional_lambda ::= expr jmp_false expr return_if_lambda
+        stmt               ::= if_expr_lambda
+        stmt               ::= conditional_not_lambda
+        if_expr_lambda     ::= expr jmp_false expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
         conditional_not_lambda
                            ::= expr jmp_true expr return_if_lambda
@@ -216,7 +216,7 @@ class Python27Parser(Python2Parser):
         self.check_reduce['raise_stmt1'] = 'AST'
         self.check_reduce['list_if_not'] = 'AST'
         self.check_reduce['list_if'] = 'AST'
-        self.check_reduce['conditional_true'] = 'AST'
+        self.check_reduce['if_expr_true'] = 'tokens'
         self.check_reduce['whilestmt'] = 'tokens'
         return
 
@@ -229,6 +229,12 @@ class Python27Parser(Python2Parser):
             return invalid
 
         if rule == ('and', ('expr', 'jmp_false', 'expr', '\\e_come_from_opt')):
+            # If the instruction after the instructions formin "and"  is an "YIELD_VALUE"
+            # then this is probably an "if" inside a comprehension.
+            if tokens[last] == 'YIELD_VALUE':
+                # Note: We might also consider testing last+1 being "POP_TOP"
+                return True
+
             # Test that jmp_false jumps to the end of "and"
             # or that it jumps to the same place as the end of "and"
             jmp_false = ast[1][0]
@@ -268,11 +274,8 @@ class Python27Parser(Python2Parser):
             while (tokens[i] != 'JUMP_BACK'):
                 i -= 1
             return tokens[i].attr != tokens[i-1].attr
-        # elif rule[0] == ('conditional_true'):
-        #     # FIXME: the below is a hack: we check expr for
-        #     # nodes that could have possibly been a been a Boolean.
-        #     # We should also look for the presence of dead code.
-        #     return ast[0] == 'expr' and ast[0] == 'or'
+        elif rule[0] == 'if_expr_true':
+            return (first) > 0 and tokens[first-1] == 'POP_JUMP_IF_FALSE'
 
         return False
 

uncompyle6/parsers/parse3.py

@@ -324,9 +324,9 @@ class Python3Parser(PythonParser):
 
     def p_stmt3(self, args):
         """
-        stmt               ::= conditional_lambda
+        stmt               ::= if_expr_lambda
         stmt               ::= conditional_not_lambda
-        conditional_lambda ::= expr jmp_false expr return_if_lambda
+        if_expr_lambda     ::= expr jmp_false expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
         conditional_not_lambda
                            ::= expr jmp_true expr return_if_lambda
@@ -406,11 +406,11 @@ class Python3Parser(PythonParser):
         # a JUMP_ABSOLUTE with no COME_FROM
         conditional    ::= expr jmp_false expr jump_absolute_else expr
 
-        # conditional_true are for conditions which always evaluate true
+        # if_expr_true are for conditions which always evaluate true
         # There is dead or non-optional remnants of the condition code though,
         # and we use that to match on to reconstruct the source more accurately
-        expr             ::= conditional_true
-        conditional_true ::= expr JUMP_FORWARD expr COME_FROM
+        expr           ::= if_expr_true
+        if_expr_true   ::= expr JUMP_FORWARD expr COME_FROM
         """
 
     @staticmethod
@@ -584,9 +584,9 @@ class Python3Parser(PythonParser):
               stmt ::= assign2_pypy
               assign3_pypy       ::= expr expr expr store store store
               assign2_pypy       ::= expr expr store store
-              stmt               ::= conditional_lambda
+              stmt               ::= if_expr_lambda
               stmt               ::= conditional_not_lambda
-              conditional_lambda ::= expr jmp_false expr return_if_lambda
+              if_expr_lambda     ::= expr jmp_false expr return_if_lambda
                                      return_lambda LAMBDA_MARKER
               conditional_not_lambda
                                  ::= expr jmp_true expr return_if_lambda
@@ -783,8 +783,8 @@ class Python3Parser(PythonParser):
                 custom_ops_processed.add(opname)
             elif opname == 'DELETE_SUBSCR':
                 self.addRule("""
-                    del_stmt ::= delete_subscr
-                    delete_subscr ::= expr expr DELETE_SUBSCR
+                    del_stmt ::= delete_subscript
+                    delete_subscript ::= expr expr DELETE_SUBSCR
                    """, nop_func)
                 custom_ops_processed.add(opname)
             elif opname == 'GET_ITER':

uncompyle6/parsers/parse36.py

@@ -188,21 +188,14 @@ class Python36Parser(Python35Parser):
                     self.add_unique_doc_rules(rules_str, customize)
             elif opname == 'FORMAT_VALUE':
                 rules_str = """
-                    expr            ::= fstring_single
-                    fstring_single  ::= expr FORMAT_VALUE
-                    expr            ::= fstring_expr
-                    fstring_expr    ::= expr FORMAT_VALUE
-
-                    str             ::= LOAD_CONST
-                    formatted_value ::= fstring_expr
-                    formatted_value ::= str
-
+                    expr              ::= formatted_value1
+                    formatted_value1  ::= expr FORMAT_VALUE
                 """
                 self.add_unique_doc_rules(rules_str, customize)
             elif opname == 'FORMAT_VALUE_ATTR':
                 rules_str = """
-                expr            ::= fstring_single
-                fstring_single  ::= expr expr FORMAT_VALUE_ATTR
+                expr              ::= formatted_value2
+                formatted_value2  ::= expr expr FORMAT_VALUE_ATTR
                 """
                 self.add_unique_doc_rules(rules_str, customize)
             elif opname == 'MAKE_FUNCTION_8':
@@ -246,16 +239,12 @@ class Python36Parser(Python35Parser):
                 """
                 self.addRule(rules_str, nop_func)
 
-            elif opname == 'BUILD_STRING':
+            elif opname.startswith('BUILD_STRING'):
                 v = token.attr
-                joined_str_n = "formatted_value_%s" % v
                 rules_str = """
-                    expr                 ::= fstring_multi
-                    fstring_multi        ::= joined_str BUILD_STRING
-                    joined_str           ::= formatted_value+
-                    fstring_multi        ::= %s BUILD_STRING
-                    %s                   ::= %sBUILD_STRING
-                """ % (joined_str_n, joined_str_n, "formatted_value " * v)
+                    expr                 ::= joined_str
+                    joined_str           ::= %sBUILD_STRING_%d
+                """ % ("expr " * v, v)
                 self.add_unique_doc_rules(rules_str, customize)
                 if 'FORMAT_VALUE_ATTR' in self.seen_ops:
                     rules_str = """

uncompyle6/parsers/parse37.py

@@ -73,8 +73,8 @@ class Python37Parser(Python36Parser):
                                POP_TOP POP_TOP POP_TOP POP_EXCEPT POP_TOP POP_BLOCK
                                else_suite COME_FROM_LOOP
 
-        # Is there a pattern here?
         attributes ::= IMPORT_FROM ROT_TWO POP_TOP IMPORT_FROM
+        attributes ::= attributes ROT_TWO POP_TOP IMPORT_FROM
 
         attribute37   ::= expr LOAD_METHOD
         expr          ::= attribute37
@@ -88,26 +88,50 @@ class Python37Parser(Python36Parser):
 
         compare_chained37   ::= expr compare_chained1a_37
         compare_chained37   ::= expr compare_chained1b_37
+        compare_chained37   ::= expr compare_chained1c_37
+
         compare_chained37_false  ::= expr compare_chained1_false_37
+        compare_chained37_false  ::= expr compare_chained2_false_37
 
         compare_chained1a_37      ::= expr DUP_TOP ROT_THREE COMPARE_OP POP_JUMP_IF_FALSE
         compare_chained1a_37      ::= expr DUP_TOP ROT_THREE COMPARE_OP POP_JUMP_IF_FALSE
                                       compare_chained2a_37 ELSE POP_TOP COME_FROM
         compare_chained1b_37      ::= expr DUP_TOP ROT_THREE COMPARE_OP POP_JUMP_IF_FALSE
                                       compare_chained2b_37 POP_TOP JUMP_FORWARD COME_FROM
+        compare_chained1c_37      ::= expr DUP_TOP ROT_THREE COMPARE_OP POP_JUMP_IF_FALSE
+                                      compare_chained2a_37 POP_TOP
 
         compare_chained1_false_37 ::= expr DUP_TOP ROT_THREE COMPARE_OP POP_JUMP_IF_FALSE
                                       compare_chained2c_37 POP_TOP JUMP_FORWARD COME_FROM
+        compare_chained2_false_37 ::= expr DUP_TOP ROT_THREE COMPARE_OP POP_JUMP_IF_FALSE
+                                      compare_chained2a_false_37 ELSE POP_TOP JUMP_BACK COME_FROM
 
         compare_chained2a_37       ::= expr COMPARE_OP POP_JUMP_IF_TRUE JUMP_FORWARD
-        compare_chained2a_false_37 ::= expr COMPARE_OP POP_JUMP_IF_FALSE JUMP_FORWARD
+        compare_chained2a_37       ::= expr COMPARE_OP POP_JUMP_IF_TRUE JUMP_BACK
+        compare_chained2a_false_37 ::= expr COMPARE_OP POP_JUMP_IF_FALSE jf_cfs
 
         compare_chained2b_37       ::= expr COMPARE_OP come_from_opt POP_JUMP_IF_FALSE JUMP_FORWARD ELSE
+        compare_chained2b_37       ::= expr COMPARE_OP come_from_opt POP_JUMP_IF_FALSE JUMP_FORWARD
 
         compare_chained2c_37       ::= expr DUP_TOP ROT_THREE COMPARE_OP come_from_opt POP_JUMP_IF_FALSE
                                        compare_chained2a_false_37 ELSE
+        compare_chained2c_37       ::= expr DUP_TOP ROT_THREE COMPARE_OP come_from_opt POP_JUMP_IF_FALSE
+                                       compare_chained2a_false_37
 
-        _ifstmts_jump        ::= c_stmts_opt come_froms
+        jf_cfs                     ::= JUMP_FORWARD _come_froms
+        ifelsestmt                 ::= testexpr c_stmts_opt jf_cfs else_suite opt_come_from_except
+
+        jmp_false37                ::= POP_JUMP_IF_FALSE COME_FROM
+        list_if                    ::= expr jmp_false37 list_iter
+
+        _ifstmts_jump              ::= c_stmts_opt come_froms
+
+        and_not                    ::= expr jmp_false expr POP_JUMP_IF_TRUE
+
+        expr                       ::= if_exp_37a
+        expr                       ::= if_exp_37b
+        if_exp_37a                 ::= and_not expr JUMP_FORWARD COME_FROM expr COME_FROM
+        if_exp_37b                 ::= expr jmp_false expr POP_JUMP_IF_FALSE jump_forward_else expr
         """
 
     def customize_grammar_rules(self, tokens, customize):

uncompyle6/scanners/scanner3.py

@@ -811,7 +811,14 @@ class Scanner3(Scanner):
                             self.fixed_jumps[offset] = fix or match[-1]
                             return
                     else:
-                        self.fixed_jumps[offset] = match[-1]
+                        if self.version < 3.6:
+                            # FIXME: this is putting in COME_FROMs in the wrong place.
+                            # Fix up grammar so we don't need to do this.
+                            # See cf_for_iter use in parser36.py
+                            self.fixed_jumps[offset] = match[-1]
+                        elif target > offset:
+                            # Right now we only add COME_FROMs in forward (not loop) jumps
+                            self.fixed_jumps[offset] = target
                         return
             # op == POP_JUMP_IF_TRUE
             else:
@@ -916,7 +923,7 @@ class Scanner3(Scanner):
                     # Python 3.5 may remove as dead code a JUMP
                     # instruction after a RETURN_VALUE. So we check
                     # based on seeing SETUP_EXCEPT various places.
-                    if self.version < 3.8 and code[rtarget] == self.opc.SETUP_EXCEPT:
+                    if self.version < 3.6 and code[rtarget] == self.opc.SETUP_EXCEPT:
                         return
                     # Check that next instruction after pops and jump is
                     # not from SETUP_EXCEPT

uncompyle6/scanners/scanner36.py

@@ -33,6 +33,8 @@ class Scanner36(Scanner3):
                  t.op == self.opc.CALL_FUNCTION_EX and t.attr & 1):
                 t.kind = 'CALL_FUNCTION_EX_KW'
                 pass
+            elif t.op == self.opc.BUILD_STRING:
+                t.kind = 'BUILD_STRING_%s' % t.attr
             elif t.op == self.opc.CALL_FUNCTION_KW:
                 t.kind = 'CALL_FUNCTION_KW_%s' % t.attr
             elif t.op == self.opc.FORMAT_VALUE:

uncompyle6/semantics/consts.py

@@ -27,75 +27,85 @@ else:
     maxint = sys.maxint
 
 
-# Operator precidence
-# See https://docs.python.org/2/reference/expressions.html
-# or https://docs.python.org/3/reference/expressions.html
-# for a list.
+# Operator precidence See
+# https://docs.python.org/2/reference/expressions.html#operator-precedence
+# or
+# https://docs.python.org/3/reference/expressions.html#operator-precedence
+# for a list. We keep the same top-to-botom order here as in the above links,
+# so we start with low precedence (high values) and go down in value.
 
-# Things at the top of this list below with low-value precidence will
-# tend to have parenthesis around them. Things at the bottom
+# Things at the bottom of this list below with high precedence (low value) will
+# tend to have parenthesis around them. Things at the top
 # of the list will tend not to have parenthesis around them.
-PRECEDENCE = {
-    'list':                   0,
-    'dict':                   0,
-    'unary_convert':          0,
-    'dict_comp':              0,
-    'set_comp':               0,
-    'set_comp_expr':          0,
-    'list_comp':              0,
-    'generator_exp':          0,
 
-    'attribute':              2,
-    'subscript':              2,
-    'subscript2':             2,
-    'store_subscript':        2,
-    'delete_subscr':          2,
+# Note: The values in this table are even numbers. Inside
+# various templates we use odd values. Avoiding equal-precedent comparisons
+# avoids ambiguity what to do when the precedence is equal.
+
+
+PRECEDENCE = {
+    'yield':                 102,
+    'yield_from':            102,
+
+    '_mklambda':              30,
+
+    'conditional':            28, # Conditional expression
+    'conditional_lamdba':     28, # Lambda expression
+    'conditional_not_lamdba': 28, # Lambda expression
+    'conditionalnot':         28,
+    'if_expr_true':           28,
+    'ret_cond':               28,
+
+    'or':                     26, # Boolean OR
+    'ret_or':                 26,
+
+    'and':                    24, # Boolean AND
+    'compare':                20, # in, not in, is, is not, <, <=, >, >=, !=, ==
+    'ret_and':                24,
+    'unary_not':              22, # Boolean NOT
+
+    'BINARY_AND':             14, # Bitwise AND
+    'BINARY_OR':              18, # Bitwise OR
+    'BINARY_XOR':             16, # Bitwise XOR
+
+    'BINARY_LSHIFT':          12, # Shifts <<
+    'BINARY_RSHIFT':          12, # Shifts >>
+
+    'BINARY_ADD':             10, # -
+    'BINARY_SUBTRACT':        10, # +
+
+    'BINARY_DIVIDE':          8,  # /
+    'BINARY_FLOOR_DIVIDE':    8,  # //
+    'BINARY_MATRIX_MULTIPLY': 8,  # @
+    'BINARY_MODULO':          8,  # Remainder, %
+    'BINARY_MULTIPLY':        8,  # *
+    'BINARY_TRUE_DIVIDE':     8,  # Division /
+
+    'unary_expr':             6,  # +x, -x, ~x
+
+    'BINARY_POWER':           4,  # Exponentiation, *
+
+    'attribute':              2,  # x.attribute
+    'buildslice2':            2,  # x[index]
+    'buildslice3':            2,  # x[index:index]
+    'call':                   2,  # x(arguments...)
+    'delete_subscript':       2,
     'slice0':                 2,
     'slice1':                 2,
     'slice2':                 2,
     'slice3':                 2,
-    'buildslice2':            2,
-    'buildslice3':            2,
-    'call':                   2,
+    'store_subscript':        2,
+    'subscript':              2,
+    'subscript2':             2,
 
-    'BINARY_POWER':           4,
-
-    'unary_expr':             6,
-
-    'BINARY_MULTIPLY':        8,
-    'BINARY_DIVIDE':          8,
-    'BINARY_TRUE_DIVIDE':     8,
-    'BINARY_FLOOR_DIVIDE':    8,
-    'BINARY_MODULO':          8,
-
-    'BINARY_ADD':             10,
-    'BINARY_SUBTRACT':        10,
-
-    'BINARY_LSHIFT':          12,
-    'BINARY_RSHIFT':          12,
-
-    'BINARY_AND':             14,
-    'BINARY_XOR':             16,
-    'BINARY_OR':              18,
-
-    'compare':                20,
-    'unary_not':              22,
-    'and':                    24,
-    'ret_and':                24,
-
-    'or':                     26,
-    'ret_or':                 26,
-
-    'conditional':            28,
-    'conditional_lamdba':     28,
-    'conditional_not_lamdba': 28,
-    'conditionalnot':         28,
-    'ret_cond':               28,
-
-    '_mklambda':              30,
-
-    'yield':                 101,
-    'yield_from':            101
+    'dict':                   0,  # {expressions...}
+    'dict_comp':              0,
+    'generator_exp':          0,  # (expressions...)
+    'list':                   0,  # [expressions...]
+    'list_comp':              0,
+    'set_comp':               0,
+    'set_comp_expr':          0,
+    'unary_convert':          0,
 }
 
 LINE_LENGTH = 80
@@ -210,7 +220,7 @@ TABLE_DIRECT = {
 
     'IMPORT_FROM':              ( '%{pattr}', ),
     'attribute':	        ( '%c.%[1]{pattr}',
-                                (0, 'expr')),
+                                  (0, 'expr')),
     'LOAD_FAST':	            ( '%{pattr}', ),
     'LOAD_NAME':	            ( '%{pattr}', ),
     'LOAD_CLASSNAME':	        ( '%{pattr}', ),
@@ -221,7 +231,7 @@ TABLE_DIRECT = {
     'DELETE_FAST':	        ( '%|del %{pattr}\n', ),
     'DELETE_NAME':	        ( '%|del %{pattr}\n', ),
     'DELETE_GLOBAL':	        ( '%|del %{pattr}\n', ),
-    'delete_subscr':            ( '%|del %p[%c]\n',
+    'delete_subscript':         ( '%|del %p[%c]\n',
                                   (0, 'expr', PRECEDENCE['subscript']), (1, 'expr') ),
     'subscript':                ( '%p[%c]',
                                       (0, 'expr', PRECEDENCE['subscript']),
@@ -252,10 +262,11 @@ TABLE_DIRECT = {
 
     'list_iter':	    ( '%c', 0 ),
     'list_for':		    ( ' for %c in %c%c', 2, 0, 3 ),
-    'list_if':		    ( ' if %c%c', 0, 2 ),
-    'list_if_not':		( ' if not %p%c',
-                                  (0, 'expr', PRECEDENCE['unary_not']),
-                                  2 ),
+    'list_if':		    ( ' if %p%c',
+                              (0, 'expr', 27), 2 ),
+    'list_if_not':	    ( ' if not %p%c',
+                              (0, 'expr', PRECEDENCE['unary_not']),
+                              2 ),
     'lc_body':		    ( '', ),	# ignore when recursing
 
     'comp_iter':	    ( '%c', 0 ),
@@ -281,19 +292,19 @@ TABLE_DIRECT = {
     'and2':          	( '%c', 3 ),
     'or':           	( '%c or %c', 0, 2 ),
     'ret_or':           ( '%c or %c', 0, 2 ),
-    'conditional':      ( '%p if %p else %p', (2, 27), (0, 27), (4, 27) ),
-    'conditional_true': ( '%p if 1 else %p', (0, 27), (2, 27) ),
+    'conditional':      ( '%p if %c else %c',
+                          (2, 'expr', 27), 0, 4 ),
+    'if_expr_lambda':   ( '%p if %c else %c',
+                          (2, 'expr', 27), (0, 'expr'), 4 ),
+    'if_expr_true':     ( '%p if 1 else %c', (0, 'expr', 27), 2 ),
     'ret_cond':         ( '%p if %p else %p', (2, 27), (0, 27), (-1, 27) ),
     'conditional_not':  ( '%p if not %p else %p',
                           (2, 27),
                           (0, "expr", PRECEDENCE['unary_not']),
                           (4, 27) ),
-    'conditional_lambda':
-                        ( '%c if %c else %c',
-                          (2, 'expr'), 0, 4 ),
     'conditional_not_lambda':
-                        ( '%c if not %c else %c',
-                          (2, 'expr'), 0, 4 ),
+                        ( '%p if not %c else %c',
+                          (2, 'expr', 27), 0, 4 ),
 
     'compare_single':	    ( '%p %[-1]{pattr.replace("-", " ")} %p', (0, 19), (1, 19) ),
     'compare_chained':	    ( '%p %p', (0, 29), (1, 30)),

uncompyle6/semantics/customize35.py

@@ -119,6 +119,12 @@ def customize_for_version35(self, version):
     def n_function_def(node):
         if self.version >= 3.6:
             code_node = node[0][0]
+            for n in node[0]:
+                if hasattr(n, 'attr') and iscode(n.attr):
+                    code_node = n
+                    break
+                pass
+            pass
         else:
             code_node = node[0][1]
 

uncompyle6/semantics/customize36.py

@@ -17,7 +17,9 @@
 
 from spark_parser.ast import GenericASTTraversalPruningException
 from uncompyle6.scanners.tok import Token
-from uncompyle6.semantics.helper import flatten_list
+from uncompyle6.semantics.helper import (
+    flatten_list, escape_string, strip_quotes
+    )
 from uncompyle6.semantics.consts import (
     INDENT_PER_LEVEL, PRECEDENCE, TABLE_DIRECT, TABLE_R)
 
@@ -31,28 +33,19 @@ def escape_format(s):
 #######################
 
 def customize_for_version36(self, version):
-    # Value 100 is important; it is exactly
-    # module/function precidence.
-    PRECEDENCE['call_kw']     = 100
-    PRECEDENCE['call_kw36']   = 100
-    PRECEDENCE['call_ex']     = 100
-    PRECEDENCE['call_ex_kw']  = 100
-    PRECEDENCE['call_ex_kw2'] = 100
-    PRECEDENCE['call_ex_kw3'] = 100
-    PRECEDENCE['call_ex_kw4'] = 100
+    PRECEDENCE['call_kw']     = 0
+    PRECEDENCE['call_kw36']   = 1
+    PRECEDENCE['call_ex']     = 1
+    PRECEDENCE['call_ex_kw']  = 1
+    PRECEDENCE['call_ex_kw2'] = 1
+    PRECEDENCE['call_ex_kw3'] = 1
+    PRECEDENCE['call_ex_kw4'] = 1
     PRECEDENCE['unmap_dict']  = 0
+    PRECEDENCE['formatted_value1'] = 100
 
     TABLE_DIRECT.update({
         'tryfinally36':     ( '%|try:\n%+%c%-%|finally:\n%+%c%-\n\n',
                               (1, 'returns'), 3 ),
-        'fstring_expr':     ( "{%c%{conversion}}",
-                              (0, 'expr') ),
-        # FIXME: the below assumes the format strings
-        # don't have ''' in them. Fix this properly
-        'fstring_single':   ( "f'''{%c%{conversion}}'''", 0),
-        'formatted_value_attr': ( "f'''{%c%{conversion}}%{string}'''",
-                                  (0, 'expr')),
-        'fstring_multi':    ( "f'''%c'''", 0),
         'func_args36':      ( "%c(**", 0),
         'try_except36':     ( '%|try:\n%+%c%-%c\n\n', 1, -2 ),
         'except_return':    ( '%|except:\n%+%c%-', 3 ),
@@ -67,9 +60,6 @@ def customize_for_version36(self, version):
         'call_ex' : (
             '%c(%p)',
             (0, 'expr'), (1, 100)),
-        'call_ex_kw' : (
-            '%c(%p)',
-            (0, 'expr'), (2, 100)),
 
     })
 
@@ -80,20 +70,28 @@ def customize_for_version36(self, version):
         })
 
     def build_unpack_tuple_with_call(node):
-
-        if node[0] == 'expr':
-            tup = node[0][0]
+        n = node[0]
+        if n == 'expr':
+            n = n[0]
+        if n == 'tuple':
+            self.call36_tuple(n)
+            first = 1
+            sep = ', *'
+        elif n == 'LOAD_CONST':
+            value = self.format_pos_args(n)
+            self.f.write(value)
+            first = 1
+            sep = ', *'
         else:
-            tup = node[0]
-            pass
-        assert tup == 'tuple'
-        self.call36_tuple(tup)
+            first = 0
+            sep = '*'
 
         buwc = node[-1]
         assert buwc.kind.startswith('BUILD_TUPLE_UNPACK_WITH_CALL')
-        for n in node[1:-1]:
-            self.f.write(', *')
+        for n in node[first:-1]:
+            self.f.write(sep)
             self.preorder(n)
+            sep = ', *'
             pass
         self.prune()
         return
@@ -119,45 +117,41 @@ def customize_for_version36(self, version):
         return
     self.n_build_map_unpack_with_call = build_unpack_map_with_call
 
+    def call_ex_kw(node):
+        """Handle CALL_FUNCTION_EX 1 (have KW) but with
+        BUILD_MAP_UNPACK_WITH_CALL"""
+
+        expr = node[1]
+        assert expr == 'expr'
+
+        value = self.format_pos_args(expr)
+        if value == '':
+            fmt = "%c(%p)"
+        else:
+            fmt = "%%c(%s, %%p)" % value
+
+        self.template_engine(
+            (fmt,
+            (0, 'expr'), (2, 'build_map_unpack_with_call', 100)), node)
+
+        self.prune()
+    self.n_call_ex_kw = call_ex_kw
+
     def call_ex_kw2(node):
         """Handle CALL_FUNCTION_EX 2  (have KW) but with
         BUILD_{MAP,TUPLE}_UNPACK_WITH_CALL"""
 
-        # This is weird shit. Thanks Python!
-        self.preorder(node[0])
-        self.write('(')
-
         assert node[1] == 'build_tuple_unpack_with_call'
-        btuwc = node[1]
-        tup = btuwc[0]
-        if tup == 'expr':
-            tup = tup[0]
-
-        if tup == 'LOAD_CONST':
-            self.write(', '.join(['"%s"' % t.replace('"','\\"') for t in tup.attr]))
+        value = self.format_pos_args(node[1])
+        if value == '':
+            fmt = "%c(%p)"
         else:
-            assert tup == 'tuple'
-            self.call36_tuple(tup)
+            fmt = "%%c(%s, %%p)" % value
 
-        assert node[2] == 'build_map_unpack_with_call'
+        self.template_engine(
+            (fmt,
+            (0, 'expr'), (2, 'build_map_unpack_with_call', 100)), node)
 
-        self.write(', ')
-        d = node[2][0]
-        if d == 'expr':
-            d = d[0]
-        assert d == 'dict'
-        self.call36_dict(d)
-
-        args = btuwc[1]
-        self.write(', *')
-        self.preorder(args)
-
-        self.write(', **')
-        star_star_args = node[2][1]
-        if star_star_args == 'expr':
-            star_star_args = star_star_args[0]
-        self.preorder(star_star_args)
-        self.write(')')
         self.prune()
     self.n_call_ex_kw2 = call_ex_kw2
 
@@ -166,14 +160,13 @@ def customize_for_version36(self, version):
         BUILD_MAP_UNPACK_WITH_CALL"""
         self.preorder(node[0])
         self.write('(')
-        args = node[1][0]
-        if args == 'expr':
-            args = args[0]
-        if args == 'tuple':
-            if self.call36_tuple(args) > 0:
-                self.write(', ')
-                pass
+
+        value = self.format_pos_args(node[1][0])
+        if value == '':
             pass
+        else:
+            self.write(value)
+            self.write(', ')
 
         self.write('*')
         self.preorder(node[1][1])
@@ -226,6 +219,25 @@ def customize_for_version36(self, version):
         self.prune()
     self.n_call_ex_kw4 = call_ex_kw4
 
+    def format_pos_args(node):
+        """
+        Positional args should format to:
+        (*(2, ), ...) -> (2, ...)
+        We remove starting and trailing parenthesis and ', ' if
+        tuple has only one element.
+        """
+        value = self.traverse(node, indent='')
+        if value.startswith('('):
+            assert value.endswith(')')
+            value = value[1:-1].rstrip(" ") # Remove starting '(' and trailing ')' and additional spaces
+            if value == '':
+                pass # args is empty
+            else:
+                if value.endswith(','): # if args has only one item
+                    value = value[:-1]
+        return value
+    self.format_pos_args = format_pos_args
+
     def call36_tuple(node):
         """
         A tuple used in a call, these are like normal tuples but they
@@ -331,81 +343,8 @@ def customize_for_version36(self, version):
         return
     self.call36_dict = call36_dict
 
-
-    FSTRING_CONVERSION_MAP = {1: '!s', 2: '!r', 3: '!a', 'X':':X'}
-
-    def n_except_suite_finalize(node):
-        if node[1] == 'returns' and self.hide_internal:
-            # Process node[1] only.
-            # The code after "returns", e.g. node[3], is dead code.
-            # Adding it is wrong as it dedents and another
-            # exception handler "except_stmt" afterwards.
-            # Note it is also possible that the grammar is wrong here.
-            # and this should not be "except_stmt".
-            self.indent_more()
-            self.preorder(node[1])
-            self.indent_less()
-        else:
-            self.default(node)
-        self.prune()
-    self.n_except_suite_finalize = n_except_suite_finalize
-
-    def n_formatted_value(node):
-        if node[0] == 'LOAD_CONST':
-            value = node[0].attr
-            if isinstance(value, tuple):
-                self.write(node[0].attr)
-            else:
-                self.write(escape_format(node[0].attr))
-            self.prune()
-        else:
-            self.default(node)
-    self.n_formatted_value = n_formatted_value
-
-    def f_conversion(node):
-        fmt_node = node.data[1]
-        if fmt_node == 'expr' and fmt_node[0] == 'LOAD_CONST':
-            data = fmt_node[0].attr
-        else:
-            data = fmt_node.attr
-        node.conversion = FSTRING_CONVERSION_MAP.get(data, '')
-
-    def fstring_expr(node):
-        f_conversion(node)
-        self.default(node)
-    self.n_fstring_expr = fstring_expr
-
-    def fstring_single(node):
-        f_conversion(node)
-        self.default(node)
-    self.n_fstring_single = fstring_single
-
-    def formatted_value_attr(node):
-        f_conversion(node)
-        fmt_node = node.data[3]
-        if fmt_node == 'expr' and fmt_node[0] == 'LOAD_CONST':
-            node.string = escape_format(fmt_node[0].attr)
-        else:
-            node.string = fmt_node
-
-        self.default(node)
-    self.n_formatted_value_attr = formatted_value_attr
-
-    # def kwargs_only_36(node):
-    #     keys = node[-1].attr
-    #     num_kwargs = len(keys)
-    #     values = node[:num_kwargs]
-    #     for i, (key, value) in enumerate(zip(keys, values)):
-    #         self.write(key + '=')
-    #         self.preorder(value)
-    #         if i < num_kwargs:
-    #             self.write(',')
-    #     self.prune()
-    #     return
-    # self.n_kwargs_only_36 = kwargs_only_36
-
     def n_call_kw36(node):
-        self.template_engine(("%c(", 0), node)
+        self.template_engine(("%p(", (0, 100)), node)
         keys = node[-2].attr
         num_kwargs = len(keys)
         num_posargs = len(node) - (num_kwargs + 2)
@@ -442,6 +381,137 @@ def customize_for_version36(self, version):
         return
     self.n_call_kw36 = n_call_kw36
 
+
+    FSTRING_CONVERSION_MAP = {1: '!s', 2: '!r', 3: '!a', 'X':':X'}
+
+    def n_except_suite_finalize(node):
+        if node[1] == 'returns' and self.hide_internal:
+            # Process node[1] only.
+            # The code after "returns", e.g. node[3], is dead code.
+            # Adding it is wrong as it dedents and another
+            # exception handler "except_stmt" afterwards.
+            # Note it is also possible that the grammar is wrong here.
+            # and this should not be "except_stmt".
+            self.indent_more()
+            self.preorder(node[1])
+            self.indent_less()
+        else:
+            self.default(node)
+        self.prune()
+    self.n_except_suite_finalize = n_except_suite_finalize
+
+    def n_formatted_value(node):
+        if node[0] == 'LOAD_CONST':
+            value = node[0].attr
+            if isinstance(value, tuple):
+                self.write(node[0].attr)
+            else:
+                self.write(escape_string(node[0].attr))
+            self.prune()
+        else:
+            self.default(node)
+    self.n_formatted_value = n_formatted_value
+
+    def n_formatted_value_attr(node):
+        f_conversion(node)
+        fmt_node = node.data[3]
+        if fmt_node == 'expr' and fmt_node[0] == 'LOAD_CONST':
+            node.string = escape_format(fmt_node[0].attr)
+        else:
+            node.string = fmt_node
+        self.default(node)
+    self.n_formatted_value_attr = n_formatted_value_attr
+
+    def f_conversion(node):
+        fmt_node = node.data[1]
+        if fmt_node == 'expr' and fmt_node[0] == 'LOAD_CONST':
+            data = fmt_node[0].attr
+        else:
+            data = fmt_node.attr
+        node.conversion = FSTRING_CONVERSION_MAP.get(data, '')
+        return node.conversion
+
+    def n_formatted_value1(node):
+        expr = node[0]
+        assert expr == 'expr'
+        value = self.traverse(expr, indent='')
+        conversion = f_conversion(node)
+        f_str = "f%s" % escape_string("{%s%s}" % (value, conversion))
+        self.write(f_str)
+        self.prune()
+
+    self.n_formatted_value1 = n_formatted_value1
+
+    def n_formatted_value2(node):
+        p = self.prec
+        self.prec = 100
+
+        expr = node[0]
+        assert expr == 'expr'
+        value = self.traverse(expr, indent='')
+        format_value_attr = node[-1]
+        assert format_value_attr == 'FORMAT_VALUE_ATTR'
+        attr = format_value_attr.attr
+        if attr == 4:
+            assert node[1] == 'expr'
+            fmt = strip_quotes(self.traverse(node[1], indent=''))
+            conversion = ":%s" % fmt
+        else:
+            conversion = FSTRING_CONVERSION_MAP.get(attr, '')
+
+        f_str = "f%s" % escape_string("{%s%s}" % (value, conversion))
+        self.write(f_str)
+
+        self.prec = p
+        self.prune()
+    self.n_formatted_value2 = n_formatted_value2
+
+    def n_joined_str(node):
+        p = self.prec
+        self.prec = 100
+
+        result = ''
+        for expr in node[:-1]:
+            assert expr == 'expr'
+            value = self.traverse(expr, indent='')
+            if expr[0].kind.startswith('formatted_value'):
+                # remove leading 'f'
+                assert value.startswith('f')
+                value = value[1:]
+                pass
+            else:
+                # {{ and }} in Python source-code format strings mean
+                # { and } respectively. But only when *not* part of a
+                # formatted value. However in the LOAD_CONST
+                # bytecode, the escaping of the braces has been
+                # removed. So we need to put back the braces escaping in
+                # reconstructing the source.
+                assert expr[0] == 'LOAD_CONST'
+                value = value.replace("{", "{{").replace("}", "}}")
+
+            # Remove leading quotes
+            result += strip_quotes(value)
+            pass
+        self.write('f%s' % escape_string(result))
+
+        self.prec = p
+        self.prune()
+    self.n_joined_str = n_joined_str
+
+
+    # def kwargs_only_36(node):
+    #     keys = node[-1].attr
+    #     num_kwargs = len(keys)
+    #     values = node[:num_kwargs]
+    #     for i, (key, value) in enumerate(zip(keys, values)):
+    #         self.write(key + '=')
+    #         self.preorder(value)
+    #         if i < num_kwargs:
+    #             self.write(',')
+    #     self.prune()
+    #     return
+    # self.n_kwargs_only_36 = kwargs_only_36
+
     def starred(node):
         l = len(node)
         assert l > 0

uncompyle6/semantics/customize37.py

@@ -22,8 +22,13 @@ def customize_for_version37(self, version):
     # Python 3.7+ changes
     #######################
 
-    PRECEDENCE['attribute37'] = 2
+    PRECEDENCE['attribute37'] =  2
+    PRECEDENCE['if_exp_37a']   = 28
+    PRECEDENCE['if_exp_37b']   = 28
+
     TABLE_DIRECT.update({
+        'and_not':  ( '%c and not %c',
+                      (0, 'expr'), (2, 'expr') ),
         'async_forelse_stmt':  (
             '%|async for %c in %c:\n%+%c%-%|else:\n%+%c%-\n\n',
             (7, 'store'), (1, 'expr'), (17, 'for_block'), (25, 'else_suite') ),
@@ -40,9 +45,15 @@ def customize_for_version37(self, version):
         'compare_chained1_false_37': (
             ' %[3]{pattr.replace("-", " ")} %p %p',
             (0, 19), (-4, 19)),
+        'compare_chained2_false_37': (
+            ' %[3]{pattr.replace("-", " ")} %p %p',
+            (0, 19), (-5, 19)),
         'compare_chained1b_37': (
             ' %[3]{pattr.replace("-", " ")} %p %p',
             (0, 19), (-4, 19)),
+        'compare_chained1c_37': (
+            ' %[3]{pattr.replace("-", " ")} %p %p',
+            (0, 19), (-2, 19)),
         'compare_chained2a_37': (
             '%[1]{pattr.replace("-", " ")} %p',
             (0, 19) ),
@@ -54,5 +65,7 @@ def customize_for_version37(self, version):
             (0, 19 ) ),
         'compare_chained2c_37': (
             '%[3]{pattr.replace("-", " ")} %p %p', (0, 19), (6, 19) ),
+        'if_exp_37a': ( '%p if %p else %p', (1, 'expr', 27), (0, 27), (4, 'expr', 27) ),
+        'if_exp_37b': ( '%p if %p else %p', (2, 'expr', 27), (0, 'expr', 27), (5, 'expr', 27) ),
 
         })

uncompyle6/semantics/helper.py

@@ -50,7 +50,7 @@ def find_globals_and_nonlocals(node, globs, nonlocals, code, version):
 #         # print("XXX", n.kind, global_ops)
 #         if isinstance(n, SyntaxTree):
 #             # FIXME: do I need a caser for n.kind="mkfunc"?
-#             if n.kind in ("conditional_lambda", "return_lambda"):
+#             if n.kind in ("if_expr_lambda", "return_lambda"):
 #                 globs = find_globals(n, globs, mklambda_globals)
 #             else:
 #                 globs = find_globals(n, globs, global_ops)
@@ -68,6 +68,36 @@ def find_none(node):
             return True
     return False
 
+def escape_string(str, quotes=('"', "'", '"""', "'''")):
+    quote = None
+    for q in quotes:
+        if str.find(q) == -1:
+            quote = q
+            break
+        pass
+    if quote is None:
+        quote = '"""'
+        str = str.replace('"""', '\\"""')
+
+    for (orig, replace) in (('\t', '\\t'),
+                            ('\n', '\\n'),
+                            ('\r', '\\r')):
+        str = str.replace(orig, replace)
+    return "%s%s%s" % (quote, str, quote)
+
+def strip_quotes(str):
+    if str.startswith("'''") and str.endswith("'''"):
+        str = str[3:-3]
+    elif str.startswith('"""') and str.endswith('"""'):
+        str = str[3:-3]
+    elif str.startswith("'") and str.endswith("'"):
+        str = str[1:-1]
+    elif str.startswith('"') and str.endswith('"'):
+        str = str[1:-1]
+        pass
+    return str
+
+
 def print_docstring(self, indent, docstring):
     try:
         if docstring.find('"""') == -1:

uncompyle6/semantics/pysource.py

@@ -611,13 +611,13 @@ class SourceWalker(GenericASTTraversal, object):
         # LOAD_CONST is a terminal, so stop processing/recursing early
         self.prune()
 
-    def n_delete_subscr(self, node):
+    def n_delete_subscript(self, node):
         if node[-2][0] == 'build_list' and node[-2][0][-1].kind.startswith('BUILD_TUPLE'):
             if node[-2][0][-1] != 'BUILD_TUPLE_0':
                 node[-2][0].kind = 'build_tuple2'
         self.default(node)
 
-    n_store_subscript = n_subscript = n_delete_subscr
+    n_store_subscript = n_subscript = n_delete_subscript
 
     # Note: this node is only in Python 2.x
     # FIXME: figure out how to get this into customization
@@ -1168,6 +1168,7 @@ class SourceWalker(GenericASTTraversal, object):
             self.write(' if ')
             if have_not:
                 self.write('not ')
+            self.prec = 27
             self.preorder(if_node)
             pass
         self.prec = p
@@ -1432,13 +1433,13 @@ class SourceWalker(GenericASTTraversal, object):
             assert node[n].kind.startswith('CALL_FUNCTION')
 
             if node[n].kind.startswith('CALL_FUNCTION_KW'):
-                # 3.6+ starts does this
+                # 3.6+ starts doing this
                 kwargs = node[n-1].attr
                 assert isinstance(kwargs, tuple)
                 i = n - (len(kwargs)+1)
                 j = 1 + n - node[n].attr
             else:
-                start = n-2
+                i = start = n-2
                 for i in range(start, 0, -1):
                     if not node[i].kind in ['expr', 'call', 'LOAD_CLASSNAME']:
                         break
@@ -1836,11 +1837,7 @@ class SourceWalker(GenericASTTraversal, object):
             typ = m.group('type') or '{'
             node = startnode
             if m.group('child'):
-                try:
-                    node = node[int(m.group('child'))]
-                except:
-                    from trepan.api import debug; debug()
-                    pass
+                node = node[int(m.group('child'))]
 
             if   typ == '%':	self.write('%')
             elif typ == '+':

uncompyle6/version.py

@@ -12,4 +12,4 @@
 #  along with this program.  If not, see <http://www.gnu.org/licenses/>.
 # This file is suitable for sourcing inside bash as
 # well as importing into Python
-VERSION='3.3.2'  # noqa
+VERSION='3.3.3'  # noqa