AtHeartEngineer/pyan
1
1
Fork 0
mirror of https://github.com/davidfraser/pyan.git synced 2026-01-09 15:37:57 -05:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #6 from Technologicat/master

Browse Source 
Pyan3 bugfixes and enhancements
This commit is contained in:
David Fraser
2018-01-25 11:26:34 +02:00
committed by GitHub
parent a2ff065536 410d863717
commit 8edf01843a
10 changed files with 979 additions and 639 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
README.md
View File

@@ -82,7 +82,7 @@ Currently Pyan always operates at the level of individual functions and methods;
- MRO is (statically) respected in looking up inherited attributes and `super()` ☆
- Assignment tracking with lexical scoping
- E.g. if `self.a = MyFancyClass()`, the analyzer knows that any references to `self.a` point to `MyFancyClass`
- All binding forms are supported (assign, augassign, for, comprehensions, generator expressions) ☆
- All binding forms are supported (assign, augassign, for, comprehensions, generator expressions, with) ☆
- Name clashes between `for` loop counter variables and functions or classes defined elsewhere no longer confuse Pyan.
- `self` is defined by capturing the name of the first argument of a method definition, like Python does. ☆
- Simple item-by-item tuple assignments like `x,y,z = a,b,c` ☆
@@ -94,9 +94,20 @@ Currently Pyan always operates at the level of individual functions and methods;
## TODO
- Determine confidence of detected edges (probability that the edge is correct). Start with a binary system, with only values 1.0 and 0.0.
- A fully resolved reference to a name, based on lexical scoping, has confidence 1.0.
- A reference to an unknown name has confidence 0.0.
- Attributes:
- A fully resolved reference to a known attribute of a known object has confidence 1.0.
- A reference to an unknown attribute of a known object has confidence 1.0. These are mainly generated by imports, when the imported file is not in the analyzed set. (Does this need a third value, such as 0.5?)
- A reference to an attribute of an unknown object has confidence 0.0.
- A wildcard and its expansions have confidence 0.0.
- Effects of binding analysis? The system should not claim full confidence in a bound value, unless it fully understands both the binding syntax and the value. (Note that this is very restrictive. A function call or a list in the expression for the value will currently spoil the full analysis.)
- Confidence values may need updating in pass 2.
- Make the analyzer understand `del name` (probably seen as `isinstance(node.ctx, ast.Del)` in `visit_Name()`, `visit_Attribute()`)
- Prefix methods by class name in the graph; create a legend for annotations. See the discussion [here](https://github.com/johnyf/pyan/issues/4).
- Improve the wildcard resolution mechanism, see discussion [here](https://github.com/johnyf/pyan/issues/5).
- Could record the namespace of the use site upon creating the wildcard, and check any possible resolutions against that (requiring that the resolved name is in scope at the use site)?
- Add an option to visualize relations only between namespaces, useful for large projects.
- Scan the nodes and edges, basically generate a new graph and visualize that.
- Publish test cases.
@@ -107,17 +118,21 @@ Currently Pyan always operates at the level of individual functions and methods;
The analyzer **does not currently support**:
- Tuples/lists as first-class values (will ignore any assignment of a tuple/list to a single name).
- Tuples/lists as first-class values (currently ignores any assignment of a tuple/list to a single name).
- Support empty lists, too (for resolving method calls to `.append()` and similar).
- Starred assignment `a,*b,c = d,e,f,g,h`
- Slicing and indexing in assignment (`ast.Subscript`)
- Additional unpacking generalizations ([PEP 448](https://www.python.org/dev/peps/pep-0448/), Python 3.5+).
- Any **uses** on the RHS *at the binding site* in all of the above are already detected by the name and attribute analyzers, but the binding information from assignments of these forms will not be recorded (at least not correctly).
- Enums; need to mark the use of any of their attributes as use of the Enum. Need to detect `Enum` in `bases` during analysis of ClassDef; then tag the class as an enum and handle differently.
- Resolving results of function calls, except for a very limited special case for `super()`.
- Any binding of a name to a result of a function (or method) call - provided that the binding itself is understood by Pyan - will instead show in the output as binding the name to that function (or method). (This may generate some unintuitive uses edges in the graph.)
- Distinguishing between different Lambdas in the same namespace (to report uses of a particular `lambda` that has been stored in `self.something`).
- Type hints ([PEP 484](https://www.python.org/dev/peps/pep-0484/), Python 3.5+).
- Type inference for function arguments
- Either of these two could be used to bind function argument names to the appropriate object types, avoiding the need for wildcard references (especially for attribute accesses on objects passed in as function arguments).
- Type inference could run as pass 3, using additional information from the state of the graph after pass 2 to connect call sites to function definitions. Alternatively, no additional pass; store the AST nodes in the earlier pass. Type inference would allow resolving some wildcards by finding the method of the actual object instance passed in.
- Must understand, at the call site, whether the first positional argument in the function def is handled implicitly or not. This is found by looking at the flavor of the Node representing the call target.
- Async definitions are detected, but passed through to the corresponding non-async analyzers; could be annotated.
- Cython; could strip or comment out Cython-specific code as a preprocess step, then treat as Python (will need to be careful to get line numbers right).

2
pyan.py
View File

@@ -1,4 +1,4 @@
#!/usr/bin/python3
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import re

7
pyan.txt
View File

@@ -1,7 +0,0 @@
Original version by Edmund Horner, from:
http://code.google.com/p/ejrh/source/browse/trunk/utils/pyan.py
Explanation:
http://ejrh.wordpress.com/2012/01/31/call-graphs-in-python-part-2/

2
pyan/__init__.py
View File

@@ -3,4 +3,4 @@
from .main import main
__version__ = "1.0.0"
__version__ = "1.0.2"

1235
pyan/analyzer.py
View File

File diff suppressed because it is too large Load Diff

248
pyan/anutils.py Normal file
View File

@@ -0,0 +1,248 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Utilities for analyzer."""
import os.path
import ast
from .node import Flavor
def head(lst):
if len(lst):
return lst[0]
def tail(lst):
if len(lst) > 1:
return lst[1:]
else:
return []
def get_module_name(filename):
"""Try to determine the full module name of a source file, by figuring out
if its directory looks like a package (i.e. has an __init__.py file)."""
if os.path.basename(filename) == '__init__.py':
return get_module_name(os.path.dirname(filename))
init_path = os.path.join(os.path.dirname(filename), '__init__.py')
mod_name = os.path.basename(filename).replace('.py', '')
if not os.path.exists(init_path):
return mod_name
if not os.path.dirname(filename):
return mod_name
return get_module_name(os.path.dirname(filename)) + '.' + mod_name
def format_alias(x):
"""Return human-readable description of an ast.alias (used in Import and ImportFrom nodes)."""
if not isinstance(x, ast.alias):
raise TypeError("Can only format an ast.alias; got %s" % type(x))
if x.asname is not None:
return "%s as %s" % (x.name, x.asname)
else:
return "%s" % (x.name)
def get_ast_node_name(x):
"""Return human-readable name of ast.Attribute or ast.Name. Pass through anything else."""
if isinstance(x, ast.Attribute):
# x.value might also be an ast.Attribute (think "x.y.z")
return "%s.%s" % (get_ast_node_name(x.value), x.attr)
elif isinstance(x, ast.Name):
return x.id
else:
return x
# Helper for handling binding forms.
def sanitize_exprs(exprs):
"""Convert ast.Tuples in exprs to Python tuples; wrap result in a Python tuple."""
def process(expr):
if isinstance(expr, (ast.Tuple, ast.List)):
return expr.elts # .elts is a Python tuple
else:
return [expr]
if isinstance(exprs, (tuple, list)):
return [process(expr) for expr in exprs]
else:
return process(exprs)
def resolve_method_resolution_order(class_base_nodes, logger):
"""Compute the method resolution order (MRO) for each of the analyzed classes.
class_base_nodes: dict cls: [base1, base2, ..., baseN]
where dict and basej are all Node objects.
"""
# https://en.wikipedia.org/wiki/C3_linearization#Description
class LinearizationImpossible(Exception):
pass
from functools import reduce
from operator import add
def C3_find_good_head(heads, tails): # find an element of heads which is not in any of the tails
flat_tails = reduce(add, tails, []) # flatten the outer level
for hd in heads:
if hd not in flat_tails:
break
else: # no break only if there are cyclic dependencies.
raise LinearizationImpossible("MRO linearization impossible; cyclic dependency detected. heads: %s, tails: %s" % (heads, tails))
return hd
def remove_all(elt, lst): # remove all occurrences of elt from lst, return a copy
return [x for x in lst if x != elt]
def remove_all_in(elt, lists): # remove elt from all lists, return a copy
return [remove_all(elt, lst) for lst in lists]
def C3_merge(lists):
out = []
while True:
logger.debug("MRO: C3 merge: out: %s, lists: %s" % (out, lists))
heads = [head(lst) for lst in lists if head(lst) is not None]
if not len(heads):
break
tails = [tail(lst) for lst in lists]
logger.debug("MRO: C3 merge: heads: %s, tails: %s" % (heads, tails))
hd = C3_find_good_head(heads, tails)
logger.debug("MRO: C3 merge: chose head %s" % (hd))
out.append(hd)
lists = remove_all_in(hd, lists)
return out
mro = {} # result
try:
memo = {} # caching/memoization
def C3_linearize(node):
logger.debug("MRO: C3 linearizing %s" % (node))
seen.add(node)
if node not in memo:
# unknown class or no ancestors
if node not in class_base_nodes or not len(class_base_nodes[node]):
memo[node] = [node]
else: # known and has ancestors
lists = []
# linearization of parents...
for baseclass_node in class_base_nodes[node]:
if baseclass_node not in seen:
lists.append(C3_linearize(baseclass_node))
# ...and the parents themselves (in the order they appear in the ClassDef)
logger.debug("MRO: parents of %s: %s" % (node, class_base_nodes[node]))
lists.append(class_base_nodes[node])
logger.debug("MRO: C3 merging %s" % (lists))
memo[node] = [node] + C3_merge(lists)
logger.debug("MRO: C3 linearized %s, result %s" % (node, memo[node]))
return memo[node]
for node in class_base_nodes:
logger.debug("MRO: analyzing class %s" % (node))
seen = set() # break cycles (separately for each class we start from)
mro[node] = C3_linearize(node)
except LinearizationImpossible as e:
logger.error(e)
# generic fallback: depth-first search of lists of ancestors
#
# (so that we can try to draw *something* if the code to be
# analyzed is so badly formed that the MRO algorithm fails)
memo = {} # caching/memoization
def lookup_bases_recursive(node):
seen.add(node)
if node not in memo:
out = [node] # first look up in obj itself...
if node in class_base_nodes: # known class?
for baseclass_node in class_base_nodes[node]: # ...then in its bases
if baseclass_node not in seen:
out.append(baseclass_node)
out.extend(lookup_bases_recursive(baseclass_node))
memo[node] = out
return memo[node]
mro = {}
for node in class_base_nodes:
logger.debug("MRO: generic fallback: analyzing class %s" % (node))
seen = set() # break cycles (separately for each class we start from)
mro[node] = lookup_bases_recursive(node)
return mro
class UnresolvedSuperCallError(Exception):
"""For specifically signaling an unresolved super()."""
pass
class Scope:
"""Adaptor that makes scopes look somewhat like those from the Python 2
compiler module, as far as Pyan's CallGraphVisitor is concerned."""
def __init__(self, table):
"""table: SymTable instance from symtable.symtable()"""
name = table.get_name()
if name == 'top':
name = '' # Pyan defines the top level as anonymous
self.name = name
self.type = table.get_type() # useful for __repr__()
self.defs = {iden:None for iden in table.get_identifiers()} # name:assigned_value
def __repr__(self):
return "<Scope: %s %s>" % (self.type, self.name)
# A context manager, sort of a friend of CallGraphVisitor (depends on implementation details)
class ExecuteInInnerScope:
"""Execute a code block with the scope stack augmented with an inner scope.
Used to analyze lambda, listcomp et al. The scope must still be present in
analyzer.scopes.
!!!
Will add a defines edge from the current namespace to the inner scope,
marking both nodes as defined.
!!!
"""
def __init__(self, analyzer, scopename):
"""analyzer: CallGraphVisitor instance
scopename: name of the inner scope"""
self.analyzer = analyzer
self.scopename = scopename
def __enter__(self):
# The inner scopes pollute the graph too much; we will need to collapse
# them in postprocessing. However, we must use them during analysis to
# follow the Python 3 scoping rules correctly.
analyzer = self.analyzer
scopename = self.scopename
analyzer.name_stack.append(scopename)
inner_ns = analyzer.get_node_of_current_namespace().get_name()
if inner_ns not in analyzer.scopes:
analyzer.name_stack.pop()
raise ValueError("Unknown scope '%s'" % (inner_ns))
analyzer.scope_stack.append(analyzer.scopes[inner_ns])
analyzer.context_stack.append(scopename)
return self
def __exit__(self, errtype, errvalue, traceback):
# TODO: do we need some error handling here?
analyzer = self.analyzer
scopename = self.scopename
analyzer.context_stack.pop()
analyzer.scope_stack.pop()
analyzer.name_stack.pop()
# Add a defines edge, which will mark the inner scope as defined,
# allowing any uses to other objects from inside the lambda/listcomp/etc.
# body to be visualized.
#
# All inner scopes of the same scopename (lambda, listcomp, ...) in the
# current ns will be grouped into a single node, as they have no name.
# We create a namespace-like node that has no associated AST node,
# as it does not represent any unique AST node.
from_node = analyzer.get_node_of_current_namespace()
ns = from_node.get_name()
to_node = analyzer.get_node(ns, scopename, None, flavor=Flavor.NAMESPACE)
if analyzer.add_defines_edge(from_node, to_node):
analyzer.logger.info("Def from %s to %s %s" % (from_node, scopename, to_node))
analyzer.last_value = to_node # Make this inner scope node assignable to track its uses.

4
pyan/main.py
View File

@@ -57,6 +57,9 @@ def main():
parser.add_option("-c", "--colored",
action="store_true", default=False, dest="colored",
help="color nodes according to namespace [dot only]")
parser.add_option("-G", "--grouped-alt",
action="store_true", default=False, dest="grouped_alt",
help="suggest grouping by adding invisible defines edges [only useful with --no-defines]")
parser.add_option("-g", "--grouped",
action="store_true", default=False, dest="grouped",
help="group nodes (create subgraphs) according to namespace [dot only]")
@@ -85,6 +88,7 @@ def main():
'draw_defines': options.draw_defines,
'draw_uses': options.draw_uses,
'colored': options.colored,
'grouped_alt' : options.grouped_alt,
'grouped': options.grouped,
'nested_groups': options.nested_groups,
'annotated': options.annotated}

84
pyan/node.py
View File

@@ -2,6 +2,8 @@
# -*- coding: utf-8 -*-
"""Abstract node representing data gathered from the analysis."""
from enum import Enum
def make_safe_label(label):
"""Avoid name clashes with GraphViz reserved words such as 'graph'."""
unsafe_words = ("digraph", "graph", "cluster", "subgraph")
@@ -10,19 +12,83 @@ def make_safe_label(label):
out = out.replace(word, "%sX" % word)
return out.replace('.', '__').replace('*', '')
class Flavor(Enum):
"""Flavor describes the kind of object a node represents."""
UNSPECIFIED = "---" # as it says on the tin
UNKNOWN = "???" # not determined by analysis (wildcard)
NAMESPACE = "namespace" # node representing a namespace
ATTRIBUTE = "attribute" # attr of something, but not known if class or func.
IMPORTEDITEM = "import" # imported item of unanalyzed type
MODULE = "module"
CLASS = "class"
FUNCTION = "function"
METHOD = "method" # instance method
STATICMETHOD = "staticmethod"
CLASSMETHOD = "classmethod"
NAME = "name" # Python name (e.g. "x" in "x = 42")
# Flavors have a partial ordering in specificness of the information.
#
# This sort key scores higher on flavors that are more specific,
# allowing selective overwriting (while defining the override rules
# here, where that information belongs).
#
@staticmethod
def specificity(flavor):
if flavor in (Flavor.UNSPECIFIED, Flavor.UNKNOWN):
return 0
elif flavor in (Flavor.NAMESPACE, Flavor.ATTRIBUTE):
return 1
elif flavor == Flavor.IMPORTEDITEM:
return 2
else:
return 3
def __repr__(self):
return self.value
class Node:
"""A node is an object in the call graph. Nodes have names, and are in
namespaces. The full name of a node is its namespace, a dot, and its name.
If the namespace is None, it is rendered as *, and considered as an unknown
node. The meaning of this is that a use-edge to an unknown node is created
when the analysis cannot determine which actual node is being used."""
"""A node is an object in the call graph.
def __init__(self, namespace, name, ast_node, filename):
Nodes have names, and reside in namespaces.
The namespace is a dot-delimited string of names. It can be blank, '',
denoting the top level.
The fully qualified name of a node is its namespace, a dot, and its name;
except at the top level, where the leading dot is omitted.
If the namespace has the special value None, it is rendered as *, and the
node is considered as an unknown node. A uses edge to an unknown node is
created when the analysis cannot determine which actual node is being used.
A graph node can be associated with an AST node from the analysis.
This identifies the syntax object the node represents, and as a bonus,
provides the line number at which the syntax object appears in the
analyzed code. The filename, however, must be given manually.
Nodes can also represent namespaces. These namespace nodes do not have an
associated AST node. For a namespace node, the "namespace" argument is the
**parent** namespace, and the "name" argument is the (last component of
the) name of the namespace itself. For example,
Node("mymodule", "main", None)
represents the namespace "mymodule.main".
Flavor describes the kind of object the node represents.
See the Flavor enum for currently supported values.
"""
def __init__(self, namespace, name, ast_node, filename, flavor):
self.namespace = namespace
self.name = name
self.ast_node = ast_node
self.filename = filename
self.flavor = flavor
self.defined = namespace is None # assume that unknown nodes are defined
def get_short_name(self):
@@ -53,9 +119,9 @@ class Node:
else:
if self.get_level() >= 1:
if self.ast_node is not None:
return "%s\\n\\n(%s:%d,\\nin %s)" % (self.name, self.filename, self.ast_node.lineno, self.namespace)
return "%s\\n\\n(%s:%d,\\n%s in %s)" % (self.name, self.filename, self.ast_node.lineno, repr(self.flavor), self.namespace)
else:
return "%s\\n\\n(in %s)" % (self.name, self.namespace)
return "%s\\n\\n(%s in %s)" % (self.name, repr(self.flavor), self.namespace)
else:
return self.name
@@ -109,4 +175,4 @@ class Node:
return make_safe_label(self.namespace)
def __repr__(self):
return '<Node %s>' % self.get_name()
return '<Node %s:%s>' % (repr(self.flavor), self.get_name())

9
pyan/visgraph.py
View File

@@ -71,16 +71,13 @@ class Colorizer:
class VisualNode(object):
"""
A node in the output graph: colors, internal ID, human-readable label, ...
flavor is meant to be used one day for things like 'source file', 'class',
'function'...
"""
def __init__(
self, id, label='', flavor='',
fill_color='', text_color='', group=''):
self.id = id # graphing software friendly label (no special chars)
self.label = label # human-friendly label
self.flavor = ''
self.flavor = flavor
self.fill_color = fill_color
self.text_color = text_color
self.group = group
@@ -130,6 +127,7 @@ class VisualGraph(object):
def from_visitor(cls, visitor, options=None, logger=None):
colored = options.get('colored', False)
nested = options.get('nested_groups', False)
grouped_alt = options.get('grouped_alt', False)
grouped = nested or options.get('grouped', False) # nested -> grouped
annotated = options.get('annotated', False)
draw_defines = options.get('draw_defines', False)
@@ -183,6 +181,7 @@ class VisualGraph(object):
visual_node = VisualNode(
id=node.get_label(),
label=labeler(node),
flavor=repr(node.flavor),
fill_color=fill_RGBA,
text_color=text_RGB,
group=idx)
@@ -223,7 +222,7 @@ class VisualGraph(object):
subgraph.nodes.append(visual_node)
# Now add edges
if draw_defines or not grouped:
if draw_defines or grouped_alt:
# If grouped, use gray lines so they won't visually obstruct
# the "uses" lines.
#

8
visualize_pyan_architecture.sh
View File

@@ -1,6 +1,4 @@
#!/bin/bash
./pyan.py pyan/*.py --no-uses --defines --grouped --nested-groups --colored --dot --annotated >defines.dot
./pyan.py pyan/*.py --uses --no-defines --grouped --nested-groups --colored --dot --annotated >uses.dot
dot -Tsvg defines.dot >defines.svg
dot -Tsvg uses.dot >uses.svg
echo -ne "Pyan architecture: generated defines.svg and uses.svg\n"
echo -ne "Pyan architecture: generating architecture.{dot,svg}\n"
./pyan.py pyan/*.py --no-defines --uses --colored --annotate --dot -V >architecture.dot 2>architecture.log
dot -Tsvg architecture.dot >architecture.svg