Static Intersections With Pytype

Python has had a static type system for quite some time as initially defined in PEP 484 (with additions in future PEPs). Types allow one to statically verify various aspects of a program - that a value conforms to some set of constraints (methods, property, assertions).

The most popular and well known type checker for Python has been mypy, but an interesting quality of Python is that, because types are so separate from the language and runtime, there are multiple competing type systems. This means that one can run multiple type checkers, each with their own strengths and weaknesses, on a single python project. I’m aware of 3 different type checkers for Python at this time.

This has been extremely useful for me due to mypy lacking intersection types, a feature that I very much miss from Rust where these are trivially implemented with traits.

An intersection type is simple - it is a type that implements interface A and interface B at the same time. This is really helpful when you want to extend an existing type that you don’t “own” (ie: comes from a library) - you can just attach a new interface to it. In Rust this is as simple as importing a trait for that type, in Python things are not so simple.

The goal I had was to write code like this:

    class A(object):
      def foo(self):
        print('foo')
    
    class B(object):
      def bar(self):
        print('bar')
    
    def extend_a(b: Type[_]) -> Type[A + _]: # (This is fake mypy)
        pass # ...
    
    A = extend_a(B)  # Type[A + B]
    A().foo()
    A().bar()
    # A().baz() # This should not type check!

This code is possible to express at runtime fairly easily. With Python very little isn’t possible, after all we can just monkey patch the methods of one class directly onto the other, or form a metaclass from the two classes we wish to combine.

Here’s the metaclass implementation. If you run this code with the above class definitions it will print ‘foo’ followed by ‘bar’. A pretty crazy power of Python’s metaclasses.

    def extend_a(b):
        return type('A', (A, b), {})
    
    A = extend_a(B)
    A().foo()
    A().bar()

Metaclasses give us this incredible power to extend one type with another, but there’s no way to express this in a way that mypy can understand. There is no way, in mypy, to ‘name’ the class A + B and so we can not write the annotation that would allow mypy to know that we can call foo and bar on A.

Thankfully there’s pytype, a type system from Google that takes a fairly different approach from mypy. Whereas mypy relies heavily on the definitions of types in various places, pytype is driven from type inference, only using PEP 484 annotations as assertions that must be upheld during inference.

Here’s the example that pytype provides in their repo:

    from typing import List
    def get_list() -> List[str]:
        lst = ["PyCon"]
        lst.append(2019)
        return [str(x) for x in lst]
    
    # mypy: line 4: error: Argument 1 to "append" of "list" has
    # incompatible type "int"; expected "str"

As we can see, pytype looks a lot further than just the type annotations, or at just declarations. It leverages contextual information to determine types.

Incredibly, pytype is even able to infer metaclasses!

    def extend_a(b): # (pytype will infer the types!)
        return type('A', (A, b), {})
    
    A = extend_a(B)
    
    A().foo()
    A().bar()

This code actually, incredibly, type checks with pytype!

pytype is able to generate a .pyi like the following:

    class A(A, B): ...
    
    class B:
        def bar(self) -> None: ...
    
    def extend_a(b) -> type: ...

It’s a bit strange - A appears to be a recursive type on itself… but it works. And we can even compose multiple types:

    class A(object):
      def foo(self):
        print('foo')
    
    class B(object):
      def bar(self):
        print('bar')
    
    class C(object):
      def baz(self):
        print('baz')
    
    def extend_a(*b): # (pytype will infer the return type!)
        return type('A', (A, *b), {})
    
    A = extend_a(B, C)
    
    A().foo()
    A().bar()
    A().baz()

Generating a .pyi of:

class A(A, B, C): ...

class B:
    def bar(self) -> None: ...

class C:
    def baz(self) -> None: ...

def extend_a(*b) -> type: ...

And just to prove it really works let’s call a method that doesn’t exist on our combined A:

    A().bop()

    No attribute 'bop' on A [attribute-error]

This is pretty incredible! It’s not quite as powerful as a trait system like Rust’s where we can just import a trait and start using methods from it on a type, but this gets us fairly close. We can attach interfaces at runtime to a type but reason about it statically.

One caveat is you’ll probably want to have mypy ignore that file. If we run mypy against this code, here’s what we get:

    scratch.py:19: error: Cannot assign to a type
    scratch.py:22: error: "A" has no attribute "bar"
    scratch.py:23: error: "A" has no attribute "baz"

Eventually, if mypy gets intersection types, this will be possible with just a single tool. But for now this is not an unreasonable workaround.

I think it’s fascinating that Python has multiple implementations of a type checker, allowing for them to grow in different directions, and even for one part of a project to type check with one tool but another part with another tool - it’s something I have never experienced with another language.

Pytests technique of leveraging inference is obviously very powerful. What I’d love to see is something that allows me to leverage that a bit more explicitly, giving me the ability to get that fake mypy code I’d written to work.