» CS 330: Lecture 38 – Automatic Test Running Via Metaprogramming

CS 330: Lecture 38 – Automatic Test Running Via Metaprogramming

May 9, 2018 by Chris Johnson. Filed under cs330, lectures, spring 2018.

Dear students,

Last time we saw how we could add a hook in Ruby so that when a non-existent method is called on a object, we can still execute the desired action. method_missing lets us write really virtual methods—ones that don’t even exist. Our code effectively used information about the method that would have been called and hobbled together an implementation of it on the fly.

Could we do something similar in C? Probably not. In statically-typed languages, the compiler generally wants to know everything it can at compile time. In particular, if you try to call a function in C, that function will need a declaration and definition long before the program is ever run. There’s no way we dynamically define functions.

Can we do any metaprogramming in statically-typed languages? Can we have code act on other code? Yes, we can—at least in Java. This language provides a reflection API, a set of classes and methods that abstract away the pieces of a program. For instance, we can list all the methods that a class provides:

Arrays.asList(String.class.getDeclaredMethods())
  .stream()
  .forEach(System.out::println);

We might as well use functional programming features, right?

Can you think of any reasons why you’d want to ask a class about its methods? I can think of a few. First, when I extend a superclass, I occasionally desire to override some superclass method. But as you know, for a method to override a superclass method, what has to be true? The name and parameters must be the same. (Technically the return type of the subclass method must be a subtype—a covariant type—of the superclass return type. This means Square.clone can return Square even though Rectangle.clone returns Rectangle. The parameters of the subclass method are allowed to be supertypes—contravariant types—of the parameters of the superclass method.) It’s really easy to get the signature wrong. When you go to run your code, the overriding doesn’t seem to happen! Like here:

public class Super {
  public void foo(int i, int j) {
    System.out.println("super");
  }
}

public class Sub {
  public void foo(int i) {
    System.out.println("sub");
  }
}

public class Main {
  public static void main(String[] args) {
    Super s = new Sub();
    s.foo(10, 11);
  }
}

What can we do? We can write a tool that investigates all the methods that are supposed to override a superclass method. If it can’t find a superclass method with that exact same signature, let’s have it yell at us. But how do we know which methods are supposed to override a superclass method? We need to provide a hint to our tool. Java calls these annotations. Let’s write one called Usurps:

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Retention(RetentionPolicy.RUNTIME)
public @interface Usurps {
}

Then we can annotate our inheritance hierarchy:

public class Sub {
  @Usurps
  public void foo(int i) {
    System.out.println("sub");
  }
}

With this hint in place, we can write our UsurpsChecker:

public class UsurpsChecker {
  public static void main(String[] args) {
    assertUsurps(Sub.class); 
  }

  public static void assertUsurps(Class<?> subclass) {
    Class<?> superclass = subclass.getSuperclass();
    Arrays.asList(subclass.getDeclaredMethods())
      .stream()
      .filter(m -> m.getAnnotation(Usurps.class) != null)
      .forEach(m -> {
        try {
          Method superMethod = superclass.getMethod(m.getName(), m.getParameterTypes());
        } catch (Exception e) {
          System.err.println(m.getName() + " says it usurps the superclass, but it really doesn't. Sorry, pal.");
        }
      });
  }
}

To make this a little more reusable, we can let the class name come in as a command-line parameter. To turn a String into a Class, we can use Class.forName:

public static void main(String[] args) {
  try {
    assertUsurps(Class.forName(args[0])); 
  } catch (ClassNotFoundException e) {
    e.printStackTrace();
  }
}

This is useful. Of course, we probably would want to make a plugin out of this and have it automatically run inside our IDE. That’s exactly what Eclipse does with the @Override annotation.

Now let’s do another metaprogramming-ish thing. You may have heard that some people test their software with unit tests. What exactly are unit tests? They are small and independent assertions that an abstraction’s behavior matches its specification. How are they made independent? Each test is thrown into a method. We might have a bunch of methods like this:

public static void testDefaultCtor() {
  ...
}

public static void testExplicitCtor() {
  ...
}

public static void testMagnitude() {
  ...
}

public static void testNormalize() {
  ...
}

Inside each we’ll do something like this:

instance = new object to check
actual = instance.method(params)
assertEqual("expected and actual don't match!", expected, actual)

If the expected and actual values aren’t equal, we’ll generate an exception. Let’s start by writing an assertion:

public class Certain {
  public static void assertEqualEnough(String message, double expected, double actual, double threshold) {
    if (Math.abs(expected - actual) > threshold) {
      throw new TestFailedException(String.format("%s. Expected: %f. Actual: %f", message, expected, actual));
    }
  }
}

And here’s our exception class:

public class TestFailedException extends RuntimeException {
  public TestFailedException(String message) {
    super(message);
  }
}

Now we can start writing unit tests for our Vector2 class, whose implementation does not need to be figured out before we write the unit tests:

public class TestVector2 {
  public static void testDefaultCtor() {
    Vector2 v = new Vector2();
    Certain.assertEqualEnough("Bad x after default ctor", 0, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after default ctor", 0, v.y(), 1.0e-3);
  }
}

After we write this test and others, then we’ll need a main to kick them all off:

public static void main(String[] args) {
  TestVector2.testDefaultCtor(); 
  TestVector2.testExplicitCtor(); 
  TestVector2.testMagnitude(); 
  TestVector2.testNormalize(); 
}

I don’t know about you, but that’s a lot of code to write and maintain. I could very easily see myself adding a new test and forgetting to call it in main. Or worse yet, not writing a unit test at all because it’s too much work. This is where metaprogramming shines. We can eliminate that main so that testing is much less annoying.

Instead of a custom main, we can use reflection to bootstrap the testing process. As with Usurps, we’ll figure out which methods are tests using an annotation:

@Retention(RetentionPolicy.RUNTIME)
public @interface CertainTest {
}

And then we mark all of our unit test methods:

public class TestVector2 {
  @CertainTest
  public static void testDefaultCtor() {
    // ...
  }
}

Our main should find all the annotated methods and run them:

public static void main(String[] args) {
  try {
    Class<?> clazz = Class.forName(args[0]);
    Arrays.asList(clazz.getDeclaredMethods())
      .stream()
      .filter(m -> m.getAnnotation(CertainTest.class) != null)
      .forEach(m -> {
        try {
          m.invoke(null);
        } catch (IllegalAccessException e) {
          System.out.println("e: " + e);
        } catch (InvocationTargetException e) {
          System.out.println(e.getCause().getMessage());
        }
      });
  } catch (ClassNotFoundException e) {
    System.out.println("e: " + e);
  }
}

This is pretty good. Let’s make sure it scales by writing a few more unit tests and make sure that our only new work has to occur in the unit tests themselves:

public class TestVector2 {
  @CertainTest
  public static void testDefaultCtor() {
    Vector2 v = new Vector2();
    Certain.assertEqualEnough("Bad x after default ctor", 0, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after default ctor", 0, v.y(), 1.0e-3);
  }

  @CertainTest
  public static void testExplicitCtor() {
    Vector2 v = new Vector2(10, 20);
    Certain.assertEqualEnough("Bad x after default ctor", 10, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after default ctor", 20, v.y(), 1.0e-3);
  }

  @CertainTest
  public static void testNormalize() {
    Vector2 v = new Vector2(10, 0);
    v.normalize();
    Certain.assertEqualEnough("Bad x after normalize", 1, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after normalize", 0, v.y(), 1.0e-3);

    v = new Vector2(0, 10);
    v.normalize();
    Certain.assertEqualEnough("Bad x after normalize", 0, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after normalize", 1, v.y(), 1.0e-3);

    v = new Vector2(100, 100);
    v.normalize();
    Certain.assertEqualEnough("Bad x after normalize", Math.sqrt(2) / 2, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after normalize", Math.sqrt(2) / 2, v.y(), 1.0e-3);
  }
}

We should find that our metaprogrammed main handles our changes automatically. Woo! Let’s do one more thing, however. Sometimes our unit tests can be made shorter through something called a fixture—some object that is set up to be available to every method in the test suite. It is fixed in the sense that every method should see it in the same known state at the beginning of every test. Perhaps a companion object is needed to test your class. Like a random number generator. Or you find that a bunch of tests make use of a particular instance. You could make that companion object in every test method, or you could factor that out. If we’re going to factor out data, we’ll need it to either be a static or instance variable. Since static stuff is more verbose in Java, let’s make everything instance-y.

public class TestVector2 {
  private Random fixture;

  public TestVector2() {
    fixture = new Random();
  }

  @CertainTest
  public void testDefaultCtor() {
    Vector2 v = new Vector2();
    Certain.assertEqualEnough("Bad x after default ctor", 0, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after default ctor", 0, v.y(), 1.0e-3);
  }

  @CertainTest
  public void testExplicitCtor() {
    Vector2 v = new Vector2(10, 20);
    Certain.assertEqualEnough("Bad x after default ctor", 10, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after default ctor", 20, v.y(), 1.0e-3);
  }

  @CertainTest
  public void testNormalize() {
    Vector2 v = new Vector2(10, 0);
    v.normalize();
    Certain.assertEqualEnough("Bad x after normalize", 1, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after normalize", 0, v.y(), 1.0e-3);

    v = new Vector2(0, 10);
    v.normalize();
    Certain.assertEqualEnough("Bad x after normalize", 0, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after normalize", 1, v.y(), 1.0e-3);

    v = new Vector2(100, 100);
    v.normalize();
    Certain.assertEqualEnough("Bad x after normalize", Math.sqrt(2) / 2, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after normalize", Math.sqrt(2) / 2, v.y(), 1.0e-3);
  }
}

But removing static breaks our main. Each method must be called on an instance. No problem, reflection can take care of that too:

Class<?> clazz = Class.forName(args[0]);
Object tester = clazz.newInstance();

Arrays.asList(clazz.getDeclaredMethods())
  .stream()
  .filter(m -> m.getAnnotation(CertainTest.class) != null)
  .forEach(m -> {
    try {
      m.invoke(tester);
    } catch (IllegalAccessException e) {
      System.out.println("e: " + e);
    } catch (InvocationTargetException e) {
      System.out.println(e.getCause().getMessage());
    }
  });

Does our our fixture work? Is it in the same state at the beginning of each our test methods? Sadly, no. Because it’s instance-y and we initialize it only in the constructor, its state persists across tests, and we don’t have the clean room environment that we wanted. What could we do to fix our fixture?

Let’s add a couple extra methods for setting up and tearing down our fixture, events that should sandwich each test:

public void setup() {
  fixture = new Random();
  System.out.println("pre");
}

public void teardown() {
  System.out.println("post");
}

We must tweak our main to invoke these. We could add a new annotation that marked these methods, or we could just assume they are named exactly setup and teardown:

Method setup = clazz.getMethod("setup");
Method teardown = clazz.getMethod("teardown");
...
setup.invoke(tester);
m.invoke(tester);
teardown.invoke(tester);

This concludes another example of metaprogramming, a feature which allows our code to essentially write itself and be self-aware. Metaprogramming largely exists to help us write better tools that make us more productive. But a nice byproduct is that they make us feel magical.

See you next time!

Sincerely,

P.S. It’s time for a haiku!

I like writing code
But not that mindless grinding
Just code that needs me

P.P.S. Here’s the code we wrote together:

Usurps.java

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Retention(RetentionPolicy.RUNTIME)
public @interface Usurps {
}

UsurpTest.java

import java.lang.reflect.Method;
import java.util.Arrays;
import java.util.Arrays;

public class UsurpTest {
  public static void main(String[] args) {
    Super s = new Sub();
    /* System.out.println("s.foo(0): " + s.foo(0.0f)); */

    assertUsurps(Sub.class);
  } 

  public static void assertUsurps(Class<?> subclass) {
    Class<?> superclass = subclass.getSuperclass();
    Arrays.asList(subclass.getDeclaredMethods())
      .stream()
      .filter(m -> m.getAnnotation(Usurps.class) != null)
      .forEach(m -> {
        try {
          superclass.getMethod(m.getName(), m.getParameterTypes());
        } catch (NoSuchMethodException e) {
          System.err.println("U don't surp with method " + m.toString() + ".");
        }
      });
  }
}

class Super {
  public int foo(double x) {
    return 5;
  }
  public int foo(float x) {
    return 5;
  }
}

class Sub extends Super {
  @Usurps
  public int foo(float x) {
    return 15;
  }

  public void bag() {
  }

  public void cough() {
  }
}

Vector2.java

public class Vector2 {
  private double x; 
  private double y; 

  public Vector2() {
    this(0, 0);
  }

  public Vector2(double x, double y) {
    this.x = x;
    /* this.x = y; */
    this.y = y;
  }

  public double x() {
    return x;
  }

  public double y() {
    return y;
  }

  public double magnitude() {
    return Math.sqrt(x * x + y * y);
  }

  public void normalize() {
    double mag = magnitude();
    x /= mag;
    y /= mag;
  }
}

Certain.java

public class Certain {
  public static void assertEqualEnough(String message, double expected, double actual, double threshold) {
    if (Math.abs(expected - actual) > threshold) {
      throw new RuntimeException(String.format("%s. Expected: %f. Actual: %f", message, expected, actual));
    }
  }
}

TestSuite.java

public class TestSuite {
  public static void testDefaultCtor() {
    Vector2 v = new Vector2();
    Certain.assertEqualEnough("x on default is bad bad bad", 0.0, v.x(), 1.0e-3);
    Certain.assertEqualEnough("y on default is bad bad bad", 0.0, v.y(), 1.0e-3);
  } 

  public static void testExplicitCtor() {
    Vector2 v = new Vector2(10, 20);
    Certain.assertEqualEnough("Bad x after default ctor", 10, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after default ctor", 20, v.y(), 1.0e-3);
  }

  public static void testNormalize() {
    Vector2 v = new Vector2(10, 0);
    v.normalize();
    Certain.assertEqualEnough("Bad x after normalize", 1, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after normalize", 0, v.y(), 1.0e-3);

    v = new Vector2(0, 10);
    v.normalize();
    Certain.assertEqualEnough("Bad x after normalize", 0, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after normalize", 1, v.y(), 1.0e-3);

    v = new Vector2(100, 100);
    v.normalize();
    Certain.assertEqualEnough("Bad x after normalize", Math.sqrt(2) / 2, v.x(), 1.0e-3);
    Certain.assertEqualEnough("Bad y after normalize", Math.sqrt(2) / 2, v.y(), 1.0e-3);
  }
}

TestRunner.java

import java.lang.reflect.InvocationTargetException;
import java.util.Arrays;

public class TestRunner {
  public static void main(String[] args) {
    /* TestSuite.testDefaultCtor(); */
    /* TestSuite.testExplicitCtor(); */
    runTests(TestSuite.class);
  }

  public static void runTests(Class<?> tester) {
    Arrays.asList(tester.getDeclaredMethods())
      .stream()
      .filter(m -> m.getName().startsWith("test"))
      .forEach(m -> {
        /* obj.method(param1); */
        // method.invoke(obj, param1, ...)
        try {
          m.invoke(null);
        } catch (IllegalAccessException|InvocationTargetException e) {
          throw new RuntimeException(e);
        }
      });
  }
}