Support different test configurations

Summary

/cc @mlinhoff