Snapshot Testing For OpenVADL Frontend: A Comprehensive Guide
In this comprehensive guide, we'll dive into the world of snapshot testing and how it can revolutionize frontend testing in OpenVADL. We'll explore the current challenges, draw inspiration from Kotlin and Gleam, propose a solution, and address potential concerns. So, buckle up and let's get started, guys!
The Case for Snapshot Testing
Frontend testing can often feel like a tedious and error-prone task. The traditional approach involves writing numerous tests that are not only labor-intensive but also susceptible to human error. Snapshot testing offers a promising alternative by automating the process of verifying the correctness of your UI components. By capturing a snapshot of the rendered output and comparing it against a known good state, you can quickly identify any unexpected changes or regressions. This approach not only saves time and effort but also improves the overall reliability of your tests. Snapshot testing is particularly useful for testing complex UIs with dynamic content, where manual verification can be challenging and time-consuming.
Converting to snapshot tests promises to solve these problems by making the process less repetitive and less error-prone. Let's delve into the details.
The Status Quo in OpenVADL
Currently, writing tests for the frontend in OpenVADL is quite labor-intensive, repetitive, and still error-prone. To illustrate this, let's examine a simple typechecker test:
@Test
public void invalidBooleanType() {
var prog = """
constant b: Bool<1> = true """;
var ast = Assertions.assertDoesNotThrow(() -> VadlParser.parse(prog), "Cannot parse input");
var typechecker = new TypeChecker();
var diags = Assertions.assertThrows(DiagnosticList.class, () -> typechecker.verify(ast));
Assertions.assertEquals(1, diags.items.size());
}
The problem with this approach is that it's not always clear where the error is thrown or what it contains. Even if the test confirms that an error is thrown, the specifics of the error might change without being explicitly caught by the test. This lack of precision can lead to false positives and make it difficult to pinpoint the root cause of a bug. Furthermore, the setup for these tests is often quite similar, involving parsing the input, creating a typechecker, and asserting that a diagnostic list is thrown. This repetitive setup adds to the overall overhead of writing and maintaining these tests.
The challenge lies in creating tests that are both precise and easy to maintain. We need a way to verify not only that an error is thrown but also that the error message and location are exactly as expected. This level of detail is crucial for ensuring that our typechecker is behaving correctly and that any changes to the code don't introduce unexpected errors.
FIXME: What to do about positive cases, to verify that an expression is a certain type. Maybe dumps?
Case Studies: Learning from Kotlin and Gleam
To gain insights into how other languages handle diagnostic testing, let's explore two case studies: Kotlin and Gleam. These languages offer different approaches to the problem, each with its own set of advantages and disadvantages. By examining these approaches, we can identify best practices and avoid potential pitfalls when designing our own snapshot testing solution for OpenVADL.
Kotlin
In Kotlin, they use a custom syntax to highlight diagnostics inline:
val prop1 = <!DIVISION_BY_ZERO!>1 / 0<!>
From experience, this is quite handy. However, one pet peeve I have with this approach is that the test file is no longer valid Kotlin code. It can no longer be pasted directly into the awesome Playground without first removing the diagnostic syntax by hand. The need to run the code in another context doesn't happen too often, but it can be quite handy to better understand a test or to adapt an existing one.
The downside of Kotlin's approach is that it makes the test files non-standard, requiring extra steps to run the code outside of the testing environment. While the inline diagnostics provide immediate feedback, they also introduce a barrier to quick experimentation and adaptation of existing tests.
Gleam
Gleam takes a different approach, printing the whole error to text, which makes more sense for more complex errors (which we also have in OpenVadl). Here's an example:
----- SOURCE CODE
fn wibble() { 1 }
fn wibble() { 2 }
----- ERROR
error: Duplicate definition
┌─ /src/one/two.gleam:2:1
│
1 │ fn wibble() { 1 }
│ ----------- First defined here
2 │ fn wibble() { 2 }
│ ^^^^^^^^^^^ Redefined here
`wibble` has been defined multiple times.
Names in a Gleam module must be unique so one will need to be renamed.
Again, the test file is not valid Gleam syntax, and so highlighting in the IDEs will only work with limited features. Unlike Kotlin, the errors are only shown at the end. And while the printing of the errors does include the lines, it might still be a lot of scrolling having the errors so far from the code (especially in longer examples).
However, one advantage over Kotlin is that it's easy to copy the source code into the Gleam compiler as it's not poisoned by custom syntax.
Gleam's approach offers a good balance between readability and usability. By printing the full error message, it provides detailed information about the error, including the location and cause. However, the separation of code and errors can make it harder to quickly understand the context of the error.
Proposed Solution: A Hybrid Approach
I propose to replace all our current tests that produce diagnostics with a single test runner that loads VADL files in the following format, runs them, and ensures that the generated errors are the same.
constant a: Bool = 1 as Bits<4>
// ^^^^^^^^^^^^ Error1: Type Mismatch
[Rest of the code here]
// # Errors:
//
// Error1:
// ```
// error: Type Mismatch
// â•â”€â”€[file:///Users/flo/src/open-vadl/tmp.vadl:1:20]
// │
// 1 │ constant a: Bool = 1 as Bits<4>
// │ ^^^^^^^^^ Expected `Bool` but got `Bits<4>`.
// │
// ```
//
What I like about this approach is that it's clear at a glance where errors occur, and we still get the full error message. Furthermore, the file is a valid VADL file, so you can put it into the compiler and it's easy to test (unlike Kotlin).
This solution combines the best aspects of both Kotlin and Gleam. It provides inline diagnostics to quickly identify the location of errors, while also preserving the full error message for detailed analysis. By using valid VADL files, it ensures that the tests can be easily run and adapted outside of the testing environment.
However, there is one disadvantage. You need to run it twice at the beginning because once the lines are inserted once the line numbers in the diagnostics will change. This could be confusing to new users. To avoid this, we could strip all line numbers from the code before we run the tests. But then the line numbers in the test file and the diagnostics won't line up (but they will still be shown at the correct line and we could put an explanation into the error section at the bottom).
Addressing Concerns and Future Directions
We currently also have positive tests that parse and check the input and then inspect AST nodes to verify that they have the correct type.
@Test
public void booleanConstant() {
var prog = """
constant b: Bool = true """;
var ast = Assertions.assertDoesNotThrow(() -> VadlParser.parse(prog), "Cannot parse input");
var typechecker = new TypeChecker();
Assertions.assertDoesNotThrow(() -> typechecker.verify(ast), "Program isn't typesafe");
var typeFinder = new AstFinder();
Assertions.assertEquals(Type.bool(), typeFinder.getConstantType(ast, "b"));
}
With the attempt above, the files would just run through, but we couldn't be sure that the type of the constant actually is Bool. I haven't yet found out how Kotlin or Gleam solve that problem.
One solution would be to attach an AST dump with types at the bottom, but this could get quite verbose for non-trivial solutions.
To be fair, I would also be happy not testing this exactly but instead only verifying that the generated VIAM is correct, but again, this isn't possible at the moment since there is no text representation for the VIAM.
Conclusion
Snapshot testing offers a promising approach to improve the efficiency and reliability of frontend testing in OpenVADL. By learning from the experiences of Kotlin and Gleam, we can design a solution that combines the best aspects of both approaches. While there are still some challenges to address, such as handling positive tests and AST node type verification, the potential benefits of snapshot testing are significant. By embracing this technique, we can streamline our testing process, reduce the risk of errors, and ultimately deliver a better user experience. So, let's get started and make our frontend testing great again!