AirSpec: Testing Framework

AirSpec is a functional testing framework for Scala, Scala.js, and Scala Native.

AirSpec uses test("...") { ... } syntax for writing test cases. This style requires no extra learning cost if you already know Scala. For advanced users, dependency injection to test cases and property-based testing are supported optionally.

Features

  • Simple to use: Just import wvlet.airspec._ and extend AirSpec trait.
  • Tests can be defined with test(...) functions.
    • No annotation is required as in JUnit5.
  • Support basic assertion syntaxes: assert(cond), x shouldBe y, etc.
    • No need to learn other complex DSLs.
  • Nesting and reusing test cases with test(...).
  • Async testing support for scala.concurrent.Future[ ] and Rx.
  • Lifecycle management with Airframe DI:
    • DI will inject the arguments of test methods based on your custom Design.
    • The lifecycle (e.g., start and shutdown) of the injected services will be properly managed.
  • Handy keyword search for sbt: > testOnly -- (a pattern for class or method names)
  • Property-based testing integrated with ScalaCheck
  • Scala 2.12, 2.13, Scala 3, Scala.js, Scala Native support

To start using AirSpec, read Quick Start.

Quick Start

To use AirSpec, add "org.wvlet.airframe" %% "airspec" to your test dependency and add wvlet.airspec.Framework as a TestFramework.

AirSpec uses (year).(month).(patch) versioning scheme. For example, version 19.8.x means a version released on August, 2019:

Maven Central

build.sbt

libraryDependencies += "org.wvlet.airframe" %% "airspec" % "(version)" % "test"
testFrameworks += new TestFramework("wvlet.airspec.Framework")

If you have multiple sub projects, add the above testFramework setting to each sub project.

For Scala.js and Scala Native, use %%%:

libraryDependencies += "org.wvlet.airframe" %%% "airspec" % "(version)" % "test"
testFrameworks += new TestFramework("wvlet.airspec.Framework")

For Scala Native, only Scala Native + Scala 3 is supported.

Writing Unit Tests

In AirSpec, you can describe test cases with test(...) { body } in a class (or an object) extending AirSpec.

import wvlet.airspec._

class MyTest extends AirSpec {
  test("empty Seq size should be 0") {
    Seq.empty.size shouldBe 0
  }

  test("Seq.empty.head should fail") {
    intercept[NoSuchElementException] {
      Seq.empty.head
    }
  }
}

This test syntax is useful for writing nested tests or customizing the design of DI for each test.

Assertions in AirSpec

AirSpec supports basic assertions listed below:

syntaxmeaning
assert(x == y)check x equals to y
assertEquals(a, b, delta)check the equality of Float (or Double) values by allowing some delta difference
intercept[E] { ... }Catch an exception of type E to check an expected exception is thrown
x shouldBe ycheck x == y. This supports matching collections like Seq, Array (with deepEqual)
x shouldNotBe ycheck x != y
x shouldNotBe nullshouldBe, shouldNotBe supports null check
x shouldBe definedcheck x.isDefined == true, when x is Option or Seq
x shouldBe emptycheck x.isEmpty == true, when x is Option or Seq
x shouldBeTheSameInstanceAs ycheck x eq y; x and y are the same object instance
x shouldNotBeTheSameInstanceAs ycheck x ne y; x and y should not be the same instance
x shouldMatch { case .. => }check x matches given patterns
x shouldContain ycheck x contains given value y
x shouldNotContain ycheck x doesn't contain a given value y
fail("reason")fail the test if this code path should not be reached
ignore("reason")ignore this test execution.
cancel("reason")cancel the test (e.g., due to set up failure)
pending("reason")pending the test execution (e.g., when hitting an unknown issue)
pendingUntil("reason")pending until fixing some blocking issues
skip("reason")Skipping unnecessary tests (e.g., tests that cannot be supported in Scala.js)

AirSpec is designed to use pure Scala syntax as much as possible so as not to introduce any complex DSLs, which are usually hard to remember.

Examples

Here are examples of using shouldBe matchers:

import wvlet.airspec._

class MyTest extends AirSpec {
  test("assertion examples") {
    // checking the value equality with shouldBe, shouldNotBe:
    1 shouldBe 1
    1 shouldNotBe 2
    List().isEmpty shouldBe true

    // For optional values, shouldBe defined (or empty) can be used:
    Option("hello") shouldBe defined
    Option(null) shouldBe empty
    None shouldNotBe defined

    // null check
    val s:String = null
    s shouldBe null
    "s" shouldNotBe null

    // For Arrays, shouldBe checks the equality with deep equals
    Array(1, 2) shouldBe Array(1, 2)

    // Collection checker
    Seq(1) shouldBe defined
    Seq(1) shouldNotBe empty
    Seq(1, 2) shouldBe Seq(1, 2)
    (1, 'a') shouldBe (1, 'a')

    // Object equality checker
    val a = List(1, 2)
    val a1 = a
    val b = List(1, 2)
    a shouldBe a1
    a shouldBeTheSameInstanceAs a1
    a shouldBe b
    a shouldNotBeTheSameInstanceAs b
    
    // Patten matcher
    Seq(1, 2) shouldMatch {
      case Seq(1, _) => // ok 
    }
    
    // Containment check
    "hello world" shouldContain "world"
    Seq(1, 2, 3) shouldContain 1
    
    "hello world" shouldNotContain "!!"
    Seq(1, 2, 3) shouldNotContain 4
  }
}

Running Tests in sbt

AirSpec supports pattern matching for running only specific tests:

$ sbt

> test                                   # Run all tests
> testOnly -- (pattern)                  # Run all test matching the pattern 
> testOnly -- (pattern)/(pattern)        # Run nested tests matching the nested pattern (/ is a dlimiter)

# Configure log levels of airframe-log
> testOnly -- -l (level)                 # Set the log level for the test target classes
> testOnly -- -L (package)=(level)       # Set log level for a specific package or class

# sbt's default test functionalities:
> testQuick                              # Run only previously failed test specs
> testOnly (class name)                  # Run tests only in specific classes matching a pattern (wildcard is supported)

The pattern is a slash (/)-separated test names. You can also use wildcard (*) and regular expressions in the pattern. If the test cases are nested, AirSpec represents test names as (parent test name)/(child test name)/.... so you can run specific nested test with patterns like:

test("test A") { // Matches with 'test A'. All child tests will be executed
  test("1") { ... }  // Matches with 'test A/1' 
  test("2") { ... } // Matches with 'test A/2'
}

Test names will be checked as case-insensitive partial match, so you only need to specify substrings of test names like A, A/1, 'a/2', etc. to simplify the pattern matching.

image

Disable Parallel Test Execution

sbt 1.x or higher runs tests in parallel. This is fast, but it messes up console log messages. If you prefer sequential test execution, set parallelExecution in Test to false:

parallelExecution in Test := false

Running Tests with scala-cli

For quickly writing tests, you can use scala-cli. Add AirSpec configurations with //> using directive as follows:

MyApp.test.scala

//> using test.dep org.wvlet.airframe::airspec::(version)
//> using testFramework "wvlet.airspec.Framework"
import wvlet.airspec.AirSpec

class MyAppTest extends AirSpec {
  test("foo") {
    // ...
  }
}

You can run the above test with scala-cli test command:

$ scala-cli test MyApp.test.scala
MyAppTest:
 - foo 83.42ms

Logging Your Tests

To debug your tests, showing console logs with info, debug, trace, warn, error functions will be useful. AirSpec is integrated with airframe-log to support handly logging messages with these functions.

import wvlet.airspec._

package org.mydomain.myapp

class MyTest extends AirSpec {
  info(s"info log")
  debug(s"debug log") // Will not be shown by default
  trace(s"trace log") // To show this level of log, trace log level must be set
}

Similarly, if you include wvlet.log.LogSupport to your application classes, you can add log messages to these classes.

Configure Log Levels

AirSpec natively supports airframe-log for logging. To temporally change the log level, use -l (log level) option:

> testOnly -- -l debug

This will set the log level to debug for all test classes.

To change the log level only for a specific package or a class, use -L (package or class)=(log level) option:

> testOnly -- -L org.mydomain.myapp=debug

You can use multiple -L options to set different log levels for multiple packages.

You can also use wildcard * with -L option for the ease of setting log levels for specific classes:

> testOnly -- -L myapp.*=debug
> testOnly -- -L *.MyClass=debug

Configure Log Levels in log-test.properties

To change the log level for your packages and classes, add log-test.properties file to your test resource folder src/test/resources. For multi-module projects, put this file under (project folder)/src/test/resources folder.

This is an example of changing log levels of your packages and classes:

src/test/resources/log-test.properties

# Show debug logs of all classes under org.mydomain.myapp package
org.mydomain.myapp=debug
# Show all logs including trace-level logs
org.mydomain.myapp.MyTest=trace

As you modify this property file, a background thread automatically reads this log file and refreshes the log level accordingly.

For more details, see the documentation of airframe-log.

Detecting Runtime Environment

For detecting the running environment of tests, the following methods are available:

  • inCI: Boolean
  • inTravisCI: Boolean
  • inCircleCI: Boolean
  • inGitHubAction: Boolean
  • isScalaJVM: Boolean
  • isScalaJS: Boolean
  • isScalaNative: Boolean
  • isScala2: Boolean
  • isScala3: Boolean
  • scalaMajorVersion: Int

For example:

test("scala-3 specific tests") {
  if(isScala3) {
    scalaMajorVersion shoudlBe 3 
  }  
}

Async Testing

Since the version 22.5.0, AirSpec supports tests returning Future[_] values and Rx. Such async tests are useful, especially if your application needs to wait the completion of network requests, such as Ajax responses in Scala.js, RPC responses from a server, etc. If test specs returns Future or Rx values, AirSpec awaits the completion of async tests, so you don't need to write synchronization steps in your test code.

import wvlet.airspec._
import scala.concurrent.{Future,ExecutionContext}

class AsyncTest extends AirSpec {
  // Use the default ExecutionContext for running Future tasks.
  private implicit val ec: ExecutionContext = defaultExecutionContext

  // AirSpec awaits the completion of the returned Future value
  test("async test") {
    Future.apply {
       println("hello async test")
    }
  }
}

Flaky Tests

If your test cases are flaky, i.e., failing occasionally, you can wrap the text code with flaky block to mark failures inside the block as 'skipped'. You may need to check the skipped test reports to see if they are really flaky or not:

import wvlet.airspec._

class MyTest extends AirSpec {
  test("performance test") {
    val elaplsed_time = xxx // measure something
    flaky {
       elapsed_time < 100 shouldBe true
    }
  }
}

Dependency Injection with Airframe DI

AirSpec can pass shared objects to your test cases by using function arguments. This is useful for sharing objects initialized only once at the beginning with test cases.

Global and Local Sessions

AirSpec manages two types of sessions: global and local:

  • For each AirSpec instance, a single global session will be created.
  • For each test method in the AirSpec instance, a local (child) session that inherits the global session will be created.
    • It is possible to override the local design by using test(..., design = ...) function.

To configure the design of objects that will be created in each session, configure design by calling initDesign(d: Design => Design) (global design) or initLocalDesign(d: Design => Design). It is also possible to customie design by overriding protected def design:Design or protected def localDesign: Design methods in AirSpec. initDesign and initLocalDesign methods are shortcuts for configuring design and localDesign methods.

Session LifeCycle

AirSpec manages global/local sessions in this order:

  • Create a new instance of AirSpec
    • Run beforeAll
    • Call initDesign(design) to prepare a new global design
    • Start a new global session
      • for each test method m:
        • Call before
        • Call initLocalDesign(localDesign) to prepare a new local design
        • Start a new local session
          • Call the test method m by building method arguments using the local session (and the global session). See also Airframe DI: Child Sessions to learn more about the dependency resolution order.
        • Shutdown the local session. All data in the local session will be discarded
        • Call after
      • Repeat the loop
    • Shutdown the global session. All data in the global session will be discarded.
    • Call afterAll

DI Example

This is an example to utilize a global session to share the same service instance between test methods:

import wvlet.airspec._

case class ServiceConfig(port:Int)
class Service(val config:ServiceConfig)

class ServiceSpec extends AirSpec {
  initDesign { design =>
    design
      .bind[ServiceConfig].toInstance(ServiceConfig(port=8080))
      .bind[Service].toSingleton
      .onStart{x => info(s"Starting a server at ${x.config.port}")}
      .onShutdown{x => info(s"Stopping the server at ${x.config.port}")}
  }

  test("test1") { (service:Service) =>
     info(s"server id: ${service.hashCode}")
  }

  test("test2") { (service:Service) =>
     info(s"server id: ${service.hashCode}")
  }
}

This test shares the same Service instance between two test methods test1 and test2, and properly start and closes the service before and after running tests.

> testOnly -- ServiceSpec
2019-08-09 17:24:37.184-0700  info [ServiceSpec] Starting a server at 8080  - (ServiceSpec.scala:33)
2019-08-09 17:24:37.188-0700  info [ServiceSpec] test1: server id: 588474577  - (ServiceSpec.scala:42)
2019-08-09 17:24:37.193-0700  info [ServiceSpec] test2: server id: 588474577  - (ServiceSpec.scala:46)
2019-08-09 17:24:37.194-0700  info [ServiceSpec] Stopping the server at 8080  - (ServiceSpec.scala:34)
[info] ServiceSpec:
[info]  - test1 13.94ms
[info]  - test2 403.41us
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2

It is also possible to reuse the same injected instance by nesting test methods:

class ServiceSpec extends AirSpec {
  initDesign { _.bind[Service].toInstance(...) }
  test("server test") { (service:Service) =>
    test("test 1") {
      info(s"server id: ${service.hashCode}")
    }

    test("test 2") {
     info(s"server id: ${service.hashCode}")
    }
  }
}

Overriding Design At Test Methods

If you need to partially override a design in a test method, use test(..., design = ...) to provide a custom child design:

import wvlet.airspec._
import wvlet.airframe._

class OverrideTest extends AirSpec {

  initDesign {
    _.bind[String].toInstance("hello")
  }

  test("before overriding the design") { (s:String) =>
    s shouldBe "hello"

    // Override the design only for this test method
    test("override the design", design = _.bind[String].toInstance("hello child")) { (cs: String) =>
      cs shouldBe "hello child"
    }
  }
}

FAQs

Q: MissingTestDependency error is shown for a test without any argument block

For example, this test throws an error:

// wvlet.airspec.spi.MissingTestDependency: Failed to call `return a function`. Missing dependency for Int:
test("return a function") {
  val f = { (i: Int) => s"count ${i}" }
  f
}

This is because the above code is equivalent to having one dependency argument like this:

test("return a function") { (i: Int) =>
  // ...
}

Workaround: Specify [Unit] type:

test("return a function")[Unit] {
  val f = { (i: Int) => s"count ${i}" }
  f
}

Pro Tips

Designs of Airframe DI are useful for defining modules of your application. If you need to switch some implementations or configurations for your tests, override your design as shown below:

import wvlet.airframe._

object AppModule {
  // Define your application design
  def serviceDesign: Design = {
    newDesign
        .bind[Service].to[ServiceImpl]
        .bind[ServiceConfig].toInstance(new ServiceConfig(...))
  }

  // Define designs of other compoments as you need
  def componentDesign: Design = ...
}

// Design for testing
object AppTestModule {
  def serviceDesignForTests: Design = {
    AppModule.serviceDesign  // Reuse your application design for tests
     .bind[ServiceConfig].toInstnce(new ServiceConfig(...)) // Override the config for tests
  }
}

import wvlet.airspec._
class AppTest extends AirSpec {
  // Use the testing design
  initDesign(_ + AppTestModule.serviceDesignForTests)

  // Inject a Service object initialized with a test configuration
  test("start up test") { (service:Service) =>
     // test your service here
  }
}

If you are already familiar to dependency injection using Airframe DI or Google Guice, it would not be so difficult to split your application into some units of testable modules. This is generally a good practice to minimize the scope of tests only for specific components.

This code will run the same Fixture two times using different data sets: image

AirSpecContext also contains the name of test classes and method names, which would be useful to know which tests are currently running.

Property Based Testing with ScalaCheck

AirSpec supports property-based testing using ScalaCheck, a framework for writing tests for a wide range of input values. First, add wvlet.airspec.spi.PropertyCheck trait to your spec, and use forAll methods:

import wvlet.airspec._

class PropertyBasedTest extends AirSpec with PropertyCheck {
  test("testAllInt") {
    // Run tests for a wide range of Int values
    forAll{ (i:Int) => i.isValidInt shouldBe true }
  }

  test("testCommutativity") {
    forAll{ (x:Int, y:Int) => x+y shouldBe y+x }
  }

  // Using custom genrators of ScalaCheck
  test("useGenerator") {
    import org.scalacheck.Gen
    forAll(Gen.posNum[Long]){ x: Long => x > 0 shouldBe true }
  }
}

Background & Motivation

In Scala there are several rich testing frameworks like ScalaTests, Specs2, uTest, etc. We also have a simple testing framework like minitest. In 2019, Scala community has started an experiment to creating a nano-testing framework nanotest-strawman based on minitest so that Scala users can have some standards for writing tests in Scala without introducing third-party dependencies.

A problem here is, in order to write tests in Scala, we usually have only two choices: learning DSLs or being a minimalist:

  • Complex DSLs:

    • ScalaTests supports various writing styles of tests, and assersions. We had no idea how to choose the best style for our team. To simplify this, uTest has picked only one of the styles from ScalaTests, but it's still a new domain-specific language on top of Scala. Specs2 introduces its own testing syntax, and even the very first example can be cryptic for new people, resulting in high learning cost.
    • With these rich testing frameworks, using a consistent writing style is challenging as they have too much flexibility in writing test; Using rich assertion syntax like x should be (>= 0), x.name must_== "me", Array(1, 2, 3) ==> Array(1, 2, 3) needs practices and education within team members. We can force team members so as not to use them, but having a consensus on what can be used or not usually takes time.

  • Too minimalistic framework:

    • On the other hand, a minimalist approach like minitest, which uses a limited set of syntax like asserts and test("...."), is too restricted. For example, I believe assertion syntax like x shouldBe y is a good invention in ScalaTest to make clear the meaning of assertions to represent (value) shoudlBe (expected value). In minitest assert(x == y) has the same meaning, but the intention of test writers is not clear because we can write it in two ways: assert(value == expected) or assert(expected == value). Minitest also has no feature for selecting test cases to run; It only supports specifying class names to run, which is just a basic functionality of sbt.
    • A minimalist approach forces us to be like Zen mode. We can extend minitest with rich assertions, but we still need to figure out the right balance between a minimalist and developing a DSL for our own teams.

AirSpec: Writing Tests As Plain Functions In Scala

So where is a middle ground in-between these two extremes? We usually don't want to learn too complex DSLs, and also we don't want to be a minimalist, either.

Why can't we use plain Scala functions to define tests? ScalaTest already has RefSpec to write tests in Scala functions. Unfortunately, however, its support is limited only to Scala JVM as Scala.js does not support runtime reflections to list function names. Scala.js is powerful for developing web applications in Scala, so we don't want to drop it, and the lack of runtime reflection in Scala.js is probably a reason why existing testing frameworks needed to develop their own DSLs like test(....) { test body }.

Now listing functions in Scala.js is totally possible by using airframe-surface, which is a library to inspect parameters and methods in a class by using reflection (in Scala JVM) or Scala macros (in Scala.js). So it was a good timing for us to develop a new testing framework, which has more Scala-friendly syntax.

And also, if we define tests by using functions, it becomes possible to pass test dependencies through function arguments. Using local variables in a test class has been the best practice of setting up testing environments (e.g., database, servers, etc.), but it is not always ideal as we need to properly initialize and clean-up these variables for each test method by using setUp/tearDown (or before/after) methods. If we can simply pass these service instances to function arguments using Airframe DI, which has a strong support of life-cycle management, we no longer need to write such setUp/tearDown steps for configuring testing environments. Once we define a production-quality service with proper lifecycle management hooks (using Airframe design and onStart/onShutdown hooks), we should be able to reuse these lifecycle management code even in test cases.

AirSpec was born with these ideas in mind by leveraging Airframe modules like Airframe Surface and DI. After implementing basic features of AirSpec, we've successfully migrated all of the test cases in 20+ Airframe modules into AirSpec, which were originally written in ScalaTest. Rewriting test cases was almost straightforward as AirSpec has handy shouldBe syntax and property testing support with ScalaCheck.

In the following sections, we will see how to write tests in a Scala-friendly style with AirSpec.

← airframe-rx: ReactiveX interfaceairframe-codec: Schema-On-Read Object Serializer →