Embedded In-Memory Implementation of OJAI Driver for Testing

Contributed by

9 min read

Writing the appropriate tests should always be a priority during the development cycle of any team. We really encourage to test first (TDD) and to test all, or at least as much as we can.

When writing applications that rely on MapR Database, we might encounter testing that is not out of the box and not as easy as it should be. In order to test under these settings, we need a way to communicate with the MapR Database.

In many cases, we don't really need to connect to MapR Database because we are testing an application component that only depends on it. That means that we should have a way to mock or simulate calls to the MapR Database.

Let's look at one example.

trait Repository[A] {
    def findAll(): Stream[A]

    def findById(id: String): Option[A]
}

trait Filter[A] {
    def apply(fn: A: => Boolean): Boolean
}

class PredicateAll(repository: Repository[Document]) extends Filter[Document] {
    override def apply(fn: Document => Boolean): Boolean =
        repository.findAll().forall(fn)
}

class PredicateAny(repository: Repository[Document]) extends Filter[Document] {
    override def apply(fn: Document => Boolean): Boolean =
        repository.findAll().exist(fn)
}

In the previous case, the class under test is 'PredicateAll'. It depends on 'Repository' to get the data it needs and then evaluate itself on it. However, if we think about, we don't really need a real implementation of MapR Database to accomplish this task. We could, somehow, simulate and inject it into the class under test, so the corresponding tests are executed.

As of today, that is not possible using any of the libraries offered by MapR Database. Even though they present an inconvenience, at this point we are forced to have MapR Database ready in all the environments where we are testing.

Common patterns like using mocking can take only so far, but having a full in memory implementation is not only useful for testing, but to run fully functional applications completely in memory.

Using OJAI-Testing Library.

'ojai-testing' is an embedded, in-memory implementation of OJAI driver to be used in testing. The main feature supporting this library is the ability to have a fully functional interface for MapR Database that lives in memory during our testing time.

Linking

We can get 'ojai-testing' binaries directly from Maven Central in the following way.

mvn
<dependency>
  <groupId>com.github.anicolaspp</groupId>
  <artifactId>ojai-testing_2.X</artifactId>
  <version>1.0.6</version>
</dependency>
sbt
libraryDependencies += "com.github.anicolaspp" % "ojai-testing_2.X" % "1.0.6"

Please, make sure to get the right binaries for the Scala version being used, since this library has been cross built for Scala 2.11.x and 2.12.x.

Once we have linked the library binaries, we can access the main abstractions needed for testing.

There are two entry points to the library. First, we can manually register the corresponding 'Driver' to be used, then we can create a 'Connection' and a 'DocumentStore,' as we do when talking to MapR Database, as we show next.

class OptionOneTest extends FlatSpec with Matchers {
    DriverManager.registerDriver(InMemDriver)

    it should "have registered the driver" in {
        val connection = DriverManager.getConnection("ojai:anicolaspp:mem")

        connection.isInstanceOf[InMemoryConnection] should be (true)

        connection.getStore("anicolaspp/my_store").isInstanceOf[InMemoryStore] should be (true)
    }
}

Notice that after we have registered the 'InMemoryDriver' with the 'DriverManager,' everything we ask from it will be 'InMemory…,' indicating that we don't need access to any on-disk storage.

The second and most convenient option is to use the 'OjaiTesting' trait as follows.

class BestOptionTest extends FlatSpec with OjaiTesting with Matchers {
    it should "have registered the driver" in {
        connection.isInstanceOf[InMemoryConnection] should be (true)

        documentStore("anicolaspp/my_store").isInstanceOf[InMemoryStore] should be (true)
    }

    it should "offer same entry points" in {
        connection.getClass() should be (DriverManager.getConnection("ojai:anicolaspp:mem").getClass)

        documentStore("anicolaspp/store") should be (connection.getStore("anicolaspp/store"))
    }
}

Notice that by using 'OjaiTesting,' we don't need to manually register the 'InMemoryDriver'; we automatically get access to 'connection' and to 'documentStore(...).' Also, there is only one 'documentStore' in memory per store name. The following test shows this last statement.

class OneStorePerNameTest extends FlatSpec with OjaiTesting with Matchers {
    it should "keep track of the stores" in {
        val myStore = documentStore("anicolaspp/my_store")

        val otherStore = connection.getStore("anicolaspp/my_store")

        myStore should be (otherStore)
    }

    it should "not give the same stores" in {
        documentStore("anicolaspp/my_store") should not be(documentStore("anicolaspp/another_store"))

    }
}

As the tests show, no matter how many times we ask for the same store (identified by name), we shall provide the same store every time. Again, there is only one store per store identifier.

Testing Using 'ojai-testing'

Once we have gained access to the in-memory resource this library offers, we should be able to write OJAI queries in the same way we use the MapR Database living on disk.

class ConnectionTest extends FlatSpec
  with OjaiTesting
  with Matchers
  with BeforeAndAfterEach {

    override def beforeEach(): Unit = connection.close()  

    "Connection" should "create empty document" in {

        connection.newDocument().asJsonString() should be("{}")
    }

    it should "mutation set existing doc" in {
        val store = documentStore("anicolaspp/mem")

        store.insert(
            connection
                .newDocument()
                .set("_id", "1")
                .set("name", "pepe")
                .set("age", 20))

        val mutation = connection
            .newMutation()
            .set("name", "nico")

        store.update("1", mutation)

        val doc = store.findById("1")

        doc.getIdString should be("1")
        doc.getString("name") should be("nico")
        doc.getInt("age") should be(20)
  }

    it should "create a query from json" in {

        val query = connection
          .newQuery()
          .where(connection
            .newCondition()
            .is("name",  QueryCondition.Op.EQUAL, "pepe")
            .build())
          .select("a", "b", "c")
          .build()

        connection.newQuery(query.asJsonString()).asJsonString() should be (query.asJsonString())
  }

  it should "create a document from json" in {

    val document = connection
      .newDocument()
      .set("name", "pepe")
      .set("value", 5)

    connection.newDocument(document.asJsonString()).asJsonString() should be (document.asJsonString())
  }
}

As we can appreciate, we are able to write queries using the in-memory store without any issues, and by having these capabilities, we completely removed the need for having to access the real MapR Database for most cases.

What About the Java Folks?

If we are using Java, then we are also covered. In the same way we mixed in 'OjaiTesting,' there is an interface that can be used when testing OJAI applications in Java. The following example shows how it can be done.

public class JavaTesting implements JavaOjaiTesting {

    @Test
    public void testGetConnection() {
        assert connection() instanceof InMemoryConnection;
    }

    @Test
    public void testGetStore() {
        assert documentStore("anicolaspp/java_store") instanceof InMemoryStore;
    }
}

As noticed, by implementing the 'JavaOjaiTesting' interface, we get access to the same abstractions that Scala users get. That is 'connection()' and 'documentStore(...).' From here, we just write our normal Java tests, using the testing framework of our choice.

Testing Our Predicates

In order to conclude by giving a more realistic example, let's implement some possible tests for our previously defined predicates.

class PredicateAllTest extends FlatSpec
      with OjaiTesting
      with Matchers {

    it should "eval to false when no docs" in {

        val repository: Repository[Document] = DocumentRepository(documentStore("anicolaspp/my_store"))

        PredicateAll(repository).apply(_ => true) should be (false)
    }

    it should "eval to true when all documents" in {
        val store = documentStore("anicolaspp/my_store")

        val doc1 = connection.newDocument().set("_id", "1").set("name", "pepe").set("age", 20)
        val doc2 = connection.newDocument().set("_id", "2").set("age", 30)

        store.insert(doc1)
        store.insert(doc2)

        val repository = DocumentRepository(store)

        PredicateAll(repository).apply(_.getInt("age") >= 0) should be (true)
    }
}

Notice that for testing 'PredicateAll,' we don't need to interact with a real MapR Database, since the data is being provided by 'repository,' and it uses a 'DocumentStore,' no matter how it is actually implemented or provided. In our case, we are using 'ojai-testing' 'InMemoryStore.' Once again, we can totally avoid setting up a MapR cluster only to run these tests.

Conclusions

Sometimes it makes little sense to have to implement one of the patterns described at the beginning of the post, just to have application tests. Most of the time, they require extra set up and resources, yet we don't want to ignore testing; it is very important.

On the other hand, we can use the in-memory advantages that 'ojai-testing' offers to overcome the challenges presented in this post, while testing applications that somehow interact with OJAI.

Even when using Scala or Java, the library should cover most of the use cases, basic and advanced, while helping out to achieve our testing goals.


This blog post was published May 01, 2019.
Categories

50,000+ of the smartest have already joined!

Stay ahead of the bleeding edge...get the best of Big Data in your inbox.


Get our latest posts in your inbox

Subscribe Now