ETL Testing Basics

    12 September 2009

    When doing developer testing outside of the database, it is always a good idea to start with simple unit tests that stub or mock any dependencies. If your test calls a method that calls other objects, a database, a service, etc., then you have entered the realm of integration tests. Integration tests are much harder to design, setup, and maintain. Unfortunately, when you start testing an ETL process, you are going to have to test the end-to-end integration of multiple databases. This makes an already difficult problem exponentially more difficult. For starters, your data needs to start in an expected state, and there can only be one test running against the database(s) at a given time or the data will be corrupted. To make the testing of ETL processes more manageable, this article introduces some general concepts that should be followed.