A mild introduction to unit testing, mocking and patching for novices

On this story, I wish to elevate a dialogue about unit testing in information engineering. Though there are many articles on Python unit testing on the web, the subject appears to be like a bit imprecise and uncovered. We’ll discuss information pipelines, the components they encompass and the way we will check them to make sure steady supply. Every step of the information pipeline will be thought-about as a perform or course of and ideally, it needs to be examined not solely as a unit however all collectively, built-in into one single information stream course of. I’ll attempt to summarize the methods that I exploit usually to mock, patch and check information pipelines together with integration and automatic exams.

What’s unit testing within the information world?

Testing is an important a part of any software program growth lifecycle and helps builders ensure the code is dependable and will be simply maintained sooner or later. Take into account our information pipeline as a set of processing steps or capabilities. On this case, unit testing will be thought-about as a method of writing exams to make sure that every unit of our code, or every step of our information pipeline doesn’t produce unintended outcomes and is match for goal.

In a nutshell, every step of an information pipeline is a technique or perform which must be examined.

Knowledge pipelines could be totally different. Actually, they usually range tremendously when it comes to information sources, processing steps and ultimate locations for our information. At any time when we remodel the information from level A to level B, there’s a information pipeline. There are totally different design patterns [1] and methods to construct these information processing graphs and I wrote about it in one in all my earlier articles.

Check out this easy information pipeline instance beneath. It demonstrates a typical use case state of affairs when information is being processed within the multi-cloud. Our information pipeline begins from the…