Infrastructure Testing for Decentralized Applications built on Blockchain or Distributed Ledger Platform

Background: Ethereum is the first programmable blockchain platform provided to the developer community to build business logic in the form of a Smart Contract that eventually helps developers build decentralized applications for any business use case. Once Smart Contracts are developed, they need to be registered on the blockchain, followed by deploying to the blockchain. After deploying the contract, the contract address gets assigned through which contract methods can be executed by the abstract layer built over ABI. Web3 is the module which is the most popular implementation to interact with local or remote node participating in the underlying blockchain network, built over Ethereum.

Define Decentralized Application Architecture for Testing: Needless to say that testing of any Decentralized application built over the blockchain platform is not only highly complex but also requires a specialized skill set with the most analytical mind of white box testers. At Magic FinServ, we possess rich real-time experience of some of the most complicated concepts of testing Blockchain-based decentralized applications. Based on this experience, our strategy divides Blockchain-based decentralized application into three isolated layers from a testing perspective –

1. Lowest-Layer - Blockchain platform to provide a platform on which smart contracts can be executed. 

a. Ethereum Virtual Machine

b. Encryption (Hashing & Digital Signature by using cryptography)

c. P2P Networking

d. Consensus Algorithm

e.  Blockchain Data & State of the network (Key-Value storage)

2.  Middle-Layer - Business Layer (Smart Contract) to build business logic for business use cases

a. Smart Contract development - Smart Contract Compilation, Deployment & Execution in Test Network

b. Smart Contract Audit

3. Upper-Layer - API Layer for Contracts, Nodes, Blocks, Messages, Accounts, Key Management & Miscellaneous endpoints to provide an interface to execute business logic and get updates on the state of the network at any given point in time. These interfaces can be embedded between upstream & downstream as well.

Based on these defined components of blockchain, we build an encompassing generic testing strategy for each layer in 2 broad categories –

1. Functional: As the category name suggests, this category ensures that all components that belong to each layer should function as per defined acceptance criteria by the business user/technical analyst/business analyst. We prefer to include System/Integration testing under this category to ensure that all components within each layer work as defined, but also as a complete system, it should accomplish the total business use case. 

2. Non-Functional: This category covers all kinds of testing other than functional testing like Performance, Security, Usability, Volatility & Resiliency testing not only at Node level but container level as well if Docker is being used to host any service.

In defining the generic testing strategy for these two broad categories, we surmise that infrastructure needs to be set up first & it will not be the same all the time. Before moving ahead on this, we need to answer other questions –

Question1: Why is the setting up of infrastructure the most critical & essential activity to start strategizing blockchain-based application testing?  

Question2: What all potential challenges do testers face while setting up infrastructure?

Question3: What all solutions do Testers have to overcome with Infrastructure setup?

To answer the first question:

We need to take a few steps backward to understand what we do for testing any software application in the traditional approach. For starting any software application testing, an environment has to be set up, but that is always a one-time activity until the development team does any significant change to the underlying platforms or technology. However, that happens very rarely. So testers can continue with testing without much worry about infrastructure.

The two core concepts of Blockchain technology are P2P networking & consensus algorithms. Testing these two components is heavily dependent on infrastructure setup, meaning how many nodes we need to test one feature or the other.

For P2P, we need a different number of connected nodes with an automated way to kill nodes randomly & then observe the behavior in each condition.

For Consensus, it depends on what kind of consensus is being used & based on the nature of consensus, different types of nodes, each with a different number of nodes will be needed.

Another differentiating factor that is not applicable for public blockchain but has a significant impact on a private blockchain is different types of permission to different nodes.

There is a frequent requirement to keep changing network topology for verifying most of the features of decentralized applications.

By now, we know how important it is to change network topology for each feature; otherwise, testing would not be as effective.  As we all know now, the blockchain network is a collection of many machines (a.k.a. nodes) connected with Peer To Peer networking. It is always a priority to automate the infrastructure required for mimicking the network topology that is needed for testing.

To answer the second question:

1. If we manually spin-off instances, let’s assume five instances, with required software & user setup, we need to spend almost 2–3 hours per instance.

2. Manually setting up machines is highly error-prone & mundane. Even simple automation does not help until the automation framework is intelligent enough to understand the need for different network topologies.

3. Due to agile methodology adoption, spending so much time setting up just infrastructure is not acceptable as the testing team usually does not have that length of time to complete testing for any given sprint.

4. All testers have to understand infrastructure setup as all need to change network topology to test most of the features. Finding a tester with good infrastructure knowledge is also challenging. 

5. Invalid network topology, most of the time, does not show an immediate failure due to the blockchain concept’s inherent nature. Eventually, an incorrect network topology leads to time and effort spent on non-productive testing without finding any potential bugs.

6. High defect rejection ratio by Dev, either due to incorrect network topology or due to incorrect peering among nodes

To answer the third question:

There are four ways to set up a network of nodes –

1. Physical machines with LAN

2. Virtual Machines

3. Docker Containers on the same machine can be considered as an isolated machine

4. Cloud Platform

We use Docker containers & cloud platforms to set up the infrastructure for testing blockchain-based applications as setting up physical machines, or virtual machines is not viable from a maintenance perspective. 

Physical machines with LAN: To set up a blockchain network with physical devices is tough & scalability is challenging since we need additional devices to achieve the desired testing quality. During Infrastructure testing, we need to make machines (a.k.a. Nodes) go up & down to verify the volatility. This is a cumbersome process for the tester as well. Setting up the network with physical devices requires physical space and need maintenance of these machines at regular intervals. We usually do not recommend going for this option; however, if a customer requires the testing to be done in such a manner, we can define a detailed strategy to execute it. 

Virtual Machines: Compared to a network of physical machines, virtual machines do have a lot of advantages. Increasing the number of VMs on an underlying device also highly complicates the matter since maintaining VMs is not user friendly. Another disadvantage is that we need to hardcode the limit of all the resources beforehand. However, combining Option1 and Option2  (multiple physical machines with multiple VMs on a single machine) seems to be a better choice, although it still requires lots of maintenance and carries overheads that act as a time sink for the tester. As reducing time to test is a critical aspect of quality delivery, we focus on saving as much time as possible to be invested in other higher-value elements of testing.

The advantage of using a cloud platform lies in the ability to spin off as many machines as needed without the overheads of maintenance or any other physical activity. However, it is still an uphill task to maintain such a network with multiple machines on the cloud too. Eventually, we thought of option 4 with Docker, and then we concluded that by combining option3 with option4, we could create a very solid strategy to perform infrastructure testing by overcoming various problems. 

Based on our real-world experiences, we tried various options individually, and in combination, our recommended approach is this process. 

Always perform Sanity/Smoke testing for any build with docker containers. Once all sanity/smoke tests finish without any failure, then switch to replicate the required network topology for functional testing of new enhancements & features.

The advantages of our approach are –

1. Build failure issues can be found in less time & can be reported back to dev without any delay that cloud infrastructure introduces. Before taking this approach, we had to spend 2-3 hours to report any build failure bug, whereas the same can be caught in 5-10 minutes as we always run selective test cases under sanity/smoke. 

2. Saved the cost of cloud infrastructure in case of failure in the build, as there is no uptime in case of build failure. 

3. Saved a lot of time for the testing team to spend in infrastructure setup. 

4. Dev team gets more time to fix issues found in sanity/smoke testing as it gets reported in just a few minutes. 

5. Significant reduction in rejection of bugs by the development team. 

6. Timely delivery percentage for the build, without any major bug, has increased significantly. 

To dive into the testing strategy, using docker containers with cloud platform shall be covered in an upcoming blog, followed by our automation framework for the infrastructure setup testing. We will also try to define the answer for the most frequently asked questions by the customer –

Question1: Why should customers be concerned about Infrastructure testing for decentralized applications?

Question2: Why should customers look for an automated way of Infrastructure Setup testing for blockchain-based decentralized applications?

Stay tuned for the upcoming blogs and follow us on LinkedIn for timely updates.