ElasTest is a platform/tool that aims to ease end to end testing of distributed systems. The two key features of the platform are:
1) Provide an easy deployment process and easy access to the necessary services usually involved in an end to end test and
2) Provide easy-to-use tools to show and analyze logs and metrics of all elements involved in an end to end test.
ElasTest offers an open source cloud-based testing service platform OS independent, very flexible and compatible with current Continuous Integration (CI) tools such as Kubernetes or Jenkins and methodologies so that testers/developers can use it without disrupting their common practices. It offers capabilities to test end-to-end different type of applications including web, mobile, real-time video communications and Internet-of-Things under different configurations and environments allowing testers to reproduce real world conditions. It aims to provide advanced testing capabilities to increase the scalability, robustness, security and quality of experience of large distributed systems.
Because end-to-end tests are hard to develop, they need to gather information from the system at runtime, and doing so requires a lot of effort (i.e., time, investment and costs). At the end, ElasTest’s aim is to allow any software development team to deliver software faster and with fewer defects.
In few words:
ElasTest is an open source integrated end-to-end testing platform to test distributed systems in large (SiL) by:
- Running end-to-end tests under real world conditions.
- Gathering information from the tests and the software under test.
- Presenting the information unified, integrated and enabling easy comparison of tests results from different executions.
The main features of the platform are:
- Easy to deploy, OS independent, and flexible.
- Easy to use and to configure all the components of the tested application. wherever they are and run the tests in a few clicks.
- Get useful information from all the components.
- Gather and analyse logs.
- Ensure that the tested applications work properly in all browsers and versions.
ElasTest platform is conceived as an open source cloud-based testing platform aimed to simplify the end-to-end testing processes for different types of applications, including web, mobile, real-time video communications and Internet-of-Things. Such a platform co-exists within the software lifecycle process integrated with other tools, and taking care of the three tasks depicted above. ElasTest provides facilities to leverage specific test services as device emulators, browsers, security tools, and monitoring probes, and operates in different environments. ElasTest provides advance testing capabilities aimed to increase the scalability, robustness, security and quality of experience of large distributed systems, and new capabilities are on under development. All in one, ElasTest will make any software development team capable of delivering software faster and with fewer defects.
The core of ElasTest platform is a solution for test automation all along the test process cycle, including SuT deployment, test execution, SuT monitoring during test run, and test results reporting to testers. Moreover, the intent is to make ElasTest flexible and extensible to collaborative teams and to also include more tools to cover other testing tasks, such as test generation, or test reliability assessment.
The overall architecture distinguishes between a set of core services (i.e., the ElasTest core-platform), and a set of other pluggable services (i.e., test support services) that either are optional, or provide some domain-specific/legacy feature. Among the others, the ElasTest core-platform includes the ElasTest Platform Manager (EPM) that abstracts to the ElasTest services the underlying cloud infrastructure where they are actually deployed. In other words, the EPM is the interface that makes the core-platform fully technologically agnostic and enables ElasTest to be deployed and executed seamlessly in several target cloud infrastructures (e.g., Docker, OpenStack, Kubernetes, AWS, etc.).
ElasTest is unique and fosters collaborative team development in all stages, for people who are in different oganisations as they have the feeling to be part of one single integrated team that is focused on the software’s quality. ElasTest is perfect for all different roles and that are even in different places, organisations and locations. In the near future we hope that these groups have different access rights depending on their role such as researchers or project managers or different technological profiles, and they have access only to the part of the system they need to access.
Collaboration between Quality Assurance (QA) teams and development teams is essential for ensuring the success of a software project, and ElasTest easies this collaboration. Testers and developers work in good synergy helping software projects reach both speed and quality.
ElasTest is unique because provides all in one single dashboard, all it’s centralized in ElasTest platform facilitating collaboration among teams. All logs which are placed in different systems are available to the different teams who were involved in the development such as testing teams, in different organization or groups. These different roles (i.e researchers, testers, managers) may check the status, log analysis and can access the system to analyse their software or to understand it. Because we know teamwork between testers, developers, etc is crucial for successfully deploying and maintaining a high-quality software product, and testing is crucial in pre-production stage.
Also, future plans are that ElasTest incorporates two new collaborative features: ElasTest will allow you to “export t-jobs” to use in your new projects, and to “export the reports” of the testing results, to be used in for example sales or publications.
ElasTest has four demonstrators which cover different application domains: 5G networks, web applications, WebRTC, and IoT.
To verify whether ElasTest’s ambitious goals are achieved and to monitor progress, ElasTest has set precise validation metrics on both the efficiency and the effectiveness of the testing process. The key metrics target time-to-market, tester's productivity and satisfaction, maintenance effort and QoE. ElasTest will assess these and several other Key-Performance Indicators in several iterations over four vertical demonstrators. All the assessments will be made publically available to guide ElasTest adopters.
The four ElasTest demonstrators cover different application domains: 5G networks, web applications, WebRTC, and IoT.
- The Web Vertical Demonstrator is focused on using ElasTest in an E-commerce platform and is provided by Atos Worldline. One of the most important aspects for an E-commerce client is to be sure the platform is working correctly and -as expected- from all kind of browsers and versions of those browsers. With ElasTest it is possible to do this without any additional installation. At the same time ElasTest allows the Quality Analyst (QA) to test the application manually or automatically leveraging (e.g., on Selenium[1]) the most widespread technology for automated testing. Once ElasTest has been installed and configured, visualizing different kind of metrics of the server, like memory consumption or CPU usage, is really straightforward for the QA. Saving logs from all executions, having all the execution information correctly saved for further analysis for the near future is done automatically and makes different deployments and result comparison really easy.
- The 5G Networks Vertical Demonstrator is driven by the research institute Fraunhofer FOKUS. It addresses the challenge of deploying innovative services in the telecommunication domain. The expected KPIs for 5G networks include ultra-low latency, high availability, high capacity and ultra-reliability. The Fraunhofer FOKUS Open5GCore toolkit[2] is an implementation of the carrier-grade network towards the 5G environment and its management. It mirrors, in a prototypical form, the pre-standard advancements on the core network, radio network integration, distributed management and virtualization. Its aim is to accelerate research and development as well as knowledge transfer towards partners. A core network consists of multiple components with specific functionalities, therefore a testing platform able to facilitate complex SuT deployments as well as end-to-end performance tests is required. By making use of the Elastest SuT deployment, monitoring and test orchestration features, the task of analyzing the performance of the Open5GCore network deployment becomes straightforward and uncomplicated. This allows the development efforts to remain focused on 5G core network features while the 5G KPIs monitoring for the targeted use-cases is handled in multiple deployment scenarios using the ElasTest Platform.
- FullTeaching is an open-source application to make online classes easy for teachers as well as students. It supports multiple communication channels between teachers and students (forums, chats, and real-time video sessions) and is provided as a WebRTC demonstrator for the ElasTest project by Naeva Tec. “Traditional testing” for this kind of web application that includes real-time WebRTC communications is difficult and won’t provide a completely tested system. We need to be able to test what the user is experiencing during the communication session, not only how the system and the application behaves under specific circumstances. We call this Quality of Experience (QoE) testing. QoE is defined by the International Telecommunication Union (ITU) as “The overall acceptability of an application or service, as perceived subjectively by the end-user” It is a quite abstract definition, but the key is that QoE is subjective to the user and this is the main challenge for automatic QoE testing. The approach we follow, when testing QoE in WebRTC is defining some objective parameters such as transmission rate, delay, jitter, bit error rate, packet loss rate, etc. and assigning them some thresholds by which we consider the communication successful. Currently these parameters are not easy to measure on automatic tests and nearly impossible to use as acceptance metrics. ElasTest is improving the capability to take these measures and is expected to allow us to use them as acceptance metrics on QoE tests. This will make a big difference and bring to the table a whole new set of possibilities for testing QoE on WebRTC communications. It will also give us the ability to increase the overall quality of this kind of applications while reducing time to market.
- The IoT vertical from TUB (Technische Universitaet Berlin) demonstrates the rapid prototyping and testing of IoT applications. IoT applications are built upon the concept of using data from one or more sensors, collectively applying a logic to make a decision and based on the decision signalling an actuator to take action. In the real world, sensors sample physical quantities such as temperature, humidity, and pressure to name a few, while actuators represent a class of devices that receive the order to start or stop or continue operating on other devices. For example, in an application monitoring the temperature of a room, a temperature sensor must be connected to the application via appropriate electronics. Suppose the logic in application is to check whether the temperature is too hot. The sensor data is compared against a threshold, if it is above the threshold, an actuator is signalled, and in its turn the actuator signals an alarm. Composing an IoT application in real world requires the use of real sensors and actuators. Testing IoT applications thus requires the capability to alter operations and conditions of the IoT application and thereby to verify the robustness of application. IoT applications may need to be replicated and scaled up to increase their coverage as a user wants. However, building and testing scalable applications can be expensive and time consuming because of the various physical parts and connections that need to be taken care. ElasTest overcomes the challenge of building and testing scalable IoT applications. The ElasTest Device Emulator Service (EDS) provides emulated sensors and emulated actuators, which imitate their real world counterparts. ElasTest addresses the challenges in scaling up IoT testing by using the emulated sensors and actuators provided by EDS to build and rapidly prototype complex scalable applications. Users can ask for multiple emulated sensors and actuators from the EDS. Then, they wire these in a desired configuration and run them. Thanks to ElasTest, the IoT application can be deployed as a SuT and tested against with test jobs. The tester is capable of injecting faults or test conditions in a running SuT, while the EDS can alter the behavior of any emulated sensor or actuator.
Considering the metrics that have to be assessed, the validation methodology adopted inside the ElasTest project includes three different types of empirical studies that complement each other in evaluating ElasTest results:
- Comparative Case Study (CCS): i.e., an examination undertaken over time useful for a comparison within and across contexts. It involves the analysis and synthesis of the similarities, differences and patterns across two or more cases (in our case two testing processes) that share a common focus or goal.
-Empirical Survey (ES): i.e., a collection of information from a sample of individuals through interviews performed via computer-assisted questionnaires.
- Quasi Experiment (QE): i.e., a controlled study used to estimate the causal impact of an intervention on its target population without random assignment to treatment or control.
Today end-to-end testing of large distributed applications remains effort-prone and mostly manual. ElasTest aims at improving the efficiency and effectiveness of the testing process and ultimately improve the quality of large software systems in the Cloud. ElasTest offers a flexible open source testing platform for rapid and accurate end-to-end testing. ElasTest will reduce the time-to-market of software projects, increasing the quality of the resulting software product, reducing the possibility of failures. It will improve the perception of the software both by the end user and the developers who can perform more complex testing in less time. ElasTest will be demonstrated on four different types of applications, including web, 5G mobile, real-time video communications, and Internet-of-Things.
The ElasTest project is carried out by an European consortium composed of the following partners and institutions. The project is led by the Spanish University Rey Juan Carlos and involves the research Centre Consiglio Nazionale delle Ricerche, the Technische Universität Berlin, Zürich University for Applied Sciences and IMDEA Software Institute. It counts with the participation of different institutions from industrial domain such as Atos Spain & Atos Worldline, Fraunhofer, IBM, NAEVATEC and Relational.
The ElasTest Community supports and fosters the interactions between the core ElasTest members and the community. Anyone can join the ElasTest Community to connect and interact for making ElasTest a reality. The ElasTest community welcomes anyone who want to participate, we expect contributors to the Open Source Community working on the ElasTest platform to make the unique e2e platform!
