Dwarfguard testing
At Dwarf Technologies we know that testing is inseparable part of development and no release we dare to designate production can make it without release and performance testing.
There are primarily two testing types done for Dwarfguard.
- Functional testing - these tests focus on validating that Dwarfguard does what is expected for a scenario being tested. To look under the hood what end-to-end scenarios are being tested, inspect the table below.
- Performance testing - by saying performance testing, we actually mean three different types of so-called non-functional testing. It is called this way because that tests are not focusing on validating some specific functionality but measure other metrics. More on that below.
Dwarfguard End-to-End scenario test table
NOTE Any release testing must be done on production deployment of two types:
- DT servers with domain, DNS, certificate and HTTPS handled by DT-internal proxy.
- Stand-alone installation with proper domain, DNS and certificate setup on-box.
| ID | Components | Scenario | Covered versions | Scenario description link |
| 01 | M--- | Download and install agent - Advantech routers | 0.3+ | details |
| 02 | MA--- | Setup monitoring group | 0.3+ | details |
| 03 | M-M-- | Add firmware - manual | 0.3+ | details |
| 04 | --MAS | Add firmware - automatic + FW flash | 0.3+/0.5+ | details |
| 05 | -A--- | Setup alert | 0.3+ | details |
| 06 | --MA- | Clone configuration (router to router) | 0.3+ | details |
| 07 | M---- | Using labels | 0.4+ | details |
| 08 | M---- | Naming and searching devices | 0.4+ | details |
| 09 | -A--S | Disabling alerts | 0.4+ | details |
| 10 | --M-S | Open, use and close webtunnel | 0.4+ | details |
| 11 | -A--S | E-mail and slack notifications | 0.5+ | details |
| 12 | ----S | Access FRW and help | 0.5+ | details |
| 13 | MA--S | Dashboard | 0.5+ | details |
| 14 | ----S | Access log information | 0.7+ | details |
| 15 | ----S | License management | 0.5+ | details |
| 16 | M---- | Changes Tracker | 0.7+ | details |
| 17 | M---- | Agent profiles - assign and edit | 0.8+ | details |
| 18 | M---- | Server command-line tests | 0.8+ | details |
| 19 | M---- | Traffic dashboard in Grafana | 1.0+ | details |
| 20 | M---- | Send configuration SMS via Dwarfguard | 1.0+ | details |
| 21 | M---- | Send SMS to arbitrary MSISDN via Dwarfguard | 1.0+ | details |
Performance testing
As noted above, there are three separate test types in this area:
- Stability tests - these tests try to emulate real-world environment as close as possible, answering how stable is Dwarfguard deployment on a particular setup.
- Benchmark tests - here we try to measure performance between releases and between different setups (e.g. HW)
- Peak tests - these tests prove that Dwarfguard is able to serve a data peak of defined amplitude and size and also help to estimate what are the boundaries by looking for critical thresholds for a given setup.
As the results could be very interesting when we need to find out what HW or what container memory/CPU we need to dedicate to be able to handle a particular number of devices, the performance tests do have dedicated page.