Help

Dwarfguard Administrator's notes

Structure of Dwarfguard deployment

Every deployment is composed of two important locations on the filesystem:

  1. /opt/dwarfg_<domain> This directory contains the installation itself including configuration files and certificates.
  2. /srv/dwarfg_<domain> Here you can find all log files and also any application variable data

The <domain> is the address used during deployment (the --domain parameter value) and it should correspond to the external URL of the server - the URL your devices are using to contact the server. To complete the example, if your domain is mydomain.com the directories will be like

  1. /opt/dwarfg_mydomain.com
  2. /srv/dwarfg_mydomain.com

Another important location is a server-shared all-deployment space. That resides again in /opt:

  • /opt/cache_dwarfg - this directory is shared between all deployments on the server.

This directory contains a few important things:

  • /opt/cache_dwarfg/conelos - downloaded firmware files of Conel OS (it "caches" FW download - hence the name "cache")
  • /opt/cache_dwarfg/home - home for the dwarfg user containing shared site configuration (like DB password) and tunnel keys for devices
  • /opt/cache_dwarfg/dwarfg_deployments - providing a list of all deployments
  • /opt/cache_dwarfg/cmdline - containing commands that help to facilitate some administration tasks

Each deployment also creates SQL database named in the same fashion as the name after /opt/ but with the distinction that dots are replaced by underscores, so the DB name for "mydomain.com" would be dwarfg_mydomain_com.


Other places of interest in the system

  1. Deployments backup (if any backup has been performed)
    1. /opt/backup_dwarfg/*
  2. Apache configuration
    1. /etc/apache2/sites-enabled/dwarfg_*
    2. /etc/apache2/sites-available/dwarfg_*
  3. Systemd configuration
    1. /etc/systemd/system/dwarfg_*
  4. crontab for RSS re-generation
    1. /etc/cron.d/dwarfg_*
  5. sudoers so that web and daemon (running under dwarfg user) can perform certain actions
    1. /etc/sudoers.d/dwarfg_


Basic operation

  • To stop Dwarfguard service, use systemctl: 
    systemctl stop dwarfg_<domain>
    OR cmdline integation: 
    dwarfg_stop
  • To start Dwarfguard service, use systemctl: 
    systemctl start dwarfg_<domain>
    OR cmdline integration: 
    dwarfg_start


Deploying Dwarfguard

Refer to Installation page


Fine-tuning Dwarfguard logging (dwarflib)

The dwarflib is configurable via configuration file on the filesystem. First, locate your Dwarfguard installation directory (/opt/dwarf_<domain>). Then you can locate the file under that directory. The file name is dwarflib_cfg.txt

During deployment, 2 files are generated in the installation directory:

  • dwarflib_cfg_prod.txt
  • dwarflib_cfg_dbg.txt

While the mentioned configuration file (dwarflib_cfg.txt) is a symbolic link to the first mentioned file - the production one. These files contains recommended dwarflib settings for production and for debugging a problem. You can simply switch between them by pointing the symbolic link to the one you intend or prepare your customized configuration (see below for more on this). Once you do any change, you need to inform Dwarfguard that it needs to reread the configuration. This can be done either with or without service interruption.

  • without service interruption: /opt/dwarfg_<domain>/dwarfg_ctl.sh --reload-cfg
  • with service interruption:
    • (A) on cmdline: systemctl restart dwarfg_<domain>
    • (B) via UI: System -> Status -> Restart
    • (C) on cmdline: dwarfg_restart

If you open up the file dwarflib_cfg.txt, you will find out it is generously commended textual file. The format resembles a bash source file and next to the comments, it contains only variable definitions. In the first part of the file, variables are documented and the default values are given so that you can return to the defaults any time later on. At the end of the file, the actual variables with values are defined. You are advised to perform your changes there.

A few remarks:

  • the DBG_GENERIC applies to all log messages (all areas). If you set this to e.g. MDL_INFO, all log messages on the MDL_INFO level and more important will be stored in the debug file, regardless of which area they belong to.
  • by using DBG_AREA_* variables you are able to set more detailed logging for a particular area than what is set via the DBG_GENERIC.
  • If you want to inspect what is happening on the system or want to do any troubleshooting, your first action should be to disable the debug message throttling - set DBG_THROTTLE=0
  • If you want to stop the debug messages being cut to a maximal length, you may set the DBG_NOLIMIT=1
  • If you want to review the data send by individual devices and responses sent by Dwarfguard back to the devices, look at DBG_DUMPDATAPUSH and DBG_DUMPDATARESP
  • Do not set DBG_MEMIN on production deployments.


Handling Dwarfguard license (UI/licman)

Dwarfguard license operations could be handled:

  • via UI under System -> License management
  • using the licman binary on commandline. The binary is part of the Dwarfguard deployment.
  • directly replacing the license file using commandline

Out of the options above, the UI is the most user-friendly. However, the license management for the UI admin may be disabled if the customer is using SaaS. In SaaS mode, only server admin is able to replace license using the other options mentioned above. Note that UI is internally calling licman as that is the most flexible tool for license management.

If you need (as a server admin) simply to replace the license file, you can do it this way:

  • backup the license file /opt/dwarfg_<your_domain>/dwarfg_license.lic
  • OPTIONAL: check the new license before replacing the old one with licman -n -c <license_file_path> as seen below.
  • overwrite existing license with new license file
  • restart Dwarfguard service using systemctl restart dwarfg_<your_domain> or simply dwarfg_restart

All other operation can be handle using the licman tool. The usual form of using the tool is providing at least one option and a path to the license file.

  • licman -d ... dump the license file provided or your actual license if no path provided. Also checks license signature.
  • licman -c ... check the provided license. Useful when used together with -u for detecting license differences before upgrading to a new license (checking if the new license is worse in some aspects).
  • licman -u ... upgrade to the new license provided. If license is perceived as worse in at least one aspect, upgrade is not performed (see force option below).
  • licman -n ... not a standalone option. Tells the tool to not attempt to find out the actual number of devices from DB. Useful prior to the Dwarfguard installation.
  • licman -f ... not a standalone option, use together with -u / -r to force overwriting the license even if the new one is worse in some aspect.
  • -b / -r ... backup and restore the license (copies the license to/from a defined backup filename)
  • licman -h ... prints help

Please do not forget to restart Dwarfguard service after exchanging the license.


Backup and restore

  1. Backup
    1. Locate your Dwarfguard deployment directory - /opt/dwarfg_<domain>. In our example, the domain is dwarfg1.dwarfgate.com
    2. Execute backup. Backup is part of the functionality of the dwarfg_upgrade.sh script. You can also use cmdline integration - just dwarfg_backup 
      root@pecka:~# /opt/dwarfg_dwarfg1.dwarfgate.com/dwarfg_upgrade.sh backup
Backing up existing Dwarfguard installation from /opt/dwarfg_dwarfg1.dwarfgate.com to /opt/backup_dwarfg/dwarfg_dwarfg1_dwarfgate_com%0.7.2
Stopping dwarfg service (dwarfg_dwarfg1.dwarfgate.com.service) ...
Preparing target directory ...
Copying dwarfg data ...
Dumping dwarfg DB ...
Restarting dwarfg service ...
    3. In case of consecutive backup, the old backup is rotated and you can see line like this after the Preparing target directory... message. 
      Rotating old backup to "/opt/backup_dwarfg/rot_1676881477_dwarfg_dwarfg1_dwarfgate_com%0.7.2"...
    4. Backup is (as you noticed probably) done into /opt/backup_dwarfg directory. Also, backup introduces a brief service disruption as he service is stopped before the backup and started again once the backup finishes.
  2. Restore
    1. Restore is also performed by the dwarfg_upgrade.sh script (or using dwarfg_restore command). However, you need to execute the script directly from the backup directory. If you attempt to execute dwarfg_upgrade.sh residing in another location, it will fail. Note that the data content of the backup is situated in the /opt/backup_dwarfg/<backup_name>/data path.
    2. Restore operation work automatically for a single-deployment scenario (up Dwarfguard version 0.7.1). If you have multiple Dwarfguard deployments on your server, the restore will work automatically only for the first deployment. As for the additional deployments, you will need to correct the port numbers after restore and restart the apache service.
    3. If you happen to execute the restore while the installation is still in place, the restore will fail early and will NOT impact the running service. Example: 
      root@pecka:~# /opt/backup_dwarfg/dwarfg_dwarfg1_dwarfgate_com%0.7.2/data/dwarfg_upgrade.sh restore
Dwarfguard version/domain: 0.7.2/dwarfg1.dwarfgate.com. Reserving deployment (dwarfg_dwarfg1_dwarfgate_com)
Reserving deployment failed (conflict: 1_dwarfg_dwarfg1_dwarfgate_com), aborting restore.
    4. NOTE that the restore operation is always interactive and waits for the admin to confirm selected deployment number. In case of a single-deployment scenario, just press enter. The successful restore looks like this: 
      root@pecka:~# /opt/backup_dwarfg/dwarfg_dwarfg1_dwarfgate_com%0.7.2/data/dwarfg_upgrade.sh restore
Dwarfguard version/domain: 0.7.2/dwarfg1.dwarfgate.com. Reserving deployment (dwarfg_dwarfg1_dwarfgate_com)
DEPS is 
Deployment 1_ not found in DEPS, selecting...
Selected deployment number: 1
Restoration deployment number: 1
Please confirm the deployment number above. Port offset and `dwarfgd` port are calculated from the above.

      Updating offset in /opt/backup_dwarfg/dwarfg_dwarfg1_dwarfgate_com%0.7.2/data/base_defs ...
Updating Dwarfguard daemon local port in /opt/backup_dwarfg/dwarfg_dwarfg1_dwarfgate_com%0.7.2/data/base_defs to 8484...
dwarfg:x:127:
Restoring Dwarfguard data... 
Restoring Dwarfguard database... 
Users: 1, resetting dwarfg user password to HQp5nSqUUbPV
Enabling site dwarfg_dwarfg1.dwarfgate.com.
Adding dwarfg service and starting it up ...
Created symlink /etc/systemd/system/multi-user.target.wants/dwarfg_dwarfg1.dwarfgate.com.service → /etc/systemd/system/dwarfg_dwarfg1.dwarfgate.com.service.
● dwarfg_dwarfg1.dwarfgate.com.service - Dwarfguard core service
Loaded: loaded (/etc/systemd/system/dwarfg_dwarfg1.dwarfgate.com.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2023-02-20 09:27:48 CET; 4s ago
Main PID: 1282545 (dwarfgd)
Tasks: 8 (limit: 37736)
Memory: 35.8M
CPU: 132ms
CGroup: /system.slice/dwarfg_dwarfg1.dwarfgate.com.service
     └─1282545 /opt/dwarfg_dwarfg1.dwarfgate.com/dwarfgd
Dwarfguard restored and on-line


Moving Dwarfguard deployment to another server

NOTE this covers only the single-deployment scenario. While you still can apply similar process for multi-deployment scenario, it will involve manual steps around ports and DB passwords.

The move of the deployment to another server can be (as of Dwarfguard v. 0.7.1) achieved using these steps:

  1. Backup of the Dwarfguard deployment on the first server
  2. Copy of the backup data to the second server
  3. Restore of the backed-up Dwarfguard on the second server

While that seems trivial, it is not. You need to consider both technical and non-technical aspects that needs to be handled:

  1. Machine preparation
    • Restore operation is not an installation operation. To prepare the second server you need first:
      • Deploy Dwarfguard there (e.g. using the free license) to get all the required packages and directories prepared. NOTE that you should use the same version you are going to restore. You can use deploy with no domain (just execute ./install.sh from the Dwarfguard package)
      • Remove the installed Dwarfguard by calling /opt/dwarfg/cleanup.sh --yes
  2. DB password
    • NOTE that the restore operation restores the dwarfg user password in DB during restore operation, which is fine when performing backup-restore operation, however you are attempting backup-move-restore operation. As the DB password on the second server was generated during the free package installation, it differs from the password on the first machine. While the restore operation restores the DB password in database, it does not (at last until version 0.7.2) overwrites the site-wide settings so you need to update it manually:
    • Look into /opt/cache_dwarfg/home/.dwarfg_site_config on the first server
    • Copy the password into the same file on the second server
  3. Restore
    • After copying the backup (e.g. for domain one.two.com), execute 
      /opt/backup_dwarfg/data/dwarfg_upgrade.sh restore
  4. DNS and Certificates
    • The restore operation naturally restores the URL/domain. Also, all the devices have agents installed and will continue using the URL defined there, so:
      1. In the case you are replacing the server/VM and it will use the same IP, you are fine.
      2. If you are also migrating to a different IP address, you need to update your DNS resolution scheme so that the requests send from the devices will end up on the new server.
      3. In case you are changing the URL completely (to another one), you will need to either perform configuration update on every agent or reinstall each of the agents.
    • Also note that in the default setting, the agent on most platforms require a valid HTTPS certificate. Follow one of the scenarios:
      1. If your certificate is handled on your SSL-termination proxy, you may need to update configuration there depending on your settings. You are done in this case, nothing more should be necessary.
      2. If you are using Dwarfguard self-signed certificate setup, the certificate is restored during the restore operation. However, do not forget that for the self-signed certificate, the public counterpart of the certificate is contained in the agents so that they could verify the server. You cannot change/update the self-signed certificate on the server only as that would render the agents refusing to send the data to the server (not matching certificate). If you update the self-signed certificate (using System -> Setup -> Server -> Server IP address or FQDN), you will need to distribute the updated certificate to all agents. Unless you already lowered security setting on your devices, you need to perform configuration change on every device. Your possible optios are:
        1. Set g_security=1 on all agents. With security level set to 1, agents will accept certificate exchange. The process is still using HTTPS (so it is encrypted) and it also utilizes certification-exchange tokens generated during device registration so it is pretty secure. However, it still constitutes a lower security level than default (2).
        2. Download pre-set agents for all devices one by one and reinstall them. The devices will keep their DeviceID and all data.
        3. Reinstall agents on all devices. The devices will be registered as new and no data will be kept. You would also need to delete all the old devices unless you want them be shown as Late all the time.
  5. Legal
    • Please note you are legally-bound by the EULA that you accepted during any Dwarfguard installation that you cannot use a license for more than one deployment at a time. Therefore, you should stop Dwarfguard service on the first server as soon as you move your deployment to the second server.


Upgrading Dwarfguard

A few important notes regarding upgrading Dwarfguard:

  1. The upgrade process is automatically preceded by backing up existing Dwarfguard deployment. In case anything goes wrong:
    1. Upgrade script attempts to cleanup the unsuccessful upgraded deployment
    2. After cleanup, a restore operation is attempted
    3. If the automatic cleanup+restore fails, don't despair. You have a fresh full backup available and thus you definitely can restore manually afterwards (see Backup and Restore chapter above).
  2. The upgrade can be triggered only from the commandline as it may require interaction with administrator.
  3. The new Dwarfguard version may or may not be supported by your license. While you can restore old version and old license from backup, it is recommended to make sure you have license that supports the version you are upgrading to at hand.
  4. A version x.y.z+1 is guaranteed to support upgrade from version x.y.z and version x.y+1.0 is guaranteed to support upgrade from x.y.z where z is the last patchlevel in x.y.* branch.

As for the process itself, the instructions to be followed:

  1. Copy the new version's installation package to the machine where the current Dwarfguard version is deployed.
    1. NOTE that since version 0.7.2 all official release packages follow the convention dwarfg_release_x.y.z.tgz (for beta builds it is dwarfg_install_x.y.z.tgz)
  2. Uncompress the package and run dwarfg_install/upgrade.sh script.
  3. You will see the information that the upgrade MUST be triggered from the installed Dwarfguard, which is true. The upgrade script from the new package will, however, scan for your existing deployments and give you the exact command you can immediately use to start the process for each one of the deployments found. In case of a single deployment, it offers to autostart the upgrade for you: 
    Found 1 Dwarfguard deployment(s).
To upgrade domain dwarfg_pecka.otn, use command on following line:
/opt/dwarfg_pecka.otn/dwarfg_upgrade.sh upgrade /tmp/dwarfg_install/appdir/
One deployment found. Do you want to autostart the upgrade? (Y for yes)
  4. Answering by Y starts the upgrade process
  5. Upgrade process always starts by performing a backup (rotating old backup if present). Also, a test for the upgrade path is executed (meaning: the upgrade process test if the upgrade from your existing version to the new version is supported) and the license is tested using the new version's licman to check if your existing license supports the new version.
  6. In case your old license does not support the new Dwarfguard version but you have a newer license prepared, the following is recommended prior upgrade (or after the first upgrade attempt fails):
    1. Stop running Dwarfguard service
    2. Overwrite the original license (after backing up of course) by the new license. The file with license resides under your deployment installation directory (depends on the domain you are using, but beginning with /opt/dwarfg_ always, so it could be /opt/dwarfg_mydomain.com) and the name of the file is dwarfg_license.lic
    3. (Re-)start the upgrade
  7. Upgrade finishes automatically, creating (or appending to already existing) upgrade log at `/opt/cache_dwarfg/home/upgrade_log.txt
  8. The newly upgraded deployment will be started automatically.
  9. Please note that some of the Dwarfguard server parameters are intentionally not kept from the old version because the new version may come with defaults better fitted for the new version. The parameters not kept in the base_defs file are these:
    1. LISTENER_THREADS
    2. DEVICE_INTERVAL_DEFAULT
  10. Also please note that other server configuration files are currently not processed during the update as they may contain no-longer valid options, the file syntax could have changed in the new version or there are simply too many new content in these files. Don't forget you have the full backup handy and in case you did some configuration there in the previous version, just compare the files in the backup and the new ones after upgrade finishes, applying your changes and going through new configuration parameters available. The files are:
    1. dwarfg.ini
    2. user.ini


Getting Firmware files into on-premise (no Internet) installation

While the on-premise deployment that has no access to Internet cannot obviously download the firmware files, you can copy them from one machine to another, using e.g. a flashdisk. Note the following instructions are for Advantech cellular router's firmware.

  1. Next to your on-premise installation, deploy Dwarfguard to another machine that has Internet access. Use the free license for this.
  2. Note down the FW version you want to make available in your on-premise deployment
    1. E.g. a router comes with not-yet-available version 6.2.9, note down 6.2.9
    2. If you want to be able to push FW version 6.3.4 to your devices, note down 6.3.4
  3. On your Internet-enabled free Dwarfguard deployment, add the version you want to download (e.g. 6.3.4)
  4. Wait until the FW version becomes available (until download is finished)
  5. Copy the downloaded Firmware from the /opt/cache_dwarfg/conelos directory by copying the complete directory from there to the medium (flashdisk). In our case, that would be the 6.3.4 directory. The command would look like 
    cp -r /opt/cache_dwarfg/conelos/6.3.4 /my/flashdisk/location
  6. Attach the medium to the on-premise machine with no Internet connection where Dwarfguard is deployed.
  7. Copy the whole directory (6.3.4) to the cache directory. E.g. 
    cp -r /flashdisk/mountpoint/6.3.4 /opt/cache_dwarfg/conelos
  8. If case there is no entry in the SW Manager -> Firmware table (yet), add the Firmware version there. This way, the firmware download attempt starts immediately, FW is found in the cache directory and set as available immediately.
  9. In the case the entry in the table was there already, the firmware version appears as available once the next download attempt is performed. Next FW download attempt may come in a few minutes or the next day (depending on the number of failed attempts, the interval for next download is prolonged)
    1. NOTE that you can simply restart Dwarfguard service (System -> Status -> Restart) to kickstart the download attempt immediately.


Cmdline integration

The command line integration is stored in /opt/cache_dwarfg/cmdline.

Provided commands are useful for regular administration tasks and then some. It is also usable in the multi-deployment scenario.

The command dwarfg_help will give you the necessary documentation on how to use it.

Copyright (c) 2023 by     Wiki is powered by Fossil SCM