Backup Troubleshooting

VSS Troubleshooting Guide

Error pattern:
The backup is aborted or is completed with errors. VSS errors/errors are displayed in the logs (e.g.: VSS -E-05408, VSS -E-05591).
VSS errors can take many different forms.

Possible causes:
The VSS providers are used by another process (other backup software, other backup jobs). The VSS providers are not fully present or are defective. The agent requires functional VSS providers. We rely 100% on Microsoft VSS for this - our Agent does not bring its own VSS providers. If there are problems on the part of Microsoft VSS, it can lead to errors occur.

Recommended Action:
1. check the schedule. It is not possible to start multiple backup jobs at the same time on the system, as the VSS components will then compete.
Restart the server once. Often a reboot helps to solve VSS problems.
3. install all available Windows updates.
Additional memory is required for VSS snapshots. Check if you have enough free memory on the system (at least 25GB).
Disable third-party backup software (Acronis, Backup Exec, etc.).
6. the size limit of the shadow storage should be adjusted. Run the following command per partition:
vssadmin resize shadowstorage /for=<drive letter:> /on=<drive letter:> /maxsize=10GB

Example:
vssadmin resize shadowstorage /for=c: /on=c: /maxsize=10GB
vssadmin resize shadowstorage /for=d: /on=d: /maxsize=10GB
7. delete old VSS snapshots with the command:

vssadmin delete shadows /all

8. run the following command and send us the output: vssadmin list writers and vssadmin list providers
After setting, you would need to restart both TERRA CLOUD BACKUP services once.
Check the Windows Event Log. Look for "vss" and "volsnap" errors that occur around the time of the backup.

Often it helps to search the Internet for the event ID of the error to get a hint for the actual cause of the error.

9a. Provided you have configured a file-based BMR job:
Install the image plug-in and create an image-based BMR job. Test whether the behavior also occurs with it. 9b. If you have configured an image-based BMR job: Uninstall the image plug-in and create a file-based BMR job.
Please include only "Bare Metal Restore" in the job. Test whether the behavior also occurs with it. 10. create VSS snapshot manually:

On Windows Server operating systems a VSS snapshot can be created via PowerShell, we recommend to test this once in case of VSS problems.

You can create a snapshot manually with the following command:

vssadmin create shadow /for=C:

As an example, we have created a snapshot for C:, if this is not possible, there are fundamental VSS problems on the system that are not caused by the TERRA CLOUD Backup Agent.

VSS graphical user interface

In addition to accessing the Microsoft Shadow Copy Service through PowerShell, you can also access it through a graphical interface.
Via this interface you can e.g: Configure the shadow storage on which the shadow copies are stored (e.g. define the size).

1. Manually create shadow copies, e.g. to check the functionality

From the properties of a volume, you can access the graphical interface via the "Shadow Copies" tab:

Force system VSS provider

Using the following registry key, TERRA CLOUD Backup Agent always uses Microsoft Shadow Copy Provider 1.0:

HKEY_LOCAL_MACHINE\SOFTWARE\EVault\InfoStage\Agent\UseSystemVSSProvider = 1 (DWORD)

Please restart both TERRA CLOUD Backup services after this configuration:

• TERRA Cloud Backup BUAgent
• TERRA Cloud Backup Agent

Hyper-V VSS Writer exclusion

Error pattern:
Backup jobs with the "BMR" option fail due to Microsoft VSS error messages on a Hyper-V host.
Problem:
Only the system partition on a physical Windows server ("host") with the Hyper-V role enabled is backed up.
The virtual machines (located on other partition) are explicitly backed up using the agent.
The Backup Agent on the HOST during the backup of drive C: also puts the VMs into the "backup in progress" state, i.e. activates a VSS functionality.

Cause:
The backup agent only triggers the Microsoft VSS provider to put the system itself (not "just" what was included) into a consistent state.
At the time of snapshot creation, the operating system does not "know" that actually only C is affected.
Only the agent reads, compresses, encrypts, and transfers from the VSS snapshot only the data included in the job.
According to current knowledge, the Hyper-V VSS Writer is responsible for the described behavior.

Solution:
A BMR backup of the Hyper-V host is not recommended, for a separate post on this see: Hyper-V Server Host BMR Backup. If you still want to backup the C partition, please exclude the Hyper-V VSS Writer from the backup:

Hyper-V VSS Writer Exclusion Guide:

1.) Stop the TERRA CLOUD Backup Agent services using Powershell:

Get-Service -DisplayName "TERRA Cloud Backup*" | Stop-Service

2.) In the installation directory you will find the job configuration files (<jobname>.vvc)

Example:

• "BMR.vvc".
  

Create a copy of the job file, e.g. with the name "BMR.vvc.backup"

Edit the job file "BMR.vvc" and add the following part to the end:
VSSWriterBackup001 {
Name = Microsoft Hyper-V VSS Writer
InstName =
ID = 66841cd4-6ded-4f4b-8f17-fd23f8ddc3de
Include = ".\*"
Options = 4
}
Provided that it is a cluster, there is certainly still a writer in the system for the cluster shared volumes (you can find out via vssadmin list writers).
Provided that this is the case, this Writer should also be excluded:

VSSWriterBackup002 {
Name = Cluster Shared Volume VSS Writer
InstName =
ID = 1072ae1c-e5a7-4ea1-9e4a-6f7964656570
Include = ".\*"
Options = 4
}

3.) Start the TERRA CLOUD Backup Agent services using Powershell:
Get-Service -DisplayName "TERRA Cloud Backup*" | Start-Service

4.) Perform a backup

5.) If the backup was not successful or the problem still occurs, please restore the previously saved job configuration file.

VSS -E-05600 - System Writer missing - BMR not available

Error pattern:

You will receive the following error message in the log file:

VSS -E-05600 Components of System Writer - no prescribed writer present (Writer ID E8132975-6F93-4464-A53E-1050253AE220)

Additionally in your backup job in the backup selection that BMR is not available:

The Microsoft VSS System Writer is also not displayed in the Writer listing.
You can view the VSS Writer either in PowerShell or in CMD with the following command:

vssadmin list writers

Cause:
The causes of a missing system writer can be very different, so each case must be considered individually.
Please note that a working Microsoft VSS service is the basis for all application consistent backups via TERRA CLOUD Backup.
This error pattern is basically a problem with the operating system itself and not with the backup agent.

Recommendations for action:

• Please reboot the system and check again the VSS Writer
• If the Writer is still not present, please check the Windows system event logs for Microsoft error codes
• Try to back up the system via Windows Server Backup
• Search with a Windows error code

Recommended Microsoft contribution to this error image:

Missing System Writer Case Explained

Microsoft error code 0x80070018 associated with a missing System Writer

Error pattern:
You will find the error code 0x80070018 VSS Error XML document is too long in the Windows Application Logs.

Recommended Action:
Please check the following article from Microsoft about this error pattern:

VSS Error XML document is too long

Note: The System Writer is a part of the operating system, so we can only give you recommendations. Please contact Microsoft Support to resolve the issue.

Backup, share is automatically included in BMR

Error pattern:
In the job configuration, a network share appears below the "BMR" option, in addition BMR backups fail with VSS errors.

Recommended Action:
A registry key can be used to exclude a network path from the VSS snapshot, for example
1. windows + R to open the task "Run" and enter them "regedit".
2. navigate to the path HKEY_LOCAL_MACHINE\SOFTWARE\EVault\InfoStage\Agent.
3. add the string named WriterSelectionToIgnore.
4. start the extension with ^
5. add an \ for each \ in the located path.
6. do not forget the * - symbol at the end.

e.g.: ^\\\\Name\\Folder1\\*.*

If no remedy is found after this configuration, it is possible that a service is stored on a network path. To do this, please check the system in question with the PowerShell query:
Get-WmiObject win32_service | select Name, PathName This command checks the storage path of the stored services, these may only be located on locally mounted volumes.

In this case, please also check the Microsoft VSS Writer details for network shares. Instructions on how to read these details can be found here:
Read details of Microsoft VSS Writer

DISK-E-04205 - Error searching for file GLOBALROOT\Device\HarddiskVolumeShadowCopyXY\windows\\systemroot\system32\drivers\*

Error pattern:
DISK-E-04205 Error searching for file GLOBALROOT\Device\HarddiskVolumeShadowCopyXY\windows\\system\root\system32\drivers\%NAME%
DISK-E-04101 Operating system message: The syntax for the file name, directory name, or volume name is incorrect.

Cause:
The entry or reference in the registry indicates a non-existent or incorrect path.

Solution approach:
An adjustment of the path in the registry is necessary, you can find the respective reference under:
Computer\HKEY_Local_Machine\System\CurrentControlSet\Services\%DATEINAMEAUSErrorMESSAGE%

We recommend that you export the corresponding entry before making any changes in the registry.

Often an adjustment to the following entry is sufficient, if necessary you have to check the existing path and adjust the entry specifically:

%systemroot%\system32\DRIVERS\%FILEINAMEAUSERMESSAGE%.sys

After the entry has been adjusted, the system must be restarted to apply the change.

VSS -E-05658 VsscBackup::createSnapshotMap 2 internal error

Error pattern:
You will get the following error image in the log file:

VSS -W-05408 Error adding volume to snapshot set: volume <\?\Volume{GUID}\> <\?\Volume{GUID}\> (BS error: 0x8004230c VSS_E_VOLUME_NOT_SUPPORTED )
VSS -E-05658 VsscBackup::createSnapshotMap 2 internal error
SV -E-09487 GetVolumeInformationW function failed for volume{GUID} (Operating system error: The device is not ready)

Cause:
The Microsoft VSS service cannot create a snapshot for the complained volume because the volume (e.g. a USB stick) is not supported. This can be caused, for example, by the "Entire server" option in the backup set of a file-based backup job, since volumes are included in the backup for which no VSS snapshots can be created. In this case, please remove the option from the backup set and add the volumes to be backed up individually.

The error image regarding the "Entire server" option in connection with a file-based job has already been forwarded to the software manufacturer..

Recommended Action:
Please check which volume it is based on the GUID, you can use the following PowerShell script:

Write-host "Please enter the GUID of the volume(the contents of the curly brackets)"
$GUID = Read-Host
$pattern = "\?\Volume{GUID}\"
$pattern = $pattern.Replace("GUID", $GUID)
Get-Volume -UniqueId $pattern | Select-Object -Property UniqueId,FileSystemLabel,DriveLetter | Out-GridView

File cannot be checked for hard links (Windows\System32\LogFiles\Sum\无ǽ)

Error pattern:
You will receive the following warning message in the log file: "File cannot be checked for hard links: \GLOBALROOT\Device\HarddiskVolumeShadowCopy14\Windows\System32\LogFiles\Sum\无ǽ (OS error (0x2): The system cannot find the specified file)."
Cause:
The operating system cannot check the listed file for hard links due to the special characters.
Solution:
These are standard files of a Microsoft installation, which are not relevant for the function of the server.
Thus, you can delete the \Sum folder (Windows\System32\LogFiles\Sum\) without any problems.
After that, the error will no longer appear in the log file and the backup will thus run through successfully again.

FAILED_AT_FREEZE - VSS_E_WRITERERROR_TIMEOUT)> 0x800423f2

Error pattern:
You will receive the following error message in the log file:

FAILED_AT_FREEZE(0x800423f2 -VSS_E_WRITERERROR_TIMEOUT)

Cause:
A VSS writer times out when trying to bring its application into a consistent state. A cause for this can be performance problems of the system.

Recommended Action: Please follow the usual steps for resolving VSS errors. If the error persists in subsequent backups, we recommend increasing the timeout for the VSS snapshot. The following article explains the increase of the timeout: https://blogs.technet.microsoft.com/asiasupp/2011/08/01/windows-server-backup-failed-to-backup-with-error-0x81000101/

VSS -E-05408 - VSS_E_UNEXPECTED_PROVIDER_ERROR

Windows Agent

Error pattern:
You will receive the following error message in the log file right after starting the backup:

VSS -E-05408 Error adding volume to snapshot set: volume <\?\XY}\> <\\?\Volume{XY}\> (BS error: 0x8004230f VSS_E_UNEXPECTED_PROVIDER_ERROR)
---
VSS -E-05408 error adding volume to snapshotset: Volume <\?\XY}\> <\?\Volume{XY}\> (OS error: 0x8004230f VSS_E_UNEXPECTED_PROVIDER_ERROR)

Cause:
TERRA CLOUD Backup Agent is trying to use an incorrect VSS provider.

Solution:
Please use the following command via CMD or PowerShell to check which VSS providers are stored on the system.

vssadmin list providers

The TERRA CLOUD Backup Agent must use Shadow Copy Provider 1.0.

Provided there are other VSS providers on the system, the following REG key can remedy the situation:
The entry ensures that the Shadow Copy Provider 1.0 is used when creating the snapshot for the TERRA CLOUD Backup Agent.
This must be created manually.

Hyper-V Agent

Error pattern:
You will receive the following error message in the log file right after starting the backup:

VSS -E-05408 Error adding volume to snapshot set: volume<\> <\>Volume{XY}\> (BS error: 0x8004230f VSS_E_UNEXPECTED_PROVIDER_ERROR)[Host1]
---
VSS -E-05408 error adding volume to snapshotset: Volume <\?\XY}\> <\?\Volume{XY}\> (OS error: 0x8004230f VSS_E_UNEXPECTED_PROVIDER_ERROR)[Host1]

Cause:
TERRA CLOUD Backup Agent is trying to use an incorrect VSS provider.

Example:

Solution:

1. Determine the provider identifier (provider ID) from the output of the following command: vssadmin list providers
2. Run the registry editor (regedit)
3. Take a backup of the affected registry entries.
4. Delete the corresponding registry key from the following path: HKLM\System\CurrentControlSet\Services\VSS\Providers
5. Scan the registry for the provider ID and remove all existing references.
6. Open Component Services by typing "dcomcnfg" at the command prompt.
7. Open Component Services -> Computer -> COM+ Applications and remove the entry of the third-party VSS provider.
8. Then restart the affected host.

Note: This customization must be done per affected host!

VSS -E-05616 - 0x8004231f VSS_E_UNKNOWN

Error pattern:
You will receive the following error message in the log file:

VSS -E-05616 snapshot could not be created (OS error: 0x8004231f VSS_E_UNKNOWN (OS error (0x8004231F): unknown error code 0x8004231F))

Cause:
This Microsoft error code can be caused by insufficient free space on a backed up volume or insufficient shadow copy space.

Recommended Action:
Please check via vssadmin list shadowstorage how much shadow copy storage is allocated to the volume and clean up or extend the volume until the free space exceeds that for shadow copy storage.

Test whether you can create a shadow copy by right-clicking on the drive in the Explorer "Configure Shadow Copies..." and creating a shadow copy for the volume with "Create Now". A functional BMR backup via TERRA CLOUD Backup can only be created if all volumes can successfully create a shadow copy.

VSS -W-05408 - 0x80042317 VSS_E_MAXIMUM_NUMBER_OF_SNAPSHOTS_REACHED

Error pattern:
You will receive the following error message in the log file: VSS -W-05408 Error adding volume to snapshot set: Volume <\?\Volume{XY}\> <\?\Volume{XY}\> (BS error: 0x80042317 VSS_E_MAXIMUM_NUMBER_OF_SNAPSHOTS_REACHED)

Cause:
Microsoft VSS allows a maximum of 64 shadow copies on one system. It is not possible to request another shadow copy for the TERRA CLOUD backup, the agent passes on the message of the operating system in the log file.

Recommended Action:
Please check how many shadow copies exist on the system, for this you can enter the following command with administrative privilege in the command line:

vssadmin list shadows

These shadow copies should either be moved to another shadow copy store or deleted.
If another software solution based on Microsoft VSS is used, it should be ensured that there is no overlap between the backup windows.

Recommended Microsoft contributions to VSS

For comprehensive documentation on the Volume Shadow Copy Service, see:

Microsoft VSS Dokumentation

Read details of Microsoft VSS Writer

Each VSS writer is responsible for its own area, e.g. the system status or the registry. With the help of the following instructions, you can find out which directories and folders the respective writers are responsible for. This evaluation can be very helpful when analyzing a VSS problem, e.g. to locate an .exe file on a network share. Please note that "Diskshadow" is only available on server operating systems.

Instruction:

1. Please open the CMD with administrative permission
2. Create the "C:\temp" directory
3. Run the following command "DISKSHADOW /l c:\temp\diskshadow.txt".
4. Execute the following command "LIST WRITERS DETAILED".
5. You can find the output in the create "Temp" directory

AFC Logs

AFC stands for Agent Forensic Collector, this tool collects backup and restore log files as well as other important information from a system on which a Windows Agent is installed.
These extensive log files are essential for the diagnosis of many error patterns.

AFC Logs Windows Agent

Download:
You can download the latest version of AFC Tool here:
https://backup.terracloud.de/download/afc.exe

Content and procedure:

1. Run the afc.exe on the system where the agent is installed

Please select the desired backup job via "Select Job".

1. Additionally select "Partition Info", "VShadows" and "Installed Applications", as shown in the screenshot
2. Upload the zip archive to TERRA Drive using the link below, no password is required:
3. AFC Logfile Upload
4. Please provide a short upload confirmation to TERRA CLOUD Support

Note:
The VShadows option is only available on server operating systems.

AFC Logs Hyper-V Agent

Please run the afc.exe on the system where the management component is installed (MGMT Agent).
In the "Custom Forensics" field, enter the path of the installation directory, as in the following screenshot:

AFC Logs Linux Agent

Download:
You can download the latest version of AFC Tool here:
https://backup.terracloud.de/download/AFCforLinux.tar

For most derivatives, you can use the following command to perform the download:
wget https://backup.terracloud.de/download/AFCforLinux.tar

Collect log files:

1. Please unpack the script with the following command: tar -xf AFCforLinux.tar
   
2. Change via cd to the directory where you have unpacked the script
3. Check your permissions, depending on the derivative with su or sudo-i
4. Please change the access rights for the afc_linux.sh script with the command chmod 777 afc_linux.sh.
5. Then run the script via ./afc_linux.sh

Follow the instruction in the script and specify your support ticket number, alternatively you can specify the order number of the backup package

1. Optionally you can add files (e.g. screenshots)
2. The AFC logs can be found in the /tmp/ directory with the name "AFC-< ticket_number >-<hostname>-<YYY-MM-DD-HH-MM-SS>.tar[.gz]".
3. AFC Logfile Upload
4. Please provide a short upload confirmation to TERRA CLOUD Support

TERRA CLOUD Backup Portal

Computer / Backup Agent is shown as offline since 01.01.0001

Error pattern:
The computer / backup agent is shown to you in the portal as "offline" since 01.01.0001, although the services of the agent are active.

The error pattern looks identical for the different agents (Windows, Linux, Hyper-V and VMware):

Cause:
The Backup Agent cannot communicate with the portal servers (AMP servers) via TCP port 8087. However, it was possible to register on the Backup Portal via TCP port 8086, otherwise the system would not be displayed at all.

Solution:
Please release the TCP port 8087 to the outside for the system on which the Backup Agent is installed. For more information, see the Network configuration.

Users can only retrieve their own Vault profiles

Error pattern:
You receive the following error message when attempting to register a system with the Vault:

Cause:
This error message appears because not all portal servers can access this Vault profile yet.
It can take up to 5 minutes to pass the information to all portal components.
Solution:
Logging in again may speed up this process and thus resolve this error message.

UPDT-E-10320 - Could not upgrade the Agent. The system must be restarted

Error pattern:
You will receive the following message in the Backup Portal after starting the upgrade process: You must restart this computer for the change to the agent software configuration to take effect.

In BUAgent logfile you can find the following error code for this:
UPDT-E-10320 Could not upgrade the Agent. The system must be restarted

Cause:
A registry key on the system to be backed up prevents the agent from being updated. This key is usually cleared by a reboot.

Recommended Action:
Please check the following article for this, as the procedure is identical:
Pending Restart

Windows Agent

Windows Agent General

No connection between agent and vault

Error:
When adding a vault to a computer, the error message appears that the agent cannot connect to the vault.

Causes:

• The data entered is not correct.
• Communication is not possible due to blocked ports.

The following ports must be enabled from the inside out for communication from the agent to the Vault to work:

• 2546 TCP
• 2547 TCP (only for replication of the satellite to the basevault).

Review:
Please run the following tests from the system on which the agent is installed:

• ping backup.terracloud.de
• ping vault-wmh1-p002.terracloud.de
• telnet vault-wmh1-p002.terracloud.de 2546

Alternatively via PowerShell:

• test-netconnection vault-wmh1-p002.terracloud.de -port 2546

(Please use the correct Vault address at this point, see email with credentials)

Solution:
Please check the access data, copy them 1:1 from the email "Your TERRA CLOUD Backup Account".
If the connection attempts fail, please release the corresponding ports.

Native protected data amount differs from actual memory consumption

Error pattern:
The natively protected amount of data is incorrectly displayed in the Backup Portal in the job details or in a report.

Unlimited VSS shadow copy storage:
For each volume shadow copy storage is defined by the operating system, the maximum size can be set either as a total value e.g. 10240MB or "unlimited".
Please check if a fixed and therefore not unlimited shadow copy storage is defined for all volumes.

'Large shadow copy memory:
The size of the shadow copy is included in the calculation of the natively protected amount of data.
Particularly large volumes that are hardly occupied create a discrepancy between the actual amount of data occupied and the amount of data measured by the agent.

'Example of a large volume:
You add a new volume with 1TB capacity to a system that you want to back up. For this volume, 50GB is set aside by default for shadow copies. When backing up the entire volume, these 50GB would be included in the calculation of the native amount of data protected. Please reduce the shadow copy storage for this volume to a much smaller value. If the volume occupancy increases, you can always increase the shadow copy storage.

'Cause of deviation for a large shadow copy memory:
Backup jobs that back up an entire volume, e.g. image jobs or BMR backups also measure the reserved space during the backup, which is released after the backup. You can observe this during the backup as the occupied memory increases for the duration of the backup. The development team of our software manufacturer is already working on an optimization of the measurement timing.

Configuration of the shadow copy memory:
You can make the shadow copy storage via the graphical interface.
The limitation for the System Reserved Partition can only be configured via the properties in the graphical user interface, since no drive letter is assigned for it.

Alternatively, you can also perform the configuration via CMD or PowerShell:

vssadmin resize shadowstorage /for=<drive letter:> /on=<drive letter:> /maxsize=10GB

Example:
vssadmin resize shadowstorage /for=c: /on=c: /maxsize=10GB
vssadmin resize shadowstorage /for=d: /on=d: /maxsize=10GB

Windows Server Deduplication:
This Windows Server role enables deduplication of data on the server. You can see the compressed and deduplicated dataset via the Windows Explorer.
However, the Backup Agent transmits the uncompressed original size of the backup set as meta information.

Please use the following PowerShell command to check if the role is installed on the server in question:
Get-WindowsFeature -Name FS-Data-Deduplication

Should the role be installed:
Please exclude the "Chunk Store" folder below the "System Volume Information" directory from the backup. After the next backup is completed, the measured natively protected data volume should be correct again.

Recommendations:
To determine and analyze the used space on a volume, we recommend, in addition to Windows Explorer, the free tool TreeSize Free.

Invalid file format or delta mapping file is corrupted

Error pattern:
Jun 07 22:15:38 DELT-E-04355 Invalid file format C:\Program Files\TERRA Cloud Backup\Agent\JOBNAME\0000xxxx.DTA
Jun 07 22:15:38 BKUP-E-08196 Your delta mapping file is corrupted. Delete the file, or rename it, and synchronize it with your director. [VV]

Cause:
The error message says that the delta file (0000000X.DTA) or the delta assignment file (0000000X.DTX) is damaged. The delta file contains checksums of the already saved blocks, the delta allocation file is an index file for the delta file. This file is needed so that the agent knows which blocks have already been saved and which are new or have changed.

Solution:
Please perform a backup reset, instructions can be found in the following entry for a very similar error: DELT-E-05782 - Delta file damaged 0000000X.DTA file is damaged: hash mismatch

A value is required for Server0.Password

Error pattern:
A backup fails immediately after startup with the following message in the log files:

PARS-E-05152 A value is required for Server0.Password

Cause:
A configuration file (global.vvc) is missing the hash value for the vault password.
Without this entry, the agent cannot authenticate to the vault system.

Solution:
Please store the Vault password in the Backup Portal again, to do this please go to "Vault Settings" and edit the existing configuration:
See also 2.1.1

DELT-E-05782 - Delta file corrupted 0000000X.DTA file is corrupted: hash mismatch

Error pattern:
You receive the following error message in the log file of a backup job:
DELT-E-05782 C:\Program Files\TERRA Cloud Backup\Agent\Jobname\0000000X.DTA file is corrupted: hash mismatch

The error message says that the delta file (0000000X.DTA) is corrupted, this file contains checksums of the backed up blocks of the last backup.
This file is needed so that the agent knows which blocks have already been backed up and which are new or have changed.

Solution:
Please run the following PowerShell script on the agent system:
Backup Reset Script
Among other things, this script deletes the corrupted delta file and performs a synchronization, in the process the delta file is recreated.
A synchronization can take some time depending on the system size.
After the execution, the encryption password must be entered again.

Illegal function

Error pattern:
The affected system is displayed as "Offline" in the Backup Portal, even though it is in use.
The "TERRA Cloud Backup BUAgent" service is not running:



When you try to start the service, you will receive the following error message:



Cause:
The "TERRA Cloud Backup BUAgent" service accesses a configuration file (global.vvc) in the installation directory of the agent, among other things, settings for the agent are stored in this file,
that you have configured in the portal (e.g. a bandwidth limit).
With this error pattern, this configuration file is empty, incomplete or corrupted.
Without a proper global.vvc file, the agent cannot back up.

Recommended Action:
For troubleshooting, the global.vvc file in the installation directory can be opened first, e.g. with the editor. In this case the file is empty.



This empty global.vvc can be removed.
After deleting the file you can perform a repair installation of the agent, to do this you need to enter the setup of the agent and select the following option:



After successful repair installation, the global.vvc file will be filled with content.
Now please store the Vault password again in the Backup Portal, you can find instructions on how to do this under the item "A value is required for Server0.Password"
If the error pattern cannot be solved with these steps, a complete uninstallation followed by an installation of the agent can help, in which the configuration files (including the global.vvc) are downloaded from the vault system and stored in the agent directory.

Hyper-V Server Host BMR Backup

Backing up physical Hyper-V hosts that are actively running VMs has been known to cause problems with Microsoft VSS. Microsoft VSS tries to put the COMPLETE system (including CSVs) into a consistent state.
To enable this, all virtual machines would therefore also have to be paused. Since no VSS snapshot is triggered via the Integration Services via the agent-based backup, there are problems here.

The question generally arises as to why hypervisor servers should be backed up at all.
Enclosed are sample articles from the web:

• http://www.altaro.com/hyper-v/hyper-v-backup-strategies-dont-worry-management-os/
  
• https://www.backupassist.com/blog/support/10-tips-for-best-practice-hyper-v-backups/
  
• https://www.veeam.com/blog/hyper-v-hosts-to-back-up-or-not-to-back-up.html
  

Individual files and folders can be backed up without problems - only BMR has problems.

For the reason it is also not supported See function overview
With host-based backup from version 8.60 ONLY the VMs can be backed up, not the host itself (also no files and folders).

'We recommend agent-based backup (from within the VM) to take full advantage of the backup solution's features.
Provided a Hyper-V host fails completely, it should be reinstalled and added back to the cluster.

Backup jobs pile up in a queue at

Error pattern:
You see several processes in the queue in the Backup Portal for one system, the current backup job no longer shows any progress.

Recommended Action:
Please restart the TERRA CLOUD Backup services so that the open processes are terminated.

Get-Service -DisplayName "TERRA Cloud Backup*" | Stop-Service
Get-Process | ? ProcessName -Match "VV|VVAgent" | Stop-Process -Force
Get-Service -DisplayName "TERRA Cloud Backup*" | Start-Service

SMTP Prober / Mail Notification

Error pattern:
The Backup Agent does not send mails according to the defined criteria.
You will receive the following or similar messages in the log file of your backup job:

Recommended Action:
You can test the stored settings via the Backup Portal. If the test is not successful, you can perform the following steps:

1. Update the agent to the latest version, as the mail notification feature has been updated to current standards in newer agent versions.
   
2. As an alternative to port 25, check port 587 or 465.
   

Please check once with SMTP Prober Tool if the mails can be sent from the system in general.

SMTP Prober
You can download the tool from the following link: http://www.alexnolan.net/software/SMTPProber.htm
If you can successfully send mails from the system with the tool, it is ensured that the communication with the mail server can be established.

GetVolumeNameForVolumeMountPointW() failed

Cause:
This can happen when database files (MDF or LDF) exist on either a mount point or a symbolic link, rather than in a normal physical location.
Example:
You have your logical location for the database files in C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\DATA, but you have created a symbolic link to the data that exists on another drive:


Solution:
The location for the MDF and LDF files must be updated.
One way to do this is to detach the database and then attach it again.
If the database is attached, make sure to select the MDF file in the current location (D:\Data\MSSQL\DATA in the above example).
Once the logical reference is updated to the correct path, the backup is successful.

Windows Agent Installation

---

The installation package is not supported by this processor type

Error pattern:
When installing the agent, you get the following error screen:

Cause:
The executed setup is not compatible with the used processor type (64 bit or 32 bit).

Solution:
Please use the appropriate setup for the processor type used.

Pending restart

Error:
When installing the agent, you will receive the following message:

Cause:
The operating system creates the "RebootRequired" registry key. TERRA CLOUD Backup Agent checks if this key exists before installation.

HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired

Recommended Action:
Please perform the reboot before installing the agent, because e.g. recommended Windows updates require a reboot after their installation.

Backup Host Agent Error 0x800706d9

Error pattern:
When registering the system, you will receive the following error message in the Backup Portal

Solution:

Please check if it is a cluster node with failover cluster role installed it is not a cluster system.

• If the system is not part of a cluster, please uninstall the Failover Cluster role
  

The agent checks:

• If no failover cluster role is installed => Connect to Hyper-V
  
• If Failover Cluster role is installed => Connect to the cluster instance
  

No connection between agent and portal

Error:
When installing the agent on a target system, the agent cannot authenticate to the portal.

Causes:

• The data entered is not correct.
• Communication is not enabled due to blocked ports.
• Ports 8086 and 8087 TCP must be enabled from the inside out.

Review:
Please run the following tests from the system on which the agent is installed:

• telnet backup.terracloud.de 8086
• telnet amp01.terracloud.de 8087
• telnet amp10.terracloud.de 8087
• telnet amp20.terracloud.de 8087

Alternatively, you can run the following commands in PowerShell:
1 .. 20 | ForEach-Object { Write-Host "amp$(($_).ToString("00")).terracloud.de" : $(Test-NetConnection "amp$(($_).ToString("00")).terracloud.de" -Port 8087).TcpTestSucceeded }
Test-NetConnection backup.terracloud.de -Port 8086

Solution:
Please check the access data, these were defined by you within the backup portal itself.
If the connection attempts fail, please release the corresponding ports.

Agent Upgrade Utility stops working / MAPI job upgrade

Error:
Agent Setup hangs and a window appears with the info "Agent Upgrade Utility stopped working".
Then as soon as you click on "Close Program", a MAPI job upgrade window opens.





Solution:
Please run the setup again and stop at the first step.

Then download the following .zip file:
Hotfix

Cause ".dll" files are detected as viruses by default in the browser, you will need to unzip the files and then rename them.

libeay32.dll.txt -> libeay32.dll
ssleay32.dll.txt -> ssleay32.dll

• Now change to Temp directory of the affected system:
• Open Explorer -> type %temp% in the address bar.
• There should be a folder named {BD4F2616-B17D-4982-815F-0C78C476839F}.
• Please now move the two ".dll" files into the directory.

Afterwards the update should be possible over the already started process.

Error 1920 during installation / update of the agent

Error:
During an installation / update of the Windows Agent, one encounters the following message:



Cause:
There are two possible causes for this:
1. required Visual C++ libraries are not available

2. local port 2548 is blocked by another application
Known applications that cause problems: Dr Tax

Solution:
1. please install the following Visual C++ cumulative update and then run the agent setup again:

https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0

2.Use the following CMD command to check if the port is blocked via a running process:

netstat -ano | findstr 2548

Example: TCP --- 0.0.0.0:2548 --- 0.0.0.0:0 --- LISTEN --- 5036

Unless the running process in Task Manager(Details) is then named VVAgent.exe, the port will be blocked.
Subsequently, it must be ensured either temporarily, or better permanently, that the port is open. Once the port is open, the installation / update can be performed successfully.

Windows Agent uninstallation

Error 1722 when uninstalling the Backup Agent

Error pattern:
You will receive the following error message when uninstalling Backup Agent:

Cause:
The error pattern occurs when agent installation (plug-ins) information is missing from the global.vvc file. Each plug-in adds an entry in this configuration file (e.g. image plug-in).
The error occurs when during uninstallation the entries in this file do not match the plug-ins stored on the system.

Solution:

1. Please rename the "global.vvc" inside the installation directory to "global.old".
   
2. Uninstall the agent afterwards

Agent scripting

---

PreshutdownTimeout value

If you have a Windows Server 2019 machine in use, separate permissions are required to change the following registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\gpsvc\PreshutdownTimeout

Please make sure you can edit the group policies with the users logged in.


So via "Advanced" please assign permissions to the respective user owner.


Needed in the case owner permissions.


After that, you should be able to adjust Registry Key accordingly.

Windows Agent file-based backup

---

Analysis point in the following path is not traversed

The error message says that you are trying to save a path that is some sort of link.
The actual dataset will probably be on another partition.
This directory was apparently not included in the backup, so the agent does not backup this as well.

Linked hard links were not selected and were not saved

Error pattern:
The backup job completes with warnings "Linked hardlinks were not selected and were not backed up". This warning looks like the following in the log file:

Cause of error:
Hardlinks are files or directories that reference an existing entry in the Master File Table and have the identical properties and contents as the original file. Original file and hardlink differ only in the path. The Configured Backup Set does not include all the hardlinks that exist for the original file and therefore cannot be backed up.

Possible solutions:

1. Once the entire NTFS partition is backed up, no more linked hardlinks can be faulted.
   
2. The backup set can be extended with all existing hardlinks, the required paths can be read out via the command below using PowerShell or CMD.

Syntax : fsutil hardlink list <filename>
Example : fsutil hardlink list c:\list.txt

OneDrive directory cannot be backed up

Error pattern:
A file-based backup job reports the following error in the logfile:
DISK-E-04205 Error searching for file C:\Users\%USER%\OneDrive\*.*
DISK-E-04101 Operating system message: The medium is write-protected.

Solution:
A setting within OneDrive ensures that the agent only sees links to files, as these are only downloaded when needed.
These 0KB files cause the error message, the files must be downloaded once, so that there are no more Verknpüfungen.
In addition, please uncheck the "Files on demand" checkbox, as shown in this screenshot:

SV -W-08372 Exclusion <Exclusion in backup job> is suppressed because it interferes with <Component of BMR backup> BMR selection

Error pattern:
You receive the following error message in the log file of a file-based backup job:
SV -W-08372 exclusion <exclusion in backup job> is suppressed because it affects <component of BMR backup> BMR selection.

Cause:
The configured exclusion would exclude files that are necessary for BMR backup. The backup agent ignores the exclusion to perform a BMR backup.

Recommended Action:
Please remove the offending exclusion from the affected backup job, otherwise the job will complete with a warning for each backup. Modify the exclusion, if needed, to exclude no system-relevant files.

Outlook Offline Data File (OST)

• .ost files are offline Outlook data files which are automatically created by IMAP accounts, Microsoft 365 accounts, Exchange accounts and Outlook.com accounts.
  

Since they cannot be moved from one computer to another, there is no need to back them up.

The following registry key excludes *.ost files from the creation of a VSS snapshot by default:
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\BackupRestore\FilesNotToSnapshot

Why should *.ost files be excluded from the backup?:
https://social.technet.microsoft.com/Forums/en-US/fc9bebff-9494-4b45-b43d-6b2f8e98add4/oops-windows-7-backup-omits-key-outlook-mail-files
https://support.office.com/en-us/article/find-and-transfer-outlook-data-files-from-one-computer-to-another

More information about files that are automatically excluded from a backup by the operating system:
https://docs.microsoft.com/en-us/windows/win32/vss/excluding-files-from-shadow-copies

It is not recommended to make any changes to the mentioned registry keys..

DISK-E-08235 Operating system error 225, message: The operation could not be completed successfully because the file contains a virus or possibly unwanted software

Error pattern:
You receive the following error message in a file-based backup job log file:

DISK-E-08235 Operating system error 225, message: The operation could not be completed successfully because the file contains a virus or possibly unwanted software.

Cause:
This is a message from the operating system to the agent. Windows Defender is responsible for checking the file.

Recommended Action:
Please check the complained file, in case it is a false report message from Windows Defender, you can release it via whitelist.
Optionally, you can use another antivirus solution to scan the file.

DISK-E-08235 Operating system error 1392, message: The file or directory is corrupted and unreadable

Error pattern:
You will receive the following error message in the log file:

DISK-E-08235 Operating system error 1392, message: The file or directory is corrupted and unreadable

Cause:
The Backup Agent passes on an error message from the operating system that the offending file is corrupted or unreadable.

Recommended Action:
Please check the corresponding file, because this error message is not caused by the backup software.
A system scan via sfc/scannow may resolve the error pattern.
Diagnosis via CHKDSK may also be helpful in this case.

Disable security data processing due to insufficient authorization

Error pattern:

You will receive the following warning message in the log file:

DISK-W-08623 Disable security data processing due to insufficient authorization

Cause:
Compress Advanced Attributes (compress content to save disk space) enabled for at least one directory.

Solution:
To locate these directories you can run the job once with a different log level. Please edit your job and select the option "Directory" as "Options for log file:". Afterwards, run the backup again, now you should find below the line shown below hints to the directories that cause this. This form of logging generates large log files therefore this type should only be used temporarily. Locate these folders and in Properties->Advanced remove the item "Compress content to save disk space". Afterwards, both TERRA CLOUD backup services should be restarted.

DISK-E-04144 Error opening file [...] $Exported$BCD for input

Error pattern:
You will receive the following warning message in the log file:

DISK-E-04144 Error opening file \GLOBALROOT\Device\HarddiskVolume2\EFI\Microsoft\Boot\$Exported$BCD for input
DISK-E-08235 Operating system error 32, message: Process cannot access the file because it is being used by another process
DISK-E-04144 Error opening file \GLOBALROOT\Device\HarddiskVolume2\EFI\Microsoft\Boot\$Exported$BCD.LOG for input
DISK-E-08235 Operating system error 32, message: Process cannot access the file because it is being used by another process

Cause:
This error pattern is caused by various antivirus software based on Bitdefender. The module "Behavioral scan" in version 6.6.18.265 is responsible.

Solution:
Please update your antivirus software, as this error pattern has been fixed in an update.

UTIL-E-04101 O/S message: The request could not be executed due to an I/O device error

Error pattern:
You will receive the following warning message in the log file:
UTIL-E-04101 O/S message: The request could not be executed due to an I/O device error.

Cause:
The Backup Agent passes on this error message of the operating system in the log file. You can also find this error message in the system event logs under ID 153. This is not a problem with the backup solution.

Recommended Action:
Please check the following Microsoft contribution for this ID.

As described in the Microsoft post, this is a timeout of a request to the storage. For further troubleshooting we recommend the CHDSK tool, for more information see: Microsoft post chdsk

ENVC-E-04096 GetProcessorInfo FindFirstObject failed

Error pattern:
You receive the following error message in the log file of a file-based backup job:

ENVC-E-04096 GetProcessorInfo FindFirstObject failed
ENVC-E-04096 GetBootConfiguration FindFirstObject failed
ENVC-E-04096 GetComputerSystem FindFirstObject failed
ENVC-E-04096 GetOSInfo FindFirstObject failed
ENVC-E-04096 CaptureSystemRestorePointInfo Env Exception: Failed to obtain EnvInfo
BMR -E-08382 BMRCaptureEnvInfo.dll could not be initialized: BMR::CaptureEnvInfoDLL::captureSystemRestorePointInfo(): BMRCaptureEnvInfo.dll DLL failed to capture System Restore Point Info

Cause:
The backup agent cannot retrieve any information of the system (e.g. version of the operating system) via the WMI interface. This error pattern is not caused by the TERRA CLOUD Backup Agent. It may be that the required files are still being accessed by other programs and therefore the information cannot be read.

Recommended Action:
Please check the following query via PowerShell as an example:
Get-WmiObject Win32_Processor

If no output is possible here either, we recommend that you check the Windows Event Logs at this time.

DISK-W-08623 - Disable security data processing due to insufficient authorization

This article was written for the job type "UNC files". For the job type "Local system" there is the following article

Error pattern:
You receive the following warning message in the log file of a UNC job:

DISK-W-08623 Disable security data processing due to insufficient authorization

Cause:
The NAS is not allowing the ACLs to be backed up.

Recommended Action:
Please go to the agent system and open the job configuration:
C:\Program Files\TERRA Cloud Backup\Agent\<JOBNAME>.vvc"
Here you will find the Ignore_Security and Ignore_AltData entries:

Ignore_Security: this switch determines whether the ACLs should be backed up or not. The NAS (e.g. QNAP) does not allow the ACLs to be backed up, so you must set the switch to TRUE.

Ignore_AltData: This switch determines whether the extended attributes should be backed up or not. Provided the Ignore_Security option is set to TRUE, or MAC files are backed up via the job, you should set the switch to TRUE.

Afterwards, both TERRA Cloud Backup services should be restarted.

If the backup job is no longer displayed in the portal after this adjustment, additional characters were probably stored in the configuration file. In this case, please check the <JOBNAME>.vvc with an editor that can display all characters, e.g. Notepad++ (show non-printable characters).

After you have set the options and a backup has been made, be sure to do a test restore.
Since backing up MAC files is not officially supported by our solution, we cannot 100% guarantee that the files can be restored without any problems.

Windows Agent Image Backup

---

VIMG-W-09841 - CBT (Changed Block Tracking) is not available

Error message:
You will receive the following warning message in the log file:

VIMG-W-09841 CBT (Changed Block Tracking) is not available.

Cause:
The installation of the CBT driver (Changed Block Tracking) requires a reboot, this was not performed at the time of the backup.

Note:
Backing up a system via an image backup job is also possible without the Changed Block Tracking, but the backup duration is much longer.

Solution:
Please perform a system reboot to be able to use the full functionality of the image plug-in.

VIMG-W-09829 - The changed block logging driver fails to log the changes for volume <X>:<volume{XY}>. (State 0, Status 0)

Error message:
You will receive the following warning message in the log file:

VIMG-W-09829 The changed block logging driver fails to log the changes for volume <X>:<volume{XY}>. (State 0, Status 0)

Cause:
The installation of the CBT driver (Changed Block Tracking) requires a reboot, this was not performed at the time of the backup.

Note:
Backing up a system via an image backup job is also possible without the change block tracking, but the backup duration is much longer.

Solution:
Please perform a system reboot to be able to use the full functionality of the image plug-in.

PLGN-E-05905 Could not load <VolumeImagePlugin.dll>

Error message:
You will receive the following error message in the log file. PLGN-E-05905 Could not load <VolumeImagePlugin.dll>
DISK-E-04793 Drive $VOLUMEIMAGE$:\ is not recognized as a persistent mapped drive (drive not found).
DISK-E-04204 Error assigning channel to $VOLUMEIMAGE$:\*.*

Cause:
The backup solution is not able to use the image plugin. Often due to the following causes:

• The image plugin was forgotten to install after a new installation / re-registration.
  
• The image plugin / agent is corrupted.
  

Solution:
By running the agent setup again, you have the option to perform a change installation. Here you can post-install the image plugin.
Please note that a reboot should take place after the installation of the image plugin.
By re-running the agent setup, you also have the option to perform a repair installation of the agent.

VIMG-E-0449 Volumeimage_plugin::VolumeImageBackupAdapter::getNextBackupObject failed

Error message:
01.ago. 07:18:47 VIMG-E-07966 Job cannot be continued.
01.ago. 07:18:59 VIMG-E-04495 volumeimage_plugin::VolumeImageBackupAdapter::getNextBackupObject failed.
01.ago. 07:18:59 BSTR-E-04144 Error opening file C:\Program Files\TERRA Cloud Backup\Agent\*
01.ago 07:18:59 BKUP-E-04706 Job is forced to fail. [VV]
01.ago. 07:18:59 BKUP-F-04110 Job could not be completed. [VV]

Cause:
The Microsoft VSS service cannot put the system in a consistent state. There are many possible causes e.g.:

1. On the system to be backed up the "Microsoft Hyper-V VSS Writer" is stored, please note that a BMR backup on Hyper-V hosts is not supported
2. VSS Writer from third party software are deposited on the system
3. A path to an .exe file in the registry points to a network share, please note that the VSS service cannot access network shares

Recommendation for action:
Please check the deposited VSS Writers via the command "vssadmin list Writers" and compare them with a system on which an image-based BMR backup is possible.

In addition, we recommend that you read out the details of the VSS Writer and check for a stored network share. The following article describes how to read the details of the Writer:
Microsoft VSS Writer details readout

Error while collecting information about the system provisioning point

Error pattern:
You get the following or similar error image in the log file:

BKUP-E-08235 Operating system error 2, message: The system cannot find the specified file
BKUP-W-08732 Mount point information for system could not be captured. [VV]
BKUP-E-09401 An error occurred while collecting system mount point information. [VV]
BKUP-E-08235 O/S error 2, message: The system cannot find the specified file
BKUP-W-08732 Failed to collect system mount point information [VV]

Recommended Action:
Please run the following command on the affected system:
Mountvol /r
(Removes volume mount point directories and registry settings for volumes that are no longer on the system, preventing them from being automatically mounted and retaining their previous volume mount points when they are added back to the system)

Rebooting the system may also be helpful.

Windows Agent SQL Backup

---

SQL Express DB backup via SQL plugin fails

Error pattern:
06.ago. 14:08:25 DISK-E-04144 Error opening file \GLOBALROOT\Device\HarddiskVolumeShadowCopy217\Programs\Microsoft SQL Server\MSSQL10_50._SQLEXPRESS\MSSQL\DATA\SQLEXPRESS\%DB%.mdf for input
06.ago. 14:08:25 VSS -E-08102 File backup error (reason: stream backup error) C:_Programs\Microsoft SQL Server\MSSQL10_50._SQLEXPRESS\MSSQL\DATA\SQLEXPRESS\%DB%.mdf component <writer SqlServerWriter (SQL Server 2008 R2:SQLWriter), component %SERVERNAME%\ SQLEXPRESS\ SQLEXPRESS> fails.
06.ago. 14:08:25 CTLG-W-09909 Incomplete backup set metadata. Information for "plugin-specific" could not be retrieved

Cause:
SQL Express instances can only be backed up with the SQL Plug-in as of agent version 8.71. An overview of the supported SQL versions can be found in the release notes.
https://drive.terracloud.de/dl/fiLpZboTGRPeqzW1cqwS6yin/Documentation_and_Release_Notes.zip

Solution:
Please update the agent version to 8.71 or higher.

SQ12-E-09295 A successful full backup has not yet been made of database XY.

Error pattern:
You receive an error message in the log file that no successful backup could be created from a database:

SQ12-E-09295 A successful full backup has not yet been made of the database database name.

Cause:

The MSSQL VSSWriter service runs under a system account.

Recommendations for action:

• The MSSQL VSSWriter service should be changed to an ActiveDirectory account with administrator privileges.
  

SQL Backup Jobs cannot be edited after credentials change

Solution:
Make a backup copy before changing the files!
1. create a new SQL job with the new Windows credentials.
2. you will find the file "JOBNAME.vvc"
in the installation directory of the agent. 3. copy the paragraph "WindowsCredentials" from your test job to the .vvc of the correct backup job

Subsequently, the backup process could be continued.

No instance list could be provided by the agent

Error pattern:
You get the error message on the following screenshot when you try to create a new SQL backup job via Backup Portal:

Cause:
The agent cannot connect to the SQL instance. Possible causes are:

1. There is no SQL instance installed on the system

The installed SQL instance is not supported by the SQL plug-in, e.g. because it is not a Microsoft SQL instance. Recommended Action:
Please check the latest Release Notes for the Windows Agent to see if the installed instance is supported.

You are shown the databases of a named instance, although you have selected "Standard" as instance

Error pattern:
When setting up a new SQL backup job, or editing an existing job, you are not shown the databases of the MSSQL instance. Additionally, you may not be able to log in with the SQL credentials of the user of the MSSQL instance, even though you have selected the default instance.

Cause:
If you select Standard as the instance in the Backup Portal, the instance for which the port 1433 is assigned is checked in the background. If this port is assigned to another instance, this instance will be displayed or you will be asked to authenticate for it.

Solution option 1:
If possible, reassign port 1433 to the default instance.

To do this, open the SQL Server Configuration Manager

1. Select the menu item SQL Server Network Configuration -> Protocols for MSSQLSERVER -> TCP/IP -> IPALL

Port 1433 should be set for this option, as shown in the following screenshot

Solution option 2:
If solution option 1 is not possible, we recommend that you implement the following steps:

Please check the Connection String first:
1.open Notepad
2.Please add the following 3 line to the text file. <hostname> stands for the hostname of your SQL server;<port_number> is the assigned port of the default instance

     set db = CreateObject("ADODB.Connection") 

     db.open "Driver={SQL Server};Server=<hostname>,<port_number>;Database=;"

     MsgBox(db.Provider)

3.save the text file as e.g. "test_db.vbs".
4.run the created script

After verifying the connection, please make the following adjustment in the SQL server registry:
1.open the registry
2.Navigate to "HKEY_LOCAL_MACHINE\SOFTWARE\EVault\InfoStage\Agent\"
3.create the entry "ADOConnectionStrings" as "REG MULTI SZ" add
4.add the following key to the created entry

     <hostname>="Driver={SQL Server};Server=<hostname>,<port_number>;Database=;" 

An example would be: "SQL01="Driver={SQL Server};Server=SQL01,1434;Database=;" In this example, SQL01 is the host name and 1434 is the assigned port.

After customization, you will see the correct instance in the Backup Portal if you select "Default".

Windows Agent Exchange Backup

---

Circular logging

Circulation logging is currently enabled. It is better to disable it in production environments. "Circulation logging is not recommended for production environments.
Source: https://technet.microsoft.com/de-de/library/bb331968%28v=exchg.80%29.aspx
The above link also describes how to disable circulation logging on an exchange server.
An informative article on the subject of circulation logging:
http://www.itadministrator.de/themen/kommunikation/123111.html
Alternatively, you can also disable the backup of transaction logs. Then only the Exchange database is backed up, transaction logs are left out.

Customization target mount path for Granular Exchange Restore

By default, it is mounted below C:\RestoreMount. Unless there is enough space on the disk, another drive letter must be used.
The registry key below can be used to customize the destination path.

Solution:
Create the following registry entry with a string:
HKEY_LOCAL_MACHINE\SOFTWARE\EVault\Infostage\Agent\MountAlternatePath=

No restart of the backup service is required.

Example:
HKEY_LOCAL_MACHINE\SOFTWARE\EVault\Infostage\Agent\MountAlternatePath=e:\mp

When e:\mp is used as a string, the release: \EX2007SERVER\e$\mp\GRT_000010_2013-06-25-10-13-43


Note:
Only possible with Granular Restore Tool 8.x or higher.

Validation failed for Exchange database Mailbox Database XYZ, reason: Timed Out

Error pattern:
You receive the following error message in the Exchange backup log files: EX10-F-10379 Validation failed for Exchange database Mailbox Database XYZ, reason: Timed Out

Cause:
Checking the database via the ESEUTIL takes too much time, this can be attributed to a heavy workload or too low performance of the system.

Recommended Action:
Please set the following registry entries to increase the timeout of the check and adjust the flow of the check (sequential):
HKEY_LOCAL_MACHINE\SOFTWARE\EVault\InfoStage\Agent\Ex2010SequentialValidation = 1 (DWORD)
HKEY_LOCAL_MACHINE\SOFTWARE\EVault\InfoStage\Agent\Ex2010ValidationTimeOut = 18000 (DWORD)

Restart both TERRA CLOUD Backup services and try to start a backup again including a database check.

Windows Agent Oracle Backup

---

The credentials for the plug-in could not be verified due to an unexpected error. Verify that the credentials are correct and try again

The Oracle plug-in uses the connection to the Oracle database EZCONNECT. Make sure that your configuration in the sqlnet.ora file is correct.
E.g.:
# SQLNET.ORA Network Configuration File: d:\oracle\ora92\network\admin\sqlnet.ora
# Generated by Oracle configuration tools.
SQLNET.AUTHENTICATION_SERVICES= (NONE)
NAMES.DIRECTORY_PATH = (EZCONNECT,TNSNAMES)

Start Oracle listener and try again with a backup.

Hyper-V Agent

Hyper-V installation of Visual C++ 2017 Runtime(64) failed

Error pattern:
You receive the following error message in Setup during Hyper-V Management Agent installation:

Cause of error:
As already described in the error message, there was a problem with the runtime download. This error pattern occurs only sporadically.

Solution:
You can either cancel the setup and perform the installation again including the download of the runtime or download and install the runtime manually.

You can download the current runtime from the following link:
https://aka.ms/vs/16/release/vc_redist.x64.exe

For more information, please visit:
https://support.microsoft.com/de-de/help/2977003/the-latest-supported-visual-c-downloads

Hyper-V VM is backed up by other process

Error code:
AGCD-E-10348 Virtual machine "XXX" ["VM-ID"] could not be backed up because it is currently being backed up by another process.

Cause:
Here, no backup of the VM's can be performed because of leftover files due to an update of the Hyper-V Host Agent from version 7.x to version 8.60,
which are misinterpreted during the backup request.

Solution:
Please change to the following directory and remove all files located there:
C:\Windows\System32\config\systemprofile\AppData\Local\EVault Software\Agent Worker\Notifications Queue\

Note:

 First, please update to the latest version of the Hyper-V Agent, as this error pattern has been fixed in version 8.84.1309. 

Hyper-V unknown exception

Error message:
"unknown exception" in back supportal during configuration/setup

Cause:
Here, no backup of the VM's can be performed because of leftover files due to an update of the Hyper-V Host Agent from version 7.x to version 8.60,
which are misinterpreted during the backup request.

Solution:
Please change to the following directory and remove all files located there:
C:\Windows\System32\config\systemprofile\AppData\Local\EVault Software\Agent Worker\Notifications Queue\

Note:

 First, please update to the latest version of the Hyper-V Agent, as this error pattern has been fixed in version 8.84.1309. 

Hyper-V backup shows no progress

Error code:
A previously working backup starts after the update to 8.60, but no progress is apparent.

Cause:
Here, no backup of the VMs can be performed because of leftover files due to an update of the Hyper-V Host Agent from version 7.x to version 8.60,
which are misinterpreted during the backup request.

Solution:

1. Stop the agent services of the MGMT agent and all host agents
   
2. Please change to the following directory (step 3) and remove all files located there:
   
3. C:\Windows\System32\config\systemprofile\AppData\Local\EVault Software\Agent Worker\Notifications Queue\
   
4. Please restart the agent services of the MGMT and all host agents.
   

Note:

 First, please update to the latest version of the Hyper-V Agent, as this error pattern has been fixed in version 8.84.1309. 

Hyper-V FAILED_GeneralFailure

Error message:
The backup job of a Hyper-V agent aborts right after startup and deposits the following error message in the log file:
AGCD-E-08383 VSS snapshot operation completed with result 'FAILED_GeneralFailure

Cause:
In this case, there is a VSS integration failure between the host and VM.

Recommended Action:
1. please remove the following option in the integration services of the VMs to be backed up and confirm with "OK":

2. lead start a backup afterwards, this backup should fail due to the lack of VSS support via the integration services.

3. please re-enable this option for all system and try to backup again.

Note:

 First, please update to the latest version of the Hyper-V Agent, as this error pattern has been fixed in version 8.84.1309. 

Hyper-V backups are no longer started via the schedule

Error pattern:
The Hyper-V Agent no longer starts a backup via the schedule.

Solution:
Please store the encryption password again in the backup job in which you edit the job.

Hyper-V update from 8.60 to 8.X fails

Error message:
You receive the following error message during the update installation of the management and host components of agent version 8.X:

Error 1316.The specified account already exists.

Cause:
One registry entry each for the MGMT and the host components is incorrect and prevents the update from 8.60 to 8.X.

Solution:

1. Please open the registry on the systems on which the individual components are installed
2. Search for the following GUID of the MGMT agent 27FB23FA45CDCCA419AFA539FAB44906 and click on the folder "SourceList".

Change the value of the PackageName key to TERRA Cloud Backup Hyper-V Agent Management.msi and save it.

1. Search the registry of the systems on which the Host Agent is installed for the GUID 2BFC6FB3019E15D449DEFDFA7BA62BA0
2. Navigate to the "SourceList" folder and change the value of the "PackageName" key to TERRA Cloud Backup Hyper-V Agent Host.msi and save it
3. After that you can perform the update

Linux Agent

DISK-F-04144 Error opening file /var/lib/lxcfs

Error pattern:
You receive an error message in the log file of a Linux BMR job that files below the /var/lib/lxcfs/ directory cannot be backed up due to lack of permission.

Cause:
It is a FUSE file system for the use of Linux containers, within this system the root user does not have the necessary permissions.

'Further information on the subject:
https://ubuntu.com/blog/introducing-lxcfs
https://linuxcontainers.org/lxcfs/introduction/

Recommended Action:
Since this directory is not necessary for a BMR restore, you can remove this directory on the backup job. After exclusion, the error message no longer appears.

Delta file not found / damaged

Error pattern:
You will receive the following error message in the log file:

BSTR-E-04166 File not found /opt/BUAgent/localjob/0000XY.DTA BKUP-E-08194 Delta file is missing. Synchronize the job to recreate the delta file. [VV]

In case of a corrupted delta file, the error pattern looks very similar.

Cause:
The agent cannot access the delta file because it either does not exist as in the example above or is corrupted and the stored checksum of the file does not match the one currently measured.

Solution:
On the protected system, check in the agent directory if there are .DTA and .DTX files in the job folder, if not you can start a synchronization of the job in the Backup Portal.
If there are files, please remove them via rm DATEINAME or e.g. via WinSCP. Please start a synchronization of the job afterwards.

An additional process starts during a backup/restore/synchronization

Error pattern:
When a Linux backup job is executed, a second "empty" process starts.

On this screenshot you can see an "empty" process that is still active after two other processes have finished.

Note:
This error pattern has already been passed on to the software manufacturer; it is merely a display error, since there is no process with the displayed process ID on the corresponding system.

Linux system is displayed as "Offline" in the portal

Error pattern:
In the Backup Portal, a Linux Agent is displayed as "Offline" even though the backed up system is active and backups are still being performed.

Cause:
The "VVAgent" service is no longer working correctly or has stopped.

Solution:
On the system in question, run the following command, for example, to restart the service:

/etc/init.d/vvagent restart

Alternative:

systemctl restart VVAgent

BMR -E-07393 Unknown message: ID = 10596 Parameter0 = /Bare_Metal_Restore_Image.iso

Error pattern:
You receive the following error message in the log file of a BMR Linux job:
BMR -E-07393 Unknown message: ID = 10596 Parameter0 = /Bare_Metal_Restore_Image.iso

If you have configured the language in the Backup Portal to English, you will receive the following error message:
BMR -E-10596 The backup agent cannot find the ISO image at the user configured location {1}.

Cause:
The system to be backed up does not have Relax and Recover version 2.5 installed.
Please note that currently only the version 2.5 is supported.(As of January 2021)

Recommended Action:
Please check the installed version first via the following command:
rear-V

If the version is lower than 2.5, you can update it to 2.5. In the case of a newer version, you would need to downgrade in the form of uninstalling and reinstalling version 2.5.

For more information and guidance on rear, please visit:
https://relax-and-recover.org/documentation/

VMware vSphere Recovery Agent

VSPH-W-10566 Failed to verify SQL Server transaction logs were truncated for instance ESXI-SQL on VM {1}.

Error pattern:
You receive the following warning message in the log file during a backup job of a vSphere Recovery Agent version 8.82 or later:

VSPH-W-10566 Failed to verify that SQL Server transaction logs were truncated for instance ESXI-SQL on VM {1}.

Cause:
The vSphere Recovery Agent cannot check if the transaction logs are truncated on a German SQL installation. Presumably, this error pattern occurs with all non-English installations.

Recommended Action:
Please change the setting for truncation of transaction logs to "Backup with copy only" (see screenshot).

The transaction logs can be optinally truncated via board means of Microsoft SQL. Alternatively, you can install a Windows agent including SQL plug-in in the VM and protect the database via an SQL backup job. In this scenario, you back up the system via a host backup and the database additionally via an application-consistent agent backup.

VSPH-W-10274 Virtual machine "Name of VM" could not be suppressed. The backup will try a crash consistent snapshot instead

Error pattern:
You will find the following error message in the log file: VSPH-W-10274 Virtual machine "Name of VM" could not be suppressed. Backup will attempt a crash consistent snapshot instead.

Cause:
The vSphere Recovery Agent cannot stop the file system inside the VM through vCenter because the Quiesce guest file system feature is not available.

Recommended Action:
Please check if the option shown below is available in vCenter when creating a snapshot. Verify that VMware Tools are installed and up to date within the VM. If you are still unable to use this feature, please contact VMware Support as it is an issue with the hypervisor.

VSPH-I-10525 An old version of VMware tools is installed on VM XYZ. Update the VMware tools to the current version

Error pattern:
You will receive information in the log file that the installed VMware tools on a VM are outdated and thus no application-consistent backup can be created.

Cause:
For an application-consistent backup, the Microsoft VSS components must be addressed within the VM. To use this feature, at least version 11 or higher must be installed.

Recommended Action:
Please check the version number of VMware Tools e.g. in vCenter and update it.

An example, after the update an application consistent backup is possible.

For more information, see the latest documentation of the vSphere Recovery Agent.

TERRA CLOUD Backup Satellite

Backup "Job cannot be registered to the Vault. It may already exist in the Vault." (Only in connection with a satellite)

Cause:
As long as the satellite is set to "Bypass inactive", a computer or a backup job cannot be registered directly to the Basevault, because the Basevault is set as the current backup destination in the Vault settings.

Solution:
Please store the local IP address of the satellite in the Vault settings and try to create the job or add the computer again.

Time in satellite is not correct

Causes:
The bandwidth limitation adheres to the receiving time. If the time is out of sync, the plan will not work correctly.
Solution:
Please share port NTP port 123/UDP from satellite to internet.

BMR recovery

Checklist troubleshooting BMR recovery

General:
In the event of an error, please go through the following checklist during or after a BMR recovery.
Based on the answers, the error pattern, if not already covered by the Troubleshooting Guide, can be narrowed down.
Furthermore, the checklist enables a faster and more efficient support process.

Checklist:

1. Is this a production restore or a test restore?
2. When was the last time the system was restored in a test recovery?
3. Which version of the Restore ISO was used? → Please always use the latest version.
4. Have you restored a file or image based backup job?
5. Does the problem occur while configuring the backsight? If Yes → Please check the procedure in the documentation for this.
6. Have you already saved log files of BMR backups? If no → A for this you find instruction under Save logfiles of a BMR backups
7. Successful recovery to a physical or virtual system? If virtual → what virtualization technology is used?

Is the source system a BIOS or UEFI installation? → The firmware used decides the VM generation, e.g. for Hyper-V.

1. Have you restored all volumes or only those necessary for the operating system?
2. Were different backup points/safesets already selected for the restore?
3. What is the role/purpose of the source system? → e.g. DC, MS-SQL, file server etc.

Tips:
A faulty production restore causes a stressful situation and downtime should be kept as short as possible. The following tips can simplify the fault clearance process and save valuable time.

Parallel recovery:
You can restore simultaneously from the primary and secondary Vault and, if necessary, a satellite. This provides you with backups from at least two sources at the same time.
The FQDNs and Public IPs can be found in the Vault overview.

Pursue multiple solution paths simultaneously:
Example:
After restoring the latest safeset, the system booted into a blue screen because there was operating system corruption at the time of the backup.
In this scenario, you could restore two older safesets from, say, three days ago and the end of last month at the same time, using Parallel Restore.
This strategy doubles the chances of hitting a backup point that contains the system's data before the operating system is corrupted.

Recovery in a disaster recovery VM:
The TERRA CLOUD offers you the possibility to provide a virtual machine or a virtual disaster recovery environment in a fully automated way.
Take advantage of this offer to initiate a secondary recovery into a completely new environment based on Microsoft Hyper-V, in addition to the primary recovery debugging.
If recovery to the original system/environment is not possible, restoring to a DRVM may increase the chance.

'Documentation of the recovery:
Please attach screenshots of error messages/incorrect behavior directly to a support ticket or request. By providing documentation in the form of the checklist and screenshots, you shorten the information gathering part of a support ticket.

---

System boots into a boot error/bluescreen/blackscreen after a restore

Error pattern:
After a successful BMR restore, the system displays a boot error after startup, such as the following:

Procedure:
Please go through the Checklist Troubleshooting BMR Restore first and provide the results to support if necessary.

Recommended actions:
Please check if you can work around the error with the following instructions.

BMR operating system volume restore:

1. Run the BMR restore again and restore only the volume for the operating system
2. Check if the system can boot after successful recovery
3. Provided the system can boot, install a Windows agent in the system and register it in the portal
4. Restore the remaining volumes via the Backup Portal, e.g. via restore from another computer

Save log files of a BMR backup

General:
To debug a BMR recovery, it may be necessary to access detailed recovery log files. The recovery software offers three different levels of detail for the log files in the settings:

• DEBUG
• LOG WARNING
• LOG ERROR

The "DEBUG" option offers the highest level of detail and is thus best suited for troubleshooting. However, there is often a problem that the log files are discarded by the reboot after the failed recovery or cannot be saved outside the system beforehand.

Best practice for providing log files of a BMR back-up:
The following instructions show you a recommended way to save log files for debugging a BMR back-up and make them available to TERRA CLOUD Support.

The log files must be stored outside the storage area relevant for the restore. This can be done on a network share or a local volume.

Network sharing procedure:
Please restart the recovery for debugging and select the log level "DEBUG" in the settings under "Log Settings". You can store your network share as the storage destination. After the restore, please send the saved log file to TERRA CLOUD Support.

Procedure "Local Disk":
Please provide the target system with another volume of at least 20 GB, as the log files can become very large depending on the size of the recovery. If this is not possible, e.g. because it is a physical server, we recommend booking a disaster recovery VM in the TERRA CLOUD including an additional VHDX for the log files.

You can set up the additional hard disk from the command prompt in the main menu of the recovery software using Diskpart. In the following screenshot, Notepad has been opened from the command line to show the volume set up via Diskpart:

After that, please navigate to the settings under "Log Settings" and select the log level "DEBUG". You can then select the extra volume set up as the storage destination. After the recovery is complete, please provide the saved log file from the volume to TERRA CLOUD support.

Failed to GetDiskDetails

Error pattern:
During a BMR recovery you will receive the following error:


Cause:
The Restore ISO is not able to process the job details from our Vault.

Solution:

Note:
In this case, if it is a disaster recovery VM within the TERRA Cloud, the required RestoreAPI_IS.dll is already stored for you in the ISO.
You can proceed directly to step 2 and start the recovery.

Step 1:
Please paste the following file into the restore iso, you can use the following software for this.

Step 2:
Start the recovery with the edited ISO.

Step 3: Select the language German, then open the menu item Command Prompt and proceed as follows:

• Type "taskmgr" to open the task manager. Then close the "Carbonite Server Backup System Restore Main..." process.
• Next, enter the following commands in the X:\Windows\System32\RestoreApp> path:
  

1. del RestoreAPI_IS.dll
   
2. copy D:\RestoreAPI_IS.dll . (Note partition name of ISO file, test if necessary. By default D:\)
   
3. RestoreApp.exe

Step 4: Afterwards, perform the recovery.

Device Manager failed to get current information

This error message appears after the password for the Restore Point backup job is entered.

Solution:
The SCSI controller driver from "Paravirtual" must be changed to "LSI Logic SAS".

Restore BIOS system on an EFI system

Error pattern:
You will receive the following error message in "Step 3 Map Volumes" during recovery:

You cannot drag the volume to this position because the new order of partitions would violate Microsoft Rules.
The partitions schould be in the following order: Recovery (if any), ESP(if any), MSRP and then primary data partition(s).
Please delete all partitions on this disk and try again.

Cause:
You see in the screenshot the error image that a Hyper-V GEN1 VM (BIOS) should be restored in a Hyper-V GEN2 VM (UEFI). This is not possible due to the different firmware of the two VM generations.

Solution:

Hyper-V:
Create a new VM in the same generation as the source system (in the example above, a GEN1 VM should be created for recovery).

VMware:
You can create a new VM via the vCenter or the Host Client and decide via the "Firmware" option whether EFI (Required for GEN2 VMs) or BIOS (Required for GEN1 VMs) is to be used. is to be used for the VM.

In the following screenshot you can see the creation process of a VM in vCenter:

'TERRA CLOUD IaaS VM:
You can decide if a GEN1 or GEN2 should be deployed when ordering an IaaS VM just before the configuration is complete, this generation must again match the firmware or generation of the source system.

Alternative solution:
Restoring a BIOS system to a UEFI system is possible, but not recommended.
To do this, convert the target volumes for the system partition to GPT (by right-clicking on the disk icon) and drag only the system partition from the source to it.
The Microsoft System Reserved partition does not need to be dragged from the backup to the target, only other data partitions if present.

Restore EFI system to a BIOS system

Error pattern:
You will get the following error message in "Step 3 Map Volumes" during recovery:

an EFI System partition can only be restored to a System using UEFI not a BIOS system

Cause:
You see in the screenshot the error image that a Hyper-V GEN2 VM (UEFI) should be restored in a Hyper-V GEN1 VM (BIOS). This is not possible due to the different firmware of the two VM generations.

Solution:

Hyper-V:
Create a new VM in the same generation as the source system (in the example above, a GEN2 VM should be created for recovery).

VMware:
You can create a new VM via the vCenter or the Host Client and decide via the "Firmware" option whether EFI (Required for GEN2 VMs) or BIOS (Required for GEN1 VMs) is to be used. is to be used for the VM.

In the following screenshot you can see the creation process of a VM in vCenter:

'TERRA CLOUD IaaS VM:
When ordering an IaaS VM, you can decide shortly before the configuration is completed whether a GEN1 or GEN2 should be provided, this generation must again correspond to the firmware or generation of the original system

Bootable Media Creator

Creating a bootable USB stick fails with error code 1

Error pattern:
You get the following error when you try to prepare a bootable stick with Bootable Media Creator:

Cause:
This error pattern only occurs on systems that are not installed in English.
We have already forwarded this to our software vendor to extend support for additional languages.

Solution:
Please create the ISO locally first. You can create a bootable stick via the software Rufus.


If you have any further concerns, please do not hesitate to contact us by email (support@terracloud.de) or by phone at (-850).
Your TERRA Cloud Team