I'm using Runtime Software's free DriveImage XML to back up several XP machines over a network to Windows 2003 server. Scheduled tasks run DriveImage XML in command-line mode and are set to try VSS before drive locking. Another scheduled task collects logs and emails them to me.

Today I noticed that one of the backups did not complete. When I tried to run the backup interactively, DriveImage XML told me that it "Could not initialize Windows Volume Shadow Copy Service (VSS)" and that the error code was 80042316. The Volume Shadow Copy service was running. I tried stopping and starting the service and re-running the backup, but it still could not proceed.

Some Google research turned up the meaning of 80042316 as "another snapshot creation is in progress. Please retry later your snapshot creation." However to the best of my knowledge there are no other backups or other VSS-aware processes running. 

I thought perhaps the recent change from Trend CS 3.6 to NOD 32 2.7 might have caused a problem, but DriveImage XML completed fine on several other machines running NOD32.

Finally I remembered that there is a command-line utility to look into VSS on a machine. I typed

vssadmin list shadows

at a command prompt and it showed that there is a shadow copy on some kind of virtual volume (\\?\Volume{2eb5f062-fd3d-11d8-8bb5-806d6172696f}\). Stopping and starting the Volume Shadow Copy service did not remove the shadow copy.

Maybe this is a remnant from an old backup? I don't know how to directly delete shadow copies under Windows XP, but let's see if a reboot helps...

Bingo! After the reboot, the "vssadmin list shadows" command showed "No shadow copies present in the system," and DriveImage XML was able to create a new shadow copy and do the backup.

Not Fixed

Alas, when DriveImage XML completed and exited, I still had a volume shadow copy on the volume, and the next time I ran DriveImage XML, it  gave me the same message but a slightly different error code:  80042317. Google isn't so helpful in finding a description for 80042317. But rebooting the machine again killed the shadow copy and let DriveImage XML run, so I'm assuming it's the same issue.

Bummer! So why can't DriveImage XML release/delete its shadow copies on this machine? The Microsoft storage team lists three recommended hotfixes for VSS in this blog entry, but they are all for Windows Server 2003, not XP.

Excluding the C:\System Volume Information folder (where volume shadow copies are stored, I believe) from NOD32 didn't help.

Excluding dixml.exe (the DriveImage XML executable) from NOD32 didn't help.

I'll update this if I find a solution (other than rebooting the machine after every backup!). 

A response to my previous blog post asked a fair question:  what sets NOD32 apart or even on par with Trend Client-Server Messaging? I decided that I would do some testing with Trend. Since I am using NOD32 without the Exchange component, I tested Trend Client-Server without the Messaging component.

KeyFinder Problems

I've been building a UBCD4WIN version 3.12 ISO file as my test case. UBCD4WIN includes a large number of plug-ins. One of them is keyfinder.exe, the Magical Jelly Bean Keyfinder version 1.51. This handy program can display and update Windows and Office installation keys.

I installed the Trend 3.5 agent on my workstation and tried the UBCD4WIN build. The build failed because the keyfinder.exe file was missing. By uninstalling and re-installing UBCD4WIN, and temporarily disabling the Trend agent, I confirmed that Trend is deleting this file without logging it as a virus or spyware and without sending it to quarantine. Trend is supposed to encrypt and save suspicious files on the client in C:\Program Files\Trend Micro\Client Server Security Agent\Suspect, but that folder is empty. I finally got Trend to leave the file alone by adding keyfinder.exe to Trend's exclusion list.

Is this a bug? I guess I'd better try the latest version, Trend 3.6 with Patch 1. I downloaded the 336MB installation file, upgraded the server, and let it push out the 3.6.1095 client. No reboot was requested.

After removing the file exclusion from the Trend configuration, I opened Windows Explorer and highlighted keyfinder.exe (but did not execute it). The Trend icon in the system tray indicated activity, then I got a message from Windows indicating that my system may be vulnerable because Trend was not running. The Trend system tray icon disappeared when I put the mouse over it. So scanning keyfinder.exe caused the Trend Real-Time scanner to crash.

I rebooted the client and did the same test, highlighting the file in Windows Explorer. This time keyfinder.exe was not deleted and the Trend real-time agent did not crash. However, the UBCD4WIN build process, which actually copies keyfinder.exe, failed again because access was denied on that file. When I went back to look at keyfinder.exe in Windows Explorer, it was deleted before my eyes. The Trend Client/Server Security Agent real-time scan window still tells me that there are 0 infected files; "Last virus/malware found" is blank. So Trend is again deleting it without any warning or logging. I had to add the file back to the exclusion list so I could complete the UBCD4WIN build.

The Numbers

Once I got the UBCD4WIN build to complete, I tested it with various levels of extension exclusions as I had NOD32. The results, along with the NOD32 2.7 results from the previous post:

Some other numbers that are interesting from a system administration point of view are installation size and memory overhead. The table below summarizes these numbers for both server and workstation installations.

The server install of Trend Client-Server includes its web-based management console. The server install of NOD32 includes the ESET Remote Administrator Server and Console 2.0.56.

Conclusions

ESET NOD32 has a reputation for being lean and fast. Compared to Trend Client-Server 3.6, NOD32 definitely looks "lean" on disk space and memory footprint. However, Trend allowed the UBCD4WIN build to proceed a little faster than NOD32.

My greatest concerns with Trend come from areas other than performance. One was the experience back in February 2007 of Trend passing on a "possible worm" to the client desktop, which would have allowed users to run the worm. I was amazed that there was no way to configure Trend to not pass possible malware to end users. The other experience is the one described above:  deleting a file without warning and without logging. I wonder if I have lost other files that way and will never know.

Clearly there is no perfect anti-virus solution. Both NOD32 and Trend CS(M) have their own configuration hassles and "gotchas." I do appreciate the reduced memory footprint of NOD32, and the fact that my SBS server no longer sends me a daily alert that it is running out of allocated memory. We'll see how well it performs in the long term.

Version 2.7:  The Fine Print

As I mentioned in the updates to my previous blog post, after encountering some issues with NOD32 version 3.0, and on the advice of experienced NOD32 system admins, I downgraded to version 2.7. 

What I didn't realize is that by doing so, I would be giving up the ability to exclude files and folders from my scheduled and on-demand virus scans. Why is that a problem? Some years ago, I wrote and sold a set of Microsoft Word macros. NOD32 calls these a "possible unknown macro virus." Fair enough, but I know what they are so I want to exclude them from scanning. In NOD32 version 3.0, I can exclude them from both real-time and on-demand scans. In NOD32 version 2.7, I can only exclude them from the real-time scanner. With 2.7, if I want the virus alerts to stop, I have to submit the macros to ESET for evaluation. I'm not too keen on sending my intellectual property to a software vendor for review.

Performance Comparison

The main reason given by other system admins for using NOD32 version 2.7 is that it performs much better than version 3.0. After seeing version 2.7 using 50% of the CPU during an in-depth server scan, I started wondering about this. I wondered more when I noticed that my desktop occasionally seemed to "stall out" for a few seconds running 2.7--an issue that I had not had with 3.0.

The final impetus for doing a real comparison was the sluggishness of the desktop when building a UBCD4WIN ISO file. I decided this might be a good test, since it involves copying a large number of files from both a CD and from disk.

About Extensions

One thing that 2.7 and 3.0 have in common is that by default, they both scan all files, regardless of extension. This turns out to make a big difference in performance. For example, during the UBCD4WIN build process, there is a step that appends data to a 450KB file called txtsetup.sif. From observing the process in FileMon, I see that this process accesses the file thousands of time, apparently working with file offsets to copy one line at a time. When NOD32 (either version) is set to its default to scan all extensions, it takes six minutes to append to this file. However, if I uncheck the "all extensions" box in the NOD32 setup dialog and allow it to scan only its pre-defined list of extensions, UBCD4WIN blows by this append step in about five seconds.

The Numbers

My comparison between the versions was not terribly rigorous. I just ran the UBCD4WIN build process multiple times, noting the start and stop times. Since I didn't note seconds, there's at least a ±1 minute margin of error. I ran the tests on an old Dell Dimension 2400 (Pentium 4 2.66GHz) with 1GB of RAM. I compared NOD32 2.70.39 to 3.0.642, both with current virus signatures.

The times to build the UBCD4WIN ISO file were as follows:

No antivirus software installed:  9 minutes. 

NOD32 version 2.7, real-time scanning disabled:  9 minutes.
NOD32 version 2.7, scanning only specific extensions:  14 minutes.
NOD32 version 2.7, scanning all extensions:  20 minutes.

NOD32 version 3.0, real-time scanning disabled:  10 minutes.
NOD32 version 3.0, scanning only specific extensions:  14 minutes.
NOD32 version 3.0, scanning all extensions:  24 minutes.

With both versions, watching Performance Monitor, I saw the antivirus process frequently exceeding 90% CPU usage, especially during the six-minute scan of txtsetup.sif.

Conclusions

Perhaps the most obvious conclusion is that, regardless of NOD32 version, it's hard to justify accepting the default to scan all extensions, since it adds 40% or more to file access times. When set to only scan extensions that ESET identifies as vulnerable to viruses, there is no significant performance difference between the versions.

If performance is about the same, what other factors might influence one's choice between 2.7 and 3.0?

• Both versions have quirks that make it difficult or impossible to deploy some configuration settings using ESET's .xml configuration files.
• Version 3.0 has the important ability to exclude files and folders from on-demand and scheduled scans. It may be possible to work around this in 2.7 using Task Scheduler and NOD32's command-line scanner.
  
• In Version 3.0, the correlation between the client and the configuration editor (used for mass deployments) is less clear than in 2.7.

As the newer product, version 3.0 may still have issues that make 2.7 the safer bet for the short term. However, it's encouraging that version 3.0's real-time performance is on par with 2.7, at least with the specific-extensions list. I'd be interested to hear if others get similar results.