Check your System Restore settings.
If you have enabled in System Properties, the System Restore tab, to monitor all drives, you might have an active monitor.
Turn off monitoring all drives, and select only your fixed disks for monitoring.
(Right click My Computer, Properties, System Restore tab)
There is no question that antivirus programs can cause this, but I suspect you would have seen this previously if this was the case. While malware can cause all sorts of issues, my gut instinct tells me it is not the cause of your problem in this case.
You should also check on the System Properties sheet that XP has not decided to use a portion of the removable drive as Virtual Storage. It should not, but check. Click the Advanced tab, and check under Performance options. At the least, make static assignments to your fixed volumes rather than let the system decide. This is a long-shot, but one worth investigating.
Finally, if the device is USB 2.0, they are designed to be hot-swappable. Meaning, that if you are absolutely sure all sessions are closed on the drive, go ahead and remove it.
My own guess: you have Most Recently Used "MRUs in the trade) active through an Office or other application, and have an MRU pointer still active to a recently used file. If so, go ahead and unplug it.