Tricky new malware unnerves security vendors