Some autorun based viruses and, more recently, a thread on the Wine developer mailing list had me thinking a bit about a reasonable way of identifying potentially dangerous Windows executables.
Background
Windows XP SP2
Recent version of Windows (from XP SP2 IIRC) has a mechanism (which seem to be based on IAttachmentExecute) to determine whether or not to prompt the user before running an application that might be consider dangerous. In Windows XP, this is at least files that were downloaded from the Internet and files on network shares under certain circumstances. Browsers, Firefox 3 and IE 6 that comes with Windows XP SP2, marks files that was downloaded from the Internet in order for this to work. Some more details about this can be found elsewhere.
For the Windows XP implementation of this, the user has an option to disable prompts to run the file, either from the file properties dialog or from the prompt that opens. IIRC signed files allow you to disable the prompt for all files from a publisher. (I cannot find any information on the handling of signed files)
This mechanism was probably added to remind the user that the program is potentially dangerous and that it should be checked before being run.
Autorun based malware
Autorun based "viruses" (trojans or worms are probably more correct) are another, but related problem. This kind of malware spread by infecting drives, including removable drives, such as external hard drives, USB flash drives, SD cards, etc. with an executable that is triggered from the autorun.inf file when the drive is connected to a computer or if autorun is triggered manually. (Double clicking on the drive in My Computer usually has this effect.) The risk of infection with this kind of malware can be mitigated by disabling outrun on removable drives.
The main factor that enable the effective distribution of autorun based malware is that autorun is enabled writable, removable devices, which allows any application that can write (on any computer where it is used) to the drive to execute arbitrary code on another computer. CDs and DVDs can also be infected, but the opportunities are limited for the read-only and non-rewritable versions, since they are normally not easily writable directly (special software are typically used to write to these media) and only a few computers have the opportunity to write to it.
Complete protection
The most common way (other ways, such as exploits are not considered here) that malware reaches a computer is from an external source, such as the Internet, e-mail attachments, network shares and removable drives. The mechanism mentioned earlier, describes the approach that Microsoft took to mitigate the risk for some cases. The risk could be mitigated further by including additional external sources of executables to the list of sources that is distrusted by default. The only completely effective way to ensure that the user is prompted before any malware is run, is to distrust anything that was not part of the operating system (these can be made identifiable with a digital signature) by default. The user would then need to specifically whitelist any other executable that he / she wants to use. A slight tradeoff would be to allow the user to whitelist the vendor of signed applications, rather each executable.
Another tradeoff would be to trust everything that was marked as trustworthy by a trusted application. This should allow, when the user trusts an installer, to run the applications installed by it without further prompts.
A problem with any system that allows executables to be marked trusted, without restricting applications from marking others as trusted, is that trusting a single malicious executable can bypass the trust system for any other files. Vista UAC seem to indicate that users do not like to be prompted about the actions of a program. Users are likely to ignore prompts anyway if they do not know what it is about.
Suggestions
The suggestions are based on my view of a theoretical ideal and do not take the feasibility of the implementation of the suggestions into account.
Security and usability was the criteria used to decide on these suggestions.
A sensible middle ground
Existing files on local, fixed drives and read-only removable drives are more trustworthy than files on a writable removable disk, or files that is known to be from an external source. (A read-only disk is not completely safe and old files on a local disk are only safe if they were checked as they arrived)
Prompting users to run files that they have just requested to run will be annoying or even confusing to some users. This is made worse by users that simply ignores anything that looks like an error message and clicks whichever option seem likely to make it go away.
Ignoring possible annoyance to users, a reasonable level of protection would involve prompting the user before running anything from a writable removable drive, writable network share or anything that was downloaded from the Internet. (The writable requirement is to find files that could have been modified by anyone)
For files with a digital signature, a SSH-like trust system should be able to reduce annoyance by allowing the user to trust a specific certificate (irrespective whether it was self-signed or signed by a "trusted" certification provider) and not prompting again from applications signed with that certificate.
If easily protectable from malicious ratings, a web based trust indicator for certificates (or individual executables for unsigned files) might optionality be used in order to guide users. (This has privacy implications, and the user should opt-in explicitly)
Mapping trustworthiness in a Wine environment
Drives that should be distrusted by default should be marked by the user or his distribution by mounting them with the noexec option. (Candidates for this should be writeable removable storage (flash drives, but not CDs) and network shares) The user can then be prompted before running the file. (The user should be able to disable the prompts in the same way as the attachment manager prompts)
Files on noexec volumes should be able to be marked as trustworthy. If the filesystem is writable and supports, extended attributes can be used, otherwise a database needs to be kept in the WINEPREFIX.
For other volumes, files can be assumed to be trustworthy if their executable flag is set and can be marked as non-trustworthy by removing the bit. Extended attributes are another option, and a database in the WINEPREFIX (a system-wide database in /etc might be useful for administrators) should always be an option as a fallback on a non-writable filesystem. Windows applications should be able to manipulate a file's trustworthiness in the same way as used on Windows XP SP2 or higher.
Prompts should at least tell the user to run an anti-virus on the file before running it, allow him to run the file, cancel running the file or add the file to a list of trusted files.
For signed files, an option to trust / distrust the certificate should be present as well.
