Exclude site from mod_cfml

Does anyone know how to exclude an Apache virtual host from being picked up by mod_cfml if a URL with one of the CFML extensions is called? I’ve tried adding:

    RemoveHandler .cfm .cfc .cfml

To the VirtualHost configuration but that didn’t help. I’ve also tried adding:

    CFMLHandlers " "

To the VirtualHost configuration but that didn’t help either.

Thanks.