IA-32e 64-bit and compatibility mode

Hi,

From Intel developer manual I see

IA-32e mode allows software to operate in one of two sub-modes:

•          64-bit mode supports 64-bit OS and 64-bit applications
•          Compatibility mode allows most legacy software to run; it co-exists with 64-bit applications under a 64-bit OS

I have a quad-core with HT enabled i.e. 8 logical cores

With KVM I want to run 32 bit and 64 bit VMs on a 64 bit Host OS. I want to allocate resources as follows

1. VM-1 64 bits 2-VCPUs (1 physical-core with HT enabled i.e. 2 logical cores/threads)
2. VM-2 32 bits 6-VCPUs (3 physical-cores with HT enabled i.e. 6 logical cores/threads)

If VM-1 2-VCPUs are assigned per-thread i.e. they can end up on 2 physical cores shared with VM-2. If this happens cpu switching between compatibility and 64-bit mode can impact performance? 

If I make sure VM-1 2-VCPUs are assigned to 1 physical-core and its enabled for 64-bit mode and VM-2 6-VCPUs are assigned to 3-physical cores and enabled for compatibility mode then we can avoid mode switching? 

In general will there be performance impact on switching between 2 CPU modes? What additional things I need to consider to run both 32-bit and 64-bit VMs without any performance impact?

Thanks,

Ravi