Mathematically a physical modular invariant is an invariant of a Lagrangian algebra in the product of (chiral) modular categories. The chiral modular category of a G -orbifold of a holomorphic conformal field theory is the so-called (twisted) Drinfeld centre Z(G,α) of the finite group G . We show that the diagonal modular invariant for Z(G) is physical if and only if the group G has a double class inverting automorphism, that is an automorphism ϕ:G→G with the property that for any commuting x,y∈G there is g∈G such that ϕ(x)=gx−1g−1, ϕ(y)=gy−1g−1.
Groups without double class inverting automorphisms are abundant and provide examples of chiral conformal field theories for which the diagonal modular invariant is unphysical.