Proof:For every homomorphism$\varphi : G \rightarrow H$ we have

1. For all$a, b \in G$$$\varphi(ab) = \varphi(a)\varphi(b)$$
2. $$\varphi(e_{G}) = e_{H}$$
3. For every$a \in G$$$\varphi(a^{-1}) = \varphi(a)^{-1}$$

Let$a \in S$ be arbitrary. Consider the set$a^{-1}S = \{ a^{-1}s : s \in S \}$. By assumption we have$$|a^{-1}S| = |S| > \frac{3}{4} |G|$$We also have$$|S \cup a^{-1}S| + |S \cap a^{-1}S| = |S| + |a^{-1}S| = 2 |S| > \frac{3}{2}|G|$$Since$S \cup a^{-1}S \subset G$,$|S \cup a^{-1}S| \le |G|$ and it follows that$$|S \cap a^{-1}S| > \frac{1}{2} |G|$$Let$s \in S \cap a^{-1}S$ be arbitrary. We have$s \in S$ and$as \in S$ and we deduce that

1. $$\phi(s) = s^{-1}$$
2. $$\phi(as) = (as)^{-1} = s^{-1} a^{-1}$$$$\phi(as) = \phi(a)\phi(s) = a^{-1} s^{-1}$$

That is,$a$ and$s$ commute. Since this holds for all$s \in S \cap a^{-1}S$ we have$$S \cap a^{-1}S \subset C_{G}(a)$$Where$C_{G}(a)$ is the centralizer of the singleton set$\{a\}$ in$G$. Clearly$C_{G}(a)$ is a subgroup of$G$ and$$|C_{G}(a)| \geq |S \cap a^{-1}S| > \frac{1}{2}|G|$$By Lagrange's theorem

$$|G| = [G: C_{G}(a)] |C_{G}(a)| > [G: C_{G}(a)] \frac{1}{2} |G| \Rightarrow \\ [G:C_{G}(a)] < 2 \Rightarrow [G:C_{G}(a)] = 1 \Rightarrow G = C_{G}(a)$$

and$a$ commutes with all members of$G$, in other words$a \in Z(G)$ the center of the group$G$.

Since$a \in S$ is arbitrary,$S \subset Z(G)$ and$|Z(G)| > \frac{3}{4}|G|$. Since$Z(G)$ is a subgroup of$G$, again by using Lagrange's theorem as above we deduce that$G = Z(G)$, which means that$G$ is an abelian group.

Finally, let$a \in G$ be arbitrary. Since$|S| = |aS| > \frac{3}{4} |G|$ a similar argument to the one given above shows that$S \cap aS \neq \emptyset$.Assume that$s \in S \cap aS$ with$s = at$ and$t \in S$. Since$G$ is abelian we have

1. $$\phi(s) = s^{-1} = (at)^{-1} = t^{-1} a^{-1} = a^{-1} t^{-1}$$
2. $$\phi(s) = \phi(at) = \phi(a) \phi(t) = \phi(a) t^{-1}$$

We conclude that$$\phi(a) = a^{-1}$$and the proof is complete.

To construct a finite group$G$ and an automorphism$\phi : G \rightarrow G$ such that for exactly$\frac{3}{4}$ of elements$g \in G$ we have$\phi(g) = g^{-1}$ first observe that$G$ needs to be non-abelian. Since otherwise a minor modification to the last part of above proof shows that$\phi$ sends all elements to their inverse. Also$|G|$ must be divisible by$4$ in order to$\frac{3}{4}|G|$ be a positive integer.

As @KeithKearnes mentioned in the comments we only need to search for an example in groups with less that$10$ elements. The only possible cardinalities divisible by$4$ are$\{4, 8\}$. But every group with$4$ elements is abelian. Hence we need to find a non-abelian group with$8$ elements. @KeithKearnes gave two examples.

First example:Let$D_{4}$ be the dihedral group of order$8$ defined by$$D_{4} = <r, s : s^{2} = e, r^{4} = e, srs = r^{-1} > = \{e, r, r^{2}, r^{3}, s, rs, r^{2}s, r^{3}s\}$$and let$\phi : D_{4} \rightarrow D_{4}$ be the identity automorphism, that is for all$g \in D_{4}$,$\phi(g) = g$. By a straightforward calculation it follows that if$$ S = \{e, r^{2}, s, rs, r^{2}s, r^{3}s\}$$then for all$s \in S$,$\phi(s) = s^{-1}$ and for all$g \in D_{4} - S$,$\phi(g) \neq g^{-1}$. Therefore$\phi$ sends exactly$\frac{3}{4}$ of elements of$D_{4}$ to their inverse.

Second example:Let$Q_{8}$ be the quaternion group defined by$$Q_{8} = <\bar{e}, i, j, k: \bar{e}^{2} = e, i^{2} = j^{2} = k^{2} = ijk = \bar{e} > = \{1, i, j, k, -1, -i, -j, -k \}$$Here$e = 1$ and$\bar{e} = -1$. Let$\phi : Q_{8} \rightarrow Q_{8}$ be the conjugation by$i$, that is for$g \in Q_{8}$ we have$\phi(g) = i^{-1} g i = -i g i$. It is clear that$\phi$ is an automorphism, it is an inner automorphism. It is straightforward to see that for all$g \in \{1, j, k, -1, -j, -k \}$ we have$\phi(g) = g^{-1}$ and for all$g \in \{i, -i\}$ we have$\phi(g) \neq g^{-1}$. Hence$\phi$ sends exactly$\frac{3}{4}$ of the elements of$Q_{8}$ to their inverse.

I am thankful to @KeithKearnes for the examples.

• gr.group-theory
• finite-groups
• combinatorial-group-theory
• automorphisms