Arithmetic mod $n$ is arithmetic with cosets of $n\mathbb{Z}$: we add classes by adding any representatives. The same works in any group, provided the subgroup is normal.
Why it works
Theorem. If $N$ is normal in $G$, the rule $(aN)(bN) = abN$ is well defined, and with it the set $G/N$ of cosets is a group, the quotient group (or factor group). Its identity is $eN = N$, the inverse of $aN$ is $a^{-1}N$, and $|G/N| = |G|/|N|$.
Proof of well-definedness. Suppose $aN = a'N$ and $bN = b'N$, so $a' = an_{1}$ and $b' = bn_{2}$ with $n_{1}, n_{2} \in N$. Then $$a'b' = a\,n_{1}\,b\,n_{2} = ab\,(b^{-1}n_{1}b)\,n_{2}.$$ Since $N$ is normal, $b^{-1}n_{1}b \in N$, so $a'b' \in abN$ and $a'b'N = abN$. The group axioms are inherited from $G$: $(aN)(bN)(cN) = abcN$ either way, $(eN)(aN) = aN$, and $(a^{-1}N)(aN) = N$. $\square$
Proof of well-definedness. Suppose $aN = a'N$ and $bN = b'N$, so $a' = an_{1}$ and $b' = bn_{2}$ with $n_{1}, n_{2} \in N$. Then $$a'b' = a\,n_{1}\,b\,n_{2} = ab\,(b^{-1}n_{1}b)\,n_{2}.$$ Since $N$ is normal, $b^{-1}n_{1}b \in N$, so $a'b' \in abN$ and $a'b'N = abN$. The group axioms are inherited from $G$: $(aN)(bN)(cN) = abcN$ either way, $(eN)(aN) = aN$, and $(a^{-1}N)(aN) = N$. $\square$
Normality is exactly what makes this work. In $S_{3}$ with $H = \{e, (1\;2)\}$, the cosets $(1\;3)H$ and $(1\;3)(1\;2)H$ are equal, yet multiplying each by $(1\;3)H$ "by representatives" gives two different cosets. With a non-normal subgroup there is no quotient group.
Worked example: Computing in $\mathbb{Z}_{12}/\langle 4 \rangle$
In $G/N = \mathbb{Z}_{12}/\langle 4 \rangle$, compute $(3 + N) + (2 + N)$ and find the order of $1 + N$.
- $N = \{0, 4, 8\}$, and $|G/N| = 12/3 = 4$: the cosets $N, 1 + N, 2 + N, 3 + N$.
- $(3 + N) + (2 + N) = 5 + N = 1 + N$, since $5 - 1 = 4 \in N$.
- $1 + N$, $2 + N$, $3 + N$, $4 + N = N$: the order of $1 + N$ is $4$, although $|1| = 12$ in $\mathbb{Z}_{12}$.
Answer: $1 + N$; order $4$.
Worked example: A quotient of $D_{4}$
Describe $D_{4}/N$ for $N = \{e, r^{2}\}$.
- $|D_{4}/N| = 8/2 = 4$: the cosets $N$, $rN = \{r, r^{3}\}$, $sN = \{s, r^{2}s\}$, $rsN = \{rs, r^{3}s\}$.
- Each non-identity coset squares to $N$: $r^{2} \in N$, $s^{2} = e$, $(rs)^{2} = e$.
- A group of order 4 in which every element has order at most 2 is not cyclic.
Answer: A non-cyclic group of order $4$, like $U(8)$.
Common mistake
Taking the order of a coset to be the order of its representative. The order of $aN$ is the least $m$ with $a^{m} \in N$, which can be much smaller than $|a|$: in $\mathbb{Z}_{12}/\langle 4 \rangle$ the coset $1 + N$ has order $4$, while $1$ has order $12$ in $\mathbb{Z}_{12}$. By Lagrange, the order of $aN$ divides $|G/N|$.
Key idea
For a normal subgroup $N$, cosets multiply by representatives, $(aN)(bN) = abN$, giving the quotient group $G/N$ of order $|G|/|N|$. The order of $aN$ is the least $m$ with $a^{m} \in N$.