Why does the X Window System use a server?

I never really understood why a window system must have a server. Why do desktop environments, display managers and window managers need xorg-server? Is it only to have a layer of abstraction on top of the graphics card? Why do window systems employ a client-server model? Wouldn’t inter-process communication via named pipes be simpler?