What keeps draining entropy?

If I do watch cat /proc/sys/kernel/random/entropy_avail I see that my systems entropy slowly increases over time, until it reaches the 180-190 range at which point it drops down to around 120-130. The drops in entropy seem to occur about every twenty seconds. I observe this even when lsof says that no process has /dev/random or /dev/urandom open. What is draining away the entropy? Does the kernel need entropy as well, or maybe it is reprocessing the larger pool into a smaller, better quality pool?