How to prevent fork bomb?

To prevent fork bomb I followed this http://www.linuxhowtos.org/Tips%20and%20Tricks/ulimit.htm

ulimit -a reflects the new settings but when I run (as root in bash) :(){ :|:&};: the VM still goes on max CPU+RAM and system will freeze.

How to ensure users will not be bring down the system by using fork bombs or running a buggy application?

OS: RHEL 6.4

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

To make this change pervasive you’ll need to add these limits to the entire environment. Changes using the ulimit command are only to the current environment.

NOTE: This will have no effect on the root user!

Example

Edit this file: vi /etc/security/limits.conf and add entries to the file limiting the number of processes (nproc) that a specific user or group of users’ is allowed to have.

vivek hard nproc 300
@student hard nproc 50
@faculty soft nproc 100
@pusers hard nproc 200

NOTE: There are more examples in that file. Be careful with using the “all” (aka. *) this will limit system accounts too.

References

• How to: Prevent a fork bomb by limiting user process

Method 2

The superuser or any process with the CAP_SYS_ADMIN or CAP_SYS_RESOURCE capabilities are not affected by that limitation, that’s not something that can be changed. root can always fork processes.

If some software is not trusted, it should not run as root anyway.

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0