Why python has limit for count of file handles?

I writed simple code for test, how much files may be open in python script:

for i in xrange(2000):
    fp = open('files/file_%d' % i, 'w')
    fp.write(str(i))
    fp.close()

fps = []
for x in xrange(2000):
    h = open('files/file_%d' % x, 'r')
    print h.read()
    fps.append(h)

and I get a exception

IOError: [Errno 24] Too many open files: 'files/file_509'

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

The number of open files is limited by the operating system. On linux you can type

ulimit -n

to see what the limit is. If you are root, you can type

ulimit -n 2048

now your program will run ok (as root) since you have lifted the limit to 2048 open files

Method 2

I see same behavior on Windows when running your code. The limit exists from C runtime. You can use win32file to change the limit value:

import win32file

print win32file._getmaxstdio()

The above shall give you 512, which explains the failure at #509 (+stdin, stderr, stdout as others have already stated)

Execute the following and your code shall run fine:

win32file._setmaxstdio(2048)

Note that 2048 is the hard limit, though (hard limit of the underlying C Stdio). As a result, executing the _setmaxstdio with a value greater than 2048 fails for me.

Method 3

To check change the limit of open file handles on Linux, you can use the Python module resource:

import resource

# the soft limit imposed by the current configuration
# the hard limit imposed by the operating system.
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print 'Soft limit is ', soft 

# For the following line to run, you need to execute the Python script as root.
resource.setrlimit(resource.RLIMIT_NOFILE, (3000, hard))

On Windows, I do as Punit S suggested:

import platform

if platform.system() == 'Windows':
    import win32file
    win32file._setmaxstdio(2048)

Method 4

Most likely because the operating system has a limit for the number of files that an application can have open.

Method 5

On Windows one can get or set the limit with the built-in ctypes library:

import ctypes
print("Before: {}".format(ctypes.windll.msvcrt._getmaxstdio()))
ctypes.windll.msvcrt._setmaxstdio(2048)
print("After: {}".format(ctypes.windll.msvcrt._getmaxstdio()))

Method 6

Since this is not a Python problem, do this:

for x in xrange(2000):
    with open('files/file_%d' % x, 'r') as h:
        print h.read()

The following is a very bad idea.

fps.append(h)

Method 7

The append is needed so the garbage collector does not clean up and close the files

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0