Looping through Dask array made of npy memmap files increases RAM without ever freeing it
Context I am trying to load multiple .npy files containing 2D arrays into one big 2D array to process it by chunk later.All of this data is bigger than my RAM so I am using the memmap storage/loading system here: pattern = os.path.join(FROM_DIR, '*.npy') paths = sorted(glob.glob(pattern)) arrays = [np.load(path, mmap_mode='r') for path in paths] … Read more