performance Archives - Page 14 of 20

Fastest save and load options for a numpy array

August 12, 2022 by Magenaut

I have a script that generates two-dimensional numpy arrays with dtype=float and shape on the order of (1e3, 1e6). Right now I’m using np.save and np.load to perform IO operations with the arrays. However, these functions take several seconds for each array. Are there faster methods for saving and loading the entire arrays (i.e., without making assumptions about their contents and reducing them)? I’m open to converting the arrays to another type before saving as long as the data are retained exactly.

How do I re.search or re.match on a whole file without reading it all into memory?

August 12, 2022 by Magenaut

I want to be able to run a regular expression on an entire file, but I’d like to be able to not have to read the whole file into memory at once as I may be working with rather large files in the future. Is there a way to do this? Thanks!

Mass string replace in python?

August 12, 2022 by Magenaut

Say I have a string that looks like this:

c++11 regex slower than python

August 12, 2022 by Magenaut

hi i would like to understand why the following code which does a split string split using regex

time.time vs. timeit.timeit

August 11, 2022 by Magenaut

Sometimes, I like to time how long it takes parts of my code to run. I’ve checked a lot of online sites and have seen, at large, two main ways to do this. One is using time.time and the other is using timeit.timeit.

Why is variable1 += variable2 much faster than variable1 = variable1 + variable2?

August 11, 2022 by Magenaut

I have inherited some Python code which is used to create huge tables (of up to 19 columns wide by 5000 rows). It took nine seconds for the table to be drawn on the screen. I noticed that each row was added using this code:

why isn’t numpy.mean multithreaded?

August 11, 2022 by Magenaut

I’ve been looking for ways to easily multithread some of my simple analysis code since I had noticed numpy it was only using one core, despite the fact that it is supposed to be multithreaded.

System lags when doing large R/W operations on external disks

August 11, 2022 by Magenaut

I am having some issues with system-wide latency/lagging when doing large disk imaging operations on an Ubuntu 18.04 system. Here’s the system specs:

Should I care about unnecessary cats?

August 11, 2022 by Magenaut

A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially if the first command would need multi-line arguments.