I have a reasonable size (18GB compressed) HDF5 dataset and am looking to optimize reading rows for speed. Shape is (639038, 10000). I will be reading a selection of rows (say ~1000 rows) many times, located across the dataset. So I can’t use x:(x+1000) to slice rows.
In solid mechanics, I often use Python and write code that looks like the following: for i in range(3): for j in range(3): for k in range(3): for l in range(3): # do stuff I do this really often that I start to wonder whether there is a more concise way to do this. The … Read more
I need to find the index of the first value in a 1d NumPy array, or Pandas numeric series, satisfying a condition. The array is large and the index may be near the start or end of the array, or the condition may not be met at all. I can’t tell in advance which is more likely. If the condition is not met, the return value should be -1. I’ve considered a few approaches.
Requirements:
I asked the most efficient method for mass dynamic string concatenation in an earlier post and I was suggested to use the join method, the best, simplest and fastest method to do so (as everyone said that). But while I was playing with string concatenations, I found some weird(?) results. I’m sure something is going on but I can’t not get it quite. Here is what I did:
I have a NumPy array with integer values. Values of matrix range from 0 to max element in matrix(in other words, all numbers from 0 to max data element presented in it). I need to build effective( effective means fast fully-vectorized solution) for searching number of elements in each row and encode them according to matrix values.
TL;DR how to link ATLAS/MKL to existing Numpy without rebuilding.
I have taken Problem #12 from Project Euler as a programming exercise and to compare my (surely not optimal) implementations in C, Python, Erlang and Haskell. In order to get some higher execution times, I search for the first triangle number with more than 1000 divisors instead of 500 as stated in the original problem.
The other day I was doing some Python benchmarking and I came across something interesting. Below are two loops that do more or less the same thing. Loop 1 takes about twice as long as loop 2 to execute.
I have an existing sqlite3 db file, on which I need to make some extensive calculations. Doing the calculations from the file is painfully slow, and as the file is not large (~10 MB), so there should be no problem to load it into memory.