I have an InnoDB table that needs to be re-populated every ten minutes within anywhere from 60k to 200k records. Our approach up to this point has been as follows:
- Turn off Autocommit
- Truncate the table
- Perform Select Queries & additional Calculations (using PHP)
- Insert new records
After the Truncate operation is performed though, the data is immediately deleted, and is no longer available from the User Interface. To our users, this has been pretty disconcerting, even though within about 30 seconds or so the script encounters the Commit operation and the table is repopulated.
I thought that perhaps I could wrap the whole operation, including the
Truncate, in a transaction, and that this might cut down on the length of time during which the table appears empty to users. So I changed
SET AUTOCOMMIT=0 to
Yikes! This had the opposite of the desired effect! Now the
TRUNCATE operation still occurs at the beginning of the script, but it takes much longer to actually execute the
INSERT operations within the transaction, so that by the time the
COMMIT operation takes place and the data in the table is available again, it has been nearly ten minutes!
What could possibly cause this? Truthfully, I wasn’t expecting any change at all, because I was under the impression that initiating a transaction basically just turns off
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
According to this URL, as of MySQL 5.1.32,
TRUNCATE TABLE is DDL and NOT DML like DELETE. This means that
TRUNCATE TABLE will cause an implicit
COMMIT in the middle of a transaction block. So, use
DELETE FROM on a table you need to empty instead of
DELETE FROM tblname; can be rolled back. It could take a while to rollback, so make sure InnoDB is properly tuned to handle the transaction time for such rollback possibilities.
A better way to accomplish this might be to insert the data into a new table, and then use rename on both tables in order to swap them. A single rename is all that’s needed for the swap, and this is an atomic action, which means the users won’t even be able to detect that it happened, except for the new data showing up. You can then truncate/delete the old data.
From your description I can’t really explain your time difference. The only thing that comes to mind is that you don’t actually wrap the inserts into one transaction, but loop it.
The key difference with SET AUTOCOMMIT=0 is that if it’s already 0, it won’t do anything, where as with START TRANSACTION you will initiate a sub transaction within the current transaction.
COMMIT so that transaction is not an ACID transaction. This is noted above.
I use your approach in MySQL to emulate an
OUTER JOIN of two tables by email address. The result stays in a table I can quickly
INNER JOIN later.
Your approach is already having out-of-date data (since you require
DELETE). So here is another approach which also uses out-of-date data, but foregoes transactions entirely. Less locking FTW.
INSERT... ON DUPLICATE KEY UPDATE and mark an “update time”. At end of your script,
DELETE anything with an old “update time”.