Have you ever run into a scenario where you have to loop through a set of data and perform database queries for each iteration? Obviously we shouldn’t run them one by one, so we end up using a TRANSACTION or even just building out one large query in the case of INSERTing a bunch of records. Now let’s say we’re not using a traditional RDBMS and instead we’re using Redis.
Pipelining accomplishes this need by allowing you execute multiple commands against the Redis server without the latency of running each one individually. You can mix and match different commands when you are pipelining, for instance, if you want to loop through an array and increment one key, decrement another and update the value of a key, you can do so.
In this example I’m working with an array of new usernames indexed by user ID. I’m going to increment a key that contains the number of times the user has changed their username, and then update the username key directly:
$redis = new Redis
// Opens up the pipeline
$pipe = $redis->multi(Redis::PIPELINE
// Loops through the data and performs actions
foreach ($users as $user_id => $username)
{
// Increment the number of times the user has changed their username
$pipe->incr('changes:' . $user_id
// Changes the username
$pipe->set('user:' . $user_id . ':username', $username
}
// Executes all of the commands in one shot
$pipe->exec
There’s no limit to what commands you can use, even if the commands return data, so let’s say we want to increment a key and grab another. In this example I’m looping through an array of user IDs and incrementing a key that tracks how many times the user key has been accessed, then grabs the user key’s value:
$redis = new Redis
// Opens up the pipeline
$pipe = $redis->multi(Redis::PIPELINE
// Loops through the data and performs actions
foreach ($users as $user_id => $username)
{
// Increment the number of times the user record has been accessed
$pipe->incr('accessed:' . $user_id
// Pulls the user record
$pipe->get('user:' . $user_id
}
// Executes all of the commands in one shot
$users = $pipe->exec
// Dumps our data
print_r($users
Keep in mind, the $users array that is returned will contain entries for the increments as well as the keys you pulled with get since every command returns a value. Additionally, the return array will be indexed with integers and are sorted in the same order that you ran the commands. You will have to loop through the array and do some data manipulation to get cleaner / well indexed results. I’ve gotten fancy with this in the past by creating a separate array that I could use with array_combine to index the results.
In case you need to cancel a transaction that you’ve started, you can simply run $pipe->discard().
Why pipeline? Well for me personally, I’ve been able to dramatically improve performance on some of my own sites by leveraging pipelining. It’s good practice to never hit a data store inside of a loop so remember to use pipeline() when you are faced with those situations! Has pipelining saved your day before? Comment below!
