In the company I work for, there was a campaign running for a group of users. Our task was to show some information to the user when they logged into our app if they were in the selected group for the campaign. The group was maintained by operators. When a user joined the campaign, they sent the user information into a messaging queue1 for us to add to our end.
We could store the campaign users in a table in a database, but that would cause a performance issue for us. In our app use case, login should be as fast as possible. So, we could benefit from a cache to check if the user is in the campaign or not. However, this would mean there would be a database query every time a user logged into our app, which would increase the login response time for all users, which was not acceptable.
Our idea was to insert every user into both the database and cache2 while consuming messages. Note that we need to keep the data in persistent storage anyway. We implemented a naive app to test the approach. We observed a few inconsistencies between the database and cache data. There were some users in the database who were not present in the cache. This could happen for a few reasons, such as the consumer inserting into the database but failing to insert into the cache, or the cache getting emptied for some infrastructure reason.
We had to handle these inconsistencies. What we did was to not acknowledge a message unless we had inserted it into both the database and cache, and somehow keep it in the messaging queue to consume later (this was a feature in Solace). Furthermore, we ran a parallel cron job that compared the timestamp of the last write in the cache with the last write in the database, and if the cache was behind, it would rebuild the cache from the database. For rebuilding, we had to have a lock to prevent updates to the cache while rebuilding. We used atomic operations3 from Redis to achieve this.
Furthermore, what was interesting to me was that bulk inserting into Redis is really fast. Bulk inserting a batch of 1 million records into Redis took 12 seconds, but if we did it in 100 batches of 10,000 records, it would take ~70 seconds. So, while rebuilding, try to make your batch sizes as large as possible.