This can happen if your time zone is different from the one configured in Task Reminder. Head to the Task Reminder settings page and make sure the time zone is set correctly.
If you have users from multiple time zones in your instance, there unfortunately is no way to make this work for all time zones. For Task Reminder to be able to send the right dates to every user from every time zone, it needs to be able to get the time zone of each user. This is currently not implemented by Atlassian.
If you want to help us fix this, please vote on this Confluence Cloud issue: CONFCLOUD-69670