Firstly, this is not a hard rule so you don't have to start over. Dates are compliant, so unless you're using them to recreate baked goods or treats (which is explicitly against the rules), then you haven't broken any rules and don't have to start over. However, it is a recommendation to not use dates to recreate your favorite sugary coffee creamer.
So I'd look at your context. Do you need something sweet in your coffee in the mornings? Did the almond milk recreate that experience for you? If you're using it as a crutch because you need to make your coffee more sweet or because you need something sweet in the morning, then I'd probably ditch it. If you've been propping up sugar cravings and feel like this might take something away from your program, perhaps start over. If not, I think you're doing just fine and should continue on!