I can totally relate as I have been there myself. I found that it all came down to technique in terms of stick grip and mechanics. The more time I spend understanding how grip affected the way I executed a stroke, the more I learned that in most cases I was doing more work than I needed to. By scaling back on the motion used between arms, wrist, and fingers, I was eventually able to establish a consistant technique that would work for all levels of playing.
When I teach now, I tell my students, “Drumming is about being lazy and we, as drummers, want to get more for less. We want the most music for the least effort.” By demonstrating how to scale back the physical motion of a stroke (arms to wrists to fingers) I can now demonstrate how it’s possible to play efficiently using smaller motions and only employing larger movements when trying to acheive very strong accents.
When dropping volume in low dynamic phrases and passages, I found that the struggle resulting in tempo loss was caused by trying to adjust the technique on the fly - essentially changing the way I execute strokes in order to reach that lighter dynamic. Now, having realized that I can use a relaxed techinque to for all dynamic levels, the change is no challenge at all. I’m ready and don’t have to alter anything besides the smallest finger motions.
By incorporating this new found approach into all my playing and practicing (especially rudiment work) a was able to gain the consistant ability I only saw in others. I strongly encourage others who are struggling with this issue to look at their overall technique and ask themselves - “Am I working to hard?”