When I had a S40, I did use the headrest for longer rides, and didn't mind it for shorter commutes.
I don't know what brand of helmet you are using, but I solved my dang-this-helmet! issue by switching to the Lazer brand, which places the retention knob at the top of the helmet rather than in the back, right where the headrest pad is. I am currently using the G1 Mips. The back of the retention band slides up and down for adjustment, and I have it adjusted flush with the lower edge of the helmet so that the back of my skull is completely unobstructed, and the slightly curved edge nestles just right with the shape of the headrest pad on my V20c--a bike for which the headrest is not optional. I had the same helmet clearance (maybe even a little bit more) on the S40 headrest.