I have managed lots of riding
I have managed lots of riding over many days, weeks, and months with a sleeping bag and bivy sack on top of my rear rack, a pannier on each side of the rack, and a tennis ball container filled with tools, first aid, flat kit, etc. mounted in a water bottle cage on the steer tube between my thighs. Try to make everything you bring multi-functional. For example, I didn't carry warm pants or a jacket. Instead I had long johns, a tight fitting dry-fit shirt, a looser warm shirt, rain jacket and rain pants (and tight shorts and shirt for warm weather riding). I layered these to suit the conditions.
Keep in mind that what you're trying to bring is the insulation in the sleeping bag and clothes, the waterproofing of the bivy sack, etc. You are trying to bring with you the ability to meet the demands of your needs and wants. So try to bring them with as little extra stuff as possible. Panniers, compression sacks, and bungee cords don't serve a direct purpose; they're simply there to contain the things you actually need. So limit the containers and restrainers as much as possible.