A longer chain stay would drop your bar some.
Not sure if you already have that or not?
Talk to Robert Holler at at Rose City Cycles since he is about the same height as you I think.
I understand your nervousness with the V at high speeds. Once I get above 30 mph on mine my confidence quickly evaporates.
The pendulum effect of the BB ahead of the wheel makes it seem like the bike could change course really quickly.
For the life of me I can't understand why they didn't spec a Viscoset headset instead of the integrated headset on the V20c. Robert said it wasn't needed on the new bike since the geometry had been tweaked. I have an older V20 so I can't really say if that's the case or not.
My S40 has the Viscoset and it seems to modulate the steering response somewhat making for more confidence inspiring downhill runs.
I would say to run the widest tires you can to see if that helps.
It also helps to have nerves of steel like Jason Perez.
Read his blog on racing the California Triple Crown to get some inspiration for your upcoming events.
At one point he was going so fast down a roller coaster mountain road he actually got airborne!
I still get airbourne regularly simply because 50+ mph and crappy road surfaces send a lot of energy through the frame. The instance Tuloose is referring to was when I floated away from the road over the top of an undulating road peak at high speed, that was quite the experience. I may have some of the most unique experiences on the V20 given I almost exclusively ride in the twisty, steep mountains, and I can descend and corner near a pro tour pace.
I couple of things that may help you wrap your brain around what you're feeling because people like me struggle to understand a feeling unless they can quantify and relate to it. A DF bike is self centering in that if you push it on it's own, it tries to straighten out on it's own even. So when you steer a DF bike you are adding inputs to steer it away from it's natural state of going straight. This is also often why when a motorcycle rider gets buckled off their motorcycle, the bike will often instantly straighten out and ghost ride away happy as can be. A V20 is opposite in that it wants to turn and as it turns it wants to exponentially turn more and more. So on the V20 your inputs are to keep it straight which is why you have to relearn how to ride a bike when you get on it, your input pressures are backwards. This may be due to all the weight out in front of the front axel. The other way to describe the feeling of steering a V20 is it feels similar to a rear wheel steering vehicle. Don't worry about the whole inputs being backwards thing, once you've learned it, you never forget it, and your brain makes the switch subconsciously.
You need to be able to see ahead if you're going to be riding in the mountains and any type of imperfect, unpaved areas. Either wedge up the rear of your seat to elevate to your shoulder and head or get a different seat that has that shape built in. My greatest struggle with the V20 initially was with the seat and headrest, which were different back then. Another thing I discovered, causing some input instability, was a lack of depth perception when cornering. That lack of depth perception was due to my head being tipped too far back and my nose blocking the apex of my path from my outside eye. That one took quite of few switchbacks to figure but once I did, I worked on getting my head more forward.
I've ridden my V20 across short sections of gravel, but I try to avoid it when possible. It's just not ideal, even if you can make adjustments to make it better. My recommended changes to make the V20 better on dirt. You'll notice the first 2 also recommended for road riding as well.
1. Practice riding the bike in general on the road to figure out its characteristics.
2. Your head angle and viewpoint adjustment
3. Bigger tires at lower PSI to reduce the road surface's impact on your steering input.
Best of luck mastering the machine, be safe, have patients.