Sounds like your starter sprag clutch is shagged dude. Very common problem with the 1100's from 99 to 2002.
Like you I got stung by this, mine's a 2002 too. Apparently they sorted the problem with the 2003 model onward. I'd always wondered why spares websites listed the XVS 1100 as 99-02 and 03-10 models.... that's the difference.
The way they sorted it was by replacing the starter motor with an "updated" version which has a "torque limiting clutch" which allows the engine kickback on shutdown. Before, when the engine shuts down, depending on where the pistons are at that moment, the crank could rotate backwards a few degrees before it finally comes to a complete stop. This causes the starter motor to be turned by the engine, causing twisted starter motor bolts you may read about on tinterwebnetz.
The reason is, the starter clutch (sprag clutch) can only turn in one direction and lock up in the other direction. So when your starter turns the sprags in the starter clutch catch the crank and turns it, and engine then starts.
When the engine starts, the crank turns faster than the starter motor and this is allowed for by the starter clutch.
It's kinda hard to get it pictured in your head.
But anyway, the thing is the engine kickback on shutdown turns the starter sprag clutch in the direction it doesn't go. So this unwanted torque is manifested in 2 ways - a) your starter motor gets twisted, b) starter clutch sprags get worn from this shearing torque turning it the way its not supposed to go. Eventually the sprags can't grip the crank anymore and BOOOYA your starter turns but your engine doesn't.
Do you remember a kind of short and sharp clutch grinding noise when you shutdown the engine? I have a video of mine doing it. I should post it...
That's the sound of your sprag clutch being worn down.
Mine began by making a whacking noise on startup, turns out it was the worn sprags engaging suddenly when the motor was turning full speed, as opposed to engaging when the motor begins to turn.
But I had the clutch grind/scrape noise on shutdown since day 1 (bought the bike last November with 3k miles on it, 12k on it now

)
So anyway Yamaha's new updated starter motor is in fact very very similar to the old one. The old one has the same torque limiting clutch in it, just it has a higher torque setting - so the new motor gives way to the kickback (lower torque setting) where the old one would hold tight and hence get twisted starter bolts on the starter housing.
I've opened up both the old and new starter motors and took pics, I really should post them too.
So in the end I replaced the starter motor, as if you replace the starter clutch on it's own, it will get worn down again due to the engine kickback not being allowed for by the starter motor.
So you need the new motor to take this punishment so the starter clutch doesn't have to. Look for part number 5EL8189001 on ebay (make sure it's not 5EL8189000, that's the old one), cheapest you'll get it. They know you need it so they charge you an arm & a leg.
You'll have to replace the starter clutch too - part number 5EL1559000. Cheapest place I found it was
here.
As for the starter idler gear part number 5EL1551711, I didn't bother getting a new one of these (more expense) but the surface of mine was scored by the sprags scraping it. So I put mine in a lathe and polished it off.
Check out the zoomable PDF
here for a picture of the parts I'm talking about.
One thing you can do in the mean time, I found from experimenting with my own, is put your bike in 2nd gear, let the clutch out and roll the bike backwards (engine off of course) before starting. This motion engages the sprags in the sprag (starter) clutch. So doing it in advance like this should help it to "catch" the motor crank and turn it to get it started.
So yes, you have been officially stung.