Aite guys, I think I should do an update on this.
I went ahead and bought a Fuji X100 Black LE for 1599. And have no regrets whatsoever. If looks sexy, the files it churns out are actually really really nice, and sometimes prefer it to the Oly files, which are really sharp and technical, but lacks that tinge of romance in the Fuji files. And the OVF and EVF is awesome, and compared to the last time I held the X100 a while back, the camera feels totally fast now, and responsive, the ability to map the RAW button, the faster AF, etc. Write times are slower than the OM-D, but honestly, the IQ still blows me away sometimes.
And the macro focusing in the X100 is vastly improved, and I think that it might actually focus closer than the Voigtlander and when stopped down, the shot becomes really sharp. Macro at f2.0 really is soft tbh. And of course the benefits of having 2 bodies are nice

(3 if you count the neglected D700, 4 if you count the M6).
Next on my list will probably be a M to M4/3 adapter to try out the Summicron, and the upcoming 60mm Macro Oly.