Only you can answer that question because I do not know exactly what you shoot and why you shoot. For me, I use the EPL1 70% of the time, my E1 20% of the time, my E620 7% of the time, and my E3 about 3% of the time. My EPL1 is my underwater camera, my travel camera, and my social camera. For everything else outside of this, I use my E1/E620/E3. For eg, if I need to shoot something with an FL50 flash ... then I take out my DSLRS. Why? Because the EPL1 is damn hard to handle with a FL50 flash on it. Or if I need to use the Olympus ring flash, then I take out the DSLRS. Why? Same reason as the FL50. Or if I want to go do a model shoot, I take out the DSLR. If I want to have AF lamp assist, then I take out the DSLR. If I think I am going into the rough and might get wet, the E3 comes out. If I need big time telezoom, 200mm or more in low light, I take out the E3/E620. Although I have the 4/3 m4/3 adaptor, I think its ridiculous to put the 50-200 on the EPL1. Also, the AF is slow with the 50-200 on the EPL1 - I think its not optimized. Or if I want to use the SHG lens, which I dun own any, I would take out the DSLRs. Same reason, ridiculous to put such a whopping big lens on a tiny body.
So where does IQ of the sensor factor in my decision process....it does not. As far as I am concerned, they are comparable. What I take out depends on what is required. Not what IQ am I getting from the camera. I mean...haven't you been taken by all the marketing by Olympus and Sony where they say DSLR quality in a compact body?? If u so concerned about IQ, then shoot RAW to get the most out of it.
Anyway, I think IQ consideration is so WAY over-rated by everyone. Particularly those who go for the Canon or Nikon FF DSLRS. As far as I am concerned, I do not find the IQ from the Olympus DSLRs the limiting factor. The limiting factor to my photography is actually to get the lighting right. In other words, it is more likely that a shot is not ideal because I din get the exposure right...ie. flash settings wrong etc, rather than the camera IQ cannot make it...which to me never ever happens. Once you get the exposure and lighting correct, everything else is OK.
So the decision between the E-Pens and the DSLRs is function...not IQ, like what everyone else here has already said.