Well yes compared to those cameras it is slow. But it's not really much slower than say the X100 or the X-pro when they were initially launched. From my limited experience handling the Epl-2 (which is already a gen 2 oly) I don't think it fares worse either. The NEX-7 is definitely faster but that's not blazing either.
One thing I will say though, the EOS-M seriously can't ai-servo to save its life.
Anyway it's a first gen camera, here is hoping Canon improves it as each year goes along. The EOSM makes sense for me as a backup to my DSLR. Easy for my kid to use too though he prefers the X100.
Add: TS - the EOSM isnt really pocketable even with the 22mm pancake. Unless you don't mind odd bulges in your pants.