YQT - I have contemplated the exact situation you have brought up when writing my POV, but as you may have noticed, I have not commented on this situation because it is indeed more complicated that the first situation (where the customer has not yet handed over the card). Now that you have specifically brought it up, I'll try to share what I think of it.
I am not 100% sure of the answer, but lets use an analogy as the starting point. If someone walks up to me and asks to borrow my mobile phone, and after using it, refuses to give it back to me, then he has committed a criminal offence.
However, this does not entitle me to bash him up just to get back my phone, in the same way, the customer cannot and should not resort to violence to get back the card. In the phone example, the best way would be to call the police to arrest him for the offence, and perhaps try to get some public passersby to assist in detaining him until then.
For the bus driver situation, it gets more problematic as now, you will have to explain to policemen on the ground the difference between contractual rights and statutory rights (and as we can see even in this thread, this is easily confused). It will then result in a lot of time spent and effort spent just to get things sorted out, even if ultimately you are in the right.
This problem becomes further exacerbated by the fact that by contract, the EzLink company, as you pointed out, owns the card in the first place. Now, can someone be accused of stealing something of which he is the owner for? This is again yet another complication thrown in the mix. I doubt there's any clear answer until the courts issue a decision on such a complicated affair.
In short, the best way when in doubt, and when you are sure you are not using a stolen card/fraudulent card etc, is still not to hand over your card, to prevent yourself from all these headaches and complications.