I too have been considering a lens to somewhat replace my kit + 55-250 at the wider focal lengths. I'm currently still torn between the 28-135 IS USM and the 24-105L, but I'm leaning towards the latter now.
Most reviews put the 28-135 IS USM just behind the 24-105L in terms of sharpness and optical quality. This would be more than sufficient for me except for the fact that 28mm at the wide end is too tele for me, with too much of a gap from the 22mm end of my 10-22. And since I tend to shoot more at wide-normal-medium tele focal ranges rather than zoomed out on my 55-250, I don't think I'll miss the 135mm end of the 28-135 lens as much as the 24mm end of the 24-105.
Perhaps because I have not handled an L lens yet, the L build/quality/sharpness/etc just seem like icing on the cake as of now. If the 28-135 IS USM just extended by 4mm at the wide end, there'd be no contest. But it doesn't, and the fantastically huge price difference is also an issue...
Will probably go down to J3:16 to try them both out 1st.