I think you are misinformed. I took a lot of classes in Physics department of IIT Delhi and the teachers were as good as the electrical engineering department. In fact, I did my B.Tech. project in the physics department.
Whether you should do your masters from IIT is entirely up to you.