[A2A] I don't know why people would frown upon you, but let me answer why most people watch sports in the first place. It is not for the love of the sports, but because they identify with a certain team. Objectively speaking, baseball is one of the most boring team sport. Still it is so popular in America, that 200 games for each of 30 teams which play almost everyday for 6 months are jam-packed. You read it right, every team plays almost 200 games (in a little over 6 months) and all games are practically sold out.
I was in Minnesota, who had a football team, called Minnesota Vikings. They practically have no players from Minnesota, no coaches from Minnesota, nor are the owners of the team from Minnesota, but all the Minnesotans support them because they have Minnesota in their name. Similarly, most people from Delhi support Delhi Daredevils, the IPL team although very few players are from Delhi because they identify with the name Delhi.
So, people are puzzled why are you watching a game when you don't have any horses in the race. I don't think they are upset, just unable to understand why you would do it.