I do not own a pair of sweatpants so debating on where I am going to wear them to is never an issue. I am probably one of the few people in the US that doesn't even own a sweatshirt. I've just never been a fan on them. I finally broke down and bought a "hoodie" this fall, but I would consider this particular one more a piece of outerwear than a sweatshirt.
My wife has tried to get me sweatshirts in the past as gifts but she has given up after finally realizing I just do not like them.
I would much rather be in a wool sweater and oxford cloth shirt when I am lounging around the house or even working outside in the fall. And instead of sweatpants, I'm usually in a pair of well-worn cords or moleskin pants which to me are just as comfortable.
I think my aversion to sweats developed when I had a job in the Marines where I traveled extensively but wasn't allowed to wear my uniform and my civilian clothes had to include a collared shirt and couldn't include jeans, sweats, or sneakers. Old habits die hard.