What i do:
One thin dry fit t shirt under layer.
cotton long sleeve tee over.
If its too warm i can easily lose the top layer, but i agree a sweatshirt is too bulky
If its too cold for the t shirt the long sleeve tee might feel warm, but its a much more pleasant experience.