Well it is mostly the compression and decompression. On a SDTV the resolution will not be upscaled and the artifacts don't show up. Here is an example of what is typical between a HDTV and a SDTV:
The photo on the left is what is expected due to the combination of decompression and resolution enlargement of a HDTV. The photo on the right is what is to be expected from just decompression on a SDTV. A lot cleaner.
Some HDTVs process SD up to HD resolutions better than others. The TV that use the Faroudja DCDi processors or the Fujitsu processors will do the best as they anticipate the noise due to decompression and eliminate some of it.
So in a way it is the HDTV processing (or lack of it) that is screwing up the picture.