My Brother who has a Chromecast, the normal 1080P module or older not sure explained he had a similar issue and it was down to buffering on wifi we believe because after he moved his wifi to only 5Ghz it became better.