Browser is loading two images - one for srcset, one for src (Chrome 41 et al)

I'm using the srcset attribute on a web page I'm developing.

<img src="img/picture-820x496.jpg" 
    srcset="img/picture-820x496.jpg 1200w, 
    img/picture-374x226.jpg 992w, 
    img/picture-305x185.jpg 768w, 
    img/picture-707x428.jpg 300w" />

If I check which resources get loaded with the page, I see that Chrome 41 as well as FF using the polyfill as well as Safari 7 always load the image twice - once the full resolution and once the according size from the srcset attribute. What am I doing wrong here?