High Resolution Formats - do you hear a difference on DirectStream?

I hear more difference between 16/44 and 24/96 than 24/96 and 24/192

24/96 and 24/192 feels to me like single to double rate DSD whereas the difference with red book is really big (even with the DS)