According to Paul, they got a bad run of these RCA jacks at one point in time. I am not sure of the age of your DS Sr., but mine was plagued with these RCA jacks. One of mine pulled off in the same way. There is nothing to worry about as long as it is securely affixed which it sounds like it is.
As wonky as it may sound, the solution is a bit of super glue along the rim (ONLY) of the sleeve and press back on firmly. That is of course if it is to ever come back off. It is is secure now, you are getting sufficient contact and it will not affect SQ. I pressed it back on a few time before finally dabbing a bit of super glue on the end of the sleeve. Paul has posted a couple time about implementing this fix using super glue, so it is PSA endorsed.
But if it cannot be pull it off, then that is a great thing! No need to worry, just enjoy the music.