The song is about a crisis in McCartney's relationship with his then-girlfriend Jane Asher. She was rejecting him by not returning phone calls and ignoring him — for once, he was in a vulnerable position. The more biting tone of the song marks a change away from his earlier, happier love songs. "You Won't See Me" was recorded during the last session for Rubber Soul on the night of November 11, 1965. The deadline for completing the album was up and the band needed to record three songs that evening to complete. As a result, they cut the song in only two takes. At 3:22, the song was the longest that The Beatles had recorded to that point and marked a trend by Bob Dylan and others at the time to start writing longer songs.
Mal "Organ" Evans (one of The Beatles' roadies throughout their career) is credited on the album sleeve as having played Hammond organ on this track, his contribution consisting solely of an A note quietly held throughout the last part of the song.
"You Won't See Me" was never a part of The Beatles' concert repertoire, but McCartney played the song live during his 2005-06 concert tour.
MacDonald commented that Starr added an overdubbed hi-hat part in addition to the hi-hat part played in the basic rhythm track, and that Evans's organ part was a single A note held throughout the final verse and chorus.