Hamlet tells Ophelia "Get thee to a nunnery"; the word "nunnery" is ambiguous because while in addition to referring to a convent, which is what the word would probably seem to mean to most people, "nunnery" was a euphemism for "brothel" in Elizabethan time, probably Hamlet was underscoring that, because beforehand, he calls Polonius a "fishmonger" which is a slang term for agent of prostitutes. Therefore, the audience is unsure whether Hamlet wishes Ophelia should save her purity or if he is insulting her. Hamlet says to Ophelia, "Get thee to a nunnery" this could be so that she will stop enabling people, like her father, Polonius, to spy on Hamlet and undermine him. Hamlet wants honesty and loving kindness from women, yet gets none of it from the women in his life. Better for them and him that they make their way to a nunnery. Hamlet suspects Ophelia has betrayed him by being willing 'bait' for her father's spy trap.
A few minutes before, Hamlet was very close to killing himself in the ‘To be or not to be’ soliloquy, so thinking about his emotional state, he's feeling a total fragile wreck. Then along comes Ophelia, his much-loved girlfriend and she finishes with him. She returns his love-letters and poems and gifts. That's not good timing which is why Hamlet rages. Then, Hamlet realises they are being spied on. He suddenly suspects Ophelia is a willing accomplice to this covert operation and that she's cheerfully dumped him in public to get a reaction.
He does what many love-sick, unhappy, rejected ex-boyfriends do when they can't have the girl they love. He rages. For some reason, we reserve the worst insults for the ones we love. He calls her names and tries to hurt her feelings. Also, he may be playing, aware that they are being spied on; he may be exaggerating his 'madness'. But considering he was moments