Born in Oberebenheit in Saxony, Grossman started his career as a bricklayer, but in 1952 he joined the Ministry for State Security where he studied political and military espionage. He rose rapidly in the ranks, being promoted to deputy division leader of military espionage in 1956 and becoming division leader in 1962.
In 1983 he was promoted to deputy head of the General Reconnaissance Administration (HVA), the foreign espionage arm of the Stasi. In 1986 when Markus Wolf retired, Großmann was promoted to head of the HVA and deputy minister of state security. In 1989 he was awarded the rank of colonel general. When the Stasi was dissolved in 1990, Großmann, like all other members, lost his position.