The remains of 45 bodies were pulled from the ground in about three hours at the site, located near the holy shrine of Hussein, the grandson of the Islamic Prophet Muhammad.
Women slapped themselves in the face in grief and men beat their chests to pay tribute to their slain countrymen.
"The blood of innocent people won't go away. Criminals should stand trial," some of the 1,000 people gathered at the site chanted. "Death to the Baath Party members."
Local residents in Karbala's Mokhayem district said they suspect as many as 5,000 sets of remains are buried in the area, though they offered no immediate proof. The mass grave is the third uncovered in Iraq this week.
"This is a material evidence of crimes committed by the tyrant Saddam," said resident Bassem al-Tamimi.
The Shiites, who make up 60 percent of Iraq's population, were persecuted and oppressed under Saddam's Sunni Muslim-dominated regime.
Many also hold deep anger against the United States, a country they believe abandoned them in 1991 after encouraging them to mount an armed revolt against Saddam after the Gulf War. Thousands of Shiites were slain by Saddam's forces after that rebellion.
The mass grave is the latest to surface as Iraqis, freed from Saddam's rule, begin to take stock of their missing and dead.
On Thursday, volunteers said after 10 days of digging that about 3,100 sets of remains of people killed after the 1991 Shiite revolt had been found at Mahaweel, 100 kilometers (60 miles) south of Baghdad. Iraqis said some had apparently been buried alive at the site, the largest mass grave found since U.S. forces overthrew Saddam's Baath party government last month.
Also this week, Iraqis pulled bodies from a newly discovered mass grave near the southern city of Basra. That site in southern Iraq was believed to contain remains of about 150 Shiite Muslims killed by Saddam's regime after another rebellion in 1999.