MPs have launched an inquiry into how the material is being managed across the UK after serious safety concerns were raised.
Exposure to asbestos can lead to mesothelioma, a type of cancer which affects the lining of some organs, including the lungs.
Health and Safety Executive data shows the disease was responsible for 684 deaths in Derbyshire between 1981 and 2019 – the latest available figures.
The High Peak's Covid 19 death toll two years on
Major changes to UK Covid travel rules come into effect
Buxton charity moves to bigger premises and sees demand for mental health help soar
Family praises staff at a Buxton care home
£154,000 fundraising target for High Peak youngster's cancer treatment is reached
Of those, 158 occurred between 2015 and 2019.
This is the highest number of deaths recorded in any five-year period since records began and nine more than in 2010-2014.
The Work and Pensions Committee, which launched the inquiry, said that despite the importation, supply and use of asbestos being banned in the UK since 1999, it remains the largest single cause of work-related fatalities.
More than 5,000 deaths each year are caused by diseases linked to asbestos exposure, including mesothelioma, lung cancer and asbestosis.
The inquiry will examine the risks posed by asbestos in the workplace, the actions taken by the HSE to mitigate them and how its approach compares to those taken in other countries.
Though traditionally, higher levels of asbestos-related illness were associated with work in industrial sites such as shipyards, in recent years that also expanded to other industries, including construction.
But campaigners say people using buildings where asbestos is poorly maintained, including some schools and hospitals, are also at risk of contracting the deadly disease and want to see action from the Government to tackle the issue.
Liz Darlison, CEO of charity Mesothelioma UK, said she was shocked at the figures, which show that 12,500 people died of mesothelioma in Great Britain in 2015-19 – the highest number for any previous five-year period.
She added poor asbestos management led to "public health disaster number one".
"Our country is riddled with the stuff and we have to address this if we want to protect future generations," she said.
"We need a long-term, government-led initiative to remove asbestos, even if it takes several generations."
Dawn McKinley, chairman of the UK Mesothelioma Alliance, which includes charities, support groups and healthcare organisations, is calling for action to protect children and staff coming into contact with asbestos in schools.
She said: "Teachers, former pupils, school janitors, cleaners and canteen staff are dying from asbestos-related cancer and mesothelioma, and the numbers are increasing.
"The dose level required to contract mesothelioma is extremely small.
"Our politicians, duty holders and decision makers must come together and use their powers to protect our children from the real dangers they face from exposure to asbestos in our schools."
The HSE said targeting asbestos risk has been and remains a priority.
A spokesman said: "Sadly, the damage from exposure to asbestos takes many decades to show itself as there is often a latency of up to 40 years before disease is detectable.
"This means that cases now, and in the recent past, normally result from exposures which predate the time during the 1980s when the regulations and work practices were significantly tightened.
"Asbestos is, however, still present in older buildings given its previous uses and it must be managed appropriately."