I think the original idiom was “mad as a hatter” which was eventually shortened to “mad hatter”, possibly due to the Alice in Wonderland character.