The Times reported that he'd called the speaker "that insignificant little person in a wig".
So not so much "Mad Jack" as "Tiresome Jack".
His wealth came from iron foundries and West Indian sugar plantations.
He made guns. He owned slaves.
On the plus side he used his money to fund lighthouses, support scientific research- in particular the work of the chemist Michael Faraday- and buy pictures off Turner. He also built follies.
It must have been the follies that earned him his nickname.
Six of them are still standing- the most accessible of which is the mausoleum in Brightling churchyard. He may (but I haven't checked) be the only British person to be buried in a pyramid.
Legend has it that he was placed inside, wearing a top hat, sitting at table with a bottle of claret in front of him. The truth is that they dug him in under the floor.