The eldest Caroll set about building wealth, a trait he passed on to his son, Charles Carroll of Annapolis, who in turn bequeathed it to Charles Carroll of Carrollton. Even during the most pivotal moments of the revolution and the lead-up to independence, correspondence between the latter two Carrolls dealt as much or more with business affairs than military of political ones. But for decades, the family labored under the fear that English authorities could at any time exercise laws that would strip them of everything. In the early 1760s, Caroll of Annapolis was so fearful that he began settling his affairs and liquidating assets in preparation for a move somewhere more hospitable.
Charles Carroll of Carrollton spent years in France and later England getting his education, not returning to Maryland until he was 27, but he nonetheless had a keen sense of his familys determination to overcome a precarious position. His father was so obsessed with the familys lineage that he did not marry Carrolls mother until the son was old enough to prove himself a suitable heir. The fathers letters to the son included reminders that those who ruled were possessed of malice that they would not only deprive us of our property but our lives.
As early as 1763, Caroll of Carrollton remarked in a letter to his father that America is a growing country. In time it will amp; must be independent, but it would be some years before he would get involved in revolutionary politics in Maryland.
He returned to the New World as the controversy over the Stamp Act was coming into full bloom. That levy on assorted uses of paper, from playing cards to newspapers, drew ire throughout the colonies in a preview of the arguments about taxation without representation that would gain their full expression in the years ahead. In her 1942 biography Charles Carroll of Carrollton, author Ellen Hart Smith found ample evidence that Caroll was able to articulate well the arguments against the Stamp Act and did so but only in his private correspondence to friends and business associates in England. Publicly, he stayed out of Maryland political life, instead devoting his attentions to marrying and managing his fathers massive estate (which included, it should be noted, some hundreds of slaves). He at least professed not to mind the exclusion, writing on various occasions that holding public positions would inevitably erode a mans virtue.
Instead, he became a noted host and fixture in the Annapolis social scene, earning a membership in the tony Homony Society, which had previously excluded Catholics and which boasted as members not only the new colonial governor, Robert Eden, but also two of Marylands other eventual signers of the Declaration, William Paca and Samuel Chase.