Talent agent James Dixon appears unflappable, which is perhaps not surprising given that he is lounging under an umbrella out by the swimming pool at his South Summer street home, every inch the Hollywood player with perfect tan, great hair and aviator shades.

It’s also part of Mr. Dixon’s job to remain unruffled — he is a professional troubleshooter for his clients who include the Daily Show’s Jon Stewart and Stephen Colbert, two of network television’s most prominent late night presenters.

But he admits on the night he watched Stephen Colbert stand just yards from George W. Bush and deliver an audacious, merciless and ultimately career-defining evisceration of the then-President, he was a little — say a smidgen — flapped.

“That was a night I’ll never forget,” he says.

The night in question was the White House Correspondents dinner in April 2006. It was three years into the war in Iraq and Mr. Bush’s ratings were tanking, fast. Mr. Colbert’s were rising nearly as sharply with his satirical news show, The Colbert Report, which first aired six months earlier.

For nearly 30 minutes after taking the stage, Mr. Colbert brazenly savaged the press corps for their coverage of the administration’s misadventures and mocked the President for just about everything.

There was scattered laughter and the tension was palpable. The following day Mr. Colbert received middling reviews for his performance.

However, footage of the speech posted online quickly became the stuff of legend. Some months later New York Times columnist Frank Rich declared the speech a turning point in the mid-term congressional elections.

“It was a watershed moment for him,” Mr. Dixon says. “He was just fearless, absolutely fearless. And I don’t think [Bush] knew for five to seven minutes, he didn’t even know he was goofing him.”

The most uncomfortable aspect to the evening, says Mr. Dixon, was that Mr. Bush was so personable, during a meet and greet in a tiny VIP room at the back of the Washington Hilton where the correspondents dinner took place. Many of Mr. Colbert’s 10 siblings were there along with his mother, and met the president and Laura Bush. “Bush was very gracious and was very nice to Colbert’s mother,” says Mr. Dixon.

In another occasion earlier in the decade, Mr. Stewart appeared on the now defunct (some say defunct because of Mr. Stewart’s appearance) political talk show Crossfire, where he insulted the bow tie of presenter Tucker Carlson and later called him a name unsuitable for publication in a family newspaper.

Thankfully, says Mr. Dixon, moments like this are few and far between. Of Stewart he says:

“Once in every 10 years Jon steps over a line, but his thing is doing comedy, making people laugh, not being confrontational.”

The occasional fracas aside, Mr. Dixon’s job is about advocating on behalf of his clients for better compensation at contract time, improved budgets for shows and handling the nitty-gritty with network and studio executives.

For about five weeks in the summer Mr. Dixon finds he can do this from his Edgartown home.

“You need to keep up with their lives and their schedule, but it’s down to a science. I’m there when I need to be there . . . I’m always looking out for the client’s interest; that’s what I’m paid to do,” he says.

Mr. Dixon started out at the William Morris Agency in what is known as the mail room training program, infamous within the industry, which, according to Mr. Dixon, has Ivy League graduates literally doing the mail.

“It’s pretty humiliating,” he says. “I went from the mail room up to agent’s assistant, which is basically a secretary. But it’s amazing exposure to all aspects of the business.”

He was soon drawn to television and comedy, and eventually set up his own department in the agency handling comedians. One of his early signs was an obscure stand-up named Jon Stewart.

Mr. Stewart took over the Daily Show in 1998 and in 2000, Mr. Dixon left Willam Morris to start Dixon Talent.

A string of Emmy award nominations followed. Mr. Colbert came on the show a few years before Mr. Stewart with a bit satirizing a reactionary news anchor, and was so popular he launched his own show in 2006. Mr. Dixon now handles four of the major late night talk show hosts: Mr. Stewart, Mr. Colbert, Jimmy Kimmel and Carlson Daly, and personally 20 clients. Several staff handle more, a mix of writers, producers and performers.

Last week Mr. Stewart was voted the most trusted man in news.

“I’d be lying if I told you I planned it,” says his agent.