Charles Dexter is an agent for the Fourth Agency.
This agency is a British intelligence organization without a real name, headed by a man named Hammell who has little love for his agents and none whatsoever for Dexter. Being the fourth, it gets the really unpleasant jobs that the others pass on and Dexter is Hammell's primary recipient of the worst of these.
What the other three agencies are that make Dexter's the fourth are not mentioned. MI-6 and MI-5 are certainly the first two but the third remains a mystery, though possibly Special Branch. What is not a mystery is the fact that since it is the low man on the totem pole, Hammell takes some unimpressive cases, aka the dregs, which nevertheless find the ability to cause Dexter an inordinate amount of trouble and danger. Not that Hammell would care, of course.
On his own, Dexter would love to continue to be a part-time lawyer, part-time private investigator, and part-time raconteour that he is when not on an assignment. Unfortunately that requires money and his various bad habits make that a rare commodity. Hence his willingness to take jobs saner men would ignore. This means leaving his very shabby private office and going places he would rather not.
It is hard to feel too sorry for Dexter, though. The man is by no means a bad man but he is far from being a good guy. He is the sort of man who, if he owed you a hundred bucks would, as the expression goes, 'rather owe for the rest of his life than cheat you out of it.' While Dexter never uses that line, it would be his attitude.
So perhaps Hammell is not all wrong.