The argument that he did not commit adultery is supported only by the belief that his extramonogamous unions were legitimate marriages (the idea that he did not have intercourse with them has since long been debunked). Joseph Smith did not admit to adultery perhaps because he felt that he was actually married to these women (Bushman, 2005) and thus, only acting within his rights. After all, they had (however reluctantly) agreed to the union (often misunderstanding its nature), and made the vows in the ceremony (under the promise of familial salvation). So it could reasonably be argued that they were married, at least according to the letter of the law.
But what about the spirit of the law? I think most people would argue that there is more to marriage than a ceremony, a piece of paper, and a sexual act. As a husband of 4 years, I certainly feel that marriage consists of emotional attachment, commitment, time, and effort, at the very least. But it appears that Smith fell short of basically all of these. He did not put much, if any, effort into courting most of his plural wives; romance had very little to do with the unions (Bushman, 2005; Compton, 2001); he cohabited with them infrequently; showed no commitment to them other than the ceremony and consummation (evidenced mostly by his growing circle of wives); spent very little time with them; gave no or minimal support to them financially or emotionally; and gave little regard to their needs overall. I need not mention the complete lack of physical and emotional fidelity he displayed.
Of course, a lot of these were complicated even more because of the large number of wives he had. 34 total wives (at a minimum, at the end of his life) left about 42 minutes for each wife every 24 hours if my math is correct. Of course, Smith was at least human enough to sleep, was in and out of prison, and had dozens of other projects going on. That left him very little time to be a husband to any one of his wives.
Lucky for these women, at least 11 of them had their first husbands (some also had staged husbands; see Compton, 2001) to be in some proximity to them, help with the chores, provide financially, listen to their woes, mend the roof, laugh and play with, protect the family, and so on. But somehow Smith was the one sealed to them for eternity. Their first husbands had no claim to them in the afterlife. They, who had actually worked on their relationships with their wives, would ultimately lose them. In what way was Smith more of a husband than these men?
Perhaps the more appropriate question is "In what way was he more than an adulterer?".
Or maybe the final decision should be up to the scripture Smith dictated with his own lips; D&C 132:61 clearly states that a polygamous wife must be a virgin and belong to no one else, for "he cannot commit adultery with that that belongeth to him and to no one else." If 11 of his wives did belong to their first husbands, then it appears Smith was an adulterer by his very own scripture.
Bushman, R. L. (2005). Joseph Smith: Rough stone rolling. New York: Alfred A. Knopf.
Compton, T. (2001). In sacred loneliness: The plural wives of Joseph Smith. Salt Lake City, UT: Signature Books.