Dealing with Your Spouse's Promiscuous Past

I know this is very shallow and I have been praying and reading my Bible and any other Christian material on this topic but I am still struggling.
I have only ever been intimate with my husband. When we got serious while dating and shared our pasts he told me he was not a virgin. I found out he had been with several women before me but I also found out that he was still devastated by it and had asked God's forgiveness. By the time we got together he had not been intimate with anyone in a couple of years.
Why is it that this bothers me? I have trouble with these images in my mind of him with these other women, but I did not even know him then! It isn't like he cheated on me. Like I said, I know this is shallow because there are so many couples out there that have cheated after they got married. That, I can imagine, is extremely awful.
But I would like to know if anyone knows of any specific verses in the Bible or any Christian books on this sort of thing? I am a big reader and I pray constantly. I know these issues I am having are things Satan is putting in my mind to try and wreck my marriage. I love my husband. Our marriage won't be wrecked, at least not because of this. I just need to overcome my... jealousy and hurt... and I am having trouble finding Christian help books. Any that someone knows of and could suggest would be appreciated.