According to the context, the "lots of things to consider" is referring to the race, background by the husband. In his opinion, beside whether love this person or not, race, ethics background is also a very important factor to consider whether they should marry or not. He stated that if two people are not from the same race, they are not in the same culture, they have different language. People from different race never know each other.
Compare the husband's actions to his wife's reactions. Are these people understanding each other? If he had said yes, would their relationship have changed? If so, how?
From the husband's action to his wife's reaction, they didn't understand each other much even though the husband was trying to show how considerate he is. But he failed to understand what his wife truly wants to hear and wants to see in the conversation on the matter of "whether white people should marry black people." In the husband's view he thinks that it was ridiculous to think this kind of question when his wife asked him whether he will marry her if she were black. He thinks that it will never happen. If it happened, his wife will be another person but not the same her anymore. In his views, as he never think that this two race should get marry, he will not allow himself to fall in love or even date a black girl. He is telling the truth and trying to explaining that to his wife. However he failed to realized that that was not what his wife wanted to hear. In fact from the context we can guessed for his wife, it's not the greatest matter on whether white people should marry black people, what she matters is whether they love each other of not. For her, she thinks that if two people love each other, everything can be solved and race is not a big deal. She wants her