In the early days of the United States, only about 120,000 people out of a total population of more than 4 million could vote. Voting was usually limited to free white men who owned property and met certain religious qualifications. Eventually the right to vote became more widespread. By 1860 almost every state allowed all white men over 21 to vote.

Yes, all "white men" could vote.  That meant that men of other races not allowed to vote.  Women were also not allowed to vote.

After the Civil War (1861–65) the 15th Amendment to the Constitution gave the right to vote to men of all races. 

Women, after a long political struggle, won the right to vote in 1920 (less than 100 years ago!) with the 19th Amendment to the Constitution.