Remember that God needed the right leaders of his nation. Though the covenant was with Abraham, God still utilized Isaac to build the nation! In Matthew 1, Abraham is the first person listed and then Isaac. The Messiah's lineage stems from Abraham and Isaac fulfilling the covenant.
Abraham was NOT perfect! He decided to take matters into his own hands and have a child with Hagar. This did not please God. BUT, where we fall short, our faith overcomes. We must strive to put our whole heart into where God leads us. Faith will overcome.