"According to the days that you spent exploring the land, forty days, a year for a day, you shall carry your sin and you will know My actions" (Bamidbar 14:34). The sin of rejecting the land of Israel was not easily forgiven and forty days would turn into forty long years, enough time for an entire generation to die. This is well known and seemingly indisputable.
However, Moshe Rabbeinu, speaking to the second generation weeks before his death, seems to contradict G-d's clear enunciation of the reason for the long, long journey in the desert. "Remember the entire path along which G-d your Lord led you for forty years in the desert in order to harden you, to test you, to know what is in your heart, to see if you will keep His commandments or not" (Devarim 8:2). Moshe claimed that our sojourn in the desert was for our own benefit, to see what we were made of and to determine if the challenges we face would bring us closer to G-d. Moshe said nary a word about the spies and made no reference to punishment.
"Why" and "for what purpose" are two very different questions. Often we cannot fathom the whys of life. We must, however, attempt to derive some lesson from that which may transpire, allowing us to glean some purpose to life's mysteries. The Torah often provides the why, allowing us to focus on the what for . In chronicling the actual story of the spies, Bamidbar answers the immediate question of why we stayed in the desert for so long. While listening to the report of the spies may have been the immediate cause of our staying in the desert, our sojourn had a much greater purpose. Just as we needed the Egyptian experience to help develop our sensitivity to others-hence the constant refrain to be "kind to strangers", because we know the pain to which foreigners are often subject -so too, we needed the desert experience to harden us as a people, to see what we were made of.
The Netziv (Bamidbar 14:34) notes that had our wandering been a punishment for the sin of the spies, it would have been unfair to punish the children and those not yet born for the sins of their parents. The guilty should have been killed immediately, as they were in other sinful episodes in the Torah such as the sin of the golden calf, the Korach rebellion, the sinning with the daughters of Moav, or the sending of the poisonous snakes. The fact that much of the punishment was borne by those who bore no responsibility for the spies indicates that our wandering served a much greater purpose.
"He made life difficult for you letting you go hungry, and then He fed you the manna ... this was to teach you that it is not by bread alone that man lives, but by all that comes out of G-d's mouth" (8:4). Our desert experience was meant to teach that it is G-d who controls the world. The chet hameraglim , the sin of the spies, demonstrated the necessity to drive home this message. Perhaps the greatest failing of the generation of the spies was that they did not rely on G-d to take them to the land of Israel . Precisely because they insisted on a natural conquest of the land it became imperative to demonstrate that it is G-d who controls nature, though He allows us to feel that we are responsible for our success.
It takes forty long years to fully develop one's way of thinking. The Talmud (Avodah Zara 5b) notes that often, we only fully comprehend what we have been taught after forty years. (I recall hearing Rav Schachter comment that some of the teachings of Rav Soloveitchik became understandable to him only forty years later.) Our Sages note that "forty is the age for understanding" (Avot 5:25); thus it is only at forty that one may be ready for the study of esoteric Jewish teachings.
These forty years in the desert were meant to develop the historic mindset of the Jewish people. Our survival in the most difficult of circumstances, due to the guiding hand of G-d, would help prepare us for much travail in our future. The success of modern man, the harnessing of technology, the ability to explain almost everything that occurs in scientific terms, has desensitized us to the fundamental principle that "man does not live by bread alone but by all that comes out of G-d's mouth". May we merit the ability to recognize this without the need of "hardships to test us".