كتاب الحج
15  
The Book of Pilgrimage
(73)
Chapter: Hajj is obligatory once in a lifetime
(73)
باب فَرْضِ الْحَجِّ مَرَّةً فِي الْعُمْرِ ‏‏
Sahih Muslim 1337

Abu Huraira (Allah be pleased with him) reported:

Allah's Messenger (ﷺ) addressed us and said: O people, Allah has made Hajj obligatory for you; so perform Hajj. Thereupon a person said: Messenger of Allah, (is it to be performed) every year? He (the Holy Prophet) kept quiet, and he repeated (these words) thrice, whereupon Allah's Messenger (ﷺ) said: If I were to say" Yes," it would become obligatory (for you to perform it every year) and you would not be able to do it. Then he said: Leave me with what I have left to you, for those who were before you were desroyed because of excessive questioning, and their opposition to their apostles. So when I command you to do anything, do it as much as it lies in your power and when I forbid you to do anything, then abandon it.
وَحَدَّثَنِي زُهَيْرُ بْنُ حَرْبٍ، حَدَّثَنَا يَزِيدُ بْنُ هَارُونَ، أَخْبَرَنَا الرَّبِيعُ بْنُ مُسْلِمٍ الْقُرَشِيُّ، عَنْ مُحَمَّدِ بْنِ زِيَادٍ، عَنْ أَبِي هُرَيْرَةَ، قَالَ خَطَبَنَا رَسُولُ اللَّهِ صلى الله عليه وسلم فَقَالَ ‏"‏ أَيُّهَا النَّاسُ قَدْ فَرَضَ اللَّهُ عَلَيْكُمُ الْحَجَّ فَحُجُّوا ‏"‏ ‏.‏ فَقَالَ رَجُلٌ أَكُلَّ عَامٍ يَا رَسُولَ اللَّهِ فَسَكَتَ حَتَّى قَالَهَا ثَلاَثًا فَقَالَ رَسُولُ اللَّهِ صلى الله عليه وسلم ‏"‏ لَوْ قُلْتُ نَعَمْ لَوَجَبَتْ وَلَمَا اسْتَطَعْتُمْ - ثُمَّ قَالَ - ذَرُونِي مَا تَرَكْتُكُمْ فَإِنَّمَا هَلَكَ مَنْ كَانَ قَبْلَكُمْ بِكَثْرَةِ سُؤَالِهِمْ وَاخْتِلاَفِهِمْ عَلَى أَنْبِيَائِهِمْ فَإِذَا أَمَرْتُكُمْ بِشَىْءٍ فَأْتُوا مِنْهُ مَا اسْتَطَعْتُمْ وَإِذَا نَهَيْتُكُمْ عَنْ شَىْءٍ فَدَعُوهُ ‏"‏ ‏.‏
Reference : Sahih Muslim 1337
In-book reference : Book 15, Hadith 461
USC-MSA web (English) reference : Book 7, Hadith 3095
  (deprecated numbering scheme)