Narrated `Imran bin Husain:

The Verse of Hajj-at-Tamatu was revealed in Allah's Book, so we performed it with Allah's Messenger (ﷺ), and nothing was revealed in Qur'an to make it illegal, nor did the Prophet (ﷺ) prohibit it till he died. But the man (who regarded it illegal) just expressed what his own mind suggested.

حَدَّثَنَا مُسَدَّدٌ، حَدَّثَنَا يَحْيَى، عَنْ عِمْرَانَ أَبِي بَكْرٍ، حَدَّثَنَا أَبُو رَجَاءٍ، عَنْ عِمْرَانَ بْنِ حُصَيْنٍ ـ رضى الله عنهما ـ قَالَ أُنْزِلَتْ آيَةُ الْمُتْعَةِ فِي كِتَابِ اللَّهِ فَفَعَلْنَاهَا مَعَ رَسُولِ اللَّهِ صلى الله عليه وسلم، وَلَمْ يُنْزَلْ قُرْآنٌ يُحَرِّمُهُ، وَلَمْ يَنْهَ عَنْهَا حَتَّى مَاتَ قَالَ رَجُلٌ بِرَأْيِهِ مَا شَاءَ‏.‏
USC-MSA web (English) reference : Vol. 6, Book 60, Hadith 43
Arabic reference : Book 65, Hadith 4518