Together with Josef Mohr (original German lyrics), a Catholic priest, Gruber composed the Christmas carol Silent Night. On December 24, 1818, Mohr, an assistant pastor at St. Nicholas, showed Gruber a six-stanza poem he had written in 1816. He asked Gruber to write a melody and guitar arrangement for the poem. At Christmas Mass, while Mohr played his guitar, the two men sang "Stille Nacht" for the first time. The St. Nicholas choir repeated the last two lines of each verse.

In later years, Gruber wrote additional arrangements of the carol for organ and for organ with orchestra. He wrote dozens of other carols and masses. Carols and masses done by Gruber are still in print and sung today in Austrian churches.