It was established in the 14th century, on the bank of the Varnelė River, near an important Samogitian castle. It was the center of Samogitian Catholic church: after the baptism of Samogitia, the Samogitian Bishop resided in the town. Around 1414–1416 the first church was built, and c. 1464 the first cathedral. Varniai was the center of Samogitian episcopate until the middle of the 19th century, when authorities of the Russian Empire moved to it Kaunas. It was also residence of Motiejus Valančius.