Do newly formed word representations encode non-criterial information?

Do newly formed word representations encode non-criterial information?