The first time I travelled overseas from Australia was to Japan. The next time to Greece. Both times, I knew the language to various degrees. The first country I visited where I didn't know the language was Turkey. It was difficult for me to not feel uneasy and uncertain, but after a day or two I was beginning to enjoy the challenge of communication over barriers.
The technology of the 21st century is making it easier and easier for travellers to communicate without having to study Swahili in Three Months before they leave. More and more people are taking their iPhone with the when they travel. Read here for some are some handy apps for travelling in Japan. I have not tried them, myself, though.