Japanese is easy in terms of sentence structure and pronunciation. The difficult parts, from what I recall, were kanji, as well as a few bizarre quirks. For example, while I didn't really get into them, since it was apparently a thing in the third and fourth semester classes, there are different, extremely confusing sets of words/numbers used for counting objects, and they can apparently shift the kana for whatever they're counting. Like, "san" is the number three, but "mitsu" (the only one I know) is a counter word for three small, round-ish objects. An example is "Mitsubishi," which is mitsu + (shifted) hishi, which means "water chestnut," which is what they call what's referred to as a diamond shape in English.
Yup, there's a million different counters out there. Particles can be a bitch, too (especially wa and ga). For me the hardest was always kanji. There's a ton and each character can have dozen of readings depending on how it's used.
I'm really tired of seeing this. I'm far from an expert, but I know enough to call this out. First, if you're formally studying and it took more than a week for the class to get to the kana, quit right now. Second, do not bother with formal study unless you're willing to augment it with lots of self-study.
I encourage you to read
Why Chinese Is So Damn Hard and look for the parallels. Moonspeak is an easy to learn, difficult to master language. Specifically, its syntactic structure, ease of pronunciation, flexible grammar, and emphasis on context makes it very easy to hit the ground running. The one caveat, and crux of its difficulty is that in basically every the way of thinking is fundamentally difference from all the big western languages. So while it's easy to learn and understand from 0, if you're learning from another language that does not think like the nips do, you're going to be constantly fighting yourself to fit your square peg thinking into that round hole. Imagine relearning maths in base 7 but everything is expressed as colored dots on a grid, no numbers exist; maths haven't changed, nor has its difficulty level, but it sure feels a lot harder.
The biggest myth is that the challenge is nothing but learning thousands of runes, but that's not really true. Learning the kanji isn't difficult per se, it just requires a lot of time and memorization, a trait shared with basically every aspect of the language. This is the difference between time as investment capital a la learning to read physics equations, versus actually understanding the physics those equations describe. To put it another way, with time enough basically everyone can learn the kanji and learn nip, no matter what /djt/ says, but there is a very large subset of the population that will
never be able to understand quantum mechanics. Once again this isn't to say that the language isn't difficult, but rather that just about every SEO-optimized listicle and armchair expert that got their knowledge from them has been conflating "difficulty" with "takes a lot of time". Strictly speaking, if I had to choose I'd probably find it easier to learn the runes than relearn all the nuances of English spelling and pronunciation.
The real difficulty goes back to the fundamental difference in thinking, which is largely inarticulable because it's far greater than the sum of its many small parts. I can offer one personal example. It's commonly described that the grammatical structure is Subject→Object→Verb, and while that's not really true―all a grammatically correct sentence needs is a verb―it will serve for our purposes. If I want to say "I go to the library", 「［私は］図書館に行く」, the direction of cause and effect is very obvious for reasons that I'm sure a proper linguist could explain. My best guess is because the inverse proposition "The library comes to me" is absurd. Mountain to Mohammad and all that. This becomes significantly muddied when that certainty is no longer exists. If I'm at the library and I want to say "I criticize Tarou's book" 「［私は］太郎の本を非難する」, the meaning may still be evident in this sanitized example, but not so once language encounters the complexity of actual use, and especially when we introduce inverters like the passive form 〜れる which can reverse the entire direction of the sentence. In a sentence, as long as you're stuck in the western mode of thinking this grammatical structure turns every sentence into a word jumble which must be solved before the meaning can be parsed. More complex sentences increase difficulty exponentially.
This is compounded by the other two key elements of the language's difficulty, politesse and nuance. I can't speak objectively, but relative to English and suchlike languages Moonspeak is far more nuanced, and has much, much more complexity in its politesse. In many ways 丁寧語 and 敬語 are like bonus nested languages. There's enough difference between writing and speech that you might be able to claim the same distinction, if you felt so inclined. The other thing is that while English is a largely neutral language by virtue of its status as global lingua franca, Moonspeak has only ever been the majority language in one country, and is therefore extremely tightly coupled to its own culture. Between that and the sheer age of the language every word has a wealth of meaning that when combined can subtly yet profoundly change the meaning of the whole sentence. I hope one day to merely grok the difference between 「僕」 and 「俺」, nevermind something as complex as 「非難」 and 「批難」. A final sidenote is that the language seems to have far greater difficulty letting go of its history. A game to play if you hate your liver: start studying city and place names and take a shot every time you encounter either an archaic rune that's no longer actually used in the language, or "merely" has an obscure reading no longer in use. If you really want to die, you can use people names instead. Counters are also annoying, but you can simply use つ for most things and people will at least understand you.
And that brings us full-circle. A lot of this difficulty is simply optional. You can easily learn the most basic vocabulary and spew word salad, ignore politesse entirely, and throw in some good old-fashioned shouting and arm waving while you're at it for the complete ugly American experience and you'll still be sufficiently intelligible. You'll be the linguistic equivalent of the bull in a china shop, but you'll at least know where the library is.
One final thing is the absence of direct translations for so many words. There are tons of English concepts that exist only in Moon as loanwords, and the inverse would be true as well if we'd bothered to import as many loanwords. One example is green and blue. As best I can tell, until the end of sakoku the word for blue 「青」 encompassed both blue and green, and that meaning still lingers at least in part IE traffic lights turn 青、 not green. Another is familial relations. There is no word for "brother" or "sister", only words that relate the sibling as older or younger, or a general word for siblings. Once again, there are inversely familial relations for which the equivalent words don't exist in English.
Experts can and will correct me, I'm sure, but
tl;dr Moonspeak isn't hard because of kanji or different alphabets or counters or any of that unskilled blogger listicle shit. Moonspeak is hard because it forces your アホ外人 square peg brain into a round language hole and that's hard to describe. 
