To be honest, a lot of English speakers seem to have a decent time understanding broken English. I know a French guy and used to know a Finnish guy who could barely understand native speakers of their own language talking in broken slang but could easily understand broken English.
I'd say tenses are the biggest struggle for non-native speakers. I learned Spanish for a while and man did I absolutely hate how many tenses there were. Some tenses I thought just shouldn't even exist, like different tenses for doing something once and doing something multiple times. In the end, we can still understand you perfectly, though it is a bit amusing to listen to I'll admit.
My favorite are insults from non-English speakers. They may not know English well, but it does help them come up with some very creative and hilarious insults.
Back to the topic, I suppose that tenses are a big one but also using proper conjugations. You know, things like "Me am going to go home" instead of "I am going to go home." I've heard things like that a few times, and am guilty of it myself in Spanish.
The other big challenge would probably be encountering more eloquent words that you aren't used to in normal conversation, such as the word eloquent since there are much simpler words to describe something than using a word like eloquent I could just say "more difficult language" instead. Slang terms are also hard, even for me as a native English speaker. I have a hard time understanding Gen Alpha and even Gen Z (my generation) slang.