Yeah its pretty hard for a lot of reasons. First, the grammar is inherently different from English. English is a typologically fusional language that is borderline analytic. Analytic languages don't really have moving pieces to their words. English says, I go, he goes, I went. The verb changes a bit. In a true analytic language, nothing changes. I go. I go already. He go. She go. Not nice aspect (unwholesomeness). Similarly you can say things like un-whole-some-ness or anti-dis-establish-ment-arian-ism which is characteristic of Tagalog that is typologically agglutinative (like glue or gluten, things stick together like legos).
The verbal system is difficult for English speakers since we have a borderline analytic (verbs are simple and don't change much) system. Similarly Tagalog has "focus" also called "aspect" or "voice". This is similar to having cases for nouns (like Russian or german), but on the verb instead. The verbs are a real ballache for most English speakers.
Thirdly, there is a huge vocab under the surface and it is hard to actually acquire any of it. Tagalog has several layers of vocab. The oldest being native Tagalog or Kapampangan words. The next oldest being Malay and Sanskrit words (wika, mukha, dokha). Then Spanish loans, and finally at the top in the most informal register new English loans. Unless you do the frequency lists here (based on pretty formal Tagalog) and read lots of news, you won't encounter much of the formal Tagalog/Malayo-Sanskrit words. The Spanish and English loans are at the most common spoken register so they are easy to learn.
Lastly, everyone speaks English. Even people who don't speak English, speak English better than most Tagalog learners speak Tagalog (ouch, sorry for the reality check guys). A day laborer with no education working construction who says he doesn't speak English, can still probably speak English and communicate better than he thinks he can. You will find a lot of speakers attempt to accommodate you by using more English words which makes conversation easier (their goal is to communicate, not teach you for free) for both parties, but stunts your overall vocabulary.
English speakers and Chinese speakers tend to sound like cavemen in Tagalog. Ako gutom ako gusto punta sa tindahan para bili foods! The same stereotypes in teleserye fil-chi Tagalog applies to most foreigners too ; )