Part of the answer is that math is abstract, and is basically a rigorous, internally consistent human-invented language for expressing relationships in symbolic terms. This is especially true of higher level math, where concrete applications of sub-disciplines such as real analysis or abstract algebra are not at all obvious to a layperson.

Another (and more important) part of the answer is that math is a cumulative discipline, and each successive subject in math builds on mastery from the last one. For an interactive representation of this, look at Khan Academy's knowledge map.

See also this chart comparing American math curricula with those of those of top-performing school systems:

You can see from the chart that in many American school systems, math is taught in a poorly organized, scattershot manner that doesn't drill students in a focused manner and doesn't facilitate deep comprehension step-by-step. You can imagine that some students might wind up attempting to learn multiplication while only having a shallow understanding of addition and subtraction, or attempting to learn addition without even knowing how to count. This deficit keeps piling up over time.

The only way to remedy this is to identify and fill the gaps in your knowledge.