The first reason is their size. They are very small and are very difficult to see. Special technology needed to be developed in order to observe them and their behavior and reach the conclusion that they were in fact infectious agents that could cause diseases. This wasn't possible before because the microscopes weren't as well developed.
Another is that they have a very long incubation period. This means that if you have them, a long time will pass before they start harming you which meant that many people believed them to be harmless since they couldn't track their behavior over a longer time period, also due to the lack of good technology.
