One of the common arguments people have against unit testing, is that it slows them down. And they're right - unit testing will slow you down - if you're doing it wrong.
Brittle Me This
The easiest way to be sure that your tests become a burden, is to write brittle tests. And the easiest way to write brittle tests, is to write tests that make assumptions about the internal implementation of whatever they're testing. For example, by calling private methods from unit tests.
The Fear of Refactoring
One of the huge advantages of having a set of unit tests, is the safety net they provide which allows you to refactor code without the fear of breaking stuff. If however by refactoring your code you break the tests and thus also have to refactor them all the time, not only are you compromising the value of the tests, you're also introducing a lot of extra work for yourself.
The Value of Resilience
Whether you're writing state or behavior based tests, try to keep knowledge about internal implementations of the thing you're testing out of the test, and you'll find that your tests become much more resilient to change, and thus much more valuable.