How should I behave as a developer in a project that's headed for failure?

Take the steps required to see to your own future.

Do not get caught up in owing anything to your manager or your employer. Your manager transferred a failing project to your group. He is more responsible for you having this failure hang over your head than you are.

To your employer you are simply a resource, not a person. Do not give them the loyalty you would give a deserving person. It will not be reciprocated. In a world where all employees are basically extended freelancers, you need to put yourself first.

Failing to plan is planning to fail.

> Application is unstable and very difficult to use.

Could be that there are a couple of "killer bugs", fix them and it might not seem so bad. Difficult to use is also subjective, people get use to stuff.

> System is very convoluted, code very hard to understand, very difficult to change.

This could be "I don't understand the code, or know how to change it".

> Datamodel is too driven by a complex relational database (100+ tables).

OK, this one I can't defend :-P

> Unclear leadership, the manager responds to every new information with significant change in plans.

Well, if the plans are adding new features, these should go to the "nice to have" list. If it's "we need to rewrite this" then that's not good.

> Almost no automated tests or unit test.

No autowhatnow? :-D I don't think a project is necessarily "headed for failure" just because there are no automated/unit tests.

> Heavily depends on other systems, but no integration tests yet.

Yet! That's good, there's still time to test this before shipping.

> In fact, we actually just inherited this project (along with the mess) around 1-2 month ago from another dev team under the same manager, who have worked on it for a few month.

Aha! The root of the problem "I don't feel comfortable with other people's code"?

I disagree. If the company in question is large enough having documentation that you were actively trying to avoid disaster, were competent enough to raise concerns and suggest solutions, etc. can be helpful when people come trying to assign blame after the fact.

If the company as a whole will survive the impending train wreck then having evidence of competence, however meager, is certainly useful in many situations short of criminal prosecution.

I can practically guarantee (and the OP strongly implies it) that the problem with this project is not that it is hard to implement, but that its requirements are unspecified.

Rewriting it in a fancy feel-good language changes exactly nothing and should be treated as an act of sabotage, because it is exactly that.

What is the definition of "fail"? Maybe, given that the project is a hand me down, just barely working is good enough to the client. Maybe there is accepted scope for an over run.

How do you know it will fail? Given my questioning of what a fail is, clearly knowing it will fail is up for grabs.

Will the OP's negativity cause it to fail? This bothers me. An employee who is dooming a project is in danger of dragging the whole thing down.

Perhaps the OP is prejudging things too much?

If the OP "knows" it will fail, the likely hood is that so does every one else.

What seems odd to me is the OP's seemingly isolated position, where it looks like he doesn't get to talk to his co-workers or management. If he knows it will fail, then surely so do his co-workers. Management must also know its not progressing. But, he seems unable to discuss this with any of them, even casually. If this is the case, then my concern would be for the future of the whole company.