My heart is not at peace after istikharah. May I break off the engagement?

Yes, if the doubt is serious and persistent, but weigh it against the facts and the advice of people you trust.
Istikharah is not merely waiting for a feeling; scholars explain that Allah eases the good path and turns one away from the bad. Unease may come from ordinary nerves before marriage, or from real signs such as questionable character. Repeat the istikharah, ask people who know your fiancé(e) (istisyarah), and discuss with your parents. If the decision to break off is made, do it promptly and courteously, because delay adds to the loss on both sides. Settle the return of gifts and expenses fairly. See khitbah (proposal).