`Delete` key doesn't work on vim insert mode for deleting previously typed content

I opened vim in my iterm2. Firstly, I typed something on insert mode, like

Hello

At this stage, if I didn't exit from insert mode, delete key works and can delete the whole word if I want.

If I quit insert mode, and open insert mode again, this Hello can't be removed by delete key. But the newly typed-in content can be removed.

I've renamed my .vimrc, problem still exits.

In the shell command-line, my delete key works well.

And I did some test on remote server through iterm, didn't encounter the same issue.

What could be the cause of this problem? How to fix it?

---

PS: As I'm using macbook, the delete key is corresponding for backspace in PC.