The moment the clone wakes up, it starts having different experiences than you. Two streams of consciousness can't be the same person — if it were you, you'd see through its eyes.
Yeah, even with your memories copied, the clone's brain starts as a separate physical thing, so its consciousness branches off immediately.
