"To be fair, I thought he'd move on. Like you did," he said. "I didn't mean to like him."
Other options:
no subject
"To be fair, I thought he'd move on. Like you did," he said. "I didn't mean to like him."