You’re going to live/grow up with your grandpa.(You’re referring to growing/being raised with the child’s own grandpa.)