I would have the part next to the pipe be sand such as play sand. Then I would fill up to about 3 or 4 inches of the ground surface with the dirt that you extracted. Presuming that your extracted dirt has clay or stones, I would discard the remainder. I would top that up to a little above the ground. Then fill with top soil. Water this if needed to allow settling. The dirt should settle some. In spring or summer, I would put sod on top. I am not a pro.
If this is a 16 ft long trench, I might do things a bit differently than if it is a 4 ft long hole. But to recap, I would not put big rocks next to the pipe. Presuming you want good grass, I would not top the hole up with clay or rock. I would expect settling, and water (rain or supplemented) to speed the settling process.