Like so much else in life the answer depends on the details of teh question..... How close can you get to them?
At under 50 yards I'd use a shotshell loaded with any pellets larger than #4 sized shot. So #2, T, TT, TTT, O, OO, or OOO size would do it, but they are all for close range work.
Since I would have a hard time getting within shotgunning range of a coyote unless he was in the hen yard - I'd use a rifle cartridge. I suppose you could use a slug from your shotgun out to a hundred yards or so, but I'd still use a rifle.
I have shot them with everything from #6 shot while out pheasant hunting to 0000 buck. The last few that I have shot have been with T steel shot out of a 10 gauge. All of them will work as long as you are close enough.
My experience with coyotes and shotguns has been limited and not very productive. Both coyotes that I shot was with the "T" shot dead coyote rounds. Both were at around 45yds and I never recovered them. I did have a friend that piled one up at 52yds with #4 buck. My experience with it has made me very leary to use it. But, if I take out the shotty again for yotes, it's going to be a 12ga with 3" #4 buck...
Every year my Pro-Staff and I head to the woods to hunt the old wise Tom. We travel thousands of miles covering numerous states hunting these old guys. Lots of times we end up hunting public lands or private property that have been hunted already. That’s the last thing you want to hear when you just traveled 6 hours that someone else has already hunted the property. We had to come up with a game plan and one real fast.
The first thing we did was ask the farmer were they usually see the birds in...