Like so much else in life the answer depends on the details of teh question..... How close can you get to them?
At under 50 yards I'd use a shotshell loaded with any pellets larger than #4 sized shot. So #2, T, TT, TTT, O, OO, or OOO size would do it, but they are all for close range work.
Since I would have a hard time getting within shotgunning range of a coyote unless he was in the hen yard - I'd use a rifle cartridge. I suppose you could use a slug from your shotgun out to a hundred yards or so, but I'd still use a rifle.
I have shot them with everything from #6 shot while out pheasant hunting to 0000 buck. The last few that I have shot have been with T steel shot out of a 10 gauge. All of them will work as long as you are close enough.
My experience with coyotes and shotguns has been limited and not very productive. Both coyotes that I shot was with the "T" shot dead coyote rounds. Both were at around 45yds and I never recovered them. I did have a friend that piled one up at 52yds with #4 buck. My experience with it has made me very leary to use it. But, if I take out the shotty again for yotes, it's going to be a 12ga with 3" #4 buck...
I have a system I use in Whitetail hunting during the rut that seems to work for me. When I make a mock scrape in the area I wish to hunt I will place a couple of 35mm
film canisters (film removed of course) in the scrape after I have cleared out all leaves and other debris. The film canisters are filled with cotton balls and saturated with doe in heat urine. This keeps the urine from soaking in the ground and can be refreshed on each trip if needed. I will also use the tarsal gland of the buck...