Sorry. I mis-read. I saw "there was definitely dirty water getting into the guest shower drain when flushing the guest toilet" and was thinking there was dirty water coming out of the guest shower drain.
ok, thanks! wanted to clarify.
The shower drain enters the waste line very close to where the guest toilet waste pipe goes into the waste line.
As Terry pointed out, the cause would appear to be the tee from the guest toilet being a santee and not a wye, so waste could flow in both directions on the horizontal.
The S trap that is now in place on the shower drain instead of the P trap (will post a pic later tonight) may indeed "work" and not be an issue, but it's not code (which I definitely want everything to be within code even if I never plan to sell the house).
This guest shower may go unused for a couple of weeks so even a little syphoning coupled with some evaporation might cause a break in the water seal (not certain about this, just thinking logically out loud).
I'm going to get a licensed plumber in to evaluate everything and go from there with a course of action.
Thanks again!