Somehow we always come back to the same two extremes: the self-proclaimed engineers who insist you can safely tow a 21'er with a Smart Car as long as you have "the right set-up," and the fear mongers who insist you're placing the universe in danger if you try to tow a 13'er with anything less than a heavy-duty pick-up truck.
Sure, payload does matter. This chart in my Honda owner's manual shows how the tow rating is reduced as more passengers (or equivalent cargo) are loaded in the vehicle:
Those numbers take into account GCWR, GVWR, GAWR, payload, etc. Notice how the tongue weight
limits drop proportionately faster than trailer weight
limits, reinforcing my earlier statement that the rear axle weight
rating is often the problem.
Best thing, as Bobbie suggests, is to download the owner's manual for whatever vehicle you're considering and read everything it says about towing. Many are quite informative and will clear up a lot of your concerns. A Google search on "20XX <Make> <Model> Owner's Manual" will usually pull up a link to a downloadable PDF file. This is one time the internet is truly helpful!