First of all, lets clarify what we mean by shellfish. Of course, you'd think that these are fish with shells. Unfortunately it isn't that simple. Shellfish fall into two main categories, crustaceans like crabs, shrimp and lobster and mollusks like clams, oysters and squid. Squid (eaten as calamari) along with octopus is a subcategory of mollusks called cephalopods.
To get the benefit of grilling the meat of shellfish, it needs to be exposed to the direct heat and flame. For instance, when grilling shrimp you need to remove the shell or you might as well boil them. Out of the shell allows the meat to soak in that grill flavor. Similarly, clams and oysters will simply steam in their shells and won't gain the smoke flavor. Of course some things are just easier in the shell so don't worry about removing or cracking every shell.
As a general rule, shellfish like all fish, should be grilled over a medium high to high heat. You will want to cook it fast to keep in the juices and avoid drying out the meat. Small items, like medium shrimp or scallops should be placed on skewers (I prefer bamboo because it holds better) or cooking in a grill wok or similar grillware.
With a hot fire and the food secure for the grill the next challenge to grilling seafood is the timing. If you overcook the shellfish, it will become tough and rubbery. Of course, this is the challenge no matter how you prepare it. The flesh will become opaque and lose it's shiny surface. Generally, you should get just a touch of browning from the heat. With shrimp, the entire surface will turn pink. This is the time to take it off the grill. It's better to remove and check than to wait too long.