A "goomod" file is a complete packaged addin for World of Goo. Most goomods contain new levels or new ball types for you to play with.
To install a goomod, you need to use GooTool, which can be downloaded freely from this site.
Follow the installation instructions for your computer on the download page. You may need to install Java too.
The first time you run GooTool, you will be prompted to choose a location for your custom World of Goo. GooTool does not modify any original game files, so instead you must choose where to make a copy. GooTool then installs your addin into this copy.
Once GooTool is installed, simply click on "Install Now with GooTool" from any addin page on this website. When prompted, choose to "open" the file with GooTool rather than save.
GooTool will ask you to confirm you wish to install the addin. Then simply click "Save and Launch", and it will launch your custom game, complete with the new addin. The first time you save may take a few minutes while the game is copied.
If you installed a new level, it will appear in the top left of Chapter 1.
If you installed another type of addin, check the information page for that addin. For example, ball graphic addins generally change the appearance of your balls in the World of Goo Corporation.
For the technically curious, and for those interested in developing their own World of Goo addins, there is a description of the goomod file format.