I'm not sure how it works or how it's derived entirely, but the formula for triangulating any n-gon (n is an integer and n >= 3) is
Source:
Pickover, Clifford A. "Euler's Polygon Division Problem." The Math Book: From
Pythagoras to the 57th Dimension, 250 Milestones in the History of
Mathematics. New York, NY: Sterling Pub., 2009. 184. Print.
More Source Info:
Formula found in page 184 of this book.
See this page (page 184) in google books here so you can see the formula for yourself.
In the case of a triangle, T(3) = (2)/(3-1)! = 2/2 = 1, which means that there is only one way to triangulate a triangle.
In the case of a quadrilateral, T(4) = (2*6)/(4-1)! = 12/6 = 2, which means that there are two ways to triangulate a quadrilateral.
In the case of a pentagon, T(5) = (2*6*10)/(5-1)! = 120/24 = 5, which means that there are 5 ways to triangulate a pentagon.
In the case of a hexagon, T(6) = (2*6*10*14)/(6-1)! = 1680/120 = 14, which means that there are 14 ways to triangulate a hexagon.
You continue this pattern to find the triangulation for any convex n-gon.