We look at the 3rd book we pick.
 
The books are
 
1   2   3   4   5   6   7   8   9  10  11  12
 
You can only pick your 3rd book as 5, 6, 7, or 8, because if you pick your 3rd
book closer to the left end you won't be able to pick a 1st and 2nd book and if
you pick your 3rd book closer to the right end you won't be able to pick a 4th
and 5th book.
 
By symmetry, there are the same number of ways to pick 5 for your 3rd book as
there are ways to pick 8 for your 3rd book.  Similarly there are as many ways
to pick 6 for your 3rd book as there are ways to pick 7 for your 3rd book.
 
So we only need to know: 
 
1. How many ways we can pick 5 for our 3rd book and 
 
2. How many ways we can pick 6 for our 3rd book.
 
Let's do the first one, where we pick 5 for our 3rd book.
 
If we pick 5 for our 3rd book, we must pick books 1 and 3 for our 1st and 2nd
books.  So we need only to look at the number of ways to pick our 4th and 5th
books. 
 
If we pick 7 for our 4th book, we can pick 9-12 for our 5th book.
That's 4 ways. 
 
If we pick 8 for our 4th book, we can pick 10-12 for our 5th book.
That's 3 more ways.
 
If we pick 9 for our 4th book, we can pick 11-12 for our 5th book.
That's 2 more ways.
 
If we pick 10 for our 4th book, we can only pick 12 for our 5th book.
That's 1 more way.
 
So there are 4+3+2+1 or 10 ways to pick our 3rd book as 5.
 
By symmetry there are also 10 ways to pick 8 as our 3rd book.
 
-----------------------------------------
 
Let's do the other one, where we pick 6 for our 3rd book.
 
If we pick 6 for our 3rd book, we can either pick 1 and 3, 1 and 4, or 2 and 4
for our 1st and 2nd books.  That's 3 ways to pick our 1st and 2nd books.   
 
So we need only to look at the number of ways to pick our 4th and 5th books,
and multiply 3 by that number.
 
If we pick 8 for our 4th book, we can pick 10-12 for our 5th book.
That's 3 ways. 
 
If we pick 9 for our 4th book, we can pick 11-12 for our 5th book.
That's 2 more ways.
 
If we pick 10 for our 4th book, we can only pick 12 for our 5th book
That's 1 more ways.
 
So there are 3*(3+2+1) or 18 ways to pick our 3rd book as 6.
 
By symmetry there are also 18 ways to pick 7 for our 3rd book.
 
Therefore the total number of ways is
 
10 ways to pick 5 for the 3rd book
 
 plus
 
18 ways to pick  6 for the 3rd book
 
plus, by symmetry,
 
18 ways to pick 7 for the 3rd book
 
plus
 
10 ways to pick 8 for the 3rd book
 
10 + 18 + 18 + 10 = 56
 
Edwin