(1)  In all, there are 6! = 6*5*4*3*2*1 = 720 different permutations of 6 books on the shelf.


(2)  Of them, there are 2*5! = 240 permutations, where the books A and B are together.


(3)  Hence, in the rest 720 - 240 = 480 permutations, the books A and B are not together.
     It is the number of all possible permutations under the condition "If A and B were not placed together".
     So, 480 is the denominator in the future fraction for the probability.


(4)  The number of all possible permutations of 6 letters with the block 
     of 3 letters ACB in 3 positions between #1 and #6 is 4! = 24.


(5)  The number of all possible permutations of 6 letters with the block 
     of 3 letters AXB in 3 positions between #1 and #6 with X= C or D or E or F 
     is 4*4! = 4*24 = 96.


(6)  The number of all possible permutations of 6 letters with the block 
     of 3 letters BXA in 3 positions between #1 and #6 with X= C or D or E or F 
     is also 4*4! = 4*24 = 96.


(7)  So, the numerator in the fraction for probability is  96+96 = 192.


(8)  Finally, the probability under the problem's question is  

         =  =  =  = 0.4.    ANSWER

Consider putting A in one on those first 4 spaces and
B somewhere to the right of it.

_C_D_E_F_ 

Successful ways:
We can pick a letter to put A immediately before and B immediately after
in 4 ways.

_C_D_E_F_

Possible ways:
We can choose a pair of spaces from the 5 to put A in the leftmost one 
and B in the rightmost one in C(5,2) = 10 ways.

The probability is 4/10 or 2/5.  The ratio is the same for any arrangement
of CDEF, and either arrangement of AB, so that's the final answer. 

[If we like we could multiply numerator and denominator by the 4! ways to
arrange CDEF and also by 2! for the 2 arrangements of AB, but why bother?
The ratio, i.e., the probability, would be the same.]

Edwin