.
a)   there is mo restrictions
    The total number of books is 5 + 4 + 3 = 12.
    The total number of all possible arrangements is equal to the number of all permutations of 12 objects 
         12! = 12*11*10*9*8*7*6*5*4*3*2*1 = 479001600.      ANSWER
(b)   all books of the same size are together
     First, we consider the block of 5 large books, the block of 4 medium-size books and the block of 3 small books 
     as 3 separate entities. These entities can be arranged in 3! = 6 ways as the blocks.
     Now, 5! = 120 permutations are possible inside the block of 5 large books;
          4! =  24 permutations are possible inside the block of 4 medium-size books, and
          3! = 6 permutations are possible inside the block of 3 small books.
     All these permutations are INDEPENDENT; therefore, the total number arrangements of the books on the shelf 
     under the given restriction is the product
              6*120*24*6 = 103680 ways.      ANSWER
Solved.
-----------------
On Permutations,  see introductory lessons 
    - Introduction to Permutations
    - PROOF of the formula on the number of Permutations
    - Problems on Permutations
    - OVERVIEW of lessons on Permutations and Combinations
in this site.
Also,  you have this free of charge online textbook in ALGEBRA-II in this site
    - ALGEBRA-II - YOUR ONLINE TEXTBOOK.
The referred lessons are the part of this online textbook under the topic  "Combinatorics: Combinations and permutations". 
Save the link to this textbook together with its description
Free of charge online textbook in ALGEBRA-II
https://www.algebra.com/algebra/homework/complex/ALGEBRA-II-YOUR-ONLINE-TEXTBOOK.lesson
into your archive and use when it is needed.