The rules are different for fiction and non-fiction.
However, some universal rules apply. Do not pick a book no one is forcing you to read and which you also do not have much desire to read. Even if it’s a classic which you feel you “should” read for whatever reason. There are a great many classic books in the world, you will not get to all of them. Focus on the ones you expect will bring you pleasure.
If you’re not sure if you’ll like a book or not, recommendations are a good start. See what your local bookseller suggests. (Indie bookshops are good for this. I know it’s a horribly hipster thing to say, but I went in to a Barnes & Noble the other day for Christmas shopping, and the entire experience sucked—and not just because it was December 24th. On the other hand, there are a couple local bookstores that sell new books, and every time I walk in I leave with two or three. They are my kryptonite. I recommend Literati if you’re in the Ann Arbor area, and Pages if you’re near Grand River Ave in Detroit. Talk to the staff, they will help.)
Librarians can also help. I worked as a page in a library for a while, and it always made my day when someone asked for a book recommendation—particularly down in the children’s section. I’d ask what they liked, and they’d tell me, and I’d usually know something similar to direct them to.
In general, people who read like to tell other people who read what to read. Ask around, but don’t let other people pick out your entire reading list.
If you’re choosing a book from scratch:
It’s OK to pick a book because of its cover. Covers a like first impressions: if it makes a good one, go with your gut. Some people like the impression a shabby, well-worn cover makes: it shows the book has been much read and used. Others are drawn to a clean design, or appealing artwork.
On a less romantic note, publishers and marketers design covers to appeal to their ideal reader. If a cover looks interesting to you, it is likely because for a variety of reasons, many of them subliminal, some designer somewhere chose elements to attract you. It’s a very useful device to help match books with their ideal readers. Don’t fight that instinct.
On the other hand, some designers don’t understand their book’s content, and sometimes a good author lands a bad publisher. Or sometimes a publisher makes a bad author look shiny. Covers aren’t a guarantee of a book’s character (any more than first impressions are of a person’s), but they’re not a bad place to start.
For non-fiction:
This is likely to be a significant investment of time and there’s no the plot to spoil, so do your research. Read the entire cover: front, back, and overleaf. I will often read the forward or introduction. Sometimes I begin by reading the conclusion, because it helps orient me as to the rest of the book’s contents. It also gives me a sense of the author’s writing style. Are they clear and concise? A bit stodgy? Flowery and gushing? Forgive me, but I like my non-fiction to be to the point.
I find the table of contents particularly useful. The more detailed, the better. Other periphery information—such as a brief chronology of events, a series of maps or other illustrations, a list of important persons, or a glossary of terms can help guide your choice.
The litmus test: does the book make you want to start reading it right now?! If a thorough perusal of a book and its contents don’t fill me with excitement and anticipation, I doubt I’ll ever have the enthusiasm to slog through it. Does that make it not worthwhile? Not necessarily, but unless I have some external motivation I’m better off looking elsewhere.
For fiction:
Read only as much of the back cover and overleaf as necessary to convince you the book is worth your time.
For some books, the title and author will be all you need. These are often the best books.
However, other books hold unpromising titles, or are penned by unfamiliar authors. For these, read the back cover. Read the dust jacket. If neither of these convince you, the book is probably not noteworthy enough to have warranted a forward, so you are out of luck. Go ahead and read a few paragraphs, or a few pages, or more depending on where you are. Probably if you are in a book store you will not be allowed to read several chapters, but if you are in a library go for it. You may decide to resort to recommendations, but if a friend’s word did not provide enough of an incentive to begin, likely the word of strangers on the Internet will do no better.
For all books:
Read what inspires you, whatever that may be.