Dependencies: most software is dependant on other libraries for certain basic functions. This can pose a problem if these libraries are not available on the system being developed for. This might be for cost, licensing, hardware or other restrictions. Note that often the dependencies also need to be available in the correct version – possibly libraries have conflicting requirements!
Documentation is often scarce in the open source world; in particular developer documents such as API descriptions, configuration documentation, build instructions. You should also check the documents are up to date. If you need help you may also want to check whether there are any mailing lists or discussion forums associated with the software. This is often the case when the software is a product of a community effort.
Authors – it’s usually helpful to find out some background on how many authors are working on the source. Are you relying on a sole software developer or binding yourself to a single company? If the library is implementing a standard API this is not as critical as it’s easier to change the implementation w/o affecting your own software. In other cases you should always abstract from the implementation via your own API! Good places to find out about the authors are bug tracking systems and/or history of change documents, in addition to the website hosting the open source library.
Known bugs – check bug tracking systems (bugzilla[3], or whatever tracking system the open source website mentions) to find out about known bugs, get an idea of the stability of the software, frequency of updates, date of the last update (is anyone working on the software lately?), missing features, community feedback etc. The version of the software also provides an indication of stability, so beware of 0.1 versions!
Check how to build the software: the build system may require new tools and/or scripts. Possibly the build system uses a language (e.g. python) you are not familiar with. Is it configurable, do you need cross-compile support?
Take a look at the sources! Check for inline commenting, readability, structure, unit tests, all factors indicating the code quality. You may want to have a code checker (e.g. findbugs) analyse the code. Also check the warnings your compiler logs during the build process. Example applications are also useful if they are included.