Go to Edit > Find/Change, click on Object, then the little magnifying glass icon. You can then specify any attribute you wish to find, including fill and stroke colours.
If it doesn't find anything, you may have a 'ghost' swatch that refuses to delete, even when they aren't used. It's not uncommon. Exporting your document as IDML can fix that.