Accessibility is high on my priority list with every site I create. In theory if you plan your structure well and build your site using good markup you are making your site accessible to the majority of users. None of it is really rocket science...
1. use meaningful page titles
2. logical navigation system
3. structured use of heading tags
4. use pararagraphs and lists where applicable
5. provide alternative content for content images, ie alt/title attributes wher applicable
6. provide alternative content where javascript is not available
7. provide alternative content for audio/video such as a text only transcript for an audio podcast
8. correctly labelled forms and form elements
9. use css for styling and presentation
10. provide meaningfull summary and caption for tables
11. make linked items clear, ie underline linked text, use separate colours for link, visited, hover and active linked text
12. make the site navigable and understandable with images and javascript turned off
That probably takes care of most of it, you can then move into more refined areas such as...
1. provide skipinks to key page areas such as navigation and "main content"
2. provide tabindex for form elements though there's some debate as to how useful these actually are since if your html is correctly structured you could argue they're not really needed
3. provide accesskey's for key link elements, again if your html structure is good they might not be needed, plus some assitive technology uses pre determined accesskeys which can cause conflicts
Those last 3 might open a can of worms
I'd argue that it's impossible to build a *perfect* accessible site but if you make every reasonable effort to cater for as many user types as possible you shouldn't need to worry.
I have worked with a few disadvantaged web users in the past, all they want to to is to be able to navigate a site and enjoy the content using a screen reader or be able to understand audio if they're deaf - a well coded site will provide that.
It's not a crime to omit the odd image alt attribute, we're all human and we all miss things but not using them on a large scale is a crime!