We have had some late night white boarding sessions formulating both incredibly simplistic and overly complex algorithms. The middle of those two is the hard thing to get to.

The most recent white board session ended in me deciding I need to write a letter to my 9th grade math teacher. I think i gave her crap about how algorithm homework was a big waste of time, but she made me do it anyway.

The idea of considering replies is interesting. On digg however, I find the comments tend to be 98% worthless. I miss /. comments.

Probably the best/easiest thing to do is plan to be adaptive to how the community uses your site. Your algorithm will likely need to change over time.

I would also include the number of replies, although I must admit that I've never implemented such a system myself. Typically, the best articles have the most replies, and quite often, the most interesting content is the replies themselves.