Should Judge Walker's sexual orientation matter?

There has been a bit of controversy over the last couple of days regarding the revelation that Vaughn Walker, the federal judge presiding over the challenge to Proposition 8 (the California ban on SSM), is gay.  Even those who believe that Proposition 8 should be struck down on constitutional grounds confess that "If I were on the side supporting the ban and found it struck down by a supposedly gay judge, I'd have some questions about whether the judicial deck had been stacked from the start."  In the event that the ban is struck down, I really hope that the ban's supporters will not pin the outcome on the sexual orientation of Judge Walker.  I haven't followed the proceedings closely, but I know that there have been some very controversial rulings that seem to favor the plaintiffs.  Of course those rulings are fair game to raise on appeal.  But challenging the rulings is one thing; piercing the judicial veil is quite another.  One premise of the rule of law is that we maintain public confidence in the ability of judges to do their jobs impartially; even if pure impartiality is a fiction, it's a very important fiction.