The Gills, for whom on-loan Adam Nowland - who will not play at the weekend - scored a consolation goal in their 2-1 defeat at Millwall on Tuesday, are second from bottom, but Rufus says:
"The win over Stoke was a massive game for us because they are a decent side but we are playing a struggling Gillingham side now - and sometimes they are the games that can work out the hardest.
"We have to take heart from the Stoke victory and take it on to Saturday; we go into every game wanting to win, having to win, and that is the only attitude you can take.
"We stuck to it on Tuesday, kept a good clean sheet, limited their chances and took ours.
"We looked quite solid and that is a reward for all the hard work we have put in in training.
Rufus returned to the starting line-up because of injury to Chris Powell, and he adds:
"I was delighted to be back because I haven't played since we beat Sheffield United.
"All you can do is try and stay in the team, but it is up to the manager.
"My fitness is fine and I am feeling good.
"Chris is a top guy and he has been playing steadily as well.
"But he has a slight injury now and the door is open again so hopefully I can get a run in the team."
Reflecting on Alan Pardew's first year in charge, he adds:
"He has come in and changed things round, not doing it all at once but taking his time, and he has got a good squad of players, which you need."
Rufus is aware of the gap Wigan have opened up at the top of the table - six point ahead of the Hammers and three ahead of nearest challengers Reading - but he says:
"Wigan are going to be a danger - but you don't know what they have got in depth, what they have as back-up.
"I think we have got the best squad in the division without a shadow of a doubt but if Wigan can keep their players fit they are going to be very strong."