Carlo Corazzin headed the only goal of this third round tie from David Eyres' left-wing corner a minute before half-time.
It extended West Ham's failure to win at home this season to seven games and left manager Glenn Roeder more headaches ahead of two consecutive home Premiership matches against Leeds United and Manchester United.
Roeder, criticised in some quarters for sticking rigidly to 4-4-2 formation, tried 3-5-2 from the start tonight but the outcome was sadly inevitable - another embarrassing home defeat by a side that, despite injuries, West Ham should beat at Upton Park.
Without the continuing absence of strikers Fredi Kanoute and Paolo Di Canio, Hammers struggled to create any clear-cut openings against this well-organised Latics side, who justified their current status as second division title challengers, leading scorers in their division and unbeaten away from Boundary Park..
A year ago, Kanoute and Di Canio were missing again as Hammers bowed out of the League Cup early at Reading and for all Titi Camara's endeavour and Jermain Defoe's penalty box twists and turns, they were unable to conjure an equaliser.
And as the game wore on, Roeder reverted to his favoured back four. Gary Breen, who played on the right of a back three alongside Ian Pearce (centre) and Christian Dailly (left), and Scott Minto were replaced by Sebastien Schemmel and Nigel Winterburn, both filling full-backs slots.
Richard Garcia, the young midfielder, came on 10 minutes from the end for Cisse on the right, but the only out-and-out striker on the bench, young Frenchman Youssef Sofiane, remained tracksuited and still awaiting his debut.
In desperation, Ian Pearce was pushed forward to try and add a physical presence to the front line. The centre-back drove one shot low and wide and headed two crosses wide of the posts at the Bobby Moore Stand end.
The crowd of almost 22,000, swelled by kids who got in for just £1, showed remarkable patience given the absence of a home win this season and didn't begin booing the dispirited men in claret and blue until the final minutes, when hopes of an equaliser had all but gone.
Referee Uriah Rennie bore the brunt of the crowd's anger and frustration. His failure to apply the advantage rule didn't help to brighten a dark, bleak night, although Oldham had the strongest penalty appeal turned down when Wayne Andrews tumbled under challenge from keeper David James on the hour mark. Camara's two vain appeals were less convincing.
West Ham's best player? No contest, it was Joe Cole yet again. The extra man in midfield allowed him greater freedom in the first half, especially, and two typically exciting runs through the middle after the break led to half chances that Defoe couldn't convert.
Credit to Oldham, though, who were applauded off the field by home fans. Dowie's men now join a long list of lower divisions sides who have humiliated Hammers in the cups over many years.
Minute-by-minute:
2 - Defoe escapes down the left before sending a nice touch inside to Lomas, whose shot from six yards out is blocked by Oldham keeper Pogliacomi.
6 - Camara turns well on the right and fires in low cross that Pogliacomi grabs at the second attempt.
8 - Hill flicks on Hall's long throw in the penalty area, but Murray fires left-footed over the bar.
20 - After Carrick battles well in midfield, Cisse's 25-yard snapshot flies just wide of the target.
28 - A quick-thinking free-kick from Eyres, who shoots when it appears he is going to cross the ball, almost catches out James, who does well to palm the effort over the bar.
38 - Dailly's misguided header back towards goal lets Andrews in, but the striker slices his shot well wide of the target.
41 - Corazzin's fierce shot from the edge of the penalty area strikes Pearce and flies off for a corner.
42 - GOAL. From the resulting kick, Corazzin stoops at the near post and sees off two challenges to head past James and stun the Upton Park crowd. West Ham United 0 Oldham Athletic 1.
43 - Dailly booked for foul on Andrews.
Added time: 1 minute
Half-time: West Ham United 0 Oldham Athletic 1.
Second half:
46 - Cole wriggles past three defenders into the area and cuts the ball back into the six-yard box but Armstrong clears.
48 - Another tremendous run from Cole sees him burst from his own half and evade two challenges before slipping the ball into the path of Defoe, whose right-foot shot is well saved by Pogliacomi.
55 - Double substitution for West Ham, Schemmel replaces Minto and Winterburn replaces Breen.
59 - Defoe booked for diving.
60 - Andrews latches on to a loose ball in the area and touches it past James before crashing to the ground under the keeper's challenge. However, referee Rennie waves away Oldham's appeals for a penalty.
60 - Murray booked for dissent.
67 - Schemmel booked for foul on Eyres.
68 - Camara does well to jink past two defenders and find space in the area, but his low cross is deflected off a defender and lands in the arms of Pogliacomi.
70 - Cole again tries to inspire his team-mates with a surging run down the left that ends when his shot is blocked by a defender and falls to Defoe, who blasts over from 12 yards out.
71 - Camara's flick finds Defoe again in the box, and this time the young striker is denied by a fine one-handed save from Pogliacomi.
73 - Substitution for Oldham, Killen replaces Andrews.
80 - Substitution for West Ham, Garcia replaces Cisse.
83 - Camara turns well in the area and goes down somewhat theatrically under the challenge of Hill. Referee Rennie waves play on.
89 - Garcia's deep cross is met by Pearce, whose header sails miserably wide.
90 - Hill booked for time-wasting.
Added time: 3 minutes.
90 - Hall booked for ungentlemanly conduct.
90 - Camara and Pearce booked for ungentlemanly conduct.
Full-time: West Ham United 0 Oldham Athletic 1.