You can get from 30th Street Station in Philly to Penn Station in NYC as fast as just a couple of minutes more than one hour if you take the Acela Express (ie, no stops)...mostly runs in the morning and afternoon as a commuter train. It is more expensive.
Otherwise, it takes about 1.5 hours for the non-express trains. As noted above if you take the NE Corridor line which will stop along the way in Jersey, it can take much longer (although I've never seen it take 4-6 hrs unless you are changing trains and waiting for a connection).