Obviously it's gonna be nigh impossible for any single person to develop a usable 3D game engine. So basically you'll have to go 2D via XNA or Game Maker or just plain old Flash. The other option is to borrow someone else's engine and make a mod. And by someone else's I mean Valve's.
I was introduced to modding through my interest in Counter Strike. Me and Pat Bonner got all up ins with the mapping, which is basically one of three tiers of construction that go into a mod. Portal sort of stirred my interest in actually working on a real mod.
The three tiers:
Programming. Although Source's "source code" isn't available for editing, classes and header files may be edited freely. A modder can also create new classes, including new NPCs, a different GUI, new weapons, etc.
Mapping. The next level is implementing these new classes, as well as existing architecture, in a "map", or the world in which a user interacts. These includes designing the flow of a game, creating the setting, establishing the game's mood, and scripting game events. A mapper lays down the walls, ceilings and floors, textures them, inserts "entities" to produce effects, and places in lighting.
Modeling. Unlike the starve-yourself one, this Modding involves manipulating in 3D, textures and primitive objects to create the detailed representations of smaller objects or more detailed objects than the mapper can handle. They design the weightmaps for ragdoll effects. Some do double time as animators, designing the motion of the ragdolls and then creating scripted sequences in which characters interact.