To reiterate how we came to the conclusion that BamHI was used:
Because the Blot for Gene X returned a negative result (no band show; i.e. Gene X did not bind to the compliment) - BamHI must have been used to cut GeneX
This leaves us with options A or C
An important note here is that the diagram does not show the length between the BamHI and HindIII site - but we can assume it is 0.65kb since we know the total length of the plasmid and the length of all the other fragments.
To respond to your first point: the entire plasmid is 1.7kb long, no matter which endonuclease is used. Look at each lane - they all add up to 1.7kb. Even Lane 4 (one band at .8kb and the other at .9kb).
Let's focus on lane 2: You are correct with judging the lengths of the bands (one is more than 0.5kb and less than 0.8kb, and the other is slightly more that 1kb). Now visualize the action as each endonuclease cuts the plasmid. First, we know from above that BamHI must cut it, so now visualize that the plasmid is cut and becomes a straight line (I know this is biologically inaccurate but it helps to explain). Next, we must determine the next endonuclease that cuts the plasmid. We know the resultant pieces must be one of 0.5 - 0.8kb and the other must be slightly more than 1kb. Pretend that A occurs - BamHI cuts the plasmid followed by EcoRI. The result would be a fragment with a length of 0.8kb and 0.9kb - this does not match lane 2 and leaves us with C as are only remaining, and correct answer. For the sake of exercise - simulate the action of BamHI followed by HindIII: BamHI cuts the plasmid and creates one long fragment, then HindIII cuts and produces 1 fragment of 0.65 kb and another of 1.05 kb.