I don't think you should necessarily worry about whether or not to apply, but whether or not you'll succeed. If you had difficulty grasping the material in undergraduate, you'll have seriously troubles in any professional program, not just optometry.
I'm not saying you should quit, but if you are taking out loans to pay for the schooling, you may want to think about yourself. Was it trouble, or laziness? Trouble grasping material cannot be corrected without tons or work. Laziness can, but there's no room for it after your bachelor's.