I only had one pub officially published at the time of application (first lab in undergrad). I didn't ask that PI for a rec letter (even though he was more than happy to write one) because it had been years since I worked with him, and I had a LOR from my most recent PI.
In a vacuum, I don't think it's a huge issue imo. If that's the biggest "red flag" on your app, then your app is fine. I was never asked why I didn't have a LOR from my first PI that I published under. If I had been asked, I would have just said that I wanted to get rec letters that were from people who knew me more recently. As long as your PI doesn't trash you if a school calls them to verify information (which it doesn't sound like he would), you're good to go.