I'm done some shadowing, and not all OT work is physical rehab. There are a lot of psychosocial aspects to it too, such as working with drug addicts, working with autism, working in schools for kids with learning disabilities, or working with traumatic brain injury (TBI) patients. They require no physical work. I am sure there are more areas of OT that you can look into also.
I don't know where your arthritis affects you the most. If it is in the knees or back, then med school can and residency can get tough for most specialties with all the standing around; with the hands, it would get tough if you pick a speciality with lots of procedures. But there are many specialities you could probably look into such as Psych. You should pose that question to the Allo forum.