I had to give up martial arts for 9 months due to injury and coming back was like a second lease on life. However, I did have to make a realistic evaluation of my risks. I stopped training in (kick)boxing and focused on BJJ since short of getting dropped on my head, I'm not at further risk of reaggravating my neck issue. On the other hand, repeated, sharp blows to the head seem like a bad idea.
I agree with the sentiment that if it's no longer fun, you shouldn't do it. I'm coming up on my 6th year of regular training in martial arts and it's only added more and more to my life. Especially now in a particularly difficult point in my life (broken engagement, had to move in with parents, fired from job and have to pursue legal remedy, etc.). Training MA provides a respite.
If there's an alternative activity that gives you what has been termed "Flow", then please do that. There are activities that are fun, then there's those that are fulfilling.