Well, that sounds unpleasant. Nothing like a little nagging pain to ruin one's day.
The problem is a lot of conditions could cause the pain you're describing. Anything from gallstones to a pulled muscle. If it's bothering you enough, your best bet is to get an exam from your primary care provider. He or she can try to rule out anything serious.
If you spike a fever, that's an entirely different matter. In that case, you'd want to seek medical attention as soon as possible.
Meanwhile, you might want to try to figure out if anything specifically aggravates the pain or causes it to flare up. Exercise? Eating? Having a bowel movement? Drinking alcohol? These could all be important clues to help your medical provider diagnose the problem.
Important: The opinions expressed in WebMD User-generated content areas like communities, reviews, ratings, blogs, or WebMD Answers are solely those of the User, who may or may not have medical or scientific training. These opinions do not represent the opinions of WebMD. User-generated content areas are not reviewed by a WebMD physician or any member of the WebMD editorial staff for accuracy, balance, objectivity, or any other reason except for compliance with our Terms and Conditions. Some of these opinions may contain information about treatments or uses of drug products that have not been approved by the U.S. Food and Drug Administration. WebMD does not endorse any specific product, service, or treatment. Do not consider WebMD User-generated content as medical advice. Never delay or disregard seeking professional medical advice from your doctor or other qualified healthcare provider because of something you have read on WebMD. You should always speak with your doctor before you start, stop, or change any prescribed part of your care plan or treatment. WebMD understands that reading individual, real-life experiences can be a helpful resource, but it is never a substitute for professional medical advice, diagnosis, or treatment from a qualified health care provider. If you think you may have a medical emergency, call your doctor or dial 911 immediately.