A swollen and painful clitoris could be due to allergy, hormonal changes or sexual practices that are irritating it.
There is a way of examining the genital area under a type of microscope called a colposcope.
Minor changes, such as small bumps or thickening, become more apparent through this technique, and it might be that this would be useful in your case.
The kind of pain you get is also important. Burning or searing pain may be indicating vulvodynia.
This is a medical term for a difficult to diagnose and treat condition in which symptoms occur without visible surface changes. However, there are some specialist treatments available for it.
Itching pain may indicate allergy, and I note that you are already avoiding things that might aggravate it.
The skin in this area is very sensitive, and reacts to a wide range of allergens including things you have not mentioned like coloured or scented toilet paper, fabric softener, and bath additives.
It is also possible to become allergic to things you have used for years. Creams are available to treat allergies in this area.
The clitoris can also enlarge due to hormonal changes, although this is rare. There are conditions where excess male hormones are produced that can make the clitoris enlarge and feel uncomfortable.
If you had this rare condition you might notice other changes like excess hair growth, acne, weight gain and lowering of the voice.
Have you noticed any pain in your clitoris? If yes, you should see your doctor for some blood tests. These are quite easy to do.