Change the opacity of Shadow layer to 25%.
Now look at the below image carefully.
I have defined two areas, one is source and another is target. Source is the area which is perfectly exposed while target is the area with somewhat shadows. Basically what I want is taking a sample color from the Source and brush that sampled color on my Target. To do this simple put your mouse on the source and press and hold Alt/Opt. Your brush tool will temporarily be change to Eyedropper Tool as shown below. Now click on the source. Photoshop will automatically take the sample of the source color.
Now look at your background color. It should have been changed to your source color like mine.
Now apply your brush on the are which is highlighted with the red color in below image.
Your image should look like this.