At 4 weeks, I don't see it as a problem to give him the paci, especially if the paci is just used for naps.
However, no it is not cruel to take it away now if you want to nip the addiction in the bud so to speak. It is cruel to take it away for one sleep time, then return it for the next one, KWIM? It's confusing.
If you do decide to take it away, and the gradual weaning doesn't work, go cold turkey and be consistent.
I hope that I haven't confused you. Just to let you know, I started my son on a paci at 2 months or so (yes on purpose) as he really seemed to have a need to suck. As I was BF'ing at the time, I refused to be his soother (DH couldn't understand why ... I even showed him my damaged nipples :roll: ) I then took it away, with minimal fuss, at about 7-8 months old (I think), and was very lucky that pu/pd worked ... that time.
Do what you think is best and be consistent. You are a great mom, and keep up the good work!!