I started a breeder and transported them through the nether to my base. Then just kept them There, I put around 4 workblocks(maybe I need more, but then I killed everyone but one and that one still didn’t do anything. I created the world in the beta of this update with experimental gameplay. They are not nitwits, just brown robed unemployed.
UPDATE: one of the breeding villagers was using one of the workblocks all the way from my base, but that shouldn’t effect all the others ones though.