Mix together milk and peanut butter in a sauce pan. Put over low heat and stir until the peanut butter is completely dissolved, don't let it boil or get too hot. Cool mixture before adding it into other ingredients.
Add in all remaining ingredients except for jelly and pour into 2 qt. ice cream maker.
Follow manufacturer's directions.
After ice cream has set up, put a layer of ice cream into freezer container and top with 1/3 of the jelly, repeat two more times ending up with jelly layer on top.