I figured out a couple of commands for you to keep the player in a certain radius. It does this by teleporting a marker entity to your position every tick, but only within the radius of the sphere, so once the player leaves the sphere a second command can teleport the player back to the marker. These commands can be run in repeating command blocks or a function. Here's the commands necessary:

Load Commands (run once)

/tag username add constrained
Adds a tag to the player to keep them in this spot. If you have multiple players you'll need to set up a system with multiple tags or teams or something similar. you can also change the target selector to whatever you want.

/execute at username run summon marker ~ ~ ~ {Tags:["pos"]}
Summons a marker at the player that will be used to track their location. You can change the target selector as well as the tag for this one, just make sure to keep the tags consistent.

Tick Commands (run every tick)

execute positioned x y z as @e[type=marker,tag=pos,distance=..7] if entity @a[tag=constrained,distance=..7] run tp @a[tag=test,distance=..7,limit=1]
Runs as the closest marker with the tag pos inside a 7 block spherical radius centered around a chosen location. If a player with the tag constrained is found within this radius, the marker is teleported to the player. Make sure to change the x, y, and z coords to a location of your choice, you can change the radius to something other then 7, and you can change the tag names.

execute positioned x y z as @a[tag=constrained,distance=7..,limit=1] run tp @e[type=minecraft:marker,tag=pos,limit=1]
Runs as the closest player with the tag constrained outside a 7 block spherical radius. If found, it teleports the player to a marker entity with the tag pos inside of the 7 block radius. Make sure to change the x, y, and z coords to a location of your choice, you can change the radius to something other then 7, and you can change the tag names.

See if this works for you. If it doesn't, let me know, or just try a different method.