The only issue would be the pin falling out. You could drill the take down pin hole all the way through, and then use a longer pin with some sort of retention, much like the U Bolt lowers. Not optimal, but would let you test.
Now we need to figure out why the drill wandered. Was it a 3/32" bit? Too large and it could do this. I find running the bit into the hole in reverse helps as well. Only the last 0.25" of the hold needs drilling, to the rear of this area is larger.