This one might help: https://github.com/quant-sci/compute-geometry. If you have a convex object like the one in the image, I guess you'd first compute its convex hull thanks to this library, and you'd then use the resulting convex hull to determine whether an atom is inside or not.
