Your problem is that you have insufficient data to do your computation.
Latitude and longitude are positions on a sphere. These positions will be in degrees, minutes, and seconds of arc (and usually fractions of a second).
Being given coordinates on an X-Y axis implies positions on a plane. The process of converting X-Y to Lat-Lon and vice-versa is called Projection.
There are three different kinds of projections with frequent use and maybe a few others that are really out-of-the-way that nobody uses in the mainstream navigation industry.
1. Mercator - in which the flat surface is an unrolled cylinder that was based on a true cylinder. This cylinder theoretically intersects with the sphere like an apple-corer. The X-Y coordinates would be the distance from some arbitrary point on the imaginary circular intersection line or lines. There COULD be two such circles if it is not an equatorial projection or great-circle projection but most world maps are based on equatorial intersection.
2. Lambert - in which the surface is an unrolled conical section that when still conical theoretically intersects with the sphere like a pointy Chinese hat sitting on your head. This could be a single circle (more common) or, for special cases, two circles (but you still only use one of them). The coordinates will represent a distance to a reference point on one of the imaginary circular intersection lines.
3. Cassini-Soldner - in which you have what is called a pin-cushion projection, sort of like if you flattened out an older curved-screen electron-gun TV screen. (Before the days of flat-screen TVs and monitors.) The coordinates will represent a distance to the reference point which is always in the center of the projection area.
All three cases are different examples of navigational geometry. All three are attempts to do a change of dimensions - sphere (3D) to map (2D) or vice-versa - and will ALWAYS introduce distortions. To solve this kind of problem, you need to know:
a. The type of projection (Mercator/cylindrical, Lambert/conical, or Cassini-Soldner/pincushion)
b. The selected intersection point for the intersection of the sphere and the 2D surface.
c. The distance on the 2D surface from the closest intersection point to the target point.
d. In the other direction, if you have Lat-Lon of the point of question, you need the Lat-Lon of the nearest reference point.
Your question does not indicate enough for us to point you anywhere, but if you ever find out what kind of projection is involved, I've given you the names that you can use to look up projection formulas.
Just so you'll know, a LONG time ago I was a special-functions programmer for a company that made navigational devices for the oil-and-gas exploration industry. Our computers, if given enough reference points, could put you on a map within a guaranteed five meters of exact position as long as we had four radio beacons available. If we could get two different kinds of beacons (LORAN, GeoStar, TransStar, Spot-Nav, etc) we could get it down to 1.5 meters.
The math is ugly only because of all the trig to project or "unproject" coordinates between 2D and 3D, but all of those projection methods are available on-line.
