Neither one really conveys what you are trying to convey.
The first, "went to the grocery store to buy limes, but that's the least you did" implies that you did more than just go to the grocery store, or that you did something else other than just buying things (e.g. cleaned the store).
The second implies that you have done nothing since buying limes at the grocery store.
What you are looking for is something like this: "You went to the grocery store to buy limes, but that's not all you bought"
If you want to convey that they went for limes but came back with other things, but no limes, then that would be "You went to the grocery store to buy limes, but you came back with other things instead."