So this is actually because the Castle predates the modern coherent plan for the Mall.

Prior to the 1910's the Mall was actually overgrown mostly and not an open space we think of today. Full of Victorian style gardens, and winding paths, there wasnt really a clear plan to involve a coherent visual picture. Here we have a view from the top of Washington looking towards the Capitol. Notice how what is Jefferson Ave today, was winding across its whole visible length!

It was in this environment then that the Castle was built in the early 1850's. With the later additional Industrial Arts building set off to the East and Behind it.

However with the continued dredging, reclamation of land, and evolution of the Capitol, in the 1910's Congress set out on a new scheme. Known as the McMillan Plan, its goal was a coherent, beautified Mall. Part of that included a 300 ft offset so that there would be the cleared Mall as we think of it between Washington and the Hill. While it the Castle could theoretically have been moved, it wasnt a giant protrusion, and would be a costly lengthy undertaking.

However tension arose when the Department of Agriculture wanted to build their new HQ on the plot just West of the castle, after TR initially allowed them to plan for using all the space they could, the commission pointed out it would violate their plan. So for the Whitten building still in use today it was the first time the distance requirements for buildings between Independence and Constitution Ave's was enforced. And it would again when the Natural History building went up in 1910 on the North side. IfUSDA built up to the Castle's line, but the enforcement remained on the North side, the Mall would have been off center from the Capitol to Washington to Lincoln, while removing it all together would greatly reduce the open space.

While the land being reclaimed from dredging, and filling in the old canal allowed for West and East Potomac Park's to be created, and the Mall extended out to where Lincoln sits today. While Constitution Gardens today was quickly taken up during WW1 by what were known as Main Navy and the Ammunition Buildings, temp office space that ended up being around for decades.

WaPo actually also has a very neat timeline of the Mall if you are interested: https://www.washingtonpost.com/graphics/lifestyle/the-evolution-of-the-national-mall/