The Ceramic Building on Newington Causeway includes the new Travelodge Elephant & Castle hotel as well as 49 flats.
When councillors approved the development in 2017, 16 of the flats were intended as affordable housing, to be split between shared ownership and social rent.
Now developer Neobrand wants to pay Southwark Council an in-lieu contribution of £5.9 million if it can jettison the requirement for on-site affordable housing and convert the 16 flats for sale or rent on the open market.
The developer says it can't find a social housing landlord prepared to take on the flats.
On behalf of Neobrand, TMLand approached 26 housing associations, but none were willing to make an offer.
Reasons cited include the small size of the development, the 24-storey height of the building being "against HA current strategy" and the £90,000 income cap on the shared ownership homes being regarded as too low.
"It should be noted that since the discussions with the housing associations I understand the service charge has risen dramatically due to the building control requirement to have a 24-hour concierge for fire safety," wrote Tom Mulligan of TMLand.
"This would make the scheme even more unviable for any housing association."
If the proposed change of tenure is approved, the £5.9 million contribution would be added to the borough's funding pot for the construction of new council homes.
Last year Southwark Council agreed that the communal children's play space that had been constructed on the top floor of the tower could be converted into an extra home for private sale.
Approving that application, council planning officers wrote: "It is understood the placement of children's play space at this level has become unfeasible due to concerns around building regulations and fire safety compliance.
"The applicant has proposed a payment in lieu to offset the agreed provision of children's play space, as well as for affordable housing."
• For the latest application, see 22/AP/0485