The government is releasing 1,100 more prisoners early, as part of its emergency plan to ease overcrowding in jails in England and Wales.
Offenders serving more than five years are being released on licence after spending 40% of their time behind bars, a scheme that excludes those convicted of serious violence, sex crimes and terrorism.
The second tranche of emergency releases since September comes as ministers launch a major review of sentencing with a focus on new forms of punishment outside of jails.
The review aims to reduce overcrowding and could lead to judges having powers to sentence people to "prison outside prison" with a form of house arrest.
Justice Secretary Shabana Mahmood told the House of Commons on Tuesday that the government "must begin with building more prisons" but acknowledged "we cannot build our way out of this problem".
The prison population has been growing by around 4,500 a year - faster than previous governments had built new cells.
The justice secretary said the Labour government would build 14,000 spaces promised but not delivered by the Conservatives, but also wanted to change the way sentencing works to prevent the overcrowding crisis repeating.
Mahmood said that while there would always be some offenders who must be locked up, the government must "expand the range of punishments outside of prison" and consider how to punish those who are not a danger to society.
Tuesday’s early releases are the second and final part of the emergency plan announced by ministers days after the general election, which aimed to free 5,500 prison spaces across England and Wales.
Many of those leaving jail on Tuesday will have come from open prisons, meaning they have been working towards rehabilitation.
At HMP Swaleside in Kent, an inmate was seen being reunited with his children, while another was welcomed with a hug as he was picked up in a black Rolls-Royce by a group of men in matching hoodies.
At HMP Manchester, known locally as Strangeways, around a dozen prisoners emerged from behind the giant steel doors of the Victorian-era prison.
One man said he had been released nine months early from his sentence of seven years and six months thanks to the new scheme.
It was not clear if the others had also been released early or had served their time.
Sir Keir Starmer's spokesman said that while the prime minister "shares the public's anger" at the scenes of people leaving prison early, "there was no choice not to act".
"If we had not acted, we would have faced a complete paralysis of the system."
The sentencing review will be led by David Gauke, a former Conservative justice secretary.
His report, expected next spring, will look at replacing short prison sentences with new forms of community punishment and using technology to improve rehabilitation.
Officials have already been looking at expanding the use of successful “sobriety tags” that monitor if an offender drinks alcohol.
They also want to look at the case for smart watch-like devices, piloted in some parts of the US, which send "nudge" messages to offenders.
Trials have indicated they can improve how ex-offenders, who often live chaotic lives, comply with their rehabilitation.
Mahmood said the review would learn from other jurisdictions, including Texas, where the prison population has shrunk and the state government has closed prisons.
She said she would "not countenance" any further early releases and would consider a number of measures while the review was underway, including:
Extending the use of home detention curfew including increasing the maximum time someone can be on it from six months to 12 months
Reviewing the process for offenders recalled to custody while out on license
Exploring options for making the early removal scheme for foreign national prisoners more effective
Mahmood said that whatever the final proposals, there should always be a cell available for dangerous offenders - a duty the system came perilously close to failing to meet this summer when it was down to less than 100 available spaces.
The current prison population is 87,465, with 1,671 vacant places remaining.
Andrew Nielson, campaigns director of the Howard League for Penal Reform, welcomed the sentencing review.
He told the OceanNewsUK that Tuesday's emergency prisoner releases were "a really blunt tool" and "far from an ideal situation" because some inmates would not be ready to return to the community.
Ian Lawrence, general secretary of the National Association of Probation Officers (Napo), said that while he also supported the government's commitment to sentencing reform, it "comes at a cost".
"It will mean more work for probation staff," he said, adding that money would be needed to help make the probation service a "place where people want to work, rather than retiring early because they can't take the strain".
Early release scheme began in September
One of Labour's first acts after winning the election was to implement the early release scheme, drawn up by the previous government.
The first set of early releases under this strategy came in September, when 1,700 offenders left prison, reducing the overall prison population to 86,333.
However, there were some serious mistakes by the government.
Some people who should not have been released were set free. Thirty-seven inmates whose sentences had not been properly logged were let out in error. The Ministry of Justice says they have now all been returned to custody.
The OceanNewsUK discovered some released offenders who should have been fitted with an electronic tag to monitor their movements had not been fitted with the device.
Critics have previously voiced concerns about the scheme's impact on public safety, with a probation union warning it amounted to "moving the problem from one place to another without properly assessing the risks".
While the emergency release scheme has reduced the population from its peak of 89,000, officials say jails could reach another crisis point from next July if other measures fail to work.
