The flip side of the reward for success is the cost of failure. A high penalty for failure discourages risk-taking. This is, to a degree, good, because you want to discourage people from taking stupid risks. But if the penalties are too severe, then otherwise good bets will start to look bad. For example, let's say I want to invest money in an idea that I believe has a 75% chance of returning 200% on my investment, and a 25% chance of losing everything. From a pure-math standpoint, this is a rational risk. My average return would be 125% of my investment: clearly right.
But let's say that I need $1,000 to invest, and that this is my rent money and I'll be evicted if I lose it. Suddenly, the total price of failure is significantly higher. It still might be right to invest, but I'm going to be much less happy about it if I lose.
Now let's say I need $10,000 to invest, and the only way I can get it is by borrowing money that I won't be ably to pay back if I lose on the investment. Let's assume I'm facing both eviction and bankruptcy if I lose now. That extra $20,000 3/4ths of the time isn't looking so great compared to financial ruin 1/4th of the time.
As a last example: imagine that there are no bankruptcy laws and America has debtor's prison instead, with jail sentences of two months for every $1000. In this instance, borrowing the money to invest it in something so risky would be wholly irrational if not outright insanity.
In my ideal world, I think I want this investment to look, roughly, about as attractive whether I have to borrow the money to invest it or not. I don't want it to look more attractive if I'm borrowing the money (because it's not "my" money) but I don't want it to look markedly less attractive, either (because of horrific penalities for failure to repay). It's a good risk: I want to encourage people to take it.
By contrast, I do not want my ideal world to encourage people to take bad risks. A program that said "you can claim as a tax credit any losses on investments" would be horrible, because that would make otherwise stupid risks rational. A scenario where there's a 95% chance of total loss and a 5% chance of doubling the investment would suddenly become mathematically right, because the government would be covering my losses 95% of the time and I still get to keep my gains the 5% that it works out. This is bad.
I'm a libertarian in part because I believe that government intervention far more often encourages bad risks than good ones. For example, I don't like the idea of a government bailout of any of the parties (lenders, borrowers, or investors) to the real estate market meltdown, because these are all people who engaged in risky behavior that would have profited them greatly had it worked out. It did not work out, and ameliorating that risk for them means encouraging them and others after them to take similar poor risks and expect a government bailout.
But as I think of the distinction between "debtor's prison" and "bankruptcy", I wonder if there are other ways in which law could encourage responsible risk-taking. What could the government do to help people to take good risks without making bad risks attractive?